According to the docs, SQL Anywhere 12.0.1 is officially supported on Windows 8.1 starting with EBF 12.0.1.3994 (there's no particular EBF stated for Windows 10), so given you use a way older build, I'd suggest to try with a current one (such as 12.0.1.4403).

What Windows version have you used before?

Please show us the commands that DO work, so we can try debugging-by-manual-inspection, rather than debugging-by-finding-and-installing-old-build-on-Windows-10... but I will do that if necessary :)

FWIW commands similar to yours seem to work OK using SQL Anywhere 16 on Windows 10.

Logging the details of connection attempt can often be helpful in these situations. Try adding something like log=c:\temp\conlog.txt or whatever path is appropriate to your -c parameter.

I am not able to reproduce this. Perhaps you can add more detail?

I believe that dbisql -nogui should report more info than just (Not Connected) but if it does not, please add LOG=<filespec> to get client side connection debug logging as recommended by Justin. Can you also outline the start options used by the engine and whether the engine is a network or personal server.

With the -nogui option I get no log output.

With no -nogui option I get this (with secure info splatted out):

Wed Jul 27 2016 08:23:50
08:23:50 Attempting to connect using:
UID=dba;PWD=********;DBN=fred;CON=SQL_DBC_50908330;LOG=c:\temp\conlog.txt
08:23:50 Attempting to connect to a running server...
08:23:50 Trying to start SharedMemory link ...
08:23:50     SharedMemory link started successfully
08:23:50 Attempting SharedMemory connection (no sasrv.ini cached address)
08:23:50 Connected to server over SharedMemory
08:23:50 Connected to SQL Anywhere Server version 12.0.1.3554
08:23:50 Connected to the default SharedMemory server
08:23:50 Application information:
08:23:50 IP=10.10.30.172;HOST=***;OSUSER=***;OS='Unrecognized Windows version Build 9200 ';EXE='C:\Program Files\SQL Anywhere 12\bin64\dbisql.exe';PID=0x1abc;THREAD=0xa40;VERSION=12.0.1.3554;API=iAnywhereJDBC;TIMEZONEADJUSTMENT=-420
08:23:50 Connected to the server, attempting to connect to a running database...
08:23:50 [   51] Connected to database successfully
08:23:50 [   51] The number of prefetch rows has been reduced to 16 due to the prefetch buffer
08:23:50 [   51] limit. Increasing the PrefetchBuffer connection parameter may improve performance.

More info:

All of the other systems at my company with the same software and configuration and everything works just fine. We have field installations of our software--which includes this SA EBF of course--and they are all running fine. There is something on my system in particular that isn't allowing -nogui when run from cmd.exe; that includes commands run from xp_cmdshell().

Our application (Delphi, using Devexpress) connects just fine on my system. It uses a commercial DB connection package.

My previous Windows version was Windows 7. I don't know which subversion.

What startup settings? Let's make things simple and do this ...

C:\WINDOWS\system32>"%SQLANY12%\BIN32\dbinit" "c:\temp\fred.db"
...
C:\WINDOWS\system32>"%SQLANY12%\BIN32\dbsrv12" "c:\temp\fred.db"
...
C:\WINDOWS\system32>"%SQLANY12%\BIN32\dbisql" -nogui -c "uid=dba;pwd=sql;dbn=fred" select * from sys.systable
You are not connected to a database.

If I don't create an ODBC connection and I use the dsn parameter I get the "You are not connected to a database." message.

I don't know if this is related, but I created an ODBC connection for "fred" and used the dsn parameter and I got ...

table_id   file_id count                first_page last_page   primary_root creator    first_ext_page last_ext_page table_page_count ext_page_count object_id            table_name                                                                                                                       table_type    *** OutOfMemoryError ***
Request type=3
Request =[ExecuteRequest:sql=select * from 
sys.systable,containsODBCEscapeSequence=false,cachingHint=2,warning=null,error=null,errorMessage=null,resultsProcessor=sybase.isql.Console]
Error! This program has encountered an internal error.
Details have been logged to C:\ProgramData\SQL Anywhere 12\diagnostics\isqlError-2.err.xml
Exception is:
Database worker thread request failed because of a runtime error.
Request=[ExecuteRequest:sql=select * from sys.systable,containsODBCEscapeSequence=false,cachingHint=2,warning=null,error=null,errorMessage=null,resultsProcessor=sybase.isql.Console]