SQL Anywhere version 10 and above cannot start databases build in earlier versions (i.e. version 9 or below). You must rebuild the database.

See the documentation for more information on how to rebuild your database.

I tested this command file on a version 5.5 database file in c:\sybase\ksl.db

...which is on a different drive because my D: drive is a DVD :)

REM This command file runs unload-reload-all-in-one-dbunload-step, 
REM then starts dbeng16 and dbisql so you can make sure the new 
REM database is up and running.

REM Be sure to stop *all* database engines on the computer you're using, 
REM before starting the upgrade.

REM dbunload...
REM -an ...  where to put new database
REM -ap ...  new database page size
REM -c ...   old database startup connection string
REM -o ...   where to put unload console display text file
REM -v       verbose mode for console display

PAUSE MAKE SURE YOU DO NOT HAVE ANY ENGINES RUNNING.

CD /D c:\sybase

"%SQLANY16%\bin64\dbunload.exe"^
  -an kslnew.db^
  -ap 4096^
  -c "DBF=ksl.db; UID=dba; PWD=sql;"^
  -o dbunload_ksl_messages.txt^
  -v

PAUSE ...dbunload/reload is done; start dbeng16

"%SQLANY16%\bin64\dbspawn.exe"^
  -f "%SQLANY16%\bin64\dbeng16.exe"^
  -o dbeng16_kslnew_messages.txt^
  kslnew.db 

PAUSE ...dbeng16 is running; start dbisql

"%SQLANY16%\bin64\dbisql.exe"^
  -c "ENG=kslnew; DBN=kslnew; UID=dba; PWD=sql;" 

PAUSE ...dbisql is running