Here's what happens on the 32-bit version of SQL Anywhere Network Server Version 188.8.131.5298
SELECT COUNT_BIG(*) FROM SYSCOLUMN AS A CROSS JOIN SYSCOLUMN AS B CROSS JOIN SYSCOLUMN AS C; There was an error reading the results of the SQL statement. The displayed results may be incorrect or incomplete. Value SUM() out of range for destination SQLCODE=-158, ODBC 3 State="22003"
Hm, can't tell, that query really takes loooong for one of my test databases (too many columns?).
However, the following simpler query returns no error but a wrong result in dbisqlc and DBISQL 184.108.40.20669:
SELECT COUNT_BIG(*) FROM sa_rowgenerator(1, 2048) AS A CROSS JOIN sa_rowgenerator(1, 2048) AS B CROSS JOIN sa_rowgenerator(1, 2048) AS C;
It should return 2^11 * 2^11 * 2^11 = 2^33 = 8,589,934,592 but returns (after quite a while...) 0, which seems to be the result of an 2^32 overflow.
However, that query seems to be running one one thread, whereas your COUNT_BIG() query over SYSCOLUMN seems to be running parallel, which might make a difference as to the underlying query execution...