As suggested in this post http://sqlanywhere.blogspot.nl/2011/08/new-maxbps768-set-maxquerytasks-1.html you could try to set the option MAX_QUERY_TASKS to 1

Can you please attach the execution plan of your query? Or two plans even better - old and new.

Christian might be correct, but I would never use this option until I check the execution plan first.

Should "inner join C on A.id = B.id" really be coded "inner join C on C.id = B.id"?

Do any indexes exist on TableA.id, A.id, B.id, C.id, D.id, E.id, G.id and H.id?

If those indexes exist in SQL Anywhere 12 but not in 16, it might make a difference.

SELECT COUNT(id) 
  FROM TableA
 WHERE ColumnA = 0 
   AND ( (    ID in ( SELECT A.Id 
                        FROM A
                             INNER JOIN B 
                                     ON A.id = B.id 
                             inner join C 
                                     on A.id = B.id -- CODING ERROR?
                       WHERE C.ColumnA = 0
                         AND B.ColumnA = 0 ) 
           OR ID in ( SELECT D.Id 
                        FROM D
                             INNER JOIN E 
                                     ON D.id = E.id 
                             inner join C 
                                     on C.id = E.id 
                       WHERE C.ColumnA = 0
                         AND D.ColumnA = 0 ) 
           OR ID in ( SELECT G.Id 
                        FROM G
                             INNER JOIN H 
                                     ON H.id = G.id 
                             inner join C 
                                     ON C.id = H.id 
                       WHERE C.ColumnA = 0
                         AND H.ColumnA = 0 )
       ) );