What my BOFHs observe daily:
Given the extremeness of the situation (100% CPU usage), I am hoping that someone may fill me in on what we could be doing that could possibly cause this situation. We have not been able to reproduce this yet nor have we discovered any infinite loops in our code. Any pointers are welcome.
I am basically a n00b when it comes to SQL Anywhere, so I have to read up on application profiling I guess.
I had to defer the troubleshooting to my colleague, as I was otherwise engaged.
Eventually he found a query that would run all CPUs flat out and never finish. He waited for 15 minutes before killing it.
Then he looked at the query plan. In his words "it was messy".
The solution was to add an index to one of the temporary tables used. That whacked the query execution plan back into shape and we are back in business. The query now takes 20 seconds to run.
Not sure this deserves to be marked as 'an answer', but my colleague's workaround will hopefully see us through.