Hello,

we have two servers with ASA 11.0.1.2879 installed. One is a debian 6.0.7, the other a ubuntu 12.04 LTS.

Beside the ASA dbserver (64bit) we run 1-2 tomcat 7.0.27 (64bit too) instances which connect to the SQL Anywhere via the sajdb.jar interface.

On both servers we have every 1-2 months the message that the system is running out of SYSV semaphores

Failed to obtain DB connection from data source 'myDS': java.sql.SQLException: 
Could not retrieve datasource via JNDI url 'java:comp/env/jdbc/dbname' java.sql.SQLException: 
[Sybase][ODBC Driver][SQL Anywhere]Connection error: Insufficient system resources -
failed to allocate a SYSV semaphore
[See nested exception: java.sql.SQLException: Could not retrieve datasource via JNDI url
'java:comp/env/jdbc/dbname' 
java.sql.SQLException: [Sybase][ODBC Driver][SQL Anywhere]Connection error: 
Insufficient system resources - failed to allocate a SYSV semaphore]

Is there a way to know which process is eating up the SYSV semaphores ? Once this occurs we have to reboot the server, nothing else we tryed did help

asked 31 May '13, 09:37

ASchild's gravatar image

ASchild
777222740
accept rate: 14%


SQL Anywhere client libraries and servers use SYSV semaphores on most Unix platforms (Solaris is an exception). The semaphores are used to perform signalling actions across client and server processes.

There are a number of ways that semaphore can be 'leaked':

  • a process is killed using a signal that does not allow the process to 'clean up'. E.g. kill -9
  • a client application (or associated library) installs a signal handler after SA's client library has installed its handler, and the application's signal handler does not properly chain execution to SA's handler when a signal occurs... and then the client crashes with the net result that SA's client does not get a chance to cleanup.
  • a client application (or any loaded library) causes the program to halt in such a way that the library 'unload' handlers do not get executed. One known issue is if the client calls _exit() instead of exit().

There are possibly other ways that semaphores can be leaked but the above are the most common tha twe have seen. For example:

  • Apache is known to kill child processes using SIGKILL (kill -9) and therefore semaphores can be leaked and thus cause a problem over systems that run non-stop for several weeks or months.
  • We have seen some Java applications / scenarios where the the JRE (or related code) will use _exit() to terminate itself thus causing semaphores to leak if the SA JDBC was being used. (I can't remember the exact details?)
  • We have seen some third party libraries that customers have used that have installed signal handlers and these handlers were not written in such a way as to properly chain signal handlers.

I am not that familiar with Tomcat so I do not know if any of the above applies to your situation. If you are running Tomcat within an Apache process then the first known issue could be what is causing your semaphore leak.

FWIW: We have had discussions within SA engineering about possible method(s) of working around the above known issues. Many times we are able to work with the application developer(s)/customers to resolve the issue by having their application modified (e.g. correct their signal handlers to chain handlers). Sometimes there isn't any better solution other than to ensure that the semaphores periodically get cleaned up by a process that monitors the number of semaphores in use and then performs a clean up process when the limit is near. I.e. To clean up: (a) stop the Apache/httpd server, (b) delete all semaphores owned by the user that run Apache, and then (c) restart the Apache/httpd server.

permanent link

answered 31 May '13, 14:54

Mark%20Culp's gravatar image

Mark Culp
23.2k9132272
accept rate: 40%

We use the tanuki service wrapper for tomcat. The instances are terminated each night and restarted.

I have the feeling that internally it uses something like kill -s 9 when the jvm does not terminate in a "timely" fashion... I will dig deeper in that direction. Thanks

(31 May '13, 16:53) ASchild

Do you have any processes that are crashing? Those are the leading candidates for leaking SYSV semaphores.

You can check ipcs -p to get a list of the currently allocated semaphores on the system, and see which username / process id has them allocated. You can use ipcrm -s semid to remove unneeded semaphores on the system to try and avoid rebooting.

permanent link

answered 31 May '13, 10:00

Jeff%20Albion's gravatar image

Jeff Albion
10.7k171174
accept rate: 24%

1

...and the forum admins themselves might have further clues - cf. the maintenance log...

(31 May '13, 10:47) Volker Barth

ipcs -p does not give usefull info. Better: http://unix.stackexchange.com/questions/75547/how-to-get-proccesses-currently-running-semaphores-by-proc

Looks like it's the java process which causes the semaphores to be "lost". But which part of the jvm is then the question

(31 May '13, 11:24) ASchild
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×61
×22

question asked: 31 May '13, 09:37

question was seen: 5,312 times

last updated: 31 May '13, 16:53