|
We have an application which uses dbodbc12.dll --- and it stays in processes after the entire exe has finished. The exe only stays in processes when dbodbc12.dll is used. I have Process Explorer attached to it - but to me, it's just garbage information and I don't really know what it is doing. Two threads I see are: wow64cpu.dll!TurboDispatchJumpAddressEnd+0x536 wow64cpu.dll!TurboDispatchJumpAddressEnd+0x17b wow64.dll!Wow64SystemServiceEx+0x26a wow64.dll!Wow64LdrpInitialize+0x435 ntdll.dll!wcscspn+0x7f ntdll.dll!LdrInitializeThunk+0xe ntdll.dll!NtFsControlFile+0xc dbodbc12.dll!DllRegisterServer+0x25678 and wow64cpu.dll!TurboDispatchJumpAddressEnd+0x536 wow64cpu.dll!TurboDispatchJumpAddressEnd+0x3a5 wow64.dll!Wow64SystemServiceEx+0x26a wow64.dll!Wow64LdrpInitialize+0x435 ntdll.dll!wcscspn+0x7f ntdll.dll!LdrInitializeThunk+0xe ntdll.dll!NtWaitForSingleObject+0xc ntdll.dll!wcsspn+0x17a ntdll.dll!wcsspn+0x196 ntdll.dll!RtlFreeUnicodeString+0xa9 ntdll.dll!RtlExitUserThread+0x30 combase.dll!CleanupTlsOleState+0x7bc ntdll.dll!RtlInitializeExceptionChain+0x85 ntdll.dll!RtlInitializeExceptionChain+0x58 My guess is that it is in GPF mode, but it's a background process and doesn't run interactively. I wouldn't know where to begin to troubleshoot but going to Sybase/SAP support for something at this will most likely be painful. We are an OEM for Sybase SQL Anywhere --- but our REP has gone away - doesn't answer anything since SAP took over. It's been pretty painful since SAP took over Sybase --- it seems like no one really cares about the product anymore. But, that's another story - we just need this fixed and will pay for it but I want to talk to a high level developer to work directly on this issue and not go through Level 1 support. Help? |
|
This problem is related to a known "Loader Lock" problem with DllMain, which can be caused by improperly written wrapper DLLs calling functions during DLL load/unload times. See the SQL Anywhere documentation and a related Microsoft document for the underlying details. |
You could try it with an older or newer odbc client (e.g. dbodbc11.dll) if the problem persists its unlikely that it has anything to do with the Sybase odbc client
Hi Robert,
What is the full version number of the DLL (including build number)?
Does the driver otherwise function correctly in the application?
Does the problem occur with connection pooling disabled (CPOOL=NO)?
Is this a deployment or a full installation?
Is there a database file being auto-started by the ODBC driver? If you add a -o switch to this database, does the log show the database stopping successfully?
The stack traces from process explorer are showing the WOW64 translation layer that allows 32bit applications run on 64bit Windows. Unfortunately, these don't explain why the DLL is not unloading. If necessary, we could use procdump to obtain a minidump of the process (which will capture the actual 32-bit executable).
Apologies for the confusion regrading your representative. If you call 1-800-8SYBASE and ask for the sales department they should be able to help you reach the representative assigned to you.
Thanks, Mikel
I believe that the dbodbc12 dll does not do anything special to handle GPFs, so I doubt this is GPF mode.
How do you know that "the entire exe has finished" (are you sure either exit() was called or main finished)?
Why do you think this is "in GPF mode"?
What OS is this running on?
What version of SQL Anywhere is this?
Have you tried other versions of SQL Anywhere? If so did they have the same behavour?
Do you know what language is the application written in? (for example C, .NET, Java, etc.)
Is that application running as a service?
This DLL entry point is generally only called when actually registering the ODBC driver with Microsoft Windows - e.g. by
regsvr32. It generally should not be called in the course of your ODBC application, as it is not part of the Standard ODBC API. Problems will likely be related to not having Administrator permissions from your application to actually install the ODBC driver during runtime.Is it possible that your ODBC wrapper (Clarion, IIRC?) might be trying to auto-register the ODBC driver during your program run-time?
Jeff, I don't think this is the problem. The symbol is actually
dbodbc12.dll!DllRegisterServer+0x25678
Without debugging symbols, only names and addresses for exported functions are available for the stack trace. Given that the offset is very large (much larger then the length of a typical function), I'm pretty sure the thread is not actually in DllRegisterServer.
Hello,
Lot of questions and lot of answers here.
Jeff, I sent email to Eric Ocampo to open a LEVEL 2 support incident directly with you. We are more than happy to pay for the direct support on this to resolve the issue - as it is causing issues on our production server and we need it resolved.
I can get all the information required here - and/or I can get you a ZIP file with the exact tests so you can bring it to development. I have all the required files to duplicate.
Yes, the issue is only with odbc12.dll - I've tested with odbc10.dll and the issue is not there.
We duplicate the issue on all op systems (WIN7, WIN2008 Server, WIN2012 Server) - and the issue happens 100% of the time.
Can we please open a direct dialog via email - and open a support case directly with your development team right away. Please contact Eric Ocampo for our OEM contract and what is needed to do this. Again, paying for support is not an issue for me - I just need this resolved.
We have contacted Robert about his current support options separately.
We can also continue working through the forum as we always do, and only require your active input and work on the issue (and the answer to each of the above technical questions). As a reminder though, this forum is not a priority service and is the work of many volunteers.
I would also add that from a technical side, gathering an ODBC trace would also be useful in this situation and trying the latest 12.0.1.3994 EBF. If you can find threads with environments or objects that have been left open, that may lead to further clues to where the problem may be.
Hello,
The latest EBF is being used. I do know that if I use an older version of CLARION (6,7 and 8) there is no issue. The issue only appears once we go to CLARION 9. It has always never been an ODBC issue with Sybase until using the latest version of Clarion - however, the President of Soft Velocity (Clarion) has indicated the issue is on the ODBC / Sybase side and not on their side.
I can give/get a "MINI DUMP" or a "FULL DUMP" - both are large files and I can get those files (TEXT) via an FTP site or HTTP site.
This is the main thread dump:
The "MINI DUMP" is 5.3 meg and can be found here:
www.innquest.com/IQVarWeb.dmp
It may be very important to know the exact last version and build of SQL Anywhere you used prior to testing with 12.0.1! Especially when it is implied that worked.
The dump provided doesn't provide much information. It is a 64bit dump of a 32bit application running in a WOW64 vm and !not one! of the threads have an 32bit context. That would tend to support a theory that none of the 32bit libraries are even involved. It would almost appear the wow64 NTVM is actually the hung process.
How does it behave on a pure 32bit OS for you?
Further, as to the "Main thread dump" shown, most of the byte offsets are so big, you will not be able to trust any of the symbols shown. {those are just offsets relative to the 'nearest' Exported functions and not very diagnostic)
It does appear one can maybe trust this entry FreeLibrary+0x19 {and the fact that it seems reasonable that would be part of LdrUnloadDll) so the question becomes what library is being unloaded there ( x86 bin32DBODBC12.dll 12.0.1 build# 3867? possibly ) but not much else of value there.
Sooo .... . . . only as pure guess ... and IFF it is known that you are unloading our library at that point . . .
. . . you may be seeing a deadlock with the library loader "Loader Lock" here . . .
Unfortunately you won't be able to diagnose this any deeper using the current approach.
If Clarion has a 64bit binary version start with that instead. Otherwise, if you only have a 32bit binary available, find a 32bit OS and . . . Then verify the pattern exists there and get Soft Velocity/Clarion to do a detail debugging starting with that ...
That was all done on the 64-bit machine. Would it help if I have you all the call stacks and dumps on a pure 32-bit Win7 machine?
Can I give you a GotoMeeting directly to this hung process on a 32-bit machine?
The app and compilers are only 32-bit.
I don't believe so. As mentioned several times in the thread, your stack symbols for us won't line up as you don't have our debugging symbols, and we don't have yours.
No, you may not. Direct support of this nature is only available through an SAP support plan and we have discussed this option with you directly. If you are having any problems completing this engagement, please contact myself directly.
With a support plan, SAP employees are forbidden to connect to remote customer systems without using approved SAP remote connection methods. (See "SAProuter": http://help.sap.com/saphelp_nw70/helpdata/en/4f/992d39446d11d189700000e8322d00/content.htm ). "GoToMeeting" and similar methods cannot be used as they may be insecure, do not allow for auditing, and are not controlled.
We at SAP need to ensure that all remote connections from SAP support to customer systems are logged, permissible only by the customer, have the ability to be audited, and are secure.
Also, we haven't actually proven an issue with the ODBC driver yet - only that your wrapper program causes the 'hang' in the driver and changing the wrapper version, changes the behaviour. As we are continuing support via this forum currently, can you write a simple C ODBC application using the ODBC API, loading the DLL in the same way the wrapper does, and post it here to demonstrate the same issue? This would allow us to provide you with an answer here without the requirement to debug your entire application with the ODBC wrapper via remote sessions and dump files.
Yesterday, Eric (MY OEM REP said this:
I’ve connected with Jeff and am discussing what options are available. Eric Ocampo SAP BusinessObjects OEM T +1 604 647-8536 M +1 604 365-6318
I've heard nothing yet.
I've created a full ZIP file as you know that is small lines of code - it is not our entire application and it is a very small install program that shows the behavior.
This is my issue. I have an issue and I give you guys over $100K in sales a year, and it's like I'm behind a wall when I need help - once a year, if that. You want my $100K in money, but I'm not allowed to get DIRECT SUPPORT from you. I offered to pay you money or whatever, but I'm still here behind a wall trying to get this resolved. You guys want my $100K a year, but you can't talk voice to me for some reason. It's totally ridiculous. Maybe I should hold my next payment until someone decides to talk voice to me. I have a production issue here.
Sure the issue may not be on your end - but I need proof of that because the SV is saying it's your issue. I'm in the middle. I didn't write the ODBC driver and I didn't write the compiler.
Sorry, but I get so aggravated with you guys. This is the same complaint I had with Tom Maloney and Barb Penny and now Eric. All your sales people seem to want to accept our $20K and $30K sales each quarter, but the one time we need help - it's like go use the forum and go back and forth.
You don't understand we have sold over a thousand of your licenses but when it comes to the ONE TIME A YEAR (IF THAT) that I need help from you guys, it's like pulling teeth.
While you are all trying to be helpful here, this is a production issue and you need to support your customers - other than just behind a forum.
You need to provide a mechanism for general support - we do with a standalone PC for our customers. If we have an issue, we put the demo on our standalone box and we duplicate it - we go through PCI PA/DSS audits ourselves.
I can't write a simple C ODBC application because I don't know C. But, I do know that I have written a Clarion application to dynamically load the DLL, connect into the ODBC and it works fine. The issue is somewhere else and I do not know where. The vendor (Soft Velocity) is pointing fingers at you ... and yes, I understand I can use any of their runtimes from V6 to V7 to V8 without issue, and then when I get V9, it's an issue....but I'm the middle person here.
Bottom line, I can't sell your product when there is an issue and I need support to correct the issue in order to make more sales for you. As I said, I can pay for the support and I need to stop fooling around in this forum and get some real support. Otherwise, have Eric phone me directly and if you can't provide support, I look at my other options.
This is a business, we are a business - and we can't be fooling around in this forum or being told that you can't look at our application or test binaries. You surely can on a standalone PC / test box. But, we are not going to send you any more money until you help me resolve this issue - and you can let Eric know that too.