|
Hello experts, I've faced this issue for a long time and I hope you guys could help me with this. Sometimes when the sync is cancelled in the app an error of upload occurs, with the "invalid sync sequence ID" description in the server. We've been using the SQLAnywhere 16 2127 and UltraliteJ for Android. The sync is canceled using the 'syncProgress' callback from 'SyncObserver' class. Looking for evidences in the server log, I've noticed that the transaction of the canceled sync is commited, but after a bit of time a log of 'connection drop' is logged. This error is a little bit hard to reproduce, but it affects hundred of users every week in my company. After that, I'm only able to sync again with this user if I start a new sync with a new database. Any suggestions of what could be happening? A new log with verbosity -vp A new log with download/upload scenario: And a very complex scenario which I couldn't figure out what could be happening? Maybe a simultaneos sync in the same device? |
|
This issue has now been fixed and will be available in future SPs containing 17.0.0 Build 1253 or later and 16.0.0 Build 2177 or later. The details of the problem that was addressed follows: If an UltraLite client crashed or was terminated in the middle of a download-only synchronization, it was possible for the client to enter a state where all subsequent synchronizations would fail with SQLE_UPLOAD_FAILED_AT_SERVER and the MobiLink log would report mismatched sequence IDs. Thanks by this fix! Are there any plans about when it will be released?
(31 Aug '15, 18:00)
Alex
An SP has been requested for this change and QA is scheduling that work. However, we are not yet able to provide a specific timeframe for that SP to be ready.
(02 Sep '15, 09:20)
Chris Keating
Chris, the error has happened again in other scenario (download/upload configuration). Could you check the third log of my question?
(08 Sep '15, 12:10)
Alex
Hi @Chris Keating, we could reproduce this error again, but in a slight different scenario. If the sync request that is canceled is executed in a different server than the second sync (that fails because there is a pending syncing already being executed) the "sequence id error" will occur. Could you take a look on it, please?
(06 Mar '17, 09:51)
Alex
|
|
This could happen if remote database IDs are not unique. I notice that the remote ID is '2' rather than a UUID (which is the default). Since you are using non-default remote IDs, I suspect this is the cause of the problem. Further evidence of non-unique remote IDs is found in the sequence IDs. Since they are V1 UUIDs, the last 12 hex digits are a "node id" (often the network interface MAC address); the consolidated sequence ID and previous remote sequence ID came from node id d4b26ffaac50, whereas the current sequence ID came from node id 9bbc92b92a77. 3
It is worth investigating whether the remote database IDs are unique. Just a clarification for UltraLite on Android. The node id's are not MAC addresses - they are generated based on system time and randomness among other things. A node id for a given device will very likely be different after an application restart. It will stay constant for the in-memory life cycle of the application though.
(26 Aug '15, 15:19)
Andy Quick
Replies hidden
The remote database IDs are unique, for sure. In fact, I could reproduce this scenario with one single device syncing in a environment with only one user. About the node id, I've restarted my application after the cancelation (in fact, the sync is canceled prior the application closing), which can be the cause of the generation of two different "node ids", as Andy explained. Maybe the Mobilink is accepting a new request before closing the resources from the last one which can be leading to a bug.
(26 Aug '15, 16:15)
Alex
Replies hidden
Thanks Andy. In other words, the Android device would have to have be rebooted, or the application closed, cleared from memory, and then restarted, for there to be two different "node ids" from the same remote. In this case, that would have to be between 11:06:50 and 11:10:54. If that didn't happen, then it seems two remote databases have the remote ID of '2'. Remote IDs must be unique!
(26 Aug '15, 16:19)
Graham Hurst
Did you see different "node ids" (in the sequence IDs) after restarting the application? Normally an Android app stays in memory after closing, which is what Andy meant by "in-memory life cycle of the application". (A different "node id" in sequence IDs isn't a problem for MobiLink. It just seemed like evidence of non-unique remote IDs.) A new request from the same remote ID kicks out an in-progress sync, but it may be that your application is not handling failed downloads correctly.
(26 Aug '15, 16:27)
Graham Hurst
Also, since you can reproduce the scenario, please provide the steps to reproduce and/or a MobiLink server log where -vp verbosity was specified.
(26 Aug '15, 17:25)
Graham Hurst
The application doesn't stay in memory because we force kill the process. I've made some tests and the node id do change if I restart the application. About the remote IDs, they are assigned at the moment of the first application run, server-side, stored in a identity database column. We don't have any evidences of duplicated remote IDs so far.
(26 Aug '15, 17:54)
Alex
Replies hidden
I'll try to reproduce again with a higher verbosity and I'll post the log. Thanks!
(26 Aug '15, 17:57)
Alex
About the download error handling, I'm not sure what it is. Since we cancel the sync using the Ultralite callbacks and, after the Ultralite return (which its thread has been locked during the synchronize method call) we kill the process. So, where am I supposed to handle the download failure if I want to abort the process?
(26 Aug '15, 18:06)
Alex
Replies hidden
How does the remote get the remote ID from server?
(26 Aug '15, 18:09)
Graham Hurst
You shouldn't need to kill the process to cancel the sync! I think you just need to return True from SyncObserver.syncProgress. These posts might help: A partial download should be automatically rolled back as long as you have not enabled the Keep partial download synchronization parameter. But killing the process may cause problems with rollback. The reason we want to see -vp verbosity is to make sure that your application is doing a "check status" sync after a failed download (if not keeping partial downloads) or a "resume" sync (if keeping partial downloads).
(26 Aug '15, 18:18)
Graham Hurst
I'm not killing the process because the sync. I'm canceling the sync to kill the process for other reasons. I've checked here and we don't use partial downloads, since we have such a complex download preparation logics which don't fit to this feature. I couldn't reproduce the error yet (it is hard to reproduce), but as soon as I do it I update the question with this information. Thanks again!
(27 Aug '15, 08:03)
Alex
My concern was that the force kill of the process may interfere with proper clean up of the cancelled sync, eg. if it is killed after UltraLite returns, but Android may still be flushing to persistent storage, such as an SD card. (BTW the box with an arrow on a comment is for replying to that comment.)
(27 Aug '15, 11:08)
Graham Hurst
Only if the Ultralite performs IO operations asynchronously, what I doubt. We kill the process only after the Ultralite fininshes the synchronize call. (BTW, tks for the tip, but not a suggestive icon anyway)
(27 Aug '15, 14:22)
Alex
Question edited with a new log with higher verbosity, as requested.
(27 Aug '15, 15:21)
Alex
Comment Text Removed
Correction: Starting with version 16, a new request from the same remote ID does not kick out an in-progress sync. (That was the behavior in an earlier version.)
(27 Aug '15, 17:23)
Graham Hurst
We have a retry logic for checking sync status, if the ODBC driver returns a timeout ODBC state when the MobiLink server was not able to lock the remote ID. This logic seems working well with a Microsoft SQL Server. However, we did not see this retry logic shown in your MobiLink server log file. So I was wondering if you could tell us what kind of consolidated database servers you are current using?
(10 Sep '15, 12:38)
Yufei Guo
Microsoft SQL Server 2014
(10 Sep '15, 14:45)
Alex
Thanks! Hopefully, there are no other problems, because Microsoft SQL server 2014 is not officially supported by the MobiLink Server 16.0. I have two more questions: 1) the ODBC driver version; and 2) The MobiLink server command line options (Please don't let me your -c setting). Could you please give us this information?
(10 Sep '15, 15:43)
Yufei Guo
ODBC DBMS Name: Microsoft SQL Server ODBC DBMS Version: 12.00.2000 ODBC DBMS Driver Version: 11.00.3000 ODBC Version supported by the driver: 3.80 -v -c dsn=XXX;uid=XXX;pwd=XXX -x http{port=###} -o "XXX\LogML.txt" -os 10000000 -w 10 -vp -zs SQLANYm_XXX_###
(10 Sep '15, 17:47)
Alex
It looks reasonable good and the ODBC driver is the same as the one we have recommended for MobiLink server version 17.0 to use.
(11 Sep '15, 16:20)
Yufei Guo
We are not able to reproduce the problem you have seen, because the MobiLink server always retried on status check with our tests until it was able to lock the remote ID. Could you please let us know your database options? You can get the options by issuing the following query against your SQL Server database: select * from sys.databases where name = 'your_sql_server_database_name' Please give us the query result set including the column names, so that we can compare your options with ours.
(11 Sep '15, 16:46)
Yufei Guo
The properties that were not here are null "compatibility_level": "120", "collation_name": "Latin1_General_CI_AS", "user_access": "0", "user_access_desc": "MULTI_USER", "is_read_only": "0", "is_auto_close_on": "0", "is_auto_shrink_on": "0", "state": "0", "state_desc": "ONLINE", "is_in_standby": "0", "is_cleanly_shutdown": "0", "is_supplemental_logging_enabled": "0", "snapshot_isolation_state": "0", "snapshot_isolation_state_desc": "OFF", "is_read_committed_snapshot_on": "0", "recovery_model": "3", "recovery_model_desc": "SIMPLE", "page_verify_option": "2", "page_verify_option_desc": "CHECKSUM", "is_auto_create_stats_on": "1", "is_auto_create_stats_incremental_on": "0", "is_auto_update_stats_on": "1", "is_auto_update_stats_async_on": "0", "is_ansi_null_default_on": "0", "is_ansi_nulls_on": "0", "is_ansi_padding_on": "0", "is_ansi_warnings_on": "0", "is_arithabort_on": "0", "is_concat_null_yields_null_on": "0", "is_numeric_roundabort_on": "0", "is_quoted_identifier_on": "0", "is_recursive_triggers_on": "0", "is_cursor_close_on_commit_on": "0", "is_local_cursor_default": "0", "is_fulltext_enabled": "1", "is_trustworthy_on": "0", "is_db_chaining_on": "0", "is_parameterization_forced": "0"
(11 Sep '15, 17:36)
Alex
1
Your database setting is slightly different than ours. However, we are still not able to reproduce this problem, even after we reconfigured our SQL Server database setting to match yours, the MobiLink server always keeps retry on sync status, when there is a problem on lock remote ID locking. However, we have found a problem within the MobiLink server, if the retry logic does not work for sync status check. This problem will be fixed as soon as we can. To workaround this problem, please try to reduce the sync frequency for individual remotes whose last sync status was unknown. By the way, what are the OS type and OS version you are currently running your MobiLink server?
(14 Sep '15, 15:00)
Yufei Guo
Windows Server 2012
(15 Sep '15, 13:38)
Alex
More comments hidden
|
|
Hi, in the meantime , the patch that has a fix for the download_only scenario was released and can be downloaded from sap. 2
There were 2 changes made related to Invalid Sync Sequence - one was related to the download_only case and the other change was related to the non download_only behaviour reported. The second fix is Engineering Case# 789932.
(20 Oct '15, 10:10)
Chris Keating
Replies hidden
Are these two fixes in the same patch? Also, could you please inform me the build number of this patch? Tks a lot by the fixes =]
(23 Oct '15, 08:22)
Alex
Both fixes are in that SP. The second fix is Engineering Case#788991. Recall, that was the 'download-only' case.
(23 Oct '15, 09:57)
Chris Keating
Replies hidden
Got it. I downloaded the build 2193 which contains these fixes. As soon as I install and test it in the production environments I let you know. Tks a lot!
(23 Oct '15, 12:52)
Alex
Unfortunatelly the patch didn't solve the issue. The sequence id errors continue to happen. As soon as I have the logs I update the question with more informations.
(08 Dec '15, 06:13)
Alex
1
Hm, given the many nested comments here, it seems quite difficult to follow the discussion - I'd recommend to ask a separate question with your issue even if it's still the same cause... - possibly with a backlink to this one. Just my 2 cents, anyway:)
(08 Dec '15, 06:20)
Volker Barth
I too am going to suggest that a new thread be opened as this one is getting deeply nested. The software update addresses the conditions in which your sync environment can get into this state. It does not fix broken remotes. If you are seeing this problem after applying this update starting with healthy remotes, please run with -vpe level logging at minimum and capture in that log a remote that transitions from working to broken.
(10 Dec '15, 08:23)
Chris Keating
|
|
I've reproduced an scenario which leads to this issue (and I hope all the others derives from this one, for the sake of simplicity =p) Steps to reproduce (using an Android device with UltraliteJ 16 2127): 1) Synchronize the client 2) As soon as the message "COMMIT Transaction: Download" is logged on the Mobilink log, close the application 3) Start the application again the synchronize the client. The error will occurs. Note that the first synchronization hangs on the connection until the timeout of this connection is reached. I've tried different sync parameters, even enabling/disabling the download ack, but without success. Can I consider this issue as a bug? As I feared, for a download-only sync killing the app before UltraLite has finished a sync may leave the UltraLite database in a bad state. As a workaround, try disabling the Download Only sync parameter in your Android application. To avoid any changes from remotes being applied to the consolidated database, for each synchronized table define
(27 Aug '15, 16:41)
Graham Hurst
Replies hidden
We are not using the download-only parameter. In this case, the Ultralite is skipping the upload because there is no data to upload. So, this workaround is not valid for us. Any other possible workaround?
(27 Aug '15, 17:11)
Alex
2
Another developer double checked the MobiLink server source code, and you would have upload commit messages in the log if your app is doing bidirectional syncs. Look for calls to BTW, instead of recreating a remote after the problem hits, you should be able to get that remote syncing again by using the
(27 Aug '15, 17:36)
Graham Hurst
Sorry Graham, you were right. In my reproduction I was using the download-only configuration. With download/upload syncs I couldn't reproduce this issue. I've already known this procedure to reset the sync state, tks. We've been using it for months. This sequence ID issue could be reproduced more frequently in old versions of Mobilink. Subsequentials canceled syncs could lead to this issue. But we've got rid of a part of these issues (about canceled/restarted syncs) after migrating to the 2127 patch. We've still some users dealing with this sequence ID error, as reported in other questions, like the following one. I hope you guys have a better luck looking into this =] http://scn.sap.com/message/15859094#15859094
(27 Aug '15, 18:09)
Alex
Graham, the sequence ID error has happened again with a dozen of users in the last week. All the users had performed download/upload operations, instead of the download-only scenario that I've reproduced. Could you check the third code of my question to analyze it? I couldn't reproduce this scenario, but checking the logs, it seems the Ultralite is accepting a new sync request before finishing the last operation (that failed with connectivity issues).
(08 Sep '15, 12:15)
Alex
|
I don't see a COMMIT for upload. Does your application set the Download Only synchronization parameter?
We're investigating if there is a potential problem specific to cancelling download-only synchronizations. As an experiment or potential workaround, you could try disabling the Download Only sync parameter in your Android application. (If there are any changes on the remote to downloaded tables, you may also need to define ignored
upload_insertandupload_updatescripts.)There is some scenarios (not in general) where the download-only can be used. But, unfortunatelly, we cannot disable that to avoid data inconsistent (not allowed upload of data which may colide with others). I'll include this scenario in my tests. Tks.
EDIT: The sequence ID error has happened again in the production servers in the last week, as shown in the third log of my question. A little brief of one of the POSSIBLE user-case scenario (not reproduced!): the user is syncing, then the application is closed, which closes the app process too. Then (and whenever the app process is killed) our sync engine is reinitialized, which may start a new sync immediatly (if the previous one failed). Analysing the logs, it seems the first connection is lost (the process was killed, then the connection became orphan). And when a new sync is started, the Ultralite accepts it without finishing the orphan connection, which leads to the sequence ID error. Could you check the code about this scenario? Please, take in consideration that the process is killed and restarted. Maybe (and makes sense, due the other comments about the sync id and process relation) this can be the root of all the problems.
What is this connection ID (SPID)? Is it a server-side or client-side information? And what can be happening so that the log has this information of same SPID for different syncs?
EDIT 2: Added a very complex scenario in the 4th log
The connection ID identifies which MobiLink database connection is being used out of the MobiLink server's pool of database connections to the consolidated database, as returned from consolidated database.
The third log shows 3 concurrent syncs! The two things that make me uncomfortable are:
1. Killing the app with every sync cancellation.
2. Retrying sync after less time than the
timeouttime.Maybe the sync is not being canceled, because the process can be killed if the user swype the app from the "recent apps" grid. Perhaps it can be a issue? About the retry, maybe we can implement a mechanism to wait the minimum time from Mobilink at the sync engine startup, but we cannot ignore the user's request, at a button click, for example. That is the way this workaround couldn't be implemented.
How have you concluded about the concurrent syncs? When a sync is finished? There is a gap between the "End synchronization" message and the sync end that I could not understand.
The "COMMIT Transaction: End synchronization" message indicates that sync has finished with it's consolidated database connection. However the ML server worker thread from that sync still has a network connection to the remote, and is trying to send the download. Sync 1361 doesn't end till the network timeout 229 seconds later (238s after starting to send download).
If you want the network timeout to be shorter, use
SyncParms.setLivenessTimeout.If you set that to 30 seconds (the minimum recommended) and have your app wait 30 seconds on startup before enabling the sync function, then at least cancelled/killed syncs could complete before new sync requests from the same remote ID.
In case you weren't aware of it, the MobiLink Performance whitepaper provides an overview of what happens during a synchronization. The description on pages 3-6 of MobiLink 12 Performance applies to version 16 and 17.
We have already set this timeout in the Production server and avoided the sync engine start until this time (30s) is reached. However we cannot ignore the explicity user request. How should we proceed? Is there any other workaround? It is safe to assume that this is a ML issue or we are doing something wrong in the client side?
I didn't know this documentation. Tks a lot =)
Since your log showed network timeout taking the 238s and the default for Android is 240s, I assumed you were using the default.
Are those logs not from production? If not, why did you stop using -vp verbosity?
This problem could be happening if Android returns success when UltraLite commits, before writing to persistent storage is completed, and the killing of the app prevents persisting of the commit.
If the app has been cleared from memory by user clearing it away, why not show a busy animation and not enable the Sync button for 30s on initial startup?
(My Android phone has at least one app that takes a minute to start up after clearing from memory.)
We have a dozen of environments using ML 16. And almost all of them is facing this sequence ID error. The verbosity that I've turned on was in another environment (changing production configurations is not so simple, involves service unavailability and other headaches). I can wait the 30s (the app startup takes 5secs only) but let's agree that this is not the best solution for the user POV. All I'd like to know is if someone is going to investigate this problem and treat it like a bug. If so, we can wait for a better solution.
We are going to treat this as a bug but we will need to be able to reproduce it or gather sufficient logging information to understand the cause.
If your app takes 5 seconds to startup, why is there only 1 second between sync requests 1362 and 1363?
I still suspect duplicate remote IDs, since that fits the symptoms. When other customers have reported this problem, that has ultimately been determined to be the cause.
You said that remote IDs are assigned from a sequence in the consolidated. Please detail how that gets back to the remote device before that device's first MobiLink synchronization occurs.
A possible experiment would be to try using default remote IDs, and see if the problem still happens. (If you want a non-GUID identifier for each remote device, then use the synchronization user instead of the remote ID).
We assign the remote id during the device provision (first app startup in the device), performing the first authentication, creating the remote id (identity SQLServer column) and assigning to the database. So, a pair of user-device is strict related with only one database and vice-versa. There is no way in which duplicated remote ids perform syncs because our sync logic relies on it. Our product do allow the provision of two or more devices for the same user, however each pair of user/device has a different remote id configuration.
In a very simplified representation of our model we have: Entity: mobileDB Properties: idMobileDB (identity), idUser (get from login), deviceIdentifier (get from the device serial number)
All the sequence IDs seems to be related with some connection loss. The reason why the sync starts after 1 sec is explained in the use-case at the beggining of this comment thread (just after the process is killed, the sync engine is automatically initialized again - just the service, not the UI -, which may start another sync immediatly without user interaction).
For the same reason I've talked in the first part of this comment, we cannot change the way the remote ids are assigned (changing back to its default) because we use the remote id within the user id to make relations between entities during the sync download/upload (different data may be downloaded depending on which device an user is syncing or two devices of the same user performing data download). The only scenario in which two mobile DBs can sync at the same time is two device related with the same user containing the same serial number. And I've already checked it and it is not the case.
I'll work on a reproduction as I've made with the download-only scenario. Then, I let you know. Meanwhile, I hope you good luck with some insight about this issue =p
Tks, in advance.
I was puzzled by 1363 starting 1 second after 1362 because you said a few comments ago that your sync engine waits 30 seconds before retrying, but now you say the sync engine retries right away.
I've implemented this waiting in one production environment, but after the log retrieving. I have no logs after this change yet. As soon as I have it I let you know.
A quick update: in the production environment where the sync is restarted after 1 sec an iOS client is executing the sync. I'll check with the iOS team about any potential bug in which concurrent syncs could be performed.