nickva commented on issue #1002: Investigate possible race condition in replicator cluster stability check URL: https://github.com/apache/couchdb/issues/1002#issuecomment-345789213 User switched the order of cluster setup, but the issue was still there. After looking at this over the weekend noticed that it's possible there is a bug in 2.1.0 which could cause this behavior. The good news is it seems in 2.1.1 this bug was fixed inadvertently when the module was refactored. Notice the clustering module in 2.1.0: https://github.com/apache/couchdb/blob/2.1.0/src/couch_replicator/src/couch_replicator_clustering.erl#L211 `is_stable` calls ends up checking if the quiet time has expired by using `os:timestamp()` result. This means if `os:timestamp()` doesn't return monotonic results, (and in fact it is not guaranteed to) then this sequence of events is possible: * Cluster stability check fires https://github.com/apache/couchdb/blob/2.1.0/src/couch_replicator/src/couch_replicator_clustering.erl#L168 and notifies document processor that cluster is stable. * Document processor starts processing replication documents. On first update it calls `is_stable()`. However if `os:timestamp()` return a timestamp which is less than the previous one, the `is_stable()` call could return `false`. * Document update is skipped. * However cluster `stable` event already fired and there won't be another one until cluster membership changes and another quiet period passes afterward. I had a test script to check if os:timestamp() would move backwards, however I couldn't find any VMs or machines I had access to where this would happen. But it is entirely possible based on reading docs the behavior experience by the user. The good news is that this particular bug was inadvertently fixed when clustering module was refactored such that it has an explicit `stable` flag in the state: https://github.com/apache/couchdb/blob/master/src/couch_replicator/src/couch_replicator_clustering.erl#L73 This means that after notifier gen_event fires with a stable event, any calls to `is_stable` will read the state from that flag instead of going to `os:timestamp()` Contacted the user to ask them to try using the latest version 2.1.1. For now closing the issue again as it is fixed.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
