[
https://issues.apache.org/jira/browse/ZOOKEEPER-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149783#comment-17149783
]
Filip Niksic edited comment on ZOOKEEPER-2832 at 7/2/20, 3:12 AM:
------------------------------------------------------------------
Any updates on this? I noticed that the test in the attached patch doesn't
reproduce the bug in versions 3.4.10 and above. However, a slight variation
where we start servers B and C in step 2, and A and C in step 3, reproduces the
bug in version 3.5.8. -See the attached patch for this test.- (EDIT: It seems
that I can't attach a file to a comment.)
BTW, out of curiosity, why does the test explicitly call the garbage collector
after setting data?
was (Author: fniksic):
Any updates on this? I noticed that the test in the attached patch doesn't
reproduce the bug in versions 3.4.10 and above. However, a slight variation
where we start servers B and C in step 2, and A and C in step 3, reproduces the
bug in version 3.5.8. See the attached patch for this test.
BTW, out of curiosity, why does the test explicitly call the garbage collector
after setting data?
> Data Inconsistency occurs if follower has uncommitted transaction in the log
> while synchronizing with the leader that has the lower last processed zxid
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-2832
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2832
> Project: ZooKeeper
> Issue Type: Bug
> Components: quorum
> Affects Versions: 3.4.9
> Reporter: Beom Heyn Kim
> Priority: Major
> Fix For: 3.4.10
>
> Attachments: zookeeper-2832.patch
>
>
> Synchronization code may fail to truncate an uncommitted transaction in the
> follower’s transaction log. Here is a scenario:
>
> Initial condition:
> Start the ensemble with three nodes A, B and C with C being the leader
> The current epoch is 1
> For simplicity of the example, let’s say zxid is a two digit number, with
> epoch being the first digit
> Create two znodes ‘key0’ and ‘key1’ whose value is ‘0’ and ‘1’, respectively
> The zxid is 12 -- 11 for creating key0 and 12 for creating key1. (For
> simplicity of the example, the zxid gets increased only by transactions
> directly changing the data of znodes.)
> All the nodes have seen the change 12 and have persistently logged it
> Shut down all
>
> Step 1
> Start Node A and B. Epoch becomes 2. Then, a request, setData(key0, 1000),
> with zxid 21 is issued. The leader B writes it to the log but Node A is
> shutdown before writing it to the log. Then, the leader B is also shut down.
> The change 21 is applied only to B but not to A or C.
>
> Step 2
> Start Node A and C. Epoch becomes 3. Node A has the higher zxid than Node C
> (i.e. 20 > 12). So, Node A becomes the leader. Yet, the last processed zxid
> is 12 for both Node A and C. So, they are in sync already. Node A sends an
> empty DIFF to Node C. Node C takes a snapshot and creates snapshot.12. Then,
> A and C are shut down. Now, C has the higher zxid than Node B.
>
> Step 3
> Start Node B and C. Epoch becomes 4. Node C has the higher zxid than Node B
> (i.e. 30 > 21). So, Node C becomes the leader. Node B and C has the different
> last processed zxid (i.e. 21 vs 12), and the LinkedList object ‘proposals’ is
> empty. Thus, Node C sends SNAP to Node B. Node B takes a clean snapshot and
> creates snapshot.12 as the zxid 12 is the last processed zxid of the leader
> C. (Note the newly created snapshot on B is assigned the lower zxid then the
> change 21 in the log). Then, the request, setData(key1, 1001), with zxid 41
> is issued. Both B and C apply the change 41 into their logs. (Note that now B
> and C have the same last processed zxid) Then, B and C are shut down.
>
> Step 4
> Start Node B and C. Epoch becomes 5. Node B and C use their local log and
> snapshot files to restore their in-memory data tree. Node B has 1000 as the
> value of key0, because it’s latest valid snapshot is snapshot.12 and there
> was a later transaction with zxid 21 in its log. Yet, Node C has 0 as the
> value of key0, because the change 21 was never written on C. Node C is the
> leader. Node B and C have the same last processed zxid, i.e. 41. So, they are
> considered to be in sync already, and Node C sends an empty DIFF to Node B.
> So, the synchronization completes with the initially restored in-memory data
> tree on B and C.
>
> Problem
> The value of key0 on B is 1000, while the value of the key0 on Node C is 0.
> The LearnerHandler.run on C at Step 3, never sends TRUNC but just SNAP.
> So, the change 21 was never truncated on B. Also, at step 4, since B uses the
> snapshot of the lower zxid to restore its in-memory data tree, the change 21
> could get into the data tree. Then, the leader C at the step 4 did not send
> SNAP, because the change 41 made to both B and C makes the leader C think the
> B and C are already in sync. Thus, data inconsistency occurs.
>
> The attached test case can deterministically reproduce the bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
