Hi folks, Following up from the meeting yesterday, I said I would look into the code to verify the behaviour because there could be a correctness problem.
I think there could be an issue. The code is convoluted, but my understanding of it is as follows. We check all ledger, journal and index directories for a cookie. If it doesn't exist, it gets added to a missingCookieDirs list. We then iterate over this directory. If any directory in missingCookieDirs isn't listed as a ledger directory in the journal dir cookies, or isn't empty, we fail to start. The issue is that a journal dir could be emptied and we wouldn't detect it. It would be great if someone else could eyeball the code and tell me I'm wrong. The code is in Bookie#checkEnvironment. This breaks correctness. Imagine we have a ledger on b1, b2, b3. Writer w1 is writing to the ledger. The state of the ledger on the bookies is: b1: e0 Fenced: false, LAC: - b2: e0 Fenced: false, LAC: - b3: e0 Fenced: false, LAC: - w1 gets partitioned from network. w2 tries to recover the ledger, it tries to fence on all bookies. The message to b3 gets lost. b1 and b2 acknowledge the fencing, so w2 continues to recover and close the ledger with e0 as the last entry. b1: e0 Fenced: true, LAC: e0 b2: e0 Fenced: true, LAC: e0 b3: e0 Fenced: false, LAC: - If w1 became unpartitioned at this point, it wouldn't be able to add a new entry to the ledger as any quorum would see fenced on b1 or b2. However, imagine that the fenced message is only in the journal on b2, b2 crashes, something wipes the journal directory and then b2 comes back up. The new state of the ledger on the bookies will be. b1: e0 Fenced: true, LAC: e0 b2: e0 Fenced: false, LAC: - b3: e0 Fenced: false, LAC: - Now w1 can write a new entry, e1, and b2 & b3 would both acknowledge it, even though the end of the ledger is e0. It requires many planets to be aligned for it to harm us, but we must fix this. Regards, Ivan
