smengcl commented on code in PR #11190:
URL: https://github.com/apache/ozone/pull/11190#discussion_r4239855036
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -796,4 +833,53 @@ public void awaitDoubleBufferFlush() throws
InterruptedException {
public OzoneManagerDoubleBuffer getOzoneManagerDoubleBuffer() {
return ozoneManagerDoubleBuffer;
}
+
+ private void pauseApplyTransaction() {
+ synchronized (applyTransactionMonitor) {
+ if (applyTransactionPaused) {
+ return;
+ }
+ applyTransactionPaused = true;
+ boolean interrupted = false;
+ while (inFlightApplyTransactions > 0) {
+ try {
+ applyTransactionMonitor.wait();
+ } catch (InterruptedException ex) {
+ LOG.warn("Interrupted while waiting for in-flight apply transactions
to complete.");
+ interrupted = true;
+ }
+ }
+ if (interrupted) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+
+ private void resumeApplyTransaction() {
+ synchronized (applyTransactionMonitor) {
+ if (!applyTransactionPaused) {
+ return;
+ }
+ applyTransactionPaused = false;
+ applyTransactionMonitor.notifyAll();
+ }
+ }
+
+ private void enterApplyTransaction() throws InterruptedException {
+ synchronized (applyTransactionMonitor) {
+ while (applyTransactionPaused) {
+ applyTransactionMonitor.wait();
Review Comment:
I reproduced the rollback case on a three OM mini cluster (test:
https://gist.github.com/smengcl/b7c7499be16f3310693df7fca90b41fa ; it calls
installCheckpoint directly, holds the follower's flush with the double buffer
pause hook and fails the om.db backup move with the checkpoint backup
injector). Without this PR, a write that was applied but not yet flushed at the
pause is missing after the old DB is restored, and once that OM becomes leader
getKey returns KEY_NOT_FOUND; with this PR the test passes, because pause() now
flushes applied entries first. Entries applied after the pause are still
skipped after a rollback, as noted above, although that is no worse than
before, when they reached the cache and were dropped with the stopped double
buffer. In a Ratis driven install the follower refuses appendEntries while the
install is in progress, so this affects only entries already committed in its
log and not yet applied when the pause starts.
One way to close it: before installSnapshotFromLeader calls
installCheckpoint, wait until the state machine has applied and flushed up to
the follower's commit index, which cannot advance during the install, so
nothing committed is left to reject. As a backstop, record the highest index
rejected while paused and, on the path that keeps the old DB, have unpause
throw instead of resuming applies when a rejected index is above the restored
one; the existing catch then calls exitSystem, and a restart should replay
those entries from the Raft log (argued from the code, I did not run the
restart). I have this with tests and can share the patch if it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
