This is an automated email from the ASF dual-hosted git repository.

gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 16d379072f2 [fix](txn) Release the routine load job lock when a 
transaction abort fails (#70178)
16d379072f2 is described below

commit 16d379072f2447d5e7d2feefe5806d59cb70ec3e
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Nov 20 15:31:29 2026 +0800

    [fix](txn) Release the routine load job lock when a transaction abort fails 
(#70178)
    
    ### Background
    
    In shared-nothing mode, transaction jobs receive `beforeAborted` and
    `afterAborted` callbacks around an abort. A routine load job takes its
    write lock in the first callback and releases it in the second, so every
    successful before callback needs a matching after callback. Delete jobs
    also use these callbacks to release their registration, or retain it
    until a committed transaction becomes visible. Cloud transaction
    callbacks are delivered by the cloud transaction manager after its Meta
    Service RPC.
    
    ### What problem does this PR solve?
    
    Issue Number: close #69961
    Related PR: #59990 (introduced the lock leak), #68514 (restored commit
    callback pairing)
    PR Type: Bugfix
    
    Problem Summary:
    
    Since #59990, rejecting an abort of a COMMITTED routine load transaction
    or failing its edit-log write leaves the job lock held: job commands,
    scheduling and transaction publishing can hang until FE restarts. Both
    abort entry points now pair the callbacks in a `finally` outside the
    transaction monitor while preserving the original exception. A rejected
    shared-nothing abort keeps the delete callback needed for publication;
    cloud aborts keep their existing cleanup, including when Meta Service
    already aborted the transaction. Most of the diff is tests.
    
    ### Details
    
    - **Change map:** Production code: +38/-17 lines; tests: +251/-0;
    generated files and docs: 0. Callback pairing and exception handling in
    the two local abort entry points account for 49 changed lines;
    mode-specific delete callback ownership accounts for 6. The 251 test
    lines cover lock release, exception handling, publication and
    local/cloud delete callback cleanup.
    - **Frequency:** Not observed in production; found by code review.
    Triggers on a late abort of a COMMITTED routine load transaction or an
    edit-log failure.
    - **Root cause:** #59990 moved abort processing under per-transaction
    monitors and dropped the `finally` that ran `afterStateTransform`.
    Rejected state transitions and edit-log failures therefore bypassed the
    callback that releases the job lock.
    - **Why existing tests missed it:** Unit tests covered aborts without a
    routine-load callback and callback pairing only on commit. Cloud abort
    tests registered no real DeleteJob, so they could not detect retained
    delete callbacks. Regression suites for routine load and `delete_p0`
    exercise ordinary transactions and SQL results, without forcing a
    COMMITTED abort, a log failure, or checking callback ownership after an
    MS timeout abort.
    - **Fix:** Run the after callback from a `finally` outside the
    transaction monitor and log callback failures separately. Retain an
    unoperated delete callback only in shared-nothing mode: cloud
    `txnOperated=false` can mean MS has already aborted the transaction, so
    its callback must still be removed.
    - **Other paths and branches:** Both normal and 2PC local aborts are
    covered. Commit pairing was fixed in #68514; cloud already pairs abort
    callbacks in `finally`. `branch-3.1`, `branch-4.0`, `branch-4.1` and
    `branch-4.2` retain the original finally pairing and unconditional
    delete cleanup, so no backport is needed.
    - **Behavior change:** A failing after callback is logged instead of
    turning a successful abort into a failed request, as on commit and
    visible paths. The original abort failure still propagates.
    
    ### Testing
    
    - Locally ran `run-fe-ut.sh --run` over `CloudGlobalTransactionMgrTest`,
    `DatabaseTransactionMgrTest`, `GlobalTransactionMgrTest`,
    `TransactionStateTest`, `DeleteJobTest` and `RoutineLoadJobTest`: 152
    tests, 0 failures, 0 errors, 0 skipped. This includes the existing
    rejected-abort lock, exception-priority, publication and shared-nothing
    delete callback tests.
    - New `testAbortDeleteTransactionRemovesCallback` covers both `OK` and
    `TXN_ALREADY_ABORTED` through the cloud abort entry point with a real
    DeleteJob. On reviewed commit `b878321` plus the test, the
    already-aborted case fails because the callback remains registered; both
    cases pass with this fix.
    - The cloud cases mock MS RPC responses. A real FE/MS recycler timeout
    scenario was not run.
---
 .../main/java/org/apache/doris/load/DeleteJob.java |   6 +-
 .../doris/transaction/DatabaseTransactionMgr.java  |  49 +++--
 .../transaction/CloudGlobalTransactionMgrTest.java |  48 +++++
 .../transaction/DatabaseTransactionMgrTest.java    | 203 +++++++++++++++++++++
 4 files changed, 289 insertions(+), 17 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java 
b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
index 51ff705aebd..94c3cee6fda 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
@@ -241,7 +241,11 @@ public class DeleteJob extends 
AbstractTxnStateChangeCallback implements DeleteJ
 
     @Override
     public void afterAborted(TransactionState txnState, boolean txnOperated, 
String txnStatusChangeReason) {
-        // just to clean the callback
+        if (!txnOperated && !Config.isCloudMode()) {
+            // A rejected shared-nothing abort may still become visible and 
needs afterVisible().
+            return;
+        }
+        // Cloud aborts must also clean up transactions that MS already 
aborted (txnOperated=false).
         
Env.getCurrentGlobalTransactionMgr().getCallbackFactory().removeCallback(getId());
     }
 
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
 
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index 2d37d7b0bde..0107f4212f3 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -2050,17 +2050,26 @@ public class DatabaseTransactionMgr {
         transactionState.beforeStateTransform(TransactionStatus.ABORTED);
         boolean txnOperated = false;
         EditLog.EditLogItem logItem = null;
-        synchronized (transactionState) {
-            txnOperated = unprotectAbortTransaction(transactionId, reason);
-            if (txnOperated) {
-                if (Config.enable_txn_log_outside_lock) {
-                    logItem = enqueueTransactionState(transactionState);
-                } else {
-                    persistTransactionState(transactionState);
+        try {
+            synchronized (transactionState) {
+                txnOperated = unprotectAbortTransaction(transactionId, reason);
+                if (txnOperated) {
+                    if (Config.enable_txn_log_outside_lock) {
+                        logItem = enqueueTransactionState(transactionState);
+                    } else {
+                        persistTransactionState(transactionState);
+                    }
                 }
             }
+        } finally {
+            // Always pair a successful before callback outside the 
transaction monitor, also when the abort
+            // is rejected or its edit log fails: a routine load job releases 
the lock taken in beforeAborted here.
+            try {
+                
transactionState.afterStateTransform(TransactionStatus.ABORTED, txnOperated, 
reason);
+            } catch (Throwable e) {
+                LOG.warn("afterStateTransform txn {} failed. exception: ", 
transactionState, e);
+            }
         }
-        transactionState.afterStateTransform(TransactionStatus.ABORTED, 
txnOperated, reason);
         if (txnOperated) {
             awaitTransactionState(logItem, transactionState);
         }
@@ -2100,17 +2109,25 @@ public class DatabaseTransactionMgr {
         transactionState.beforeStateTransform(TransactionStatus.ABORTED);
         boolean txnOperated = false;
         EditLog.EditLogItem logItem = null;
-        synchronized (transactionState) {
-            txnOperated = unprotectAbortTransaction(transactionId, "User 
Abort");
-            if (txnOperated) {
-                if (Config.enable_txn_log_outside_lock) {
-                    logItem = enqueueTransactionState(transactionState);
-                } else {
-                    persistTransactionState(transactionState);
+        try {
+            synchronized (transactionState) {
+                txnOperated = unprotectAbortTransaction(transactionId, "User 
Abort");
+                if (txnOperated) {
+                    if (Config.enable_txn_log_outside_lock) {
+                        logItem = enqueueTransactionState(transactionState);
+                    } else {
+                        persistTransactionState(transactionState);
+                    }
                 }
             }
+        } finally {
+            // Always pair a successful before callback outside the 
transaction monitor, as in abortTransaction().
+            try {
+                
transactionState.afterStateTransform(TransactionStatus.ABORTED, txnOperated, 
"User Abort");
+            } catch (Throwable e) {
+                LOG.warn("afterStateTransform txn {} failed. exception: ", 
transactionState, e);
+            }
         }
-        transactionState.afterStateTransform(TransactionStatus.ABORTED, 
txnOperated, "User Abort");
         if (txnOperated) {
             awaitTransactionState(logItem, transactionState);
         }
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
index e7902076eea..509d9a87a5a 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
@@ -58,6 +58,7 @@ import org.apache.doris.common.jmockit.Deencapsulation;
 import org.apache.doris.common.util.PropertyAnalyzer;
 import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
 import 
org.apache.doris.job.extensions.insert.streaming.StreamingTaskTxnCommitAttachment;
+import org.apache.doris.load.DeleteJob;
 import org.apache.doris.load.routineload.RLTaskTxnCommitAttachment;
 import org.apache.doris.qe.ConnectContext;
 import org.apache.doris.qe.SessionVariable;
@@ -75,6 +76,7 @@ import org.apache.doris.transaction.GlobalTransactionMgrIface;
 import org.apache.doris.transaction.SubTransactionState;
 import org.apache.doris.transaction.TabletCommitInfo;
 import org.apache.doris.transaction.TransactionCommitFailedException;
+import org.apache.doris.transaction.TransactionNotFoundException;
 import org.apache.doris.transaction.TransactionState;
 import org.apache.doris.transaction.TransactionStatus;
 import org.apache.doris.transaction.TxnStateChangeCallback;
@@ -719,6 +721,52 @@ public class CloudGlobalTransactionMgrTest {
         }
     }
 
+    @ParameterizedTest
+    @ValueSource(booleans = {false, true})
+    public void testAbortDeleteTransactionRemovesCallback(boolean 
alreadyAborted) throws Exception {
+        long transactionId = 123535;
+        long jobId = 1002;
+        DeleteJob job = Mockito.spy(new DeleteJob(jobId, transactionId, 
"delete_abort", Map.of(), null));
+        masterTransMgr.getCallbackFactory().addCallback(job);
+
+        MetaServiceProxy proxy = Mockito.mock(MetaServiceProxy.class);
+        try (MockedStatic<MetaServiceProxy> mocked = 
Mockito.mockStatic(MetaServiceProxy.class)) {
+            mocked.when(MetaServiceProxy::getInstance).thenReturn(proxy);
+            TxnInfoPB txnInfo = buildTxnInfo(transactionId).toBuilder()
+                    .setListenerId(jobId)
+                    .setStatus(alreadyAborted ? 
Cloud.TxnStatusPB.TXN_STATUS_ABORTED
+                            : Cloud.TxnStatusPB.TXN_STATUS_PREPARED)
+                    .build();
+            
Mockito.when(proxy.getTxn(Mockito.any())).thenReturn(Cloud.GetTxnResponse.newBuilder()
+                    
.setStatus(Cloud.MetaServiceResponseStatus.newBuilder().setCode(MetaServiceCode.OK))
+                    .setTxnInfo(txnInfo).build());
+            AbortTxnResponse.Builder response = AbortTxnResponse.newBuilder()
+                    .setStatus(Cloud.MetaServiceResponseStatus.newBuilder()
+                            .setCode(alreadyAborted ? 
MetaServiceCode.TXN_ALREADY_ABORTED : MetaServiceCode.OK));
+            if (!alreadyAborted) {
+                
response.setTxnInfo(txnInfo.toBuilder().setStatus(Cloud.TxnStatusPB.TXN_STATUS_ABORTED)
+                        .setReason("User Cancelled"));
+            }
+            
Mockito.when(proxy.abortTxn(Mockito.any())).thenReturn(response.build());
+
+            if (alreadyAborted) {
+                TransactionNotFoundException exception = 
Assertions.assertThrows(TransactionNotFoundException.class,
+                        () -> 
masterTransMgr.abortTransaction(CatalogTestUtil.testDbId1, transactionId,
+                                "User Cancelled"));
+                Assertions.assertTrue(exception.getMessage().contains("is 
already aborted"), exception.getMessage());
+            } else {
+                masterTransMgr.abortTransaction(CatalogTestUtil.testDbId1, 
transactionId, "User Cancelled");
+            }
+
+            Mockito.verify(job).beforeAborted(Mockito.any());
+            Mockito.verify(job).afterAborted(Mockito.any(), 
Mockito.eq(!alreadyAborted), Mockito.anyString());
+            
Assertions.assertNull(masterTransMgr.getCallbackFactory().getCallback(jobId),
+                    "an aborted cloud DELETE must release its callback even 
when MS aborted it first");
+        } finally {
+            masterTransMgr.getCallbackFactory().removeCallback(jobId);
+        }
+    }
+
     @Test
     public void testAbortRoutineLoadTransactionWithAttachment() throws 
Exception {
         long transactionId = 123534;
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
index 3d41243955c..261413a12c9 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/transaction/DatabaseTransactionMgrTest.java
@@ -17,6 +17,7 @@
 
 package org.apache.doris.transaction;
 
+import org.apache.doris.analysis.UserIdentity;
 import org.apache.doris.catalog.BinlogConfig;
 import org.apache.doris.catalog.CatalogTestUtil;
 import org.apache.doris.catalog.ChangelogConfig;
@@ -47,6 +48,8 @@ import 
org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
 import 
org.apache.doris.job.extensions.insert.streaming.StreamingTaskTxnCommitAttachment;
 import org.apache.doris.job.offset.jdbc.JdbcOffset;
 import org.apache.doris.job.offset.jdbc.JdbcSourceOffsetProvider;
+import org.apache.doris.load.DeleteJob;
+import org.apache.doris.load.routineload.kafka.KafkaRoutineLoadJob;
 import org.apache.doris.meta.MetaContext;
 import org.apache.doris.mtmv.ivm.IvmInfo;
 import org.apache.doris.mysql.authenticate.TestLogAppender;
@@ -88,6 +91,7 @@ import java.util.Set;
 import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.FutureTask;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.locks.ReentrantReadWriteLock;
 
 public class DatabaseTransactionMgrTest {
@@ -516,6 +520,205 @@ public class DatabaseTransactionMgrTest {
                 "unexpected message: " + e.getMessage());
     }
 
+    private long beginAbortTestTransaction(String label, LoadJobSourceType 
sourceType, boolean commit)
+            throws UserException {
+        Table table = 
masterEnv.getInternalCatalog().getDbOrMetaException(CatalogTestUtil.testDbId1)
+                .getTableOrMetaException(CatalogTestUtil.testTableId1);
+        long txnId = 
masterTransMgr.beginTransaction(CatalogTestUtil.testDbId1, 
Lists.newArrayList(table.getId()),
+                label, transactionSource, sourceType, 
Config.stream_load_default_timeout_second);
+        if (commit) {
+            
masterTransMgr.commitTransactionWithoutLock(CatalogTestUtil.testDbId1, 
Lists.newArrayList(table), txnId,
+                    
GlobalTransactionMgrTest.generateTabletCommitInfos(CatalogTestUtil.testTabletId1,
 allBackends),
+                    null);
+        }
+        return txnId;
+    }
+
+    private static void abortForTest(String mode, long txnId) throws 
UserException {
+        DatabaseTransactionMgr dbTransMgr = 
masterTransMgr.getDatabaseTransactionMgr(CatalogTestUtil.testDbId1);
+        if (mode.equals("2pc")) {
+            dbTransMgr.abortTransaction2PC(txnId);
+        } else {
+            dbTransMgr.abortTransaction(txnId, "abort for test", null);
+        }
+    }
+
+    // Registers a real routine load job as the callback of the transaction, 
whose beforeAborted() takes the job's
+    // write lock and whose afterAborted() releases it. Records whether 
afterAborted() ran under the txn monitor.
+    private static KafkaRoutineLoadJob 
addRoutineLoadJobCallback(TransactionState state,
+            AtomicBoolean afterAbortedHoldsMonitor) throws UserException {
+        KafkaRoutineLoadJob job = Mockito.spy(new 
KafkaRoutineLoadJob(state.getTransactionId(), "abort_test_job",
+                CatalogTestUtil.testDbId1, CatalogTestUtil.testTableId1, 
"host:port", "topic", UserIdentity.ADMIN));
+        Mockito.doAnswer(invocation -> {
+            afterAbortedHoldsMonitor.set(Thread.holdsLock(state));
+            return invocation.callRealMethod();
+        }).when(job).afterAborted(Mockito.any(), Mockito.anyBoolean(), 
Mockito.any());
+        masterTransMgr.getCallbackFactory().addCallback(job);
+        state.setCallbackId(job.getId());
+        return job;
+    }
+
+    @ParameterizedTest
+    @CsvSource({"normal,false", "normal,true", "2pc,false", "2pc,true"})
+    public void testRejectedAbortReleasesRoutineLoadJobLock(String mode, 
boolean logOutsideLock) throws Exception {
+        boolean originalLogOutsideLock = Config.enable_txn_log_outside_lock;
+        ReentrantReadWriteLock jobLock = null;
+        try {
+            Config.enable_txn_log_outside_lock = logOutsideLock;
+            FakeEnv.setEnv(masterEnv);
+            long txnId = beginAbortTestTransaction("rejected_abort_" + mode + 
"_" + logOutsideLock,
+                    LoadJobSourceType.ROUTINE_LOAD_TASK, true);
+            TransactionState state = 
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+            Assertions.assertEquals(TransactionStatus.COMMITTED, 
state.getTransactionStatus());
+            AtomicBoolean afterAbortedHoldsMonitor = new AtomicBoolean(true);
+            KafkaRoutineLoadJob job = addRoutineLoadJobCallback(state, 
afterAbortedHoldsMonitor);
+            jobLock = Deencapsulation.getField(job, "lock");
+            Mockito.clearInvocations(masterEnv.getEditLog());
+
+            // A late abort, such as a BE rollback after its commit RPC timed 
out, is rejected.
+            UserException e = Assertions.assertThrows(UserException.class, () 
-> abortForTest(mode, txnId));
+            Assertions.assertTrue(e.getMessage().contains("is already 
COMMITTED, could not abort"), e.getMessage());
+            Assertions.assertEquals(TransactionStatus.COMMITTED, 
state.getTransactionStatus());
+            Mockito.verifyNoInteractions(masterEnv.getEditLog());
+
+            // The publish thread must still finish the transaction: 
afterVisible() takes the job's write lock.
+            Map<String, Map<Long, Long>> successTablets = new HashMap<>();
+            setSuccessTablet(successTablets, allBackends, txnId, 
CatalogTestUtil.testTabletId1,
+                    CatalogTestUtil.testStartVersion + 2);
+            setTransactionFinishPublish(state, allBackends, successTablets);
+            FutureTask<Void> publishTask = new FutureTask<>(() -> {
+                try (FakeEnv threadEnv = new FakeEnv()) {
+                    
masterTransMgr.finishTransaction(CatalogTestUtil.testDbId1, txnId, 
Maps.newHashMap(),
+                            Maps.newHashMap());
+                    return null;
+                }
+            });
+            new Thread(publishTask, "publish-after-rejected-abort").start();
+            Assertions.assertDoesNotThrow(() -> publishTask.get(10, 
TimeUnit.SECONDS),
+                    "publishing the committed transaction waits for the 
routine load job lock");
+            Assertions.assertEquals(TransactionStatus.VISIBLE, 
state.getTransactionStatus());
+
+            Mockito.verify(job).beforeAborted(state);
+            Mockito.verify(job).afterAborted(Mockito.eq(state), 
Mockito.eq(false), Mockito.any());
+            Assertions.assertFalse(afterAbortedHoldsMonitor.get(), 
"afterAborted() ran under the transaction monitor");
+            Mockito.verify(job).afterVisible(state, true);
+            Assertions.assertFalse(jobLock.isWriteLocked(), "the routine load 
job lock is still held");
+        } finally {
+            Config.enable_txn_log_outside_lock = originalLogOutsideLock;
+            // Release a lock leaked on this thread so that a blocked publish 
thread can exit.
+            while (jobLock != null && jobLock.isWriteLockedByCurrentThread()) {
+                jobLock.writeLock().unlock();
+            }
+        }
+    }
+
+    @ParameterizedTest
+    @CsvSource({"normal,false,false", "normal,true,false", "2pc,false,false", 
"2pc,true,false",
+            "normal,false,true", "normal,true,true", "2pc,false,true", 
"2pc,true,true"})
+    public void testAbortReleasesRoutineLoadJobLock(String mode, boolean 
logOutsideLock, boolean failEditLog)
+            throws Exception {
+        boolean originalLogOutsideLock = Config.enable_txn_log_outside_lock;
+        ReentrantReadWriteLock jobLock = null;
+        try {
+            Config.enable_txn_log_outside_lock = logOutsideLock;
+            FakeEnv.setEnv(masterEnv);
+            long txnId = beginAbortTestTransaction("abort_" + mode + "_" + 
logOutsideLock + "_" + failEditLog,
+                    LoadJobSourceType.ROUTINE_LOAD_TASK, false);
+            TransactionState state = 
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+            AtomicBoolean afterAbortedHoldsMonitor = new AtomicBoolean(true);
+            KafkaRoutineLoadJob job = addRoutineLoadJobCallback(state, 
afterAbortedHoldsMonitor);
+            jobLock = Deencapsulation.getField(job, "lock");
+
+            if (failEditLog) {
+                // What EditLog throws when it has no edit log stream; the 
abort is already done in memory.
+                Error editLogFailure = new Error("Fatal Error : no editLog 
stream");
+                Mockito.doThrow(editLogFailure).when(masterEnv.getEditLog())
+                        
.logInsertTransactionState(Mockito.any(TransactionState.class));
+                Mockito.doThrow(editLogFailure).when(masterEnv.getEditLog())
+                        .submitEdit(Mockito.anyShort(), Mockito.any());
+                Error thrown = Assertions.assertThrows(Error.class, () -> 
abortForTest(mode, txnId));
+                Assertions.assertSame(editLogFailure, thrown);
+            } else {
+                abortForTest(mode, txnId);
+            }
+
+            Assertions.assertFalse(jobLock.isWriteLocked(), "the routine load 
job lock is still held");
+            Assertions.assertEquals(TransactionStatus.ABORTED, 
state.getTransactionStatus());
+            Mockito.verify(job).beforeAborted(state);
+            // The transaction is aborted in memory before the edit log write, 
so txnOperated is true either way.
+            Mockito.verify(job).afterAborted(Mockito.eq(state), 
Mockito.eq(true), Mockito.any());
+            Assertions.assertFalse(afterAbortedHoldsMonitor.get(), 
"afterAborted() ran under the transaction monitor");
+        } finally {
+            Config.enable_txn_log_outside_lock = originalLogOutsideLock;
+            while (jobLock != null && jobLock.isWriteLockedByCurrentThread()) {
+                jobLock.writeLock().unlock();
+            }
+        }
+    }
+
+    @ParameterizedTest
+    @ValueSource(strings = {"normal", "2pc"})
+    public void testAbortSkipsAfterCallbackWhenBeforeCallbackFails(String 
mode) throws Exception {
+        FakeEnv.setEnv(masterEnv);
+        long txnId = beginAbortTestTransaction("before_callback_fails_" + 
mode, LoadJobSourceType.FRONTEND, false);
+        TransactionState state = 
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+        TxnStateChangeCallback callback = 
Mockito.mock(TxnStateChangeCallback.class);
+        Mockito.when(callback.getId()).thenReturn(txnId);
+        TransactionException beforeFailure = new TransactionException("before 
callback failed");
+        Mockito.doThrow(beforeFailure).when(callback).beforeAborted(state);
+        masterTransMgr.getCallbackFactory().addCallback(callback);
+        state.setCallbackId(txnId);
+
+        TransactionException thrown = 
Assertions.assertThrows(TransactionException.class,
+                () -> abortForTest(mode, txnId));
+        Assertions.assertSame(beforeFailure, thrown);
+        Assertions.assertEquals(TransactionStatus.PREPARE, 
state.getTransactionStatus());
+        Mockito.verify(callback, Mockito.never()).afterAborted(Mockito.any(), 
Mockito.anyBoolean(), Mockito.any());
+    }
+
+    @ParameterizedTest
+    @ValueSource(strings = {"normal", "2pc"})
+    public void 
testRejectedAbortKeepsAbortFailureWhenAfterCallbackFails(String mode) throws 
Exception {
+        FakeEnv.setEnv(masterEnv);
+        long txnId = beginAbortTestTransaction("after_callback_fails_" + mode, 
LoadJobSourceType.FRONTEND, true);
+        TransactionState state = 
masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, txnId);
+        TxnStateChangeCallback callback = 
Mockito.mock(TxnStateChangeCallback.class);
+        Mockito.when(callback.getId()).thenReturn(txnId);
+        Mockito.doThrow(new RuntimeException("after callback 
failed")).when(callback)
+                .afterAborted(Mockito.any(), Mockito.anyBoolean(), 
Mockito.any());
+        masterTransMgr.getCallbackFactory().addCallback(callback);
+        state.setCallbackId(txnId);
+
+        UserException e = Assertions.assertThrows(UserException.class, () -> 
abortForTest(mode, txnId));
+        Assertions.assertTrue(e.getMessage().contains("is already COMMITTED, 
could not abort"), e.getMessage());
+        Assertions.assertEquals(TransactionStatus.COMMITTED, 
state.getTransactionStatus());
+        Mockito.verify(callback).afterAborted(Mockito.eq(state), 
Mockito.eq(false), Mockito.any());
+    }
+
+    @Test
+    public void testRejectedAbortKeepsDeleteJobCallback() throws Exception {
+        FakeEnv.setEnv(masterEnv);
+        TxnStateCallbackFactory callbackFactory = 
masterTransMgr.getCallbackFactory();
+        // DeleteJob.cancel() aborts the transaction; a committed delete still 
needs afterVisible() to finish.
+        long committedTxnId = beginAbortTestTransaction("delete_committed", 
LoadJobSourceType.FRONTEND, true);
+        DeleteJob committedJob = new DeleteJob(committedTxnId, committedTxnId, 
"delete_committed",
+                Maps.newHashMap(), null);
+        callbackFactory.addCallback(committedJob);
+        masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, 
committedTxnId)
+                .setCallbackId(committedJob.getId());
+        Assertions.assertThrows(UserException.class, () -> 
abortForTest("normal", committedTxnId));
+        Assertions.assertSame(committedJob, 
callbackFactory.getCallback(committedJob.getId()));
+
+        long preparedTxnId = beginAbortTestTransaction("delete_prepared", 
LoadJobSourceType.FRONTEND, false);
+        DeleteJob abortedJob = new DeleteJob(preparedTxnId, preparedTxnId, 
"delete_prepared", Maps.newHashMap(),
+                null);
+        callbackFactory.addCallback(abortedJob);
+        masterTransMgr.getTransactionState(CatalogTestUtil.testDbId1, 
preparedTxnId)
+                .setCallbackId(abortedJob.getId());
+        abortForTest("normal", preparedTxnId);
+        Assertions.assertNull(callbackFactory.getCallback(abortedJob.getId()));
+    }
+
     @Test
     public void 
testUpdateCatalogAfterCommittedAdvancesIvmSequencePrefixForNormalCommitAndReplay()
             throws Exception {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to