Doris-Breakwater commented on issue #70586:
URL: https://github.com/apache/doris/issues/70586#issuecomment-6884967587

   Breakwater-GitHub-Analysis-Slot: slot_8a79ab2cff59
   
   **Initial assessment:** The error-classification problem is supported by 
source inspection in both public `4.1.5-release` 
(`917b20be83878e98f94e3ebf7efccc80346e3e50`) and the cited master commit. A 
failed lookup can become a false transaction-not-found response. This verifies 
the code mechanism; it does not establish the deployed transaction's final 
state, expiry, or the cause of the KV failure. The proposed fault injection has 
not been executed in this investigation. Suggested maintainer routing: cloud FE 
transaction/error handling. The issue currently has no labels.
   
   **Verified code path**
   
   - In 4.1.5, `getTransactionState` returns `null` for `RpcException`, a null 
response, any non-OK status, or missing `txn_info`. These outcomes therefore 
lose their distinction from absence. See [release lookup 
handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L2391-L2428).
   - More precisely, the exception in the issue's master lines 483–486 is 
inside `checkTransactionStateBeforeCommit`, reached through the commit path. 
The release has the same null-to-`TransactionCommitFailedException` conversion. 
On the ordinary commit path, this check is gated by a nonempty list of 
participating unique-key Merge-on-Write tables, derived from matching tablet 
commit information. A disposable table alone is therefore insufficient to 
guarantee this reproduction. See [release caller and 
check](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L430-L505),
 [table 
selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L962-L986),
 and [pinned master 
check](https://github.com/apache/doris/blob/42b148bac33f798edc9f8ae984a462b1b4a4f472
 
/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java#L478-L486).
   - Meta Service already distinguishes absence: reading a missing 
transaction-info key yields `TXN_ID_NOT_FOUND`; other read errors use the KV 
error mapping. The FE discards this distinction. See [release 
`get_txn`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/cloud/src/meta-service/meta_service_txn.cpp#L3790-L3812).
   - The server retry wrapper can translate exhausted 
`KV_TXN_STORE_GET_RETRYABLE` into `KV_TXN_GET_ERR`. The FE RPC wrapper returns 
that status without an additional status-based retry for it. Thus the 
approximately 1 ms between the FE error logs does not establish total lookup 
duration or whether server retries occurred. `RetryableNotCommitted` describes 
a KV operation, not the Doris transaction's settled state. See [server retry 
handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/cloud/src/meta-service/meta_service.h#L1072-L1132)
 and [FE RPC 
handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/MetaServiceProxy.java#L244-L310).
   
   **Information needed to verify the incident**
   
   1. Exact FE, BE, and Meta Service build hashes, including any patches or 
version skew; the version string alone cannot prove equivalence to the public 
tag.
   2. Table DDL, ingestion/commit API, transaction mode, minimal commands, and 
client retry/rollback behavior. For the identified path, confirm a 
participating unique-key MoW table and nonempty matching tablet commit 
information.
   3. Sanitized, correlated FE and Meta Service logs covering the lookup and 
all commit/abort attempts for the same database, transaction, and label, with 
timestamps/time zones, complete status messages, and exception stacks. Include 
corresponding BE/client responses and raw underlying KV/FDB error codes. 
Preserve correlation while redacting credentials and sensitive identifiers.
   4. Actual server retry settings (`enable_txn_store_retry`, 
`txn_store_retry_times`, `txn_store_retry_base_intervals_ms`), RPC/client 
deadlines, and the transaction/label state after recovery. These are diagnostic 
inputs; increasing retries or transaction timeouts is not yet justified.
   
   **Recommended next steps**
   
   - Run the proposed fault injection on an isolated cloud test cluster with a 
unique-key MoW table and actual written rows. Intercept the specific pre-commit 
lookup and return `KV_TXN_GET_ERR` while retaining the transaction record. 
Record whether this invocation sends a subsequent `commit_txn`; this failure 
branch should stop before that call. Earlier or concurrent attempts still 
require settlement checks. Compare with `TXN_ID_NOT_FOUND`, then remove the 
fault and inspect the same transaction/label.
   - Preserve a distinct lookup-failed outcome, including the original Meta 
Service status/message or transport cause. Reserve not-found for explicit 
absence; treat missing response/transaction information as lookup/protocol 
failure. Review callers before changing the shared nullable API; status-query 
and other commit entry points also need to preserve the distinction through the 
client boundary.
   - Add regression coverage for lookup failure versus definite absence, RPC 
failure, missing response information, and recovery to 
PREPARE/COMMITTED/VISIBLE/ABORTED. Preserve the existing handling of already 
committed/visible transactions. Any retry/settlement should retain the same 
transaction and label, respect the existing deadline, and avoid inferring that 
replay under a new identity is safe from this error alone.
   
   No code changes or runtime reproduction were performed. The remaining 
uncertainty is incident correlation and the underlying KV failure, not the 
demonstrated loss of error information in the inspected FE code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to