[
https://issues.apache.org/jira/browse/CAMEL-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25682 started by shashank.
----------------------------------------
> camel-resilience4j - Circuit Breaker EIP: a timeout that fires while the task
> copies its result into the exchange lets the fallback (or the timeout
> exception) and that copy change the exchange at the same time
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25682
> URL: https://issues.apache.org/jira/browse/CAMEL-25682
> Project: Camel
> Issue Type: Bug
> Components: eip
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> CAMEL-24134 added a write guard between the task of the circuit breaker (it
> runs on the timeout executor and copies its result from the correlated copy
> into the exchange) and the fallback that the caller runs after a timeout. The
> task claims the guard with {{compareAndSet(false, true)}} and then writes
> ({{ResilienceProcessor.processTask}}, lines 751-764 at main cd2ce899764e;
> async mode {{processTaskAsync}} 812-824). The fallback task only does
> {{exchangeWriteGuard.set(true)}} (line 911) and then writes the exchange. So
> when the timeout fires after the task has won the guard but before it has
> finished its write-back, both write the same exchange at the same time:
> * with {{onFallback}}: the task's {{ExchangeHelper.copyResults}} replaces the
> body of the fallback by the body of the task, or sets
> {{CamelCircuitBreakerResponseFromFallback}} back to {{false}} and
> {{CamelCircuitBreakerResponseSuccessfulExecution}} to {{true}} after the
> fallback has set them, while the exchange already continues in the route;
> * without fallback: {{copyResults}} ends with {{setException(null)}} and
> clears the {{TimeoutException}} the caller has just set (a timed-out call
> continues as a success);
> * the headers and properties maps of the exchange are written by two threads.
> The caller does not wait for the task: with a timeout the task runs in
> {{CompletableFuture.supplyAsync}} (591) behind
> {{TimeLimiter.decorateFutureSupplier}} (599); async mode uses
> {{TimeLimiter.decorateCompletionStage}} (672). The window is the time the
> task needs for {{copyResults}} and two properties, so it needs a task that
> ends at about the moment of the timeout.
> h3. Reproduction
> * New {{ResilienceTimeoutDuringWriteBackTest}} (4 tests: sync and
> {{asynchronous=true}}, with and without {{onFallback}}, timeout 500 ms). The
> task's result message holds the first read of its body by
> {{ExchangeHelper.copyResults}} (the test checks the stack) until the caller
> has returned, or until the caller has not returned 2 seconds after the
> timeout. On main all 4 fail: {{expected: <Fallback result> but was: <Task
> result>}} (sync and async), and {{expected:
> <java.util.concurrent.TimeoutException> but was: <null>}} (sync and async).
> * Without the hold: a probe (not part of the change) with 20000 calls from 32
> threads, timeout 20 ms and a task of 20 ms +/- 0.3 ms, a 64-thread timeout
> executor: on main 26 exchanges ended inconsistent (for example the fallback
> body with {{CamelCircuitBreakerResponseFromFallback=false}}); with the fix 0.
> One run each.
> h3. Proposed fix
> The guard becomes a small lock-based guard per call: the task claims it and
> writes while holding the lock; the fallback task claims it under the same
> lock, so a caller that times out while the task is writing waits until that
> write is done (bounded: {{copyResults}} and two properties). It then sets the
> timeout result or runs the fallback on the exchange, which then holds the
> result of the task (as today when the timeout fires just after the copy). The
> task writes only if it claims the exchange before the caller; the caller
> always writes after a timeout. In async mode the waiting thread is a thread
> of the time limiter's scheduler pool, which already runs the fallback. Sync
> and async mode. One {{ReentrantLock}} per call instead of one
> {{AtomicBoolean}}.
> camel-microprofile-fault-tolerance has the same {{set(true)}}
> ({{FaultToleranceProcessor}} 298) but is not affected: the same test
> (adapted) passes on main, because with SmallRye Fault Tolerance 6.11.4 the
> caller gets the {{TimeoutException}} only after the task has returned (it
> waited the whole hold in the test), so the task and the fallback do not
> overlap.
> camel-resilience4j tests with the fix: 81 run, 0 failures.
> Not changed (follow-ups): an exception thrown by {{copyResults}} after the
> task claimed the exchange is dropped and the call counts as a success (same
> on main); and whether a caller that finds the write-back already claimed
> should keep the task's result instead of running the fallback (it would
> change the outcome of such calls, so it needs a decision).
> How it was found: the backlog of an earlier review round listed the
> {{set(true)}}. A small TLA+ model of the guard (task: claim, write body,
> write properties; caller: timeout at any time before the task returned,
> claim, fallback, continue) violates "the task does not write after the caller
> took the exchange over" and "the final exchange tells one story" on main in
> 18 / 36-49 states, and both hold with the lock (28-31 states). The model is a
> sketch; the test and the probe are the evidence. The microprofile part of the
> model assumption (the caller continues while the task runs) turned out to be
> false for SmallRye, see above.
> Affected: main (cd2ce899764e; tested), camel-4.23.x and camel-4.22.x (same
> {{set(true)}}; read, not run). camel-4.18.x has no guard at all (before
> CAMEL-24134).
> Duplicate check (2026-10-15): JIRA text "circuit breaker timeout worker write
> back" (CAMEL-24134 only, resolved, the fix this completes),
> "exchangeWriteGuard" (CAMEL-25129, CAMEL-24209: other topics); GitHub pull
> requests "exchangeWriteGuard", "FaultToleranceProcessor" (#24797 =
> CAMEL-24134, merged; nothing open).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
