Omar Atie created CAMEL-24368:
---------------------------------
Summary: flaky Tests
Key: CAMEL-24368
URL: https://issues.apache.org/jira/browse/CAMEL-24368
Project: Camel
Issue Type: Test
Environment: Here is the **failing environment** from CI run
[31115312949](https://github.com/apache/camel/actions/runs/31115312949), job
[92696861359](https://github.com/apache/camel/actions/runs/31115312949/job/92696861359):
---
## CI context
| Field | Value |
|-------|-------|
| **Workflow** | `Build and test` (PR validation) |
| **PR** | [#25337](https://github.com/apache/camel/pull/25337) — CAMEL-23861
GenAI observability |
| **Branch** | `cursor/CAMEL-23861-genai-observability-c587` |
| **Commit** | `8e424d403b86a274f39e8c179d6221d33b94543c` |
| **Run attempt** | 2 |
| **Trigger** | `pull_request` against `main` |
| **Run started** | 2026-08-06 22:52 UTC |
| **Test phase failed** | 2026-08-07 00:00 UTC (~56 min test phase) |
| **Failing matrix job** | `build (25, false)` — **JDK 25** |
| **Other matrix job** | `build (17, false)` — **cancelled** (concurrency), not
the source of the failure |
---
## Hardware / OS
| Field | Value |
|-------|-------|
| **Runner** | GitHub-hosted (`ubuntu-latest`) |
| **Runner label** | `ubuntu-latest` |
| **OS** | Ubuntu 24.04 (Noble) |
| **Runner image** | `ubuntu24/20260720.247` |
| **Architecture** | **linux-amd64 (x86_64)** — *not* s390x / ppc64le / aarch64
|
| **Typical runner specs** | 4 vCPU, 16 GB RAM (standard GitHub-hosted runner) |
---
## Java / build toolchain
| Field | Value |
|-------|-------|
| **JDK** | Eclipse Temurin **25.0.3+9** (x64) |
| **JAVA_HOME** | `/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/25.0.3-9/x64` |
| **Maven Daemon** | **mvnd 1.0.5** (`linux-amd64`) |
| **Maven** | **3.9.14** (bundled with mvnd) |
| **Camel version** | **4.22.0-SNAPSHOT** |
| **Surefire** | **3.5.6** |
| **Surefire reruns** | `-Dsurefire.rerunFailingTestsCount=2` |
---
## Build / test execution profile
| Field | Value |
|-------|-------|
| **Build type** | Incremental PR build (Scalpel change detection) |
| **Scalpel result** | 605 affected modules (580 would be tested) |
| **Modules actually tested** | **606 modules** (`install` over full
incremental `-pl` list) |
| **mvnd threads** | `-Dmvnd.threads=2` |
| **Maven artifact threads** | `-Dmaven.artifact.threads=25` |
| **CI env flag** | `-Dci.env.name=github.com` |
| **Load context** | Full-reactor incremental build under CI resource
contention (~606 modules) |
---
## Test failure behavior in this environment
| Test | Result in this env |
|------|---------------------|
| `LumberjackComponentTest.shouldListenToMessages` | **Failed all 3 Surefire
runs** (Run 1/2/3) — `ConditionTimeoutException` after **30s** in
`LumberjackUtil` |
| `VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath` |
**Failed Run 1**, **passed Run 2** — `expected: <2> but was: <1>` at line 172 |
---
## Important note for JIRA
This failure happened on **standard GitHub Actions amd64 + JDK 25**, not on
s390x. So the `@DisabledOnOs(s390x)` guard from CAMEL-21438 would **not** have
skipped these tests. The environment is a heavy incremental CI build, not an
alternative-architecture Jenkins node.
Reporter: Omar Atie
h2. Summary
Flaky CI: \{{LumberjackComponentTest.shouldListenToMessages}} timeout and
\{{VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath}} race
under heavy incremental builds
h2. Issue Type
Bug
h2. Priority
Major
h2. Affects Version/s
4.22.0-SNAPSHOT
h2. Component/s
* camel-lumberjack
* camel-vertx-websocket
h2. Labels
{\{ci}}, \{{flaky-test}}, \{{test-infra}}
----
h2. Description
h3. Context
During CI validation of [PR #25337|https://github.com/apache/camel/pull/25337]
(CAMEL-23861 — GenAI observability), the build failed on *unrelated flaky
tests* while running the full incremental build (~606 modules). All AI
observability tests in the PR passed.
* CI run: [31115312949|https://github.com/apache/camel/actions/runs/31115312949]
* Failing job: \{{Build and test}} (job \{{92696861359}})
* PR analysis comment:
[5212287682|https://github.com/apache/camel/pull/25337#issuecomment-5212287682]
The PR branch was rebased onto \{{upstream/main}} and the PR-specific test
suite (\{{camel-ai/camel-ai-common}}, \{{camel-ai/camel-ai-google-genai}},
\{{camel-ai/camel-ai-openai}}, \{{components/camel-opentelemetry2}}) passes
locally with \{{BUILD SUCCESS}}.
----
h3. Failure 1 — LumberjackComponentTest (build breaker)
*Test:*
\{{org.apache.camel.component.lumberjack.LumberjackComponentTest#shouldListenToMessages}}
*Location:*
\{{components/camel-lumberjack/src/test/java/org/apache/camel/component/lumberjack/LumberjackComponentTest.java:62}}
*Result:* \{{ConditionTimeoutException}} after 30 seconds (failed all 3
Surefire reruns)
*Annotations:* \{{@Isolated}}, \{{@DisabledOnOs(OS.S390X)}} (related to
[CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438])
{code}
org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression
in org.apache.camel.component.lumberjack.LumberjackComponentTest was not
fulfilled within 30 seconds.
at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1168)
at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1141)
at
org.apache.camel.component.lumberjack.LumberjackComponentTest.shouldListenToMessages(LumberjackComponentTest.java:62)
{code}
*Likely cause:* Timing/network sensitivity when the full reactor runs under CI
load; the test waits for Lumberjack messages over TCP and may not receive them
reliably within 30s.
----
h3. Failure 2 — VertxWebsocketTest (intermittent)
*Test:*
\{{org.apache.camel.component.vertxwebsocket.VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}}
*Location:*
\{{components/camel-vertx-websocket/src/test/java/org/apache/camel/component/vertxwebsocket/VertxWebsocketTest.java:218}}
*Result:* \{{expected: <2> but was: <1>}} on first run; *passed on Surefire
rerun #2*
{code}
org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
at
org.apache.camel.component.vertxwebsocket.VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath(VertxWebsocketTest.java:218)
{code}
*Likely cause:* Race between WebSocket connection establishment and message
dispatch when multiple tests share Vert.x resources under parallel/full-reactor
load.
----
h2. Steps to Reproduce
# Run a large incremental build that includes both modules, e.g.:
{code:bash}
mvn clean install -pl
components/camel-lumberjack,components/camel-vertx-websocket -am
{code}
# Or reproduce CI-like load:
{code:bash}
mvn clean install -Dquickly
# then re-run the failing tests repeatedly
mvn test -pl components/camel-lumberjack
-Dtest=LumberjackComponentTest#shouldListenToMessages
mvn test -pl components/camel-vertx-websocket
-Dtest=VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath
{code}
# Observe intermittent timeouts or assertion failures under resource contention.
----
h2. Expected Result
Both tests pass consistently in CI and local full-reactor builds without
relying on Surefire reruns.
h2. Actual Result
* \{{LumberjackComponentTest#shouldListenToMessages}} — fails consistently in
CI (3/3 reruns)
* \{{VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}} —
fails intermittently (1/3 reruns in observed run)
----
h2. Proposed Fix (investigation starting points)
h3. Lumberjack
* Review Awaitility timeout and polling strategy in \{{shouldListenToMessages}}
* Check whether TCP listener/port binding is stable under parallel test
execution
* Cross-check [CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438]
history — test already marked \{{@Isolated}} and disabled on s390x for prior
flakiness
h3. Vert.x WebSocket
* Add synchronization or use \{{MockEndpoint}} timed assertions before counting
received messages
* Ensure WebSocket client connection is fully established before sending
* Consider \{{@Isolated}} or dedicated Vert.x instance per test if shared
lifecycle causes races
----
h2. Acceptance Criteria
* \{{LumberjackComponentTest#shouldListenToMessages}} passes reliably in CI (no
\{{ConditionTimeoutException}} under standard build load)
* \{{VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}}
passes on first run without depending on Surefire reruns
* No regression in module test coverage or build time beyond reasonable bounds
----
h2. Related Issues
* [CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438] — prior
Lumberjack flakiness (s390x / isolation)
* [PR #25337|https://github.com/apache/camel/pull/25337] — CI run where flakes
were observed (not caused by PR changes)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
