[
https://issues.apache.org/jira/browse/CAMEL-24368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omar Atie updated CAMEL-24368:
------------------------------
Affects Version/s: 4.22.0
> flaky Tests
> -----------
>
> Key: CAMEL-24368
> URL: https://issues.apache.org/jira/browse/CAMEL-24368
> Project: Camel
> Issue Type: Test
> Affects Versions: 4.22.0
> Environment: Here is the **failing environment** from CI run
> [31115312949](https://github.com/apache/camel/actions/runs/31115312949), job
> [92696861359](https://github.com/apache/camel/actions/runs/31115312949/job/92696861359):
> ---
> ## CI context
> | Field | Value |
> |-------|-------|
> | **Workflow** | `Build and test` (PR validation) |
> | **PR** | [#25337](https://github.com/apache/camel/pull/25337) — CAMEL-23861
> GenAI observability |
> | **Branch** | `cursor/CAMEL-23861-genai-observability-c587` |
> | **Commit** | `8e424d403b86a274f39e8c179d6221d33b94543c` |
> | **Run attempt** | 2 |
> | **Trigger** | `pull_request` against `main` |
> | **Run started** | 2026-08-06 22:52 UTC |
> | **Test phase failed** | 2026-08-07 00:00 UTC (~56 min test phase) |
> | **Failing matrix job** | `build (25, false)` — **JDK 25** |
> | **Other matrix job** | `build (17, false)` — **cancelled** (concurrency),
> not the source of the failure |
> ---
> ## Hardware / OS
> | Field | Value |
> |-------|-------|
> | **Runner** | GitHub-hosted (`ubuntu-latest`) |
> | **Runner label** | `ubuntu-latest` |
> | **OS** | Ubuntu 24.04 (Noble) |
> | **Runner image** | `ubuntu24/20260720.247` |
> | **Architecture** | **linux-amd64 (x86_64)** — *not* s390x / ppc64le /
> aarch64 |
> | **Typical runner specs** | 4 vCPU, 16 GB RAM (standard GitHub-hosted
> runner) |
> ---
> ## Java / build toolchain
> | Field | Value |
> |-------|-------|
> | **JDK** | Eclipse Temurin **25.0.3+9** (x64) |
> | **JAVA_HOME** |
> `/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/25.0.3-9/x64` |
> | **Maven Daemon** | **mvnd 1.0.5** (`linux-amd64`) |
> | **Maven** | **3.9.14** (bundled with mvnd) |
> | **Camel version** | **4.22.0-SNAPSHOT** |
> | **Surefire** | **3.5.6** |
> | **Surefire reruns** | `-Dsurefire.rerunFailingTestsCount=2` |
> ---
> ## Build / test execution profile
> | Field | Value |
> |-------|-------|
> | **Build type** | Incremental PR build (Scalpel change detection) |
> | **Scalpel result** | 605 affected modules (580 would be tested) |
> | **Modules actually tested** | **606 modules** (`install` over full
> incremental `-pl` list) |
> | **mvnd threads** | `-Dmvnd.threads=2` |
> | **Maven artifact threads** | `-Dmaven.artifact.threads=25` |
> | **CI env flag** | `-Dci.env.name=github.com` |
> | **Load context** | Full-reactor incremental build under CI resource
> contention (~606 modules) |
> ---
> ## Test failure behavior in this environment
> | Test | Result in this env |
> |------|---------------------|
> | `LumberjackComponentTest.shouldListenToMessages` | **Failed all 3 Surefire
> runs** (Run 1/2/3) — `ConditionTimeoutException` after **30s** in
> `LumberjackUtil` |
> | `VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath` |
> **Failed Run 1**, **passed Run 2** — `expected: <2> but was: <1>` at line 172
> |
> ---
> ## Important note for JIRA
> This failure happened on **standard GitHub Actions amd64 + JDK 25**, not on
> s390x. So the `@DisabledOnOs(s390x)` guard from CAMEL-21438 would **not**
> have skipped these tests. The environment is a heavy incremental CI build,
> not an alternative-architecture Jenkins node.
> Reporter: Omar Atie
> Priority: Major
>
> h2. Summary
>
> Flaky CI: \{{LumberjackComponentTest.shouldListenToMessages}} timeout and
> \{{VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath}} race
> under heavy incremental builds
>
> h2. Issue Type
>
> Bug
>
> h2. Priority
>
> Major
>
> h2. Affects Version/s
>
> 4.22.0-SNAPSHOT
>
> h2. Component/s
>
> * camel-lumberjack
> * camel-vertx-websocket
>
> h2. Labels
>
> {\{ci}}, \{{flaky-test}}, \{{test-infra}}
>
> ----
>
> h2. Description
>
> h3. Context
>
> During CI validation of [PR
> #25337|https://github.com/apache/camel/pull/25337] (CAMEL-23861 — GenAI
> observability), the build failed on *unrelated flaky tests* while running the
> full incremental build (~606 modules). All AI observability tests in the PR
> passed.
>
> * CI run:
> [31115312949|https://github.com/apache/camel/actions/runs/31115312949]
> * Failing job: \{{Build and test}} (job \{{92696861359}})
> * PR analysis comment:
> [5212287682|https://github.com/apache/camel/pull/25337#issuecomment-5212287682]
>
> The PR branch was rebased onto \{{upstream/main}} and the PR-specific test
> suite (\{{camel-ai/camel-ai-common}}, \{{camel-ai/camel-ai-google-genai}},
> \{{camel-ai/camel-ai-openai}}, \{{components/camel-opentelemetry2}}) passes
> locally with \{{BUILD SUCCESS}}.
>
> ----
>
> h3. Failure 1 — LumberjackComponentTest (build breaker)
>
> *Test:*
> \{{org.apache.camel.component.lumberjack.LumberjackComponentTest#shouldListenToMessages}}
> *Location:*
> \{{components/camel-lumberjack/src/test/java/org/apache/camel/component/lumberjack/LumberjackComponentTest.java:62}}
> *Result:* \{{ConditionTimeoutException}} after 30 seconds (failed all 3
> Surefire reruns)
> *Annotations:* \{{@Isolated}}, \{{@DisabledOnOs(OS.S390X)}} (related to
> [CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438])
>
> {code}
> org.awaitility.core.ConditionTimeoutException: Condition with Lambda
> expression in org.apache.camel.component.lumberjack.LumberjackComponentTest
> was not fulfilled within 30 seconds.
> at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
> at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
> at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
> at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1168)
> at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1141)
> at
> org.apache.camel.component.lumberjack.LumberjackComponentTest.shouldListenToMessages(LumberjackComponentTest.java:62)
> {code}
>
> *Likely cause:* Timing/network sensitivity when the full reactor runs under
> CI load; the test waits for Lumberjack messages over TCP and may not receive
> them reliably within 30s.
>
> ----
>
> h3. Failure 2 — VertxWebsocketTest (intermittent)
>
> *Test:*
> \{{org.apache.camel.component.vertxwebsocket.VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}}
> *Location:*
> \{{components/camel-vertx-websocket/src/test/java/org/apache/camel/component/vertxwebsocket/VertxWebsocketTest.java:218}}
> *Result:* \{{expected: <2> but was: <1>}} on first run; *passed on Surefire
> rerun #2*
>
> {code}
> org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
> at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
> at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
> at
> org.apache.camel.component.vertxwebsocket.VertxWebsocketTest.testSendWithConnectionKeyForRawParameterizedPath(VertxWebsocketTest.java:218)
> {code}
>
> *Likely cause:* Race between WebSocket connection establishment and message
> dispatch when multiple tests share Vert.x resources under
> parallel/full-reactor load.
>
> ----
>
> h2. Steps to Reproduce
>
> # Run a large incremental build that includes both modules, e.g.:
> {code:bash}
> mvn clean install -pl
> components/camel-lumberjack,components/camel-vertx-websocket -am
> {code}
> # Or reproduce CI-like load:
> {code:bash}
> mvn clean install -Dquickly
> # then re-run the failing tests repeatedly
> mvn test -pl components/camel-lumberjack
> -Dtest=LumberjackComponentTest#shouldListenToMessages
> mvn test -pl components/camel-vertx-websocket
> -Dtest=VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath
> {code}
> # Observe intermittent timeouts or assertion failures under resource
> contention.
>
> ----
>
> h2. Expected Result
>
> Both tests pass consistently in CI and local full-reactor builds without
> relying on Surefire reruns.
>
> h2. Actual Result
>
> * \{{LumberjackComponentTest#shouldListenToMessages}} — fails consistently in
> CI (3/3 reruns)
> * \{{VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}} —
> fails intermittently (1/3 reruns in observed run)
>
> ----
>
> h2. Proposed Fix (investigation starting points)
>
> h3. Lumberjack
>
> * Review Awaitility timeout and polling strategy in
> \{{shouldListenToMessages}}
> * Check whether TCP listener/port binding is stable under parallel test
> execution
> * Cross-check [CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438]
> history — test already marked \{{@Isolated}} and disabled on s390x for prior
> flakiness
>
> h3. Vert.x WebSocket
>
> * Add synchronization or use \{{MockEndpoint}} timed assertions before
> counting received messages
> * Ensure WebSocket client connection is fully established before sending
> * Consider \{{@Isolated}} or dedicated Vert.x instance per test if shared
> lifecycle causes races
>
> ----
>
> h2. Acceptance Criteria
>
> * \{{LumberjackComponentTest#shouldListenToMessages}} passes reliably in CI
> (no \{{ConditionTimeoutException}} under standard build load)
> * \{{VertxWebsocketTest#testSendWithConnectionKeyForRawParameterizedPath}}
> passes on first run without depending on Surefire reruns
> * No regression in module test coverage or build time beyond reasonable bounds
>
> ----
>
> h2. Related Issues
>
> * [CAMEL-21438|https://issues.apache.org/jira/browse/CAMEL-21438] — prior
> Lumberjack flakiness (s390x / isolation)
> * [PR #25337|https://github.com/apache/camel/pull/25337] — CI run where
> flakes were observed (not caused by PR changes)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
