[
https://issues.apache.org/jira/browse/CAMEL-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141065#comment-18141065
]
Claus Ibsen commented on CAMEL-26760:
-------------------------------------
Spike (step 1), on main be43a877 + CAMEL-26761: {{camel run shop.camel.yaml
--stub=remote --chaos}} with route {{orders}} (timer -> {{kafka:orders}}) and
route {{refunds}} ({{kafka:refunds?breakOnFirstError=true}} ->
{{kafka:refunds-done}}), no broker.
What works:
* The stubbed Kafka consumer starts without a broker, and {{camel cmd send
spike --endpoint=kafka:refunds --body=R-1}} feeds it (route refunds logs it).
* Chaos rules match stubbed endpoints: {{endpoint=kafka:orders error
rate=every:2 until=6}} fired 3 of 6, and {{route=refunds error ...}} fails the
messages entering the stubbed consumer. The report, the expectation and the
recent faults are right.
What does not:
# *chaos-suggest is blind with stubs:* "Nothing to suggest: the routes call
nothing remote, and have no input from outside". {{StubEndpoint.isRemote()}} is
always false, and {{ChaosAdvisor}} goes by {{isRemote()}}. {{camel get
endpoint}} also shows the stubbed Kafka endpoints as scope {{local}}.
({{DefaultRouteTopologyDumper}} already skips stub endpoints for the same
reason.)
# *KEPT is wrong with stubs:* route refunds has {{breakOnFirstError=true}}, so
real Kafka reads a failed record again, but the report says KEPT 0.
{{ChaosConsumers.keepsFailed}} reads the option through the configurer of the
stub component, not Kafka's, though the endpoint still has the original URI
({{kafka://refunds?brokers=...&breakOnFirstError=true}}). And the stub consumer
(a seda queue) does not read a failed message again either. So an exploration
on stubs would report "failed, not kept" for a route that keeps its messages: a
false finding.
Proposed fix, as the first PR of this ticket ("chaos sees through stubs"):
* A shadow stub (the stub component behind a {{kafka:}} URI) counts as the
original component: remote for the suggestions, and its kept options read from
the original URI.
* The report says what the stub changes: "failed, kept by kafka (stubbed: not
read again)", so neither the report nor an agent calls it lost.
* Not proposed now: making the stub consumer read a kept message again, as the
real consumer would.
_Claude Code on behalf of Claus Ibsen_
> camel-chaos - stage 3: let there be chaos, an AI agent explores chaos rules
> on a project by itself and reports back
> -------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26760
> URL: https://issues.apache.org/jira/browse/CAMEL-26760
> Project: Camel
> Issue Type: New Feature
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> The user tells an AI agent "let there be chaos" on a Camel project. Within a
> time budget, the agent runs the project, tries chaos rules by itself, sends
> traffic, reads the report, errors and log, and comes back with a report: what
> it tried, what held, what broke, and scenarios to replay.
> Most of the pieces are on main from stage 1 and 2 (CAMEL-26330, CAMEL-26607):
> chaos suggest and the tried table (what to try), chaos check (safe to add),
> expectations (a verdict that does not depend on the model), scenarios with a
> seed (replay and CI gate), and the MCP tools. The agent that wrote the chaos
> example already did this by hand, and found CAMEL-26734. This ticket makes it
> a feature.
> h3. What is missing
> # *A playbook.* An MCP prompt {{camel_chaos_explore}} (budget, focus, traffic
> allowed) next to the prompts in {{PromptDefinitions}}, with the same text
> behind {{camel ask}} and a key on the TUI chaos screen. Loop: baseline
> without rules (errors already there are not blamed on chaos), suggest, pick a
> kind not tried, state the expectation, check, add with a short until=, send
> traffic, wait for the verdict, read, clear, next. Single faults first, then
> pairs ("two failures meet").
> # *Traffic.* Timer routes feed themselves, platform-http can be fed with
> curl, but Kafka, JMS, AMQP etc. cannot. Stage 1: run with {{--stub=remote}}
> by default and feed the stubbed consumers with {{camel cmd send}}, which
> needs {{--repeat}} / {{--interval}} (today it sends one message), also on
> {{camel_runtime_send}}. The suggestions say per route how to feed it. Stubbed
> messages are not real Kafka, but close enough to test the error handling.
> # *Safety.* The faults are local, the traffic is not: by default remote
> endpoints are stubbed, and the agent asks before it sends to anything real.
> One consent covers the whole budget, instead of a confirm per rule.
> camel-chaos enforces the deadline: rules are capped to the end of the
> exploration, and cleared when it ends. The agent never edits the routes:
> hints only.
> # *One experiment per call.* {{chaos_experiment}} (rule(s), traffic, max
> wait) returns a compact result when the expectations are decided: verdict,
> per route delta, new errors, faults fired. About 10 large calls instead of
> 100 small ones, which matters for small local models.
> # *A journal and a report.* camel-chaos keeps a journal of the exploration:
> baseline, each experiment with its hypothesis, rules, seed and outcome. The
> report is text for the CLI, JSON for tools and markdown in
> {{chaos/report-<time>.md}}. Each finding is classified (route gap: no
> breaker, no DLC, not idempotent, hand-off loses; or suspected Camel bug),
> with evidence and a hint, and experiments worth keeping are saved as
> {{chaos/explore-*.chaos}} scenarios.
> # *TUI.* An Explore mode on the chaos screen: budget dialog, the experiments
> live, and the report at the end.
> h3. Stage 2 (later)
> * Realistic traffic with Citrus ({{camel test run}}): real brokers through
> test-infra, message shapes from a schema or samples.
> * A non-AI explore ({{--no-ai}}): the top suggestions in order, each with its
> own expectation and traffic, for CI and as a floor for small models.
> * Several integrations at once (a hand-off over a real broker).
> h3. Open questions
> * Do chaos rules on {{endpoint=kafka:...}} match the stubbed endpoint, and
> does {{camel cmd send}} to a stubbed consumer feed it? To verify first.
> * CLI shape: {{camel cmd chaos-explore --budget=10m}} (uses the AI of camel
> ask) or only {{camel ask "let there be chaos"}}.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
