[
https://issues.apache.org/jira/browse/CAMEL-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141045#comment-18141045
]
Claus Ibsen commented on CAMEL-26760:
-------------------------------------
On replay with a seed: the seed does not replay a run. It fixes the sequence of
random draws of each rule (the 7th match of r1 gets draw 7), not which message
is the 7th. That depends on the order of the traffic, the threads and the time
windows (until=2m, from=, burst:, within=), as "Time and the seed" in the chaos
doc says.
What has to replay is the verdict: an expectation (lost<=0, failed<=0) is
checked over all messages, so most findings show with any draws. So when the
agent saves a finding as a scenario, it:
* prefers counts over probability (rate=every:5) and over time (until=50 with
50 messages, not until=60s),
* narrows with when= a finding tied to one message,
* keeps the feed with the scenario (the sample messages and their order; new, a
.chaos file has no feed today), with one consumer thread,
* runs the scenario twice more, and the report says per finding how it replays
(always, or flaky 1 of 2).
An exact replay, times included, needs a camel-test-dst test in virtual time:
stage 2, for findings that depend on timing.
_Claude Code on behalf of Claus Ibsen_
> camel-chaos - stage 3: let there be chaos, an AI agent explores chaos rules
> on a project by itself and reports back
> -------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26760
> URL: https://issues.apache.org/jira/browse/CAMEL-26760
> Project: Camel
> Issue Type: New Feature
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> The user tells an AI agent "let there be chaos" on a Camel project. Within a
> time budget, the agent runs the project, tries chaos rules by itself, sends
> traffic, reads the report, errors and log, and comes back with a report: what
> it tried, what held, what broke, and scenarios to replay.
> Most of the pieces are on main from stage 1 and 2 (CAMEL-26330, CAMEL-26607):
> chaos suggest and the tried table (what to try), chaos check (safe to add),
> expectations (a verdict that does not depend on the model), scenarios with a
> seed (replay and CI gate), and the MCP tools. The agent that wrote the chaos
> example already did this by hand, and found CAMEL-26734. This ticket makes it
> a feature.
> h3. What is missing
> # *A playbook.* An MCP prompt {{camel_chaos_explore}} (budget, focus, traffic
> allowed) next to the prompts in {{PromptDefinitions}}, with the same text
> behind {{camel ask}} and a key on the TUI chaos screen. Loop: baseline
> without rules (errors already there are not blamed on chaos), suggest, pick a
> kind not tried, state the expectation, check, add with a short until=, send
> traffic, wait for the verdict, read, clear, next. Single faults first, then
> pairs ("two failures meet").
> # *Traffic.* Timer routes feed themselves, platform-http can be fed with
> curl, but Kafka, JMS, AMQP etc. cannot. Stage 1: run with {{--stub=remote}}
> by default and feed the stubbed consumers with {{camel cmd send}}, which
> needs {{--repeat}} / {{--interval}} (today it sends one message), also on
> {{camel_runtime_send}}. The suggestions say per route how to feed it. Stubbed
> messages are not real Kafka, but close enough to test the error handling.
> # *Safety.* The faults are local, the traffic is not: by default remote
> endpoints are stubbed, and the agent asks before it sends to anything real.
> One consent covers the whole budget, instead of a confirm per rule.
> camel-chaos enforces the deadline: rules are capped to the end of the
> exploration, and cleared when it ends. The agent never edits the routes:
> hints only.
> # *One experiment per call.* {{chaos_experiment}} (rule(s), traffic, max
> wait) returns a compact result when the expectations are decided: verdict,
> per route delta, new errors, faults fired. About 10 large calls instead of
> 100 small ones, which matters for small local models.
> # *A journal and a report.* camel-chaos keeps a journal of the exploration:
> baseline, each experiment with its hypothesis, rules, seed and outcome. The
> report is text for the CLI, JSON for tools and markdown in
> {{chaos/report-<time>.md}}. Each finding is classified (route gap: no
> breaker, no DLC, not idempotent, hand-off loses; or suspected Camel bug),
> with evidence and a hint, and experiments worth keeping are saved as
> {{chaos/explore-*.chaos}} scenarios.
> # *TUI.* An Explore mode on the chaos screen: budget dialog, the experiments
> live, and the report at the end.
> h3. Stage 2 (later)
> * Realistic traffic with Citrus ({{camel test run}}): real brokers through
> test-infra, message shapes from a schema or samples.
> * A non-AI explore ({{--no-ai}}): the top suggestions in order, each with its
> own expectation and traffic, for CI and as a floor for small models.
> * Several integrations at once (a hand-off over a real broker).
> h3. Open questions
> * Do chaos rules on {{endpoint=kafka:...}} match the stubbed endpoint, and
> does {{camel cmd send}} to a stubbed consumer feed it? To verify first.
> * CLI shape: {{camel cmd chaos-explore --budget=10m}} (uses the AI of camel
> ask) or only {{camel ask "let there be chaos"}}.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
