[
https://issues.apache.org/jira/browse/CAMEL-24834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117658#comment-18117658
]
Claus Ibsen commented on CAMEL-24834:
-------------------------------------
Analysis (2026-09-21), from the camel-jbang-mcp stepwise benchmark with the
local model (qwen3.6 35b-a3b) on the connect-service/sql example moved to H2
(three steps: register customers with an upsert, a report route, correct one
row in the database). Three runs with the 15-tool authoring set, three with the
same set plus the sql group (camel_runtime_sql, camel_runtime_datasources,
camel_runtime_sql_trace).
*Are tools missing?* Yes, on the MCP side: the server wrapped 27 of the 53
runtime tools and had no sql, datasource, sql trace, circuit breaker, metrics,
spans, EIP stats, startup, route analysis or config drift tool at all
(CAMEL-24867 adds them). The camel-jbang views have the full registry; the
benchmark harness pre-selected 15 tools, so the earlier rounds never showed the
gap.
*Is the set too big?* Not at these sizes: the 15-tool set is ~3.8k tokens, the
full view set ~9.2k (guarded by AiPanelPromptBudgetTest), +3 sql tools ~0.7k.
No measurable slowdown in the runs. Grouping is about keeping the full set out
of a local model's prefix, not about the authoring set.
*Does the model use a group when it is there?*
- Step 3 (correct a row in the database): with the sql tool 3/3, one call (an
UPDATE) and no route written; without it 1/3, the one pass wrote a throwaway
timer route with the UPDATE.
- Steps 1-2: the sql tool was never called, not once, in any run, although its
description says to try a statement against the database first. The model went
straight to the wrong dialect six times out of six (Postgres ON CONFLICT or
MySQL ON DUPLICATE KEY against H2) while saying "the database is H2", and 5 of
6 first statements used the :name parameter form camel-sql does not have
(CAMEL-24869 for a validator hint). Both sets 0/3 on step 1. The tool helps a
task that *is* a data operation; it does not make the local model verify its
own work.
*Conclusion for the design:* groups for local models only, as agreed, and the
choice of group should not be the model's: pick it from the app. At panel start
(and on reload) the registry knows the components of the running routes:
sql/jdbc/jpa present -> add the sql group; circuitBreaker present -> the
resilience group; --observe -> tracing/metrics; everything else stays in the
"full" mode a frontier model gets. That gives the local model the 15 core tools
plus the one group it can use without a meta tool it would not call. A
camel_load_tools meta tool can come later for the frontier-model path, where
the prefix does not matter.
Camel bugs found while building the experiment, all with PRs: CAMEL-24865
(--source-dir turned a missing optional classpath resource into a missing file:
no datasource auto-configure), CAMEL-24866 (a new route file next to a
multi-route file failed the reload with a duplicate route id), CAMEL-24867 (the
missing MCP tools, and nameOrPid was required in the schema although documented
as optional).
> camel-jbang-mcp: load AI tool groups (SQL, tracing, resilience) from what the
> selected app has, read-only SQL for local models
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24834
> URL: https://issues.apache.org/jira/browse/CAMEL-24834
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.23.0
>
>
> The camel-jbang-mcp server's AI tool set for local models is a fixed core set
> of 19 tools. Hosted providers get the full set of 47. A local model therefore
> cannot use the tools it would need for a question that depends on what the
> running app has: it cannot run a SQL query against the app's datasource to
> check that a message landed in a table, cannot read spans when tracing is on,
> and cannot inspect a circuit breaker or aggregator.
> A local model is typically pointed at one app. The server already knows which
> app is selected and the dev consoles say what that app has. So the tool set
> should be a function of the app, not only of the provider.
> Proposal:
> * Keep the core set as the base for local models.
> * Add tool groups that are loaded when the selected app reports the matching
> feature:
> ** SQL: execute_sql, get_datasources, get_sql_trace, when the datasource
> console reports a datasource
> ** tracing: get_spans and trace_control, when tracing is enabled
> ** resilience: get_circuit_breakers, when a circuit breaker is in the routes;
> the aggregator state likewise
> * When a group joins, add one line of guidance with it: the datasource names,
> and that table names come from the SQL trace, so the model does not guess a
> schema.
> * For local models the SQL tool is read-only (SELECT only) unless the user
> enables writes, the same way the tool mode is a setting today. The model is
> not the security boundary. Hosted providers in full mode keep the current
> behaviour.
> * The prefix changes only when the selected app changes or the app gains the
> feature after a reload, so the prompt cache is re-filled once per app.
> Later, a meta tool that lets the model ask for a named group on demand (the
> remaining tools: screen, drawing, tape, memory and threads) so a local model
> can reach the full set at the cost of one re-prefill per group. Coarse
> groups, since every change of the tool set re-fills the prompt prefix.
> The prompt budget test guards the size of the prefix in both modes and should
> cover the loaded groups too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
