[
https://issues.apache.org/jira/browse/CAMEL-25539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126988#comment-18126988
]
Federico Mariani commented on CAMEL-25539:
------------------------------------------
Numbers from the evaluation transcripts (Claude Code, Haiku 5.5 and Sonnet 5.5,
runs from 2026-10-08, camel-jbang-mcp 4.22.1): 44 of 53
{{camel_validate_yaml_dsl}} calls answered invalid. By cause: {{onException}}
as a step inside {{steps}} 10, an unknown property 9 ({{loggerName}} 3,
{{namespaces}} 2, {{onException}}, {{milliseconds}}, {{throw}}, {{prefix}}),
{{handled}} as a boolean 3. Every one was fixed in the next call, so the
validator does its job; this issue is about the doc answer that precedes the
first write (165 of 199 MCP calls in those trials came before the first route
file was written, so the model reads the doc, writes, and only then learns the
shape from the validator).
_Claude Code on behalf of Croway_
> camel-jbang-mcp - camel_catalog_doc for an EIP or language does not say where
> the element goes nor the YAML shape of expression and namespace options
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25539
> URL: https://issues.apache.org/jira/browse/CAMEL-25539
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang-mcp
> Affects Versions: 4.22.1
> Reporter: Federico Mariani
> Priority: Major
>
> In camel-jbang-mcp 4.22.1, {{camel_catalog_eip_doc}} and
> {{camel_catalog_language_doc}} return the model options and nothing about how
> the element is written in YAML or where it goes. On main both are replaced by
> {{camel_catalog_doc}} (CAMEL-24712), which has the same gap for EIPs and for
> options whose type is not a scalar. {{camel_catalog_sample}} (CAMEL-24693,
> CAMEL-25370) now answers the placement question, but only when an agent
> thinks of calling it after reading the doc.
> In an evaluation of coding agents (Claude Code and IBM Bob) on migration
> tasks with camel-jbang-mcp 4.22.1, the doc answers were read and then not
> enough to write valid YAML:
> * 4 of 20 sessions that asked for the onException doc then wrote
> {{onException}} as a step inside {{from.steps}} and got {{property
> 'onException' is not defined in the schema}}.
> * 34 sessions wrote {{handled: true}} / {{handled: false}} and got {{boolean
> found, object expected}}. The doc lists {{handled}} as {{"type":"object"}},
> with nothing that says it is an expression.
> * About 20 sessions wrote the XPath namespaces as {{namespaces:}}, as a
> string, or as a list of {{\{prefix: uri\}}} maps. The doc lists {{namespace}}
> as {{"type":"array"}} with the description "Injects the XML Namespaces of
> prefix to uri mappings."; the {{[\{key, value\}]}} shape appears nowhere in
> the answer.
> h3. Reproduction
> {noformat}
> camel_catalog_doc name=onException kind=eip
> -> kind, name, title, description, label, input, output, options
> no placement; handled:
> {"name":"handled","type":"object","required":false,"group":"common",...}
> camel_catalog_doc name=xpath kind=language
> -> namespace: {"name":"namespace","type":"array","description":"Injects the
> XML Namespaces of prefix to uri mappings."}
> {noformat}
> What the YAML DSL expects:
> {code:yaml}
> - onException: # a top-level entry, not a step
> exception: [java.net.ConnectException]
> handled:
> constant:
> expression: "true"
> - route:
> from:
> uri: direct:start
> steps:
> - setHeader:
> name: price
> expression:
> xpath:
> expression: /c:order/c:price/text()
> namespace:
> - key: c
> value: http://acme.com/cheese
> {code}
> h3. Cause
> {{dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/CatalogDocs.java}}:
> * {{eipDoc(EipModel, ...)}} puts the options only. The placement is already
> computed by {{CatalogSamples.placement(String)}} (TOP_LEVEL, ERROR_HANDLERS),
> but only {{camel_catalog_sample}} returns it.
> * {{optionToJson}} adds a YAML hint only for an element written as one of its
> kinds ({{isWrittenAsKind}}, CAMEL-25370). An option of kind {{expression}}
> ({{handled}}, {{continued}}, {{retryWhile}}, {{completionPredicate}}, ...)
> and an element list of {{PropertyDefinition}} ({{namespace}} of xpath,
> xquery, xtokenize) are shown as {{object}} / {{array}} with no shape.
> h3. Fix
> * In the EIP answer of {{camel_catalog_doc}}, add the {{placement}} from
> {{CatalogSamples.placement}} ("top-level entry: a list item at the same level
> as route or from, not a step inside a route" for onException,
> routeConfiguration, onCompletion, intercept...) and a hint that
> {{camel_catalog_sample}} returns a validated YAML example.
> * In {{optionToJson}}, add a {{yaml}} hint for an option of kind
> {{expression}} (e.g. {{handled: \{constant: \{expression: "true"\}\}}}, as
> the validator hint of CAMEL-24702 says), and for a list of
> {{PropertyDefinition}} ({{namespace: [\{key: c, value: http://...\}]}}).
> Related: CAMEL-25370 (placement in camel_catalog_sample), CAMEL-24915 (accept
> a plain value for handled). Not included: the jq functions {{header()}} /
> {{property()}} are already in the doc excerpt on main (CAMEL-25235), and the
> simple functions are listed (CAMEL-24804).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
