[
https://issues.apache.org/jira/browse/CAMEL-24844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118424#comment-18118424
]
Claus Ibsen commented on CAMEL-24844:
-------------------------------------
h3. Prior art: this has been asked for since 2008
Before filling the catalog metadata of the design note above, the tickets that
already asked for it:
* CAMEL-330 (2008-02-12) "Components / endpoints should advertise the data
types they support" - closed *Abandoned* on 2025-12-28 after 17 years. Its
wording is the same idea: "If an endpoint would publish what data types it
supports, static analysis would be possible to determine if messages could be
processed successfully or additional type converters are necessary."
* CAMEL-11132 (2017-04-10) "Provide type metadata for each
Components/DataFormats" - *still Open*, and the natural home for the static
half. Tomohisa Igarashi already named the hard part: "xslt only
consumes/produces XML so XML could be provided via annotation. But dozer
consumes/produces arbitrary data formats and the actual type to be used is
determined via configuration... Sometimes even body of the input message would
affect the possible output types. To visualize this we'll need some kind of
simulation with a test message." That last sentence is the runtime half of this
ticket, written nine years earlier.
* CAMEL-17620 (2022) "Document data type in body from the endpoints of each
component" - closed *Information Provided*; the conclusion there was to start
with the Kamelets.
* CAMEL-18698 - fixed in 4.0, and what it delivered is the
{{@DataTypeTransformer}} SPI rather than a type map: 60 transformers over 13
component groups (CloudEvents, embeddings, jackson). Useful as the conversion
mechanism the validator can suggest, not as the metadata.
State of the metadata today: the catalog carries rich header metadata (17
headers with a javaType for {{file}}) and *nothing about the body*. Of 250
Kamelets, *19* declare {{dataTypes:}}.
h3. Does the runtime half replace the static half? No - measured
The runtime facts merged on 2026-09-20 (08ee8b6113cd, 81def551326e): body type
and size per step in the message history, the failure table, the error
registry, the camel-jbang views. Four benchmark series ran with those facts
available to the model - s11, s12, s13, s14 - and the
{{contracts-openapi-server}} rung stayed *0/5* in all four. With the static
check of PR 26800 in place, s15 put it at *4/5* (series total 43/65, best of
the series; about half of the jump is the check and half a scoring fix in the
harness).
The reason is the loop: a model writes, validates, fixes, and does not run the
application between edits, so a runtime message arrives after the moment that
could have used it. The runtime half earns its place differently - it is the
ground truth that keeps generated static metadata honest, and it is the answer
to "what is the body here" on a live application.
h3. How much of the catalog has to be mapped
Measured over 1712 real YAML route files (camel-jbang-examples, camel-examples,
camel-kamelets, and the benchmark output):
|| | count ||
| components in the catalog | 407 |
| distinct schemes actually used | 122 |
| share of uses covered by the top 10 schemes | 89.5% |
| share covered by the top 14 | 92.3% |
Top 10: {{file, direct, timer, kamelet, http, rest-openapi, cron, xslt, seda,
sql}}.
Data formats are more lopsided still: {{json}} is 1214 of 1317 uses (92%), and
{{json}} + {{jacksonXml}} + {{csv}} is 98%.
So the first pass is about 15 components, 3 data formats, the EIPs and the
languages - not a campaign across 407 components. The tail is not a gap: a
component whose output type depends on its configuration or on the remote
system is exactly where the metadata should say unknown and the walk should
stop silently, which is how the phase A walk already behaves.
h3. Phase letters
The letters in this ticket have collided: the 2026-09-20 comment says "Phase B
(runtime facts) is merged... Remaining: phase A, then C and D", while PR 26800
calls the OpenAPI specification work phase B. Renaming them to what they are,
to keep the ticket readable:
# *presence* - does the body exist at this step (PR 26800, merged/open)
# *runtime facts* - body type and size per step in the message history and the
views (merged, 4.23.0)
# *catalog type metadata* - format and carrier per component, data format,
language and EIP, filled by an AI pass over the sources and verified against
the message history of the reference runs
# *type lattice* - the validator carries the type from step to step and
suggests the conversion ({{convertBodyTo}}, {{unmarshal}},
{{transformDataType}}) that makes the next step valid
> Body type flow: the validator carries the body's Java type from step to step
> and warns where a step assumes text; the message history's recorded body
> types feed the same check at runtime
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24844
> URL: https://issues.apache.org/jira/browse/CAMEL-24844
> Project: Camel
> Issue Type: Improvement
> Components: camel-core, camel-jbang, camel-yaml-dsl
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.23.0
>
>
> Camel carries the payload as the natural Java object: a GenericFile from the
> file consumer, a Map or List after {{unmarshal json}}, a byte[] or a
> StreamCache after {{marshal}}, an InputStream from http. Integration people
> think in text and bytes, and write routes as if the body were a String. Every
> failure in the round-2 local-model benchmark that was not a YAML shape error
> was that collision, and a person writing the same route gets the same runtime
> exception:
> * {{${body.orderId}}} after {{unmarshal: json}} (the body is a Map):
> MethodNotFoundException, in five examples. The runtime message now says "the
> value is a Map: a key is read with [orderId]".
> * {{body.email}} in a Groovy expression on a GenericFile, before any
> conversion: MissingPropertyException.
> * {{jsonpath}} on a body that is already a Map after unmarshal, or on a null
> body on a timer route (CAMEL-24838).
> * {{${body}}} after {{unmarshal: json}} treated as JSON text: it is the Map's
> toString ({{{id=ORD-1001, ...}}}).
> * {{new ByteArrayInputStream(body)}} in Groovy after {{marshal}}: with stream
> caching on (the Camel CLI default) the body is a StreamCache, not a byte[]:
> "could not find matching constructor". Same route, different Java type
> depending on a runtime setting.
> Measured with the CLI on 4.23.0-SNAPSHOT: logging {{${body}}} prints readable
> text at every stage (GenericFile, Map, marshal output, InputStream with
> stream caching on, twice); only with
> {{camel.main.streamCachingEnabled=false}} does a log step consume an
> InputStream and leave every later step an empty body.
> The object model must stay: it is why a Map flows into a bean and a stream
> into a file without copies. What can change is that Camel knows the type at
> every step and says nothing until the step that assumes text explodes.
> Proposal, in two halves that share one rule set (which step produces which
> type, which expressions and steps need which type):
> # *Static, in {{camel validate yaml}} and the camel-jbang-mcp validation
> tool.* Walk the route and carry the body type forward from the catalog:
> {{from: file}} gives GenericFile, {{unmarshal: json}} gives Map/List (or the
> unmarshalType), {{marshal}} gives byte[]/StreamCache, {{unmarshal:
> jacksonXml}} gives Map, {{split}} on a List gives an element, {{setBody:
> constant}} gives String, {{to: http}} gives InputStream, {{convertBodyTo}}
> gives its type, a bean gives its return type when it can be resolved. At each
> step check the expression against the type and say it in the words of the
> text world, with the line: "after unmarshal json the body is a Map:
> ${body.orderId} fails, write ${body[orderId]}"; "jsonpath needs the JSON
> text, move it before the unmarshal or use simple on the Map"; "the body is
> the file (GenericFile): convert it with convertBodyTo String or unmarshal it
> before the Groovy expression reads body.email"; "after marshal the body is
> bytes (a cached stream when stream caching is on): a Groovy script gets it
> with exchange.message.getBody(byte[])". Unknown types (a bean with no source)
> stop the flow silently, no false positives.
> # *At runtime, from the message history.* The message history already
> records, per step, what went through, including the class of the body at each
> previous step. Use it in two places: (a) the exception message of a failing
> expression or bean invocation can say "the body reaching this step was a
> java.util.LinkedHashMap (set by unmarshal at line 12)", the fact a person
> otherwise gets from a debugger; (b) {{camel trace}} and the camel-jbang-mcp
> tools that show a message's history can show the body type per step, so the
> flow is visible on a running app, and the static half can be checked against
> it (a recorded history of the reference run is the ground truth for the
> catalog's type rules).
> The static half is where most of the value is (it reaches the file before it
> runs); the runtime half makes the rule set honest and gives the "what is the
> body here" answer on a live app.
> Context: the round-2 local-model benchmark on the camel-jbang-examples ladder
> (2026-09-19); the pattern was in aggregator, order-lines, csv-to-json,
> groovy, openapi-client, filter-and-multicast, json-transform.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
