[
https://issues.apache.org/jira/browse/CAMEL-23861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101946#comment-18101946
]
Omar Atie commented on CAMEL-23861:
-----------------------------------
h3. Analysis & proposed approach for CAMEL-23861
I've reviewed the ticket description, both existing comments, CAMEL-23860
(resolved), PR #24362 (TUI AI usage prototype), and the current Camel + Spring
AI codebases. Below is an impact analysis and a concrete implementation plan
for approval before starting work.
----
h4. Current state (what we already have)
* *CAMEL-23860 (done)* — langchain4j producers already expose token usage and
finish reason as exchange headers:
** \{{camel-langchain4j-chat}} — \{{CamelLangChain4jChat*TokenCount}},
\{{CamelLangChain4jChatFinishReason}}
** \{{camel-langchain4j-tools}} — same pattern (accumulated across tool-calling
iterations)
** \{{camel-langchain4j-agent}} — same pattern via \{{Result<String>}}
** \{{camel-langchain4j-embeddings}} — same pattern
* *Route-level tracing* exists via \{{camel-opentelemetry2}} (native OTel) and
\{{camel-micrometer-observability}} (Micrometer Observation + Tracing — the
successor to deprecated \{{camel-observation}}).
* *No GenAI-specific instrumentation* — grep shows zero \{{gen_ai.*}}
attributes or metrics anywhere in Camel today.
* *TUI / developer tooling* (PR #24362, \{{AiTraceTools}}) already reads AI
headers from live exchanges — but that is message-history based, not
OTel/Micrometer backend data.
* *\{{camel ask}} / \{{LlmClient}}* — separate path (in-memory dev usage),
explicitly out of scope for this ticket per comment #2.
*Gap:* CAMEL-23860 puts data on the Exchange; this ticket is about propagating
it into production observability backends (Prometheus/Grafana, Jaeger, etc.)
using OTel GenAI semconv.
----
h4. Spring AI reference (what to align with, what to adapt)
Spring AI implements GenAI observability via *Micrometer Observation* (not raw
OTel API):
* \{{DefaultChatModelObservationConvention}} maps request/response → semconv
attributes
* \{{ChatModelMeterObservationHandler}} records metrics automatically
* Metrics: \{{gen_ai.client.operation}} (duration),
\{{gen_ai.client.token.usage}} (tokens)
* Span attributes: \{{gen_ai.operation.name}}, \{{gen_ai.system}},
\{{gen_ai.request.model}}, \{{gen_ai.response.model}},
\{{gen_ai.usage.input_tokens}}, \{{gen_ai.usage.output_tokens}},
\{{gen_ai.response.finish_reasons}}
* Each LLM call gets its own observation/span (e.g. span name \{{"chat
gpt-4o"}}), kind INTERNAL
* *Important decision (spring-ai#1174):* Spring AI pins a *stable* semconv
subset for production rather than tracking every OTel semconv draft change.
Optional opt-in module recommended for bleeding-edge semconv.
*Camel adaptation:* Camel should follow the same *semconv names and metric
names* for dashboard portability, but hook at the *Producer* level (wrapping
langchain4j \{{ChatModel.chat()}} calls) rather than inside langchain4j itself.
Camel has two tracing stacks to support:
||Stack||Typical deployment||GenAI integration path||
|\{{camel-micrometer-observability}}|Spring Boot + Micrometer|Micrometer
\{{Observation}} (closest to Spring AI)|
|\{{camel-opentelemetry2}}|Standalone Camel, Quarkus, etc.|Child span via
\{{camel-telemetry}} API → OTel attributes|
|\{{camel-micrometer}} only|Metrics without tracing|Direct \{{MeterRegistry}}
counters/timers|
----
h4. Impact analysis
*Components directly affected (Phase 1):*
* \{{camel-langchain4j-chat}}, \{{camel-langchain4j-tools}},
\{{camel-langchain4j-agent}}, \{{camel-langchain4j-embeddings}}
* New shared module (proposed): \{{camel-ai-observability}} under
\{{camel-ai-parent}}
*Downstream consumers (benefit, no code change required):*
* Grafana / Prometheus dashboards using standard \{{gen_ai_*}} metric names
* Jaeger / Tempo / any OTel backend reading \{{gen_ai.*}} span attributes
* Future TUI "AI Usage" tab (Phase 2 data source — reads OTel spans from
in-memory collector, inspired by boot-ui)
*Why child spans per LLM call (not just enriching the route span):*
* \{{langchain4j-tools}} and \{{langchain4j-agent}} can invoke the LLM
*multiple times* per single Exchange (tool-calling loops). Aggregating
everything onto the route span loses per-call latency, per-call token
breakdown, and makes cost attribution inaccurate.
* This matches Spring AI's one-observation-per-\{{ChatModel.call()}} model.
*Key challenge — model & provider resolution:*
* Langchain4j \{{ChatModel}} is an interface; model name is not uniformly
exposed.
* \{{ChatResponse}} today gives us \{{tokenUsage()}} and \{{finishReason()}}
but not consistently \{{model()}}.
* *Mitigation:* (1) optional endpoint params \{{genAiSystem}} + \{{modelName}},
(2) heuristic mapping from model impl class (\{{OpenAiChatModel}} →
\{{openai}}), (3) add \{{CamelLangChain4j*Model}} response headers as a small
follow-up if langchain4j API exposes model in response metadata.
*Risk — OTel semconv stability:*
* GenAI semconv moved to a separate repo and is still evolving. Recommend
pinning to the same stable attribute set Spring AI ships (per spring-ai#1174),
document the version, and defer unstable attributes to an optional future
module.
----
h4. Recommended architecture
*New module:* \{{components/camel-ai/camel-ai-observability}}
Provides a small, dependency-light API used by all AI producers:
{code}
GenAiObservation observation = GenAiObservationSupport.start(exchange,
GenAiOperationContext.builder()
.operationName("chat") // or "embeddings", "agent"
.system("openai") // gen_ai.system
.requestModel("gpt-4o") // gen_ai.request.model
.componentScheme("langchain4j-chat")
.build());
try {
ChatResponse response = chatModel.chat(...);
observation.recordSuccess(GenAiUsage.from(response));
return response;
} catch (Exception e) {
observation.recordError(e);
throw e;
} finally {
observation.close();
}
{code}
*Runtime backend selection (auto-detect from CamelContext, all optional):*
# *Micrometer Observation path* — if \{{ObservationRegistry}} +
\{{camel-micrometer-observability}} active:
#* Create observation named \{{"{operation} \{model}"}} (Spring AI convention)
#* Apply \{{GenAiObservationConvention}} for span attributes + low/high
cardinality key values
#* \{{DefaultMeterObservationHandler}} records \{{gen_ai.client.operation}} and
\{{gen_ai.client.token.usage}} automatically
# *OpenTelemetry path* — if \{{camel-opentelemetry2}} active (and no
ObservationRegistry):
#* Create child span via \{{SpanLifecycleManager}} / active Exchange span stack
#* Set \{{gen_ai.*}} attributes via \{{Span.setTag()}}
#* Record duration on span
# *Micrometer-only path* — if \{{MeterRegistry}} present
(\{{camel-micrometer}}) but no tracer:
#* Record counters/timers directly with same metric names as Spring AI
All paths are no-op when no observability component is on the classpath — zero
overhead for users who don't enable tracing/metrics.
*Shared semconv constants class:*
* \{{GenAiAttributes}} — attribute key constants
* \{{GenAiMetrics}} — metric name constants (aligned with Spring AI
\{{AiObservationMetricNames}})
* \{{GenAiOperationType}} — enum: CHAT, EMBEDDINGS, AGENT, TOOLS
----
h4. Phased delivery plan
*Phase 1 — langchain4j GenAI observability (this ticket, target 4.22)*
| Step | Work | Notes |
| 1 | Create \{{camel-ai-observability}} module with
\{{GenAiObservationSupport}}, conventions, constants | Optional/provided deps
on \{{camel-telemetry}}, \{{camel-micrometer}}, Micrometer Observation |
| 2 | Integrate into \{{langchain4j-chat}} producer | Single LLM call per
Exchange — simplest case |
| 3 | Integrate into \{{langchain4j-embeddings}} producer | Operation name
\{{embeddings}} |
| 4 | Integrate into \{{langchain4j-tools}} producer | *One observation per
tool-calling iteration* inside the loop |
| 5 | Integrate into \{{langchain4j-agent}} producer | One observation per
agent \{{chat()}} call |
| 6 | Unit tests | Mock \{{MeterRegistry}} + InMemory OTel exporter; assert
attribute names and metric tags |
| 7 | IT with \{{camel-opentelemetry2}} test extension | Verify child spans
appear with \{{gen_ai.*}} attributes |
| 8 | IT with \{{camel-micrometer-observability}} | Verify
\{{gen_ai_client_operation_seconds_*}} and token metrics |
| 9 | Documentation | Component docs + short upgrade-guide note (new optional
module, no breaking changes) |
*Phase 2 — broaden coverage (separate tickets)*
* Extend same \{{camel-ai-observability}} helper to \{{camel-openai}},
\{{camel-spring-ai-chat}}, AWS Bedrock, Watsonx, etc. (many already have token
headers)
* Wire TUI AI Usage tab to read GenAI spans from \{{camel-telemetry-dev}} /
OTel in-memory exporter (feeds the boot-ui-inspired panel)
*Phase 3 — optional advanced semconv (separate ticket, opt-in module)*
* \{{camel-ai-observability-otel-latest}} for unstable OTel semconv attributes,
explicit opt-in only (mirrors Spring AI recommendation)
*Explicitly NOT in scope for Phase 1:*
* \{{camel ask}} / \{{LlmClient}} / \{{AiPanel}} developer token tracking
(separate path, already in PR #24362)
* Streaming per-chunk metrics (langchain4j streaming API not uniformly
instrumented yet)
* Cost estimation / pricing dashboards (needs provider pricing tables)
----
h4. Metrics & span attributes (Phase 1 target set)
*Span attributes (stable OTel GenAI semconv subset, matching Spring AI):*
* \{{gen_ai.operation.name}} — \{{chat}}, \{{embeddings}}, \{{generate_content}}
* \{{gen_ai.system}} — \{{openai}}, \{{anthropic}}, \{{ollama}}, etc.
* \{{gen_ai.request.model}}
* \{{gen_ai.response.model}} (when available)
* \{{gen_ai.usage.input_tokens}} / \{{gen_ai.usage.output_tokens}}
* \{{gen_ai.response.finish_reasons}}
* \{{camel.component}} — \{{langchain4j-chat}} (Camel-specific, low cardinality)
*Metrics (Spring AI compatible names):*
* \{{gen_ai.client.operation}} — Timer, tags: \{{gen_ai.operation.name}},
\{{gen_ai.system}}, \{{gen_ai.request.model}}, \{{error.type}} (on failure)
* \{{gen_ai.client.token.usage}} — Counter/Distribution, tags: above +
\{{gen_ai.token.type}} (input/output)
----
h4. Testing strategy
* Unit tests in \{{camel-ai-observability}} — convention mapping, no-op when
backends absent, attribute population from mock \{{ChatResponse}}
* Component tests per langchain4j module — mock \{{ChatModel}}, verify
observation called with correct context
* Integration tests:
** \{{camel-opentelemetry2}} + InMemorySpanExporter — assert child span
hierarchy and \{{gen_ai.*}} attributes
** \{{camel-micrometer-observability}} + \{{SimpleMeterRegistry}} — assert
metric names and tag values match Spring AI export format
* Agent/tools multi-iteration test — verify *N* child spans/metrics for *N* LLM
round-trips in one Exchange
----
h4. Open questions for PMC / ticket owner
# *Scope confirmation:* Phase 1 limited to \{{langchain4j-*}} only, or include
\{{camel-openai}} in the same ticket?
# *Model name headers:* Should we add \{{CamelLangChain4j*RequestModel}} /
\{{ResponseModel}} headers alongside observability (small additive change), or
rely on endpoint config params only?
# *Configuration knob:* Global enable/disable via \{{camel.main}} property
(e.g. \{{camel.ai.observability.enabled}}, default \{{true}} when backend
present)?
# *Priority of tracing stack:* Any preference to implement Micrometer
Observation path first (Spring Boot parity) vs OTel-native path first?
----
h4. Recommendation
*Proceed with Phase 1 as described* — new \{{camel-ai-observability}} shared
module, child span/observation per LLM invocation, stable GenAI semconv subset
aligned with Spring AI, auto-detecting Camel's existing tracing/metrics
backends. This builds directly on CAMEL-23860 headers, stays separate from the
\{{camel ask}} dev tooling path, and unblocks future TUI integration without
blocking it.
Estimated complexity: *medium* — mostly new module + producer wrapping in 4
components; no core Camel API changes required. Main design work is
model/provider resolution and multi-iteration span handling in tools/agent.
Happy to start implementation once this approach is approved. Please confirm
Phase 1 scope and the open questions above.
> camel-langchain4j - Add OpenTelemetry and Micrometer observability for AI/LLM
> usage
> -----------------------------------------------------------------------------------
>
> Key: CAMEL-23861
> URL: https://issues.apache.org/jira/browse/CAMEL-23861
> Project: Camel
> Issue Type: Improvement
> Components: camel-langchain4j-agent, camel-langchain4j-chat,
> camel-langchain4j-embeddings, camel-langchain4j-tools
> Reporter: Claus Ibsen
> Priority: Major
> Labels: ai, observability
> Fix For: 4.22.0
>
>
> The camel-langchain4j components currently have no observability integration
> for AI/LLM-specific metrics and tracing. Token usage, model name, finish
> reason, and latency are not captured.
> Spring AI exposes this data through the Micrometer Observation API (which
> produces both metrics and traces) following the OpenTelemetry Semantic
> Conventions for GenAI (https://opentelemetry.io/docs/specs/semconv/gen-ai/).
> This enables dashboards showing token consumption, costs, model distribution,
> and conversation tracking.
> Camel should follow the same approach. When camel-micrometer or
> camel-opentelemetry2 is active, the langchain4j producers should capture:
> *OpenTelemetry span attributes (GenAI semantic conventions):*
> - gen_ai.system (e.g. openai, anthropic)
> - gen_ai.request.model
> - gen_ai.response.model
> - gen_ai.usage.input_tokens
> - gen_ai.usage.output_tokens
> - gen_ai.response.finish_reasons
> *Micrometer metrics:*
> - Token counters (input/output/total) with model tag
> - Request counters per model/operation
> - Latency timers/histograms
> - Error counters
> This would allow standard observability tooling (Grafana, Jaeger, Prometheus,
> etc.) and Camel TUI to visualize AI/LLM usage across routes.
> See also CAMEL-23860 which adds token data as exchange headers - this ticket
> is about propagating that data into the observability layer for monitoring
> and dashboards.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
