[
https://issues.apache.org/jira/browse/CAMEL-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096143#comment-18096143
]
Federico Mariani commented on CAMEL-24051:
------------------------------------------
Good point, but I still think the observability test infra service makes some
sense, in particular:
* Is much closer to a production (k8s/ocp) scenario
* While TUI traces and jaeger already handle the traces, with the observability
test infra we can:
** Validate that the metrics integration with prometheus works
** Validate OTLP log export
** And Perses, a unified UI
> camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces
> + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring
> OpenShift's observability stack
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24051
> URL: https://issues.apache.org/jira/browse/CAMEL-24051
> Project: Camel
> Issue Type: New Feature
> Components: test-infra
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
>
> h3. Problem / Motivation
> Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path
> already wire up {{camel-observability-services}} (Micrometer/Prometheus
> metrics, health checks, OpenTelemetry tracing via the bundled
> {{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there
> is no local backend to receive or visualize that telemetry: {{camel infra
> run}} can start backing services like Postgres or Artemis, but nothing
> analogous exists for observability.
> As a result, developers only find out whether traces, metrics, and logs
> actually work correctly (correct span correlation, meaningful dashboards,
> sane metric cardinality, etc.) once the application is deployed to a real
> OpenShift cluster, which typically provides the Red Hat build of
> OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster
> Observability Operator (Grafana). Telemetry problems surface late, in a
> shared cluster environment, rather than at dev/E2E time on a laptop or in CI.
> h3. Licensing note (why not grafana/otel-lgtm)
> The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one
> image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from
> Apache 2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party
> license policy, so an ASF project cannot depend on it even as opt-in test/dev
> tooling without a real [email protected] sign-off, and Red Hat's OSPO
> treats AGPL as off-limits even more strictly, including for test-only tooling
> in downstream builds. Only Prometheus and the OpenTelemetry Collector in that
> bundle are actually Apache 2.0.
> h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC,
> see below)
> Use an equivalent stack built entirely from Apache-2.0-licensed components:
> * *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the
> {{camel-observability-services}} default via {{/observe/metrics}}).
> * *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces]
> (Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query
> Service-compatible JSON API. Functionally a drop-in swap for Tempo.
> * *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs]
> (Apache 2.0) -- LogsQL query language, accepts OTLP logs directly via
> {{/insert/opentelemetry/v1/logs}}. Swap for Loki.
> * *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project)
> -- dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already
> powers OpenShift's own new traces UI*, which makes it a much safer downstream
> bet than Grafana. Its default plugin set already ships Prometheus, Jaeger,
> and VictoriaLogs datasource kinds with no extra plugin install.
> Adding this also revisits the fate of the existing
> {{camel-test-infra-jaeger}} module (added in CAMEL-23641): since
> VictoriaTraces exposes a Jaeger-compatible query API,
> {{camel-test-infra-jaeger}} does *not* need to be deprecated as originally
> floated -- it can keep serving its current role, or the new service can point
> at it directly, instead of being replaced outright. This should be decided
> during implementation, not assumed here.
> h3. Signal-level design rationale (matching real K8s/OCP behavior where it
> matters)
> * *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry
> pushes OTLP spans to a Collector). No divergence between local and cluster
> behavior.
> * *Metrics -- pull*: OCP's real default is Prometheus scraping via
> {{ServiceMonitor}}/{{PodMonitor}}, not app-side push.
> {{camel-observability-services}}' existing Prometheus-pull default already
> matches this -- the local service's job is to supply a scrape target pointed
> at the running app, mirroring what a {{ServiceMonitor}} does on-cluster,
> *not* to switch metrics to OTLP push.
> * *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging
> does not have the app push logs at all -- Vector tails container
> stdout/stderr and ships to LokiStack; Loki itself still doesn't natively
> ingest OTLP. Locally, pushing OTLP logs straight from the OpenTelemetry Java
> agent's logback/log4j2 instrumentation into VictoriaLogs is validated and
> simple, but it is a different data path than production and should be
> documented as a pragmatic local-dev shortcut, not a faithful mirror of the
> cluster's log-collection mechanism.
> * *Trace/log correlation*: validated to work *automatically* via the OTel
> Java agent's logback-appender instrumentation
> ({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested
> log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC
> configuration. The {{camel-mdc}}-based approach documented in
> {{camel-opentelemetry2}}'s docs is not required for this path.
> h3. Empirical validation (PoC results)
> A working proof-of-concept was built and torn down; all 4 signals (metrics,
> traces, logs, and trace/log correlation) were confirmed working end-to-end
> with real data from a real {{camel-observability-services}}-enabled Camel
> route, and Perses accepted all 3 datasource definitions with no schema
> errors. Key findings to carry into implementation:
> * VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass
> {{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
> * VictoriaTraces' Jaeger-compatible API is served at
> {{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
> * *Port collision*: Perses defaults to port 8080, the same default port used
> by Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app --
> confirmed via {{-web.listen-address :3000}} (also verified live), keeping the
> app on its normal default port. Suggest defaulting the test-infra service to
> port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may
> already be used to.
> * Minimal working docker commands (see also PoC files, not retained beyond
> this session):
> {code}
> docker network create camel-obs
> docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p
> 4317:4317 \
> victoriametrics/victoria-traces:latest -httpListenAddr=:8428
> -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
> docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
> victoriametrics/victoria-logs:latest -httpListenAddr=:9428
> docker run -d --name prometheus --network camel-obs -p 9090:9090 \
> -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
> --add-host=host.docker.internal:host-gateway \
> prom/prometheus:latest
> docker run -d --name perses --network camel-obs -p 3000:3000 \
> persesdev/perses:latest -web.listen-address :3000
> {code}
> * A minimal {{camel-observability-services}}-enabled JBang app, run via
> {{camel run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang
> run}} does not work), with the OpenTelemetry Java agent supplying traces+logs
> export, confirmed all of: Prometheus scraping {{/observe/metrics}}, a real
> Camel-level {{timer}} span in VictoriaTraces, and OTLP log ingestion into
> VictoriaLogs correlated back to that trace.
> h3. Scope
> * New test-infra module/service(s) following existing test-infra patterns
> (e.g. {{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}),
> composing the *4 separate containers* above -- there is no single prebuilt
> image bundling Prometheus + VictoriaTraces + VictoriaLogs + Perses the way
> {{grafana/otel-lgtm}} bundles its stack.
> * Wire the new service into the {{camel-jbang-core}} {{infra}} command so
> {{camel infra run observability}} works like other {{camel infra run
> <service>}} commands, including supplying the Prometheus scrape-target wiring
> against the running app.
> * Decide the relationship to the existing {{camel-test-infra-jaeger}} module
> (coexist vs. supersede) as part of implementation.
> * Out of scope for this ticket: changes to {{camel-observability-services}}'
> default export behavior itself (e.g. any future default-on
> MDC/log-correlation improvement is a separate, runtime-behavior-changing
> ticket), and cluster-side OpenShift/Kubernetes tooling (traits, {{camel
> kubernetes doctor}}, PodMonitor/PrometheusRule generation).
> h3. Open items
> * Perses' live query-through-datasource-proxy path was not confirmed during
> the PoC (datasource definitions were accepted, but querying data through
> Perses' proxy returned errors under the guessed path conventions) -- needs
> checking against Perses source/docs during implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
