Hi JB, and all, Thanks for the suggestion. Pushed efc48d429 which adds an OtelMetricsReporter section to docs/docs/metrics-reporting.md. It documents the host's responsibility for packaging the OpenTelemetry API, SDK, and a metric exporter (Gradle plus a spark-submit --packages example), the programmatic SDK registration path, exporter-wiring examples for the OpenTelemetry Collector, Prometheus (pull and push), and Amazon CloudWatch via the sigv4auth Collector extension, plus the emitted metric names and attribute set.
Verified end-to-end against the Prometheus pull pattern from the docs (host SDK with PrometheusHttpServer + OtelMetricsReporter reporting synthetic ScanReport/CommitReport, all 12 iceberg.* series visible on /metrics with the documented attribute set); each Collector YAML in the docs was otelcol-contrib validate-checked. Looking forward to your PR review. Thanks, Nori On Mon, May 25, 2026 at 3:00 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi, > > I think this is a great proposal. > > I would suggest documenting how to package the exporter, as I believe it > is up to the user to package the specific OpenTelemetry exporter they need. > > I will take a look at the PR. > > Regards, > JB > > On Thu, May 21, 2026 at 3:39 AM Noritaka Sekiyama via dev < > [email protected]> wrote: > >> Hi all, >> >> I'd like to propose adding an OpenTelemetry-based MetricsReporter to >> iceberg-core that exports ScanReport and CommitReport to any OTLP-compatible >> backend. >> >> # Background >> Iceberg ships three built-in MetricsReporter implementations today: >> LoggingMetricsReporter, InMemoryMetricsReporter (Spark-internal), and >> RESTMetricsReporter (REST catalog only). >> None of them give users an out-of-the-box way to ship scan/commit metrics to >> an external observability platform. >> The gap applies to Spark users on non-REST catalogs and to all non-Spark >> engines (Trino, Flink, etc.). >> >> # Motivation >> OpenTelemetry is the vendor-neutral CNCF standard for telemetry, >> supported by every major observability backend (Prometheus, CloudWatch, >> Datadog, Grafana Cloud, etc.). >> A single OTLP-based MetricsReporter in Iceberg lets users reach all of >> these without per-vendor integrations in the project. >> This is complementary to #14360, which adds OTel support to HTTPClient >> at the REST-catalog HTTP layer; this proposal covers the Iceberg-level >> ScanReport / CommitReport layer. >> >> # Proposal >> Issue: https://github.com/apache/iceberg/issues/16169 >> PR: https://github.com/apache/iceberg/pull/16250 >> >> The reporter follows the same SDK-ownership philosophy as #14360 - the >> host application (Spark/Flink/Trino/...) registers an OpenTelemetrySdk via >> GlobalOpenTelemetry, and the reporter just looks up a Meter from it. >> The reporter has zero Iceberg-specific catalog properties; everything >> else is owned by the host. >> >> The PR has been validated end-to-end against two unrelated OTLP backends >> (Databricks Zerobus and Amazon CloudWatch) - full procedures and queries >> are linked from the PR. >> >> # On dependencies >> Given the current sensitivity around new runtime dependencies in 1.11, >> the PR adds only opentelemetry-api to iceberg-core as compileOnly. >> The OpenTelemetry SDK and OTLP exporters are not added to the runtime >> classpath >> - they come from the host application. >> opentelemetry-sdk / -sdk-testing are testImplementation only. >> >> # Questions for the community >> >> Q1. Any objection to taking the opentelemetry-api compileOnly dependency >> in iceberg-core? >> Q2. Module placement: iceberg-core (current PR), or a >> separate iceberg-opentelemetry module? >> >> Thanks, >> Noritaka Sekiyama, Databricks >> >
