Hi Dmitri, Yufei, Thanks for pulling me in. Yes, I did raise billing/auditing as a use case for the metrics data.
Having thought about it, I agree with Dmitri that client-side retries are the wrong place to get durability. The Java client ignores errors from this endpoint, and reporting is optional, so a 500 doesn't give us a reliable audit trail either. If we need guaranteed capture, that should be handled server-side: retry or buffer the write after we've accepted the report. I'm happy to take that on as a follow-up. For #5395, I'm fine with drop-plus-counter. Failed writes are logged and counted in polaris.metrics.reports.dropped, so operators can alert on them without the failure showing up as a catalog 5xx. I'd suggest we document in the metrics docs that the persisting reporter is best-effort today. Yufei, if you still want the endpoint semantics clarified, I think raising it on the Iceberg dev list is reasonable, but I wouldn't block this PR on it. Does the counter plus the best-effort documentation work for you? Thanks, Anand On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < [email protected]> wrote: > Hi Yufei, Anand, For metrics, doing client-side retries is an overkill, > IMHO. Metric reports cannot logically conflict with any other writes. If > the server is not able to persist a piece of metrics data it already > received, retries should happen > > > Hi Yufei, Anand, > > For metrics, doing client-side retries is an overkill, IMHO. > > Metric reports cannot logically conflict with any other writes. If the > server is not able to persist a piece of metrics data it already received, > retries should happen server-side. > > Including Anand in this email directly, hoping for some background info > sharing and preferred execution approach. > > Thanks, > Dmitri. > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: > > > I think it's preferable, to return a 500 to clients. The [IRC metrics > > endpoint spec]( > > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ > > ) > > doesn't say that metrics reporting is best effort. I recall Anand > > mentioning a use case where metrics could be used for billing or auditing. > > In that case, client side retries don't seem unreasonable. > > > > I'd prefer to keep things simple and treat this as a normal REST endpoint, > > where the server clearly communicates when an operation fails. > > > > If we really want to change this behavior, I think we should first discuss > > it with the Iceberg community and clarify or update the metrics endpoint > > semantics in the spec. > > > > Yufei > > > > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Great idea about counting dropped reports. > > > > > > Yufei: Does [5395] work from your POV, given that failures are reported > > to > > > Micrometer now? > > > > > > Thanks, > > > Dmitri. > > > > > > [5395] > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> > > wrote: > > > > > > > Hi JB, Yufei, > > > > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > > > > tagged by reporter) makes sense to me. Operators can alert on it > > > > directly, and Polaris doesn't have to return a 500 on > > > > POST .../metrics. > > > > > > > > I'll add it on #5395 [1]. > > > > > > > > Thanks, > > > > Vignesh > > > > > > > > [1] > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré <[email protected]> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Nice discussion! > > > > > > > > > > I lean towards the proposal in #5395. The reasons are the ones Dmitri > > > > gave. > > > > > The metrics POST is unidirectional and telemetry-like, so clients > > gain > > > > > nothing from seeing a persistence failure. The only status the IRC > > spec > > > > > allows for it is a 5xx, which is not really actionable for them :) > > > > > > > > > > I also agree with Yufei that a silent drop should not leave operators > > > > > "blind". Today the 500 is the only server-side signal we got, and a > > > WARN > > > > > log is probably easy to miss (or ignore intentionally :)). > > > > > Maybe we could add a dedicated counter for dropped reports, for > > > instance > > > > > polaris.metrics.reports.dropped, tagged by reporter type? Operators > > > could > > > > > then alert on it directly. That would cover the visibility use case > > > > without > > > > > surfacing as catalog 5xx. > > > > > > > > > > Yufei's point about excluding the metrics endpoint from the core > > > > > availability SLO is also valid. Imho, that depends on how each > > > deployment > > > > > aggregates its metrics, so I would consider a complement more than a > > > > > replacement for the server-side change. > > > > > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > > > > > > > Thanks! > > > > > > > > > > Regards > > > > > JB > > > > > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here because > > > it > > > > > > changes REST behaviour compared to today. > > > > > > > > > > > > With this change, POST .../metrics no longer returns a 500 when the > > > > > > persisting reporter's DB write fails. The error is logged and the > > > > > > report is dropped. Errors from other reporters still propagate > > > through > > > > > > the handler as before, and authorization is unchanged. > > > > > > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. This > > > > > > change keeps a broken metrics store from showing up as catalog 5xx > > > > > > errors. > > > > > > > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > > > > behaviour. > > > > > > > > > > > > Thanks, > > > > > > Vignesh > > > > > > > > > > > > [1] > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > > > > > > > > > > > > > > > > > -- [image: Workday Logo] <https://www.workday.com/> Anand Kumar Sankaran Sr Principal, Software Engineer (Distributed Systems) Office of the CTO [email protected]
