Hi Yufei, Anand, For metrics, doing client-side retries is an overkill, IMHO.
Metric reports cannot logically conflict with any other writes. If the server is not able to persist a piece of metrics data it already received, retries should happen server-side. Including Anand in this email directly, hoping for some background info sharing and preferred execution approach. Thanks, Dmitri. On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: > I think it's preferable, to return a 500 to clients. The [IRC metrics > endpoint spec]( > > https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml > ) > doesn't say that metrics reporting is best effort. I recall Anand > mentioning a use case where metrics could be used for billing or auditing. > In that case, client side retries don't seem unreasonable. > > I'd prefer to keep things simple and treat this as a normal REST endpoint, > where the server clearly communicates when an operation fails. > > If we really want to change this behavior, I think we should first discuss > it with the Iceberg community and clarify or update the metrics endpoint > semantics in the spec. > > Yufei > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > Great idea about counting dropped reports. > > > > Yufei: Does [5395] work from your POV, given that failures are reported > to > > Micrometer now? > > > > Thanks, > > Dmitri. > > > > [5395] https://github.com/apache/polaris/pull/5395 > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> > wrote: > > > > > Hi JB, Yufei, > > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > > > tagged by reporter) makes sense to me. Operators can alert on it > > > directly, and Polaris doesn't have to return a 500 on > > > POST .../metrics. > > > > > > I'll add it on #5395 [1]. > > > > > > Thanks, > > > Vignesh > > > > > > [1] https://github.com/apache/polaris/pull/5395 > > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > Nice discussion! > > > > > > > > I lean towards the proposal in #5395. The reasons are the ones Dmitri > > > gave. > > > > The metrics POST is unidirectional and telemetry-like, so clients > gain > > > > nothing from seeing a persistence failure. The only status the IRC > spec > > > > allows for it is a 5xx, which is not really actionable for them :) > > > > > > > > I also agree with Yufei that a silent drop should not leave operators > > > > "blind". Today the 500 is the only server-side signal we got, and a > > WARN > > > > log is probably easy to miss (or ignore intentionally :)). > > > > Maybe we could add a dedicated counter for dropped reports, for > > instance > > > > polaris.metrics.reports.dropped, tagged by reporter type? Operators > > could > > > > then alert on it directly. That would cover the visibility use case > > > without > > > > surfacing as catalog 5xx. > > > > > > > > Yufei's point about excluding the metrics endpoint from the core > > > > availability SLO is also valid. Imho, that depends on how each > > deployment > > > > aggregates its metrics, so I would consider a complement more than a > > > > replacement for the server-side change. > > > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > > > > > Thanks! > > > > > > > > Regards > > > > JB > > > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here because > > it > > > > > changes REST behaviour compared to today. > > > > > > > > > > With this change, POST .../metrics no longer returns a 500 when the > > > > > persisting reporter's DB write fails. The error is logged and the > > > > > report is dropped. Errors from other reporters still propagate > > through > > > > > the handler as before, and authorization is unchanged. > > > > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. This > > > > > change keeps a broken metrics store from showing up as catalog 5xx > > > > > errors. > > > > > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > > > behaviour. > > > > > > > > > > Thanks, > > > > > Vignesh > > > > > > > > > > [1] https://github.com/apache/polaris/pull/5395 > > > > > > > > > > > > > > >
