Hi Yufei, Fair point on pre-Polaris failures. Server-side retry only covers what happens after we accept the report. Network and proxy failures would need client-side retry, which only works if clients treat the response as meaningful. Today they don't (the Java client ignores it), so I don't see #5395 making that worse. It changes what we do after receipt, not what clients can rely on.
I agree the spec question is worth asking. I'll draft a short post for the Iceberg dev list this week: Is POST .../metrics best-effort or reliable-delivery, and what should a catalog return when it can't persist a report? I'll link it here. I'd rather not hold #5395 for that. Failed writes are logged and counted, so operators can see them. If Iceberg settles on stricter semantics we can adjust. Would you be OK with that, or do you want the PR to wait? Thanks, Anand On Mon, Nov 2, 2026 at 2:37 PM Yufei Gu <[email protected]> wrote: > Hi Anand, thanks for chiming in! > If we need guaranteed capture, that > should be handled server side: retry or buffer the write after we've > accepted the report. That makes sense for failures after Polaris receives > the report. But what about > > > Hi Anand, thanks for chiming in! > > > If we need guaranteed capture, that should be handled server side: retry or > > buffer the write after we've accepted the report. > > That makes sense for failures after Polaris receives the report. But > what about failures before the request reaches Polaris, such as > network or proxy failures? Wouldn't we still need client side retries > if reliable delivery is required? > > > If you still want the endpoint semantics clarified, I think raising it on > > the Iceberg dev list is reasonable. > > Can we do that now? Since we're changing the observable REST behavior, > I'd prefer to clarify the intended semantics with the Iceberg > community first. > > On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev > <[email protected]> wrote: > > > > Hi Dmitri, Yufei, > > > > Thanks for pulling me in. Yes, I did raise billing/auditing as a use case > > for the metrics data. > > > > Having thought about it, I agree with Dmitri that client-side retries are > > the wrong place to get durability. The Java client ignores errors from this > > endpoint, and reporting is optional, so a 500 doesn't give us a reliable > > audit trail either. If we need guaranteed capture, that should be handled > > server-side: retry or buffer the write after we've accepted the report. I'm > > happy to take that on as a follow-up. > > > > For #5395, I'm fine with drop-plus-counter. Failed writes are logged and > > counted in polaris.metrics.reports.dropped, so operators can alert on them > > without the failure showing up as a catalog 5xx. I'd suggest we document in > > the metrics docs that the persisting reporter is best-effort today. > > > > Yufei, if you still want the endpoint semantics clarified, I think raising > > it on the Iceberg dev list is reasonable, but I wouldn't block this PR on > > it. Does the counter plus the best-effort documentation work for you? > > > > Thanks, > > Anand > > > > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < > > [email protected]> wrote: > > > > > Hi Yufei, Anand, For metrics, doing client-side retries is an overkill, > > > IMHO. Metric reports cannot logically conflict with any other writes. If > > > the server is not able to persist a piece of metrics data it already > > > received, retries should happen > > > > > > > > > Hi Yufei, Anand, > > > > > > For metrics, doing client-side retries is an overkill, IMHO. > > > > > > Metric reports cannot logically conflict with any other writes. If the > > > server is not able to persist a piece of metrics data it already received, > > > retries should happen server-side. > > > > > > Including Anand in this email directly, hoping for some background info > > > sharing and preferred execution approach. > > > > > > Thanks, > > > Dmitri. > > > > > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: > > > > > > > I think it's preferable, to return a 500 to clients. The [IRC metrics > > > > endpoint spec]( > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ > > <https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$%3E>> > > > ) > > > > doesn't say that metrics reporting is best effort. I recall Anand > > > > mentioning a use case where metrics could be used for billing or > > > > auditing. > > > > In that case, client side retries don't seem unreasonable. > > > > > > > > I'd prefer to keep things simple and treat this as a normal REST > > > > endpoint, > > > > where the server clearly communicates when an operation fails. > > > > > > > > If we really want to change this behavior, I think we should first > > > > discuss > > > > it with the Iceberg community and clarify or update the metrics endpoint > > > > semantics in the spec. > > > > > > > > Yufei > > > > > > > > > > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Great idea about counting dropped reports. > > > > > > > > > > Yufei: Does [5395] work from your POV, given that failures are > > > > > reported > > > > to > > > > > Micrometer now? > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > [5395] > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> > > > > wrote: > > > > > > > > > > > Hi JB, Yufei, > > > > > > > > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > > > > > > tagged by reporter) makes sense to me. Operators can alert on it > > > > > > directly, and Polaris doesn't have to return a 500 on > > > > > > POST .../metrics. > > > > > > > > > > > > I'll add it on #5395 [1]. > > > > > > > > > > > > Thanks, > > > > > > Vignesh > > > > > > > > > > > > [1] > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > > > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré > > > > > > <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Nice discussion! > > > > > > > > > > > > > > I lean towards the proposal in #5395. The reasons are the ones > > > > > > > Dmitri > > > > > > gave. > > > > > > > The metrics POST is unidirectional and telemetry-like, so clients > > > > gain > > > > > > > nothing from seeing a persistence failure. The only status the IRC > > > > spec > > > > > > > allows for it is a 5xx, which is not really actionable for them :) > > > > > > > > > > > > > > I also agree with Yufei that a silent drop should not leave > > > > > > > operators > > > > > > > "blind". Today the 500 is the only server-side signal we got, and > > > > > > > a > > > > > WARN > > > > > > > log is probably easy to miss (or ignore intentionally :)). > > > > > > > Maybe we could add a dedicated counter for dropped reports, for > > > > > instance > > > > > > > polaris.metrics.reports.dropped, tagged by reporter type? > > > > > > > Operators > > > > > could > > > > > > > then alert on it directly. That would cover the visibility use > > > > > > > case > > > > > > without > > > > > > > surfacing as catalog 5xx. > > > > > > > > > > > > > > Yufei's point about excluding the metrics endpoint from the core > > > > > > > availability SLO is also valid. Imho, that depends on how each > > > > > deployment > > > > > > > aggregates its metrics, so I would consider a complement more > > > > > > > than a > > > > > > > replacement for the server-side change. > > > > > > > > > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Regards > > > > > > > JB > > > > > > > > > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here > > > > > > > > because > > > > > it > > > > > > > > changes REST behaviour compared to today. > > > > > > > > > > > > > > > > With this change, POST .../metrics no longer returns a 500 when > > > > > > > > the > > > > > > > > persisting reporter's DB write fails. The error is logged and > > > > > > > > the > > > > > > > > report is dropped. Errors from other reporters still propagate > > > > > through > > > > > > > > the handler as before, and authorization is unchanged. > > > > > > > > > > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. > > > > > > > > This > > > > > > > > change keeps a broken metrics store from showing up as catalog > > > > > > > > 5xx > > > > > > > > errors. > > > > > > > > > > > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > > > > > > behaviour. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Vignesh > > > > > > > > > > > > > > > > [1] > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > [image: Workday Logo] <https://www.workday.com/ > > > Anand Kumar Sankaran > > Sr Principal, Software Engineer (Distributed Systems) > > Office of the CTO > > [email protected] > > -- [image: Workday Logo] <https://www.workday.com/> Anand Kumar Sankaran Sr Principal, Software Engineer (Distributed Systems) Office of the CTO [email protected]
