Started https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx.
On Mon, Nov 2, 2026 at 3:36 PM Anand Kumar Sankaran < [email protected]> wrote: > Hi Yufei, > > Fair point on pre-Polaris failures. Server-side retry only covers what > happens after we accept the report. Network and proxy failures would need > client-side retry, which only works if clients treat the response as > meaningful. Today they don't (the Java client ignores it), so I don't see > #5395 making that worse. It changes what we do after receipt, not what > clients can rely on. > > I agree the spec question is worth asking. I'll draft a short post for the > Iceberg dev list this week: Is POST .../metrics best-effort or > reliable-delivery, and what should a catalog return when it can't persist a > report? I'll link it here. > > I'd rather not hold #5395 for that. Failed writes are logged and counted, > so operators can see them. If Iceberg settles on stricter semantics we can > adjust. Would you be OK with that, or do you want the PR to wait? > > Thanks, > Anand > > On Mon, Nov 2, 2026 at 2:37 PM Yufei Gu <[email protected]> wrote: > >> Hi Anand, thanks for chiming in! > If we need guaranteed capture, that >> should be handled server side: retry or buffer the write after we've >> accepted the report. That makes sense for failures after Polaris receives >> the report. But what about >> >> >> Hi Anand, thanks for chiming in! >> >> > If we need guaranteed capture, that should be handled server side: retry >> > or buffer the write after we've accepted the report. >> >> That makes sense for failures after Polaris receives the report. But >> what about failures before the request reaches Polaris, such as >> network or proxy failures? Wouldn't we still need client side retries >> if reliable delivery is required? >> >> > If you still want the endpoint semantics clarified, I think raising it on >> > the Iceberg dev list is reasonable. >> >> Can we do that now? Since we're changing the observable REST behavior, >> I'd prefer to clarify the intended semantics with the Iceberg >> community first. >> >> On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev >> <[email protected]> wrote: >> > >> > Hi Dmitri, Yufei, >> > >> > Thanks for pulling me in. Yes, I did raise billing/auditing as a use case >> > for the metrics data. >> > >> > Having thought about it, I agree with Dmitri that client-side retries are >> > the wrong place to get durability. The Java client ignores errors from this >> > endpoint, and reporting is optional, so a 500 doesn't give us a reliable >> > audit trail either. If we need guaranteed capture, that should be handled >> > server-side: retry or buffer the write after we've accepted the report. I'm >> > happy to take that on as a follow-up. >> > >> > For #5395, I'm fine with drop-plus-counter. Failed writes are logged and >> > counted in polaris.metrics.reports.dropped, so operators can alert on them >> > without the failure showing up as a catalog 5xx. I'd suggest we document in >> > the metrics docs that the persisting reporter is best-effort today. >> > >> > Yufei, if you still want the endpoint semantics clarified, I think raising >> > it on the Iceberg dev list is reasonable, but I wouldn't block this PR on >> > it. Does the counter plus the best-effort documentation work for you? >> > >> > Thanks, >> > Anand >> > >> > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < >> > [email protected]> wrote: >> > >> > > Hi Yufei, Anand, For metrics, doing client-side retries is an overkill, >> > > IMHO. Metric reports cannot logically conflict with any other writes. If >> > > the server is not able to persist a piece of metrics data it already >> > > received, retries should happen >> > > >> > > >> > > Hi Yufei, Anand, >> > > >> > > For metrics, doing client-side retries is an overkill, IMHO. >> > > >> > > Metric reports cannot logically conflict with any other writes. If the >> > > server is not able to persist a piece of metrics data it already >> > > received, >> > > retries should happen server-side. >> > > >> > > Including Anand in this email directly, hoping for some background info >> > > sharing and preferred execution approach. >> > > >> > > Thanks, >> > > Dmitri. >> > > >> > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: >> > > >> > > > I think it's preferable, to return a 500 to clients. The [IRC metrics >> > > > endpoint spec]( >> > > > >> > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ >> > <https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$%3E>> >> > > ) >> > > > doesn't say that metrics reporting is best effort. I recall Anand >> > > > mentioning a use case where metrics could be used for billing or >> > > > auditing. >> > > > In that case, client side retries don't seem unreasonable. >> > > > >> > > > I'd prefer to keep things simple and treat this as a normal REST >> > > > endpoint, >> > > > where the server clearly communicates when an operation fails. >> > > > >> > > > If we really want to change this behavior, I think we should first >> > > > discuss >> > > > it with the Iceberg community and clarify or update the metrics >> > > > endpoint >> > > > semantics in the spec. >> > > > >> > > > Yufei >> > > > >> > > > >> > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> >> > > > wrote: >> > > > >> > > > > Hi All, >> > > > > >> > > > > Great idea about counting dropped reports. >> > > > > >> > > > > Yufei: Does [5395] work from your POV, given that failures are >> > > > > reported >> > > > to >> > > > > Micrometer now? >> > > > > >> > > > > Thanks, >> > > > > Dmitri. >> > > > > >> > > > > [5395] >> > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> > > > >> > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> >> > > > wrote: >> > > > > >> > > > > > Hi JB, Yufei, >> > > > > > >> > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, >> > > > > > tagged by reporter) makes sense to me. Operators can alert on it >> > > > > > directly, and Polaris doesn't have to return a 500 on >> > > > > > POST .../metrics. >> > > > > > >> > > > > > I'll add it on #5395 [1]. >> > > > > > >> > > > > > Thanks, >> > > > > > Vignesh >> > > > > > >> > > > > > [1] >> > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> > > > > >> > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré >> > > > > > <[email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi all, >> > > > > > > >> > > > > > > Nice discussion! >> > > > > > > >> > > > > > > I lean towards the proposal in #5395. The reasons are the ones >> > > > > > > Dmitri >> > > > > > gave. >> > > > > > > The metrics POST is unidirectional and telemetry-like, so clients >> > > > gain >> > > > > > > nothing from seeing a persistence failure. The only status the >> > > > > > > IRC >> > > > spec >> > > > > > > allows for it is a 5xx, which is not really actionable for them >> > > > > > > :) >> > > > > > > >> > > > > > > I also agree with Yufei that a silent drop should not leave >> > > > > > > operators >> > > > > > > "blind". Today the 500 is the only server-side signal we got, >> > > > > > > and a >> > > > > WARN >> > > > > > > log is probably easy to miss (or ignore intentionally :)). >> > > > > > > Maybe we could add a dedicated counter for dropped reports, for >> > > > > instance >> > > > > > > polaris.metrics.reports.dropped, tagged by reporter type? >> > > > > > > Operators >> > > > > could >> > > > > > > then alert on it directly. That would cover the visibility use >> > > > > > > case >> > > > > > without >> > > > > > > surfacing as catalog 5xx. >> > > > > > > >> > > > > > > Yufei's point about excluding the metrics endpoint from the core >> > > > > > > availability SLO is also valid. Imho, that depends on how each >> > > > > deployment >> > > > > > > aggregates its metrics, so I would consider a complement more >> > > > > > > than a >> > > > > > > replacement for the server-side change. >> > > > > > > >> > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. >> > > > > > > >> > > > > > > Thanks! >> > > > > > > >> > > > > > > Regards >> > > > > > > JB >> > > > > > > >> > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a >> > > > > > > <[email protected]> >> > > > > > wrote: >> > > > > > > >> > > > > > > > Hi all, >> > > > > > > > >> > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here >> > > > > > > > because >> > > > > it >> > > > > > > > changes REST behaviour compared to today. >> > > > > > > > >> > > > > > > > With this change, POST .../metrics no longer returns a 500 >> > > > > > > > when the >> > > > > > > > persisting reporter's DB write fails. The error is logged and >> > > > > > > > the >> > > > > > > > report is dropped. Errors from other reporters still propagate >> > > > > through >> > > > > > > > the handler as before, and authorization is unchanged. >> > > > > > > > >> > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. >> > > > > > > > This >> > > > > > > > change keeps a broken metrics store from showing up as catalog >> > > > > > > > 5xx >> > > > > > > > errors. >> > > > > > > > >> > > > > > > > Thoughts welcome, especially if anyone relies on the current >> > > > > > > > 500 >> > > > > > > > behaviour. >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Vignesh >> > > > > > > > >> > > > > > > > [1] >> > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > >> > -- >> > [image: Workday Logo] <https://www.workday.com/ > >> > Anand Kumar Sankaran >> > Sr Principal, Software Engineer (Distributed Systems) >> > Office of the CTO >> > [email protected] >> >> > > -- > [image: Workday Logo] <https://www.workday.com/> > Anand Kumar Sankaran > Sr Principal, Software Engineer (Distributed Systems) > Office of the CTO > [email protected] > -- [image: Workday Logo] <https://www.workday.com/> Anand Kumar Sankaran Sr Principal, Software Engineer (Distributed Systems) Office of the CTO [email protected]
