Re: [DISCUSS] Add Polaris Observe API

2025-04-10 Thread Eric Maynard
I think the concept is really useful. The only thing I think which would require some more investigation is how exactly we implement this API -- where the data is stored, how long it's retained, etc. We might need to consider pushing this data out into another service or at least supporting such an

Re: [DISCUSS] Release cycle for Spark Client

2025-04-10 Thread yun zou
Hi Dmitri, Thanks a lot for the feedback! For compatibility, if the client releases along with the server, one implicit guarantee we will make is that one version of the client will be compatible with the server of the same version and the versions after until there is some API field deprecation.

Re: [DISCUSS] Add Polaris Observe API

2025-04-10 Thread Jean-Baptiste Onofré
Hi Eric Thanks a lot for your feedback! As a first step, I would not store additional entities to store that, but more "querying" the existing entities (tables, etc) and the Iceberg meta (including table properties) to display that. I agree about finishing Event Listeners. In the meantime, I wou

Re: [DISCUSS] Add Polaris Observe API

2025-04-10 Thread Michael Collado
I think a simple metrics API makes a lot of sense. Decoupling this from events makes sense, as this would just be useful to query periodically for a variety of reasons not tied to event triggering. Mike On Thu, Apr 10, 2025 at 3:00 PM Jean-Baptiste Onofré wrote: > Hi Eric > > Thanks a lot for y

Re: [DISCUSS] Release cycle for Spark Client

2025-04-10 Thread Yufei Gu
I Wrote this last Friday, but forgot to send it out. Similar to what Yun said. It's true that users can mix different client and server versions in their environments. However, version interoperability should be a core design goal for Polaris or any web service. Specifically: - *Server Bac

[Discuss] Correct Handling of Service-Provided Storage Config Properties During Catalog Updates

2025-04-10 Thread Rulin Xing
Hi folks, I'd like to initiate a discussion on the expected behavior when updating catalog properties, specifically around the handling of storage configuration fields that are automatically provided by the Polaris service. *Background* When a catalog is created, certain storage configuration pro

[DISCUSS] Add In-Dev Polaris Migrator/Synchronizer Tool to apache/polaris-tools

2025-04-10 Thread Mansehaj Singh
Hi all! Nice to meet you. I opened up https://github.com/apache/polaris-tools/pull/4 recently to add a Polaris migration/synchronizer tool I've been working on to the polaris-tools repo. By request, I'm sharing a design document here detailing how the tool works and the roadmap for functionality t