Hi all, As per the latest community discussion today (6/17), there was a request to put down the intended use cases for storing Polaris Events in the persistence. There are two main use cases I am intending to support:
[From the Iceberg Events API proposal <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>] Notification of changes / workflow triggers "The endpoint enables non-catalog systems to efficiently track metadata changes without implementing the full Iceberg REST protocol. Engineers can build event-driven architectures where data quality checks, data processing pipelines, and dashboard refreshes are automatically triggered when relevant events occur. Orchestration tools only need to query the simple /events Endpoint." [Partially from the Iceberg Events API proposal <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>] Auditing "Organizations can monitor who modified objects, when events occurred, and what was changed. The [Iceberg Events API] in its current form cannot serve as a replacement to an audit endpoint, as it misses queries, permission events and other events related to the management of the catalog." While I agree with the above statements regarding auditing capabilities from the Iceberg Events API spec as it is written without Custom Operation support, I believe that we can fit auditing events into this API while still adhering to the spec. However, I am not opposed to a different API endpoint specifically for auditing purposes if the community recommends that. But the PR currently open <https://github.com/apache/polaris/pull/1844> regarding persisting Polaris Events is still required, even if that is the case. Please let me know if there are any thoughts on this. Best, Adnan Hemani > On May 13, 2025, at 6:35 PM, Adnan Hemani <adnan.hem...@snowflake.com> wrote: > > Hi all, > > I am raising a proposal to implement the proposed Iceberg REST specification > for the Events API (doc > <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>, > GH <https://github.com/apache/iceberg/pull/12584/files>). It is my > understanding that this proposal is close and that we will be required to > implement something very close to the current proposal in the near future. > > If Polaris is to implement this API, it will likely need to be through a > Persistence instance that the Polaris server can query instantly, as this API > will not be asynchronous. Please note, this proposal is not to comment on > what events we may emit today or in the future - the scope of this proposal > is solely to discuss how we plan to implement the proposed Events API. > > Changes to be made: > > Implement Event storage through the Polaris Persistence layer > > We will store events in a persistence instance of user’s choice - whether > they would like the events to be part of the same persistence instance as > their Polaris metadata or if they would like for a separate persistence > instance. Users will provide the persistence instance by configuring a JDBC > string on Polaris startup, similarly to the JDBC string we receive currently > from users for the Polaris metadata. > > For concerns regarding scaling of events in the Polaris persistence layer, we > can also implement a recommended, optional parameter for an events retention > period after which Polaris will asynchronously delete records older than that > time period. > > How to Implement Writes to the Polaris Persistence layer > > The way to implement the above proposal would be through implementation of > the `PolarisEventListener` > <https://github.com/apache/polaris/blob/main/service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java> > abstract class. In this implementation, I believe it should not be > controversial to state that we cannot block on events to be flushed to > persistence due to latency concerns - and as a result, we have two options: > 1) a simple in-memory buffer or 2) a file-based buffer. Both buffers would > flush after a certain amount of time after the first non-flushed event is > written. While option 2 offers a better event durability guarantee in case of > disaster recovery, it will come at the cost of additional latency to write to > the filesystem. If there are no security concerns regarding writing to the > filesystem, I believe this is the recommended way to implement - the > additional latency to write to filesystem should not add unreasonable > overhead given the right implementation with open filewriters. If writing to > the filesystem is not recommended, I’m not sure there is any other way to > achieve guaranteed event durability. In both options we can only achieve > eventual consistency - to get strong consistency, we will need to implement a > way to block the API call until we flush the events to persistence, which I > cannot recommend at this time due to latency concerns. > > Please reply to this thread if there are any questions and/or concerns on > this proposal. If there are no major concerns within a week, then I will > begin implementation. > > Best, > Adnan Hemani >
