Looks awesome. Thanks for taking the lead! It makes sense to use a JDBC-backed persistence layer, shared or separate. The optional retention period is a nice safeguard. I don’t see any blockers on my side. If no one raises major concerns this week, please go ahead and start the implementation. Exciting to see this coming together!
Yufei On Tue, May 13, 2025 at 6:37 PM Adnan Hemani <adnan.hem...@snowflake.com.invalid> wrote: > Hi all, > > I am raising a proposal to implement the proposed Iceberg REST > specification for the Events API (doc < > https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>, > GH <https://github.com/apache/iceberg/pull/12584/files>). It is my > understanding that this proposal is close and that we will be required to > implement something very close to the current proposal in the near future. > > If Polaris is to implement this API, it will likely need to be through a > Persistence instance that the Polaris server can query instantly, as this > API will not be asynchronous. Please note, this proposal is not to comment > on what events we may emit today or in the future - the scope of this > proposal is solely to discuss how we plan to implement the proposed Events > API. > > Changes to be made: > > Implement Event storage through the Polaris Persistence layer > > We will store events in a persistence instance of user’s choice - whether > they would like the events to be part of the same persistence instance as > their Polaris metadata or if they would like for a separate persistence > instance. Users will provide the persistence instance by configuring a JDBC > string on Polaris startup, similarly to the JDBC string we receive > currently from users for the Polaris metadata. > > For concerns regarding scaling of events in the Polaris persistence layer, > we can also implement a recommended, optional parameter for an events > retention period after which Polaris will asynchronously delete records > older than that time period. > > How to Implement Writes to the Polaris Persistence layer > > The way to implement the above proposal would be through implementation of > the `PolarisEventListener` < > https://github.com/apache/polaris/blob/main/service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java> > abstract class. In this implementation, I believe it should not be > controversial to state that we cannot block on events to be flushed to > persistence due to latency concerns - and as a result, we have two options: > 1) a simple in-memory buffer or 2) a file-based buffer. Both buffers would > flush after a certain amount of time after the first non-flushed event is > written. While option 2 offers a better event durability guarantee in case > of disaster recovery, it will come at the cost of additional latency to > write to the filesystem. If there are no security concerns regarding > writing to the filesystem, I believe this is the recommended way to > implement - the additional latency to write to filesystem should not add > unreasonable overhead given the right implementation with open filewriters. > If writing to the filesystem is not recommended, I’m not sure there is any > other way to achieve guaranteed event durability. In both options we can > only achieve eventual consistency - to get strong consistency, we will need > to implement a way to block the API call until we flush the events to > persistence, which I cannot recommend at this time due to latency concerns. > > Please reply to this thread if there are any questions and/or concerns on > this proposal. If there are no major concerns within a week, then I will > begin implementation. > > Best, > Adnan Hemani > >
