I haven’t thought of this in depth honestly - but I could see this being the case.
-Adnan > On May 21, 2025, at 4:39 PM, Eric Maynard <eric.w.mayn...@gmail.com> wrote: > > -devlist > > If we design by interface properly, it should be relatively easy to offer > both a disk buffering and an always-write implementation right? > > On Thu, May 22, 2025 at 12:12 AM Adnan Hemani > <adnan.hem...@snowflake.com.invalid> wrote: > >> Hi all, >> >> Thanks for sharing these thoughts. I’m also not completely sure about how >> much we should care about how much slower things will be if we just make a >> trip to the persistence on every write action. However, I’m building this >> feature with the intention of being able to also support read event types >> in the near future, if this is something that the customer is interested in >> enabling using the `CustomOperation` type that is defined in the Events API >> spec. Of course, this would need to be configured by the administrator, as >> maintenance of the persistence is their responsibility. >> >> Given that the Iceberg Events API spec has not yet merged and can still >> see some changes, I’m planning to begin work on the disk buffering now and >> wait for the Events API to finalize before working on the API side of the >> end-to-end implementation. >> >> Best, >> Adnan Hemani >> >>> On May 20, 2025, at 10:30 PM, Michael Collado <collado.m...@gmail.com> >> wrote: >>> >>> That’s super interesting. Glad this is being worked on. Personally, I >> don’t >>> know that the latency for writing events to a persistent storage is >> really >>> all that concerning. Looking at the enum of supported operations, only >>> write operations seem to trigger the event. It’s not like every read >>> request issues a new event. Given that the request latency here is >>> dominated by cloud storage calls, do we really care about one extra call >> to >>> Postgres? Personally, I’d skip the extra complexity of a buffer of any >> kind >>> and just write straight to the persistence store. >>> >>> Mike >>> >>> On Tue, May 20, 2025 at 9:31 AM Yufei Gu <flyrain...@gmail.com <mailto: >> flyrain...@gmail.com>> wrote: >>> >>>> Looks awesome. Thanks for taking the lead! It makes sense to use a >>>> JDBC-backed persistence layer, shared or separate. The optional >> retention >>>> period is a nice safeguard. >>>> I don’t see any blockers on my side. If no one raises major concerns >> this >>>> week, please go ahead and start the implementation. Exciting to see this >>>> coming together! >>>> >>>> Yufei >>>> >>>> >>>> On Tue, May 13, 2025 at 6:37 PM Adnan Hemani >>>> <adnan.hem...@snowflake.com.invalid> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I am raising a proposal to implement the proposed Iceberg REST >>>>> specification for the Events API (doc < >>>>> >>>> >> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli%253D1%2526tab%253Dt.0%26source%3Dgmail-imap%26ust%3D1748410327000000%26usg%3DAOvVaw0UNQYKbXoQ2YHVM7J0kB3l&source=gmail-imap&ust=1748475613000000&usg=AOvVaw38VwFquueiVwCdlo_xR2rB >>>>> , >>>>> GH < >> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://github.com/apache/iceberg/pull/12584/files%26source%3Dgmail-imap%26ust%3D1748410327000000%26usg%3DAOvVaw1AvwLK402voAm_j6zy25Mn&source=gmail-imap&ust=1748475613000000&usg=AOvVaw2IwxUir8jCegcKC47zv1Si>). >> It is my >>>>> understanding that this proposal is close and that we will be required >> to >>>>> implement something very close to the current proposal in the near >>>> future. >>>>> >>>>> If Polaris is to implement this API, it will likely need to be through >> a >>>>> Persistence instance that the Polaris server can query instantly, as >> this >>>>> API will not be asynchronous. Please note, this proposal is not to >>>> comment >>>>> on what events we may emit today or in the future - the scope of this >>>>> proposal is solely to discuss how we plan to implement the proposed >>>> Events >>>>> API. >>>>> >>>>> Changes to be made: >>>>> >>>>> Implement Event storage through the Polaris Persistence layer >>>>> >>>>> We will store events in a persistence instance of user’s choice - >> whether >>>>> they would like the events to be part of the same persistence instance >> as >>>>> their Polaris metadata or if they would like for a separate persistence >>>>> instance. Users will provide the persistence instance by configuring a >>>> JDBC >>>>> string on Polaris startup, similarly to the JDBC string we receive >>>>> currently from users for the Polaris metadata. >>>>> >>>>> For concerns regarding scaling of events in the Polaris persistence >>>> layer, >>>>> we can also implement a recommended, optional parameter for an events >>>>> retention period after which Polaris will asynchronously delete records >>>>> older than that time period. >>>>> >>>>> How to Implement Writes to the Polaris Persistence layer >>>>> >>>>> The way to implement the above proposal would be through implementation >>>> of >>>>> the `PolarisEventListener` < >>>>> >>>> >> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://github.com/apache/polaris/blob/main/service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java%26source%3Dgmail-imap%26ust%3D1748410327000000%26usg%3DAOvVaw0Z-SY-d50YHPNK38KxhHVk&source=gmail-imap&ust=1748475613000000&usg=AOvVaw0AVoXWSZgL3qK8GPbwYG9G >>>>> >>>>> abstract class. In this implementation, I believe it should not be >>>>> controversial to state that we cannot block on events to be flushed to >>>>> persistence due to latency concerns - and as a result, we have two >>>> options: >>>>> 1) a simple in-memory buffer or 2) a file-based buffer. Both buffers >>>> would >>>>> flush after a certain amount of time after the first non-flushed event >> is >>>>> written. While option 2 offers a better event durability guarantee in >>>> case >>>>> of disaster recovery, it will come at the cost of additional latency to >>>>> write to the filesystem. If there are no security concerns regarding >>>>> writing to the filesystem, I believe this is the recommended way to >>>>> implement - the additional latency to write to filesystem should not >> add >>>>> unreasonable overhead given the right implementation with open >>>> filewriters. >>>>> If writing to the filesystem is not recommended, I’m not sure there is >>>> any >>>>> other way to achieve guaranteed event durability. In both options we >> can >>>>> only achieve eventual consistency - to get strong consistency, we will >>>> need >>>>> to implement a way to block the API call until we flush the events to >>>>> persistence, which I cannot recommend at this time due to latency >>>> concerns. >>>>> >>>>> Please reply to this thread if there are any questions and/or concerns >> on >>>>> this proposal. If there are no major concerns within a week, then I >> will >>>>> begin implementation. >>>>> >>>>> Best, >>>>> Adnan Hemani >> >>
