+1 to this effort. There is value in adding support for Events - general bookkeeping and helping replay actions in the event of recovery. At the minimum we should aim to track the following all catalogs: 1. Create actions 2. Alter actions 3. Delete actions across all tables, properties and namespaces.
On Tue, Nov 30, 2021 at 9:12 PM Jack Ye <yezhao...@gmail.com> wrote: > Hi everyone, > > I would like to start some initial discussions around Iceberg event > notification support, because we might have some engineering resources to > work on Iceberg notification integration with AWS services such as SNS, > SQS, CloudWatch. > > As of today, we have a Listener interface and 3 events ScanEvent, > IncrementalScanEvent, CreateSnapshotEvent. There is a static registry > called Listeners that registers the event listeners in the JVM. > > However, when I read the related code paths, my thought is that it might > be better to register listeners per-table, based on the following > observations: > 1. Iceberg events are all table or sub-table level events. For any catalog > or global level events, the catalog service can provide notifications, > Iceberg can be out of the picture. > 2. A user might have multiple Iceberg catalogs defined, pointing to > different catalog services. (e.g. one to AWS Glue, one to a Hive > metastore). The notifications from tables of these different catalogs > should be directed to different listeners at least per catalog, instead of > the same set of listeners that are registered globally. > 3. Event listener configurations are usually static. It makes more sense > to me to define it once and then repeatedly use it, instead of > re-registering it every time I start an application. > > If we register the listeners at table level, we can add a hook in > TableOperations to get a set of listeners to emit specific events. The > listeners could be defined and serialized as a part of the table > properties, or maybe even a part of the Iceberg spec. > > This is really just my brainstorming. Maybe it's a bit overkill, maybe I > am missing the correct way to use the Listeners static registry. It would > be great if anyone could provide more contexts or thoughts around this > topic. > > Best, > Jack Ye > > > > > > > > > > -- Regards, Neelesh S. Salian
