Hi everyone,

My name is Drew Gallardo, and I’m a part of the Iceberg team at Amazon EMR
and Athena. I’m reaching out to share a proposal that introduces data
commits as a part of the RESTCatalog. The current process for data commits
lives on the client side, and by shifting this logic into the REST catalog,
we can empower the catalog service with more control of this process.

This proposal addresses specific use cases that showcase the benefits of
moving the commit logic to the service side. For instance, this shift
allows the user to refine conflict resolution mechanisms, giving precedence
to operations that modify the table state to ensure their completion
without conflict. Furthermore, our POC demonstrated an improvement in the
success rate of concurrent write operations against the GlueCatalog. This
all can be found in the detailed proposal below. Feel free to comment, and
add your suggestions!

Detailed proposal:
https://docs.google.com/document/d/1OG68EtPxLWvNBJACQwcMrRYuGJCnQas8_LSruTRcHG8/edit?usp=sharing
Github POC: https://github.com/apache/iceberg/pull/9237

Looking forward to hearing back

Thanks,

Drew Gallardo
Amazon EMR & Athena
d...@amazon.com

Reply via email to