Hi everyone, My name is Drew Gallardo, and I’m a part of the Iceberg team at Amazon EMR and Athena. I’m reaching out to share a proposal that introduces data commits as a part of the RESTCatalog. The current process for data commits lives on the client side, and by shifting this logic into the REST catalog, we can empower the catalog service with more control of this process.
This proposal addresses specific use cases that showcase the benefits of moving the commit logic to the service side. For instance, this shift allows the user to refine conflict resolution mechanisms, giving precedence to operations that modify the table state to ensure their completion without conflict. Furthermore, our POC demonstrated an improvement in the success rate of concurrent write operations against the GlueCatalog. This all can be found in the detailed proposal below. Feel free to comment, and add your suggestions! Detailed proposal: https://docs.google.com/document/d/1OG68EtPxLWvNBJACQwcMrRYuGJCnQas8_LSruTRcHG8/edit?usp=sharing Github POC: https://github.com/apache/iceberg/pull/9237 Looking forward to hearing back Thanks, Drew Gallardo Amazon EMR & Athena d...@amazon.com
