Re: Iceberg sync notes - 10 March 2021

Yan is absolutely correct. We only leverage the sort order during DELETE/UPDATE/MERGE operations in Spark for now as we handle the plan construction ourselves. There will be an API in Spark 3.2 to request a specific distribution and ordering for normal writes. There are also similar efforts in

Re: Iceberg sync notes - 10 March 2021

Hi Chen, I think currently the sort order support is mostly only on the Iceberg spec level. The user can specify sort order on table, and ideally writer should use this information on the table to determine the right sort order it should use for writing data, and persist this information to data f

Re: Iceberg sync notes - 10 March 2021

Thanks Yan. I have a question about sort order support. I saw https://iceberg.apache.org/spec/#sorting talking about support on sorting. And I found related tickets like #1373 and #1975 . However, it is not c

Re: Iceberg sync notes - 10 March 2021

Hi Chen, Here is the doc on remaining tasks for format V2 that I updated with the latest status today, including individual PRs pending review and tasks needed that are V2-blocking: https://docs.google.com/document/d/1FyLJyvzcZbfbjwDMEZd6Dj-LYCfrzK1zC-Bkb3OiICc/edit Please feel free to comment/edi

Re: Iceberg sync notes - 10 March 2021

Thanks for the summary. On V2 format. Is there a google doc to review, or any sort of backlog of tickets to track? Chen On Mon, Mar 15, 2021 at 10:34 PM Anton Okolnychyi wrote: > Hey everyone, > > Thanks to folks who attended. I added my notes from the last sync. Please > feel free to add/corre

Re: Nessie PRs

Thanks Edgar! My best guess is a temporary issue in JCenter. That brought up the point that jcenter is shutting down[1]. For the gradle plugin repo they will automatically move off jcenter w/o disruption[2] For jcenter dependencies I have removed jcenter from the build and tested on a brand new