Re: [DISCUSS] Discrepancy Between Iceberg Spec and Java Implementation for Snapshot summary's 'operation' key

2024-10-19 Thread Sung Yun
Hi Ryan, thank you for your response! That detailed context is very helpful in allowing me to understanding why the REST catalog spec has evolved the way it has, and how the Table Spec and the REST Catalog Spec should each be referenced in the sub-communities (like in PyIceberg). I'll keep thos

Re: [DISCUSS] Discrepancy Between Iceberg Spec and Java Implementation for Snapshot summary's 'operation' key

2024-10-19 Thread rdb...@gmail.com
I can provide some historical context here about how the table spec evolved and how the REST spec works with respect to table versions. We initially did not have the snapshot summary or operation. When I added the summary, the operation was intended to be required in cases where the summary is pre

Re: Spec changes for deletion vectors

2024-10-19 Thread rdb...@gmail.com
Thanks for the summary, Szehon! I would add one thing to the "minimum" for each option. Because we want to be able to seek directly to the DV for a particular data file, I think it's important to start the blob with magic bytes. That way the reader can validate that the offset was correct and that

Re: [DISCUSS] Remove iceberg-pig module ?

2024-10-19 Thread rdb...@gmail.com
+1 On Thu, Oct 17, 2024 at 11:56 PM Steve Zhang wrote: > +1 > > Thanks, > Steve Zhang > > > > On Oct 17, 2024, at 11:16 PM, roryqi wrote: > > +1. > > Péter Váry 于2024年10月18日周五 13:44写道: > >> +1 >> >> On Fri, Oct 18, 2024, 04:50 Manu Zhang wrote: >> >>> +1 >>> >>> On Fri, Oct 18, 2024 at 8:50 A