I want to follow up on this. Is there an official consolidated design doc/proposal (even wip) on V2 spec?
I saw Streaming CDC in Iceberg <https://docs.google.com/document/d/1bBKDD4l-pQFXaMb4nOyVK-Sl3N2NTTG37uOCQx8rKVc/edit#heading=h.2u29lq1ekp5r> in a few update emails related, but it only covers one part. Chen On Thu, Jul 2, 2020 at 9:53 PM OpenInx <open...@gmail.com> wrote: > Sounds good to me. > > Thanks. > > On Fri, Jul 3, 2020 at 12:58 AM Ryan Blue <rb...@netflix.com> wrote: > >> I'd like to get 0.9.0 out as soon as possible. I expect to get an early >> RC out next week, once we have more tests committed. That way, people can >> start trying it out and reporting back where it doesn't work. >> >> I'd rather not block 0.9.0 to wait on Flink connector components. There's >> still a lot of work to get in, so I think it would be good to keep these >> decoupled. That said, I think it would make sense to have a release once >> the Flink connector is ready, just like we would do for Spark 3 support. >> >> Does that sound reasonable? >> >> On Wed, Jul 1, 2020 at 7:39 PM OpenInx <open...@gmail.com> wrote: >> >>> Hi Ryan: >>> >>> Just curious when do we plan to release 0.9.0 ? I expect that the flink >>> connector could be included in release 0.9.0. >>> >>> Thanks. >>> >>> On Thu, Jul 2, 2020 at 12:14 AM Ryan Blue <rb...@netflix.com.invalid> >>> wrote: >>> >>>> Hi Chen, >>>> >>>> Right now, the main parts of the v2 spec are the addition of sequence >>>> numbers and delete files. We're also making some other requirements more >>>> strict, but those are mainly cleaning up problems and not related to >>>> row-level deletes. >>>> >>>> Upserts would be encoded as a delete and an insert. Deletes are stored >>>> in delete files, and inserts are normal data files. Delete files are valid >>>> within a partition, and apply to all data files with the same or lower >>>> sequence number. >>>> >>>> I'm planning on updating what's currently in the spec now that we have >>>> sequence numbers and delete file metadata committed in master, but right >>>> now I'm working on getting the 0.9.0 release out with support for Spark 3. >>>> The documentation should be coming in the next couple of weeks. >>>> >>>> rb >>>> >>>> On Wed, Jul 1, 2020 at 6:28 AM Chen Song <chen.song...@gmail.com> >>>> wrote: >>>> >>>>> I saw Table Spec V2 >>>>> <https://iceberg.apache.org/spec/#version-2-row-level-deletes> was >>>>> mentioned in the official iceberg doc. I know it is incomplete and wip. Is >>>>> there any to-be-reviewed or proposed version for public view? I am >>>>> interested to understand how row level upserts are supported? >>>>> >>>>> Thanks >>>>> -- >>>>> Chen Song >>>>> >>>>> >>>> >>>> -- >>>> Ryan Blue >>>> Software Engineer >>>> Netflix >>>> >>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- Chen Song
