I agree, let's create a spark-3.0 branch to start with. We've been building vectorization this way using the vectorized-reads branch.
In the long term, we may want to split Spark into separate modules for 2.x and 3.x in the same branch, but for now we can at least get everything working with a 3.0 branch. On Fri, Nov 22, 2019 at 8:34 AM John Zhuge <jzh...@apache.org> wrote: > +1 for Iceberg branch > > Thanks for the contribution from you and your team! > > On Fri, Nov 22, 2019 at 8:29 AM Anton Okolnychyi > <aokolnyc...@apple.com.invalid> wrote: > >> +1 on having a branch in Iceberg as we have for vectorized reads. >> >> - Anton >> >> On 22 Nov 2019, at 02:26, Saisai Shao <sai.sai.s...@gmail.com> wrote: >> >> Hi Ryan and team, >> >> Thanks a lot for your response. I was wondering how do we share our >> branch, one possible way s that we maintain a forked Iceberg repo with >> Spark 3.0.0-preview branch, another possible way is to create a branch in >> upstream Iceberg repo. I'm inclined to choose the second way, so that they >> community could review and contribute on it. >> >> I would like to hear your suggestions. >> >> Best regards, >> Saisai >> >> >> Ryan Blue <rb...@netflix.com.invalid> 于2019年11月20日周三 上午1:27写道： >> >>> Sounds great, thanks Saisai! >>> >>> On Mon, Nov 18, 2019 at 3:29 AM Saisai Shao <sai.sai.s...@gmail.com> >>> wrote: >>> >>>> Thanks Anton, I will share our branch soon. >>>> >>>> Best regards, >>>> Saisai >>>> >>>> Anton Okolnychyi <aokolnyc...@apple.com.invalid> 于2019年11月18日周一 >>>> 下午6:54写道： >>>> >>>>> I think it would be great if you can share what you have, Saisai. That >>>>> way, we can all collaborate and ensure we build a full 3.0 integration as >>>>> soon as possible. >>>>> >>>>> - Anton >>>>> >>>>> >>>>> On 18 Nov 2019, at 02:08, Saisai Shao <sai.sai.s...@gmail.com> wrote: >>>>> >>>>> Hi Anton, >>>>> >>>>> Thanks to bring this out. We already have a branch building against >>>>> Spark 3.0 (Master branch actually) internally, and we're actively working >>>>> on it. I think it is a good idea to create an upstream Spark 3.0 branch, >>>>> we >>>>> could share it if the community would like to do so. >>>>> >>>>> Best regards, >>>>> Saisai >>>>> >>>>> Anton Okolnychyi <aokolnyc...@apple.com.invalid> 于2019年11月18日周一 >>>>> 上午1:40写道： >>>>> >>>>>> I think it is a good time to create a branch to build our 3.0 >>>>>> integration as the 3.0 preview was released. >>>>>> What does everyone think? Has anybody started already? >>>>>> >>>>>> - Anton >>>>>> >>>>>> On 8 Aug 2019, at 23:47, Edgar Rodriguez <edgar.rodrig...@airbnb.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 8, 2019 at 3:37 PM Ryan Blue <rb...@netflix.com> wrote: >>>>>> >>>>>>> I think it's a great idea to branch and get ready for Spark 3.0.0. >>>>>>> Right now, I'm focused on getting a release out, but I can review >>>>>>> patches >>>>>>> for Spark 3.0. >>>>>>> >>>>>>> Anyone know if there are nightly builds of Spark 3.0 to test with? >>>>>>> >>>>>> >>>>>> Seems like there're nightly snapshots built in >>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/ >>>>>> - >>>>>> I've started setting something up with these snapshots so I can probably >>>>>> start working on this. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Cheers, >>>>>> -- >>>>>> Edgar Rodriguez >>>>>> >>>>>> >>>>>> >>>>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> >> > > -- > John Zhuge > -- Ryan Blue Software Engineer Netflix
