Re: Branch 2.4 is cut

Since it's not a clean revert, I've sent a PR to revert it from 2.4, please take a look, thanks! https://github.com/apache/spark/pull/22388 On Tue, Sep 11, 2018 at 1:16 AM Ryan Blue wrote: > SPARK-24882 was committed in order to make some progress, with a note > about following up with separate

Re: Branch 2.4 is cut

changes in 2.4 are adding now? From: Wenchen Fan Sent: Monday, September 10, 2018 12:35 AM To: ar...@apache.org Cc: Ryan Blue; skn...@berkeley.edu; Dongjoon Hyun; joshro...@databricks.com; Sean Owen; Spark dev list Subject: Re: Branch 2.4 is cut There are a lot of

Re: Branch 2.4 is cut

There are a lot of "breaking" changes we made in 2.4 for data source v2, while I agree SPARK-24882 is "breaking" most. I don't agree SPARK-24882 is half-baked. But I'm willing to revert it if we have a bunch of data source v2 users and they are not willing to update their implementation intensely

Re: Branch 2.4 is cut

Ryan's proposal makes a lot of sense. Its better not to release half-baked changes in 2.4 which not only breaks a lot of the APIs released in 2.3, but also expected to change further due redesigns before 3.0 so don't see much value releasing it in 2.4. On Sun, 9 Sep 2018 at 22:42, Wenchen Fan wro

Re: Branch 2.4 is cut

Strictly speaking, data source v2 is always half-finished until we mark it as stable. We need some small milestones to move forward step by step. The redesign also happens in an incremental way. SPARK-24882 mostly focus on the "RDD" part of the API: the separation of reader factory and input parti

Re: Branch 2.4 is cut

Wenchen, can you hold off on the first RC? The half-finished changes from the redesign of the DataSourceV2 API are in master, added in SPARK-24882 , and are now in the 2.4 branch. We've had a lot of good discussion since that PR was merged to update and

Re: Branch 2.4 is cut

++joshrosen (thanks for the help w/deploying the jenkins configs) the basic 2.4 builds are deployed and building! i haven't created (a) build(s) yet for scala 2.12... i'll be coordinating this w/the databricks folks next week. On Fri, Sep 7, 2018 at 9:53 AM, Dongjoon Hyun wrote: > Thank you,

Re: Branch 2.4 is cut

Thank you, Shane! :D Bests, Dongjoon. On Fri, Sep 7, 2018 at 9:51 AM shane knapp wrote: > i'll try and get to the 2.4 branch stuff today... > >

Re: Branch 2.4 is cut

i'll try and get to the 2.4 branch stuff today... On Fri, Sep 7, 2018 at 8:15 AM, Wenchen Fan wrote: > I've reached to Shane, but he is busy recently. I'll figure it out with > Josh soon. Will post update to this thread later. > > Thanks, > Wenchen > > On Fri, Sep 7, 2018 at 11:01 PM Sean Owen

Re: Branch 2.4 is cut

I'm just using 3.0 but would not hurt to create 2.5.0. If 2.5 doesn't happen then we just move those to 3.0.0 later. On Fri, Sep 7, 2018, 9:40 AM Holden Karau wrote: > Was doing my weekly code review > and went to close an issue, > but since it wasn'

Re: Branch 2.4 is cut

Was doing my weekly code review and went to close an issue, but since it wasn't one of the categories listed wasn't going to merge into the 2.4 branch but we need a new version in JIRA for us to close issues to that are going to merge into master but no

Re: Branch 2.4 is cut

I've reached to Shane, but he is busy recently. I'll figure it out with Josh soon. Will post update to this thread later. Thanks, Wenchen On Fri, Sep 7, 2018 at 11:01 PM Sean Owen wrote: > CC Shane who might have the permission to do so. > > If master is going to be 3.0, we can remove the Hadoo

Re: Branch 2.4 is cut

CC Shane who might have the permission to do so. If master is going to be 3.0, we can remove the Hadoop 2.6 builds soon for master, note. We could remove them now, honestly. On Thu, Sep 6, 2018 at 10:09 AM Dongjoon Hyun wrote: > Great for branch cut and Scala 2.12 build. > > We also need to add

Re: Branch 2.4 is cut

Great for branch cut and Scala 2.12 build. We also need to add `branch-2.4` to our Jenkins dashboard to prevent any regression. https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ Bests, Dongjoon. On Thu, Sep 6, 2018 at 6:56 AM Wenchen Fan wrote: > Good news! I'll tr

Re: Branch 2.4 is cut

Good news! I'll try and update you later. Thanks! On Thu, Sep 6, 2018 at 9:44 PM Sean Owen wrote: > BTW it does appear the Scala 2.12 build works now: > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/229/ > > Let's try als

Re: Branch 2.4 is cut

BTW it does appear the Scala 2.12 build works now: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/229/ Let's try also producing a 2.12 build with this release. The machinery should be there in the release scripts, but let me k

Re: Branch 2.4 is cut

Thanks, Wenchen. 2018년 9월 6일 (목) 오후 3:32, Wenchen Fan 님이 작성: > Hi all, > > I've cut the branch-2.4 since all the major blockers are resolved. If no > objections I'll shortly followup with an RC to get the QA started in > parallel. > > Committers, please only merge PRs to branch-2.4 that are bug f