Supporting Python 3.8.0 sounds like a new feature, and doesn't qualify a backport. But I'm open to other opinions.
On Wed, Jul 15, 2020 at 11:24 PM Ismaël Mejía <ieme...@gmail.com> wrote: > Any chance that SPARK-29536 PySpark does not work with Python 3.8.0 > can be backported to 2.4.7 ? > This was not done for Spark 2.4.6 because it was too late on the vote > process but it makes perfect sense to have this in 2.4.7. > > On Wed, Jul 15, 2020 at 9:07 AM Wenchen Fan <cloud0...@gmail.com> wrote: > > > > Yea I think 2.4.7 is good to go. Let's start! > > > > On Wed, Jul 15, 2020 at 1:50 PM Prashant Sharma <scrapco...@gmail.com> > wrote: > >> > >> Hi Folks, > >> > >> So, I am back, and searched the JIRAS with target version as "2.4.7" > and Resolved, found only 2 jiras. So, are we good to go, with just a couple > of jiras fixed ? Shall I proceed with making a RC? > >> > >> Thanks, > >> Prashant > >> > >> On Thu, Jul 2, 2020 at 5:23 PM Prashant Sharma <scrapco...@gmail.com> > wrote: > >>> > >>> Thank you, Holden. > >>> > >>> Folks, My health has gone down a bit. So, I will start working on this > in a few days. If this needs to be published sooner, then maybe someone > else has to help out. > >>> > >>> > >>> > >>> > >>> > >>> On Thu, Jul 2, 2020 at 10:11 AM Holden Karau <hol...@pigscanfly.ca> > wrote: > >>>> > >>>> I’m happy to have Prashant do 2.4.7 :) > >>>> > >>>> On Wed, Jul 1, 2020 at 9:40 PM Xiao Li <lix...@databricks.com> wrote: > >>>>> > >>>>> +1 on releasing both 3.0.1 and 2.4.7 > >>>>> > >>>>> Great! Three committers volunteer to be a release manager. Ruifeng, > Prashant and Holden. Holden just helped release Spark 2.4.6. This time, > maybe, Ruifeng and Prashant can be the release manager of 3.0.1 and 2.4.7 > respectively. > >>>>> > >>>>> Xiao > >>>>> > >>>>> On Wed, Jul 1, 2020 at 2:24 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >>>>>> > >>>>>> https://issues.apache.org/jira/browse/SPARK-32148 was reported > yesterday, and if the report is valid it looks to be a blocker. I'll try to > take a look sooner. > >>>>>> > >>>>>> On Thu, Jul 2, 2020 at 12:48 AM Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >>>>>>> > >>>>>>> Thanks Holden -- it would be great to also get 2.4.7 started > >>>>>>> > >>>>>>> Thanks > >>>>>>> Shivaram > >>>>>>> > >>>>>>> On Tue, Jun 30, 2020 at 10:31 PM Holden Karau < > hol...@pigscanfly.ca> wrote: > >>>>>>> > > >>>>>>> > I can take care of 2.4.7 unless someone else wants to do it. > >>>>>>> > > >>>>>>> > On Tue, Jun 30, 2020 at 8:29 PM Jason Moore < > jason.mo...@quantium.com.au> wrote: > >>>>>>> >> > >>>>>>> >> Hi all, > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Could I get some input on the severity of this one that I found > yesterday? If that’s a correctness issue, should it block this patch? Let > me know under the ticket if there’s more info that I can provide to help. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> https://issues.apache.org/jira/browse/SPARK-32136 > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Thanks, > >>>>>>> >> > >>>>>>> >> Jason. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> From: Jungtaek Lim <kabhwan.opensou...@gmail.com> > >>>>>>> >> Date: Wednesday, 1 July 2020 at 10:20 am > >>>>>>> >> To: Shivaram Venkataraman <shiva...@eecs.berkeley.edu> > >>>>>>> >> Cc: Prashant Sharma <scrapco...@gmail.com>, 郑瑞峰 < > ruife...@foxmail.com>, Gengliang Wang <gengliang.w...@databricks.com>, > gurwls223 <gurwls...@gmail.com>, Dongjoon Hyun <dongjoon.h...@gmail.com>, > Jules Damji <dmat...@comcast.net>, Holden Karau <hol...@pigscanfly.ca>, > Reynold Xin <r...@databricks.com>, Yuanjian Li <xyliyuanj...@gmail.com>, " > dev@spark.apache.org" <dev@spark.apache.org>, Takeshi Yamamuro < > linguin....@gmail.com> > >>>>>>> >> Subject: Re: [DISCUSS] Apache Spark 3.0.1 Release > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> SPARK-32130 [1] looks to be a performance regression introduced > in Spark 3.0.0, which is ideal to look into before releasing another bugfix > version. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> 1. https://issues.apache.org/jira/browse/SPARK-32130 > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Wed, Jul 1, 2020 at 7:05 AM Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >>>>>>> >> > >>>>>>> >> Hi all > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> I just wanted to ping this thread to see if all the outstanding > blockers for 3.0.1 have been fixed. If so, it would be great if we can get > the release going. The CRAN team sent us a note that the version SparkR > available on CRAN for the current R version (4.0.2) is broken and hence we > need to update the package soon -- it will be great to do it with 3.0.1. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Thanks > >>>>>>> >> > >>>>>>> >> Shivaram > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Wed, Jun 24, 2020 at 8:31 PM Prashant Sharma < > scrapco...@gmail.com> wrote: > >>>>>>> >> > >>>>>>> >> +1 for 3.0.1 release. > >>>>>>> >> > >>>>>>> >> I too can help out as release manager. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Thu, Jun 25, 2020 at 4:58 AM 郑瑞峰 <ruife...@foxmail.com> > wrote: > >>>>>>> >> > >>>>>>> >> I volunteer to be a release manager of 3.0.1, if nobody is > working on this. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> ------------------ 原始邮件 ------------------ > >>>>>>> >> > >>>>>>> >> 发件人: "Gengliang Wang"<gengliang.w...@databricks.com>; > >>>>>>> >> > >>>>>>> >> 发送时间: 2020年6月24日(星期三) 下午4:15 > >>>>>>> >> > >>>>>>> >> 收件人: "Hyukjin Kwon"<gurwls...@gmail.com>; > >>>>>>> >> > >>>>>>> >> 抄送: "Dongjoon Hyun"<dongjoon.h...@gmail.com>;"Jungtaek Lim"< > kabhwan.opensou...@gmail.com>;"Jules Damji"<dmat...@comcast.net>;"Holden > Karau"<hol...@pigscanfly.ca>;"Reynold Xin"<r...@databricks.com>;"Shivaram > Venkataraman"<shiva...@eecs.berkeley.edu>;"Yuanjian Li"< > xyliyuanj...@gmail.com>;"Spark dev list"<dev@spark.apache.org>;"Takeshi > Yamamuro"<linguin....@gmail.com>; > >>>>>>> >> > >>>>>>> >> 主题: Re: [DISCUSS] Apache Spark 3.0.1 Release > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> +1, the issues mentioned are really serious. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 7:56 PM Hyukjin Kwon < > gurwls...@gmail.com> wrote: > >>>>>>> >> > >>>>>>> >> +1. > >>>>>>> >> > >>>>>>> >> Just as a note, > >>>>>>> >> - SPARK-31918 is fixed now, and there's no blocker. - When we > build SparkR, we should use the latest R version at least 4.0.0+. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> 2020년 6월 24일 (수) 오전 11:20, Dongjoon Hyun < > dongjoon.h...@gmail.com>님이 작성: > >>>>>>> >> > >>>>>>> >> +1 > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Bests, > >>>>>>> >> > >>>>>>> >> Dongjoon. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 1:19 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >>>>>>> >> > >>>>>>> >> +1 on a 3.0.1 soon. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Probably it would be nice if some Scala experts can take a look > at https://issues.apache.org/jira/browse/SPARK-32051 and include the fix > into 3.0.1 if possible. > >>>>>>> >> > >>>>>>> >> Looks like APIs designed to work with Scala 2.11 & Java bring > ambiguity in Scala 2.12 & Java. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Wed, Jun 24, 2020 at 4:52 AM Jules Damji < > dmat...@comcast.net> wrote: > >>>>>>> >> > >>>>>>> >> +1 (non-binding) > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> Sent from my iPhone > >>>>>>> >> > >>>>>>> >> Pardon the dumb thumb typos :) > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Jun 23, 2020, at 11:36 AM, Holden Karau < > hol...@pigscanfly.ca> wrote: > >>>>>>> >> > >>>>>>> >> +1 on a patch release soon > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 10:47 AM Reynold Xin < > r...@databricks.com> wrote: > >>>>>>> >> > >>>>>>> >> Error! Filename not specified. > >>>>>>> >> > >>>>>>> >> +1 on doing a new patch release soon. I saw some of these > issues when preparing the 3.0 release, and some of them are very serious. > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 8:06 AM, Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >>>>>>> >> > >>>>>>> >> +1 Thanks Yuanjian -- I think it'll be great to have a 3.0.1 > release soon. > >>>>>>> >> > >>>>>>> >> Shivaram > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 3:43 AM Takeshi Yamamuro < > linguin....@gmail.com> wrote: > >>>>>>> >> > >>>>>>> >> Thanks for the heads-up, Yuanjian! > >>>>>>> >> > >>>>>>> >> I also noticed branch-3.0 already has 39 commits after Spark > 3.0.0. > >>>>>>> >> > >>>>>>> >> wow, the updates are so quick. Anyway, +1 for the release. > >>>>>>> >> > >>>>>>> >> Bests, > >>>>>>> >> Takeshi > >>>>>>> >> > >>>>>>> >> On Tue, Jun 23, 2020 at 4:59 PM Yuanjian Li < > xyliyuanj...@gmail.com> wrote: > >>>>>>> >> > >>>>>>> >> Hi dev-list, > >>>>>>> >> > >>>>>>> >> I’m writing this to raise the discussion about Spark 3.0.1 > feasibility since 4 blocker issues were found after Spark 3.0.0: > >>>>>>> >> > >>>>>>> >> [SPARK-31990] The state store compatibility broken will cause a > correctness issue when Streaming query with `dropDuplicate` uses the > checkpoint written by the old Spark version. > >>>>>>> >> > >>>>>>> >> [SPARK-32038] The regression bug in handling NaN values in > COUNT(DISTINCT) > >>>>>>> >> > >>>>>>> >> [SPARK-31918][WIP] CRAN requires to make it working with the > latest R 4.0. It makes the 3.0 release unavailable on CRAN, and only > supports R [3.5, 4.0) > >>>>>>> >> > >>>>>>> >> [SPARK-31967] Downgrade vis.js to fix Jobs UI loading time > regression > >>>>>>> >> > >>>>>>> >> I also noticed branch-3.0 already has 39 commits after Spark > 3.0.0. I think it would be great if we have Spark 3.0.1 to deliver the > critical fixes. > >>>>>>> >> > >>>>>>> >> Any comments are appreciated. > >>>>>>> >> > >>>>>>> >> Best, > >>>>>>> >> > >>>>>>> >> Yuanjian > >>>>>>> >> > >>>>>>> >> -- > >>>>>>> >> --- > >>>>>>> >> Takeshi Yamamuro > >>>>>>> >> > >>>>>>> >> > --------------------------------------------------------------------- To > unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> -- > >>>>>>> >> > >>>>>>> >> Twitter: https://twitter.com/holdenkarau > >>>>>>> >> > >>>>>>> >> Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >>>>>>> >> > >>>>>>> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Twitter: https://twitter.com/holdenkarau > >>>>>>> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >>>>>>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>> > >>>> -- > >>>> Twitter: https://twitter.com/holdenkarau > >>>> Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
