Re: Spark branch-3.1

2020-12-04 Thread Hyukjin Kwon
Shane, do you mind setting up Jenkins jobs for branch-3.2 please? On Sat, 5 Dec 2020, 08:14 Hyukjin Kwon, wrote: > Great, thank you for doing this. > > On Sat, 5 Dec 2020, 02:02 Dongjoon Hyun, wrote: > >> Thank you so much, Hyukjin Kwon. >> >> I made a PR for updating the `master` branch to 3.2

Re: Spark branch-3.1

2020-12-04 Thread Hyukjin Kwon
Great, thank you for doing this. On Sat, 5 Dec 2020, 02:02 Dongjoon Hyun, wrote: > Thank you so much, Hyukjin Kwon. > > I made a PR for updating the `master` branch to 3.2.0-SNAPSHOT. > > https://github.com/apache/spark/pull/30606 > [SPARK-33662][BUILD] Setting version to 3.2.0-SNAPSHOT

Re: [build system] WE'RE LIVE!

2020-12-04 Thread shane knapp ☠
ok, it's broken on the new nodes, so i tied the project to ubuntu16. i'll create a jira and investigate further at a later date. On Fri, Dec 4, 2020 at 8:58 AM shane knapp ☠ wrote: > no, it isn't but i'll try and take a look at this later today. > > On Fri, Dec 4, 2020 at 7:12 AM Tom Graves wr

Re: Spark branch-3.1

2020-12-04 Thread Dongjoon Hyun
Thank you so much, Hyukjin Kwon. I made a PR for updating the `master` branch to 3.2.0-SNAPSHOT. https://github.com/apache/spark/pull/30606 [SPARK-33662][BUILD] Setting version to 3.2.0-SNAPSHOT Bests, Dongjoon. On Fri, Dec 4, 2020 at 7:05 AM Tom Graves wrote: > Can we update the ver

Re: [build system] WE'RE LIVE!

2020-12-04 Thread shane knapp ☠
no, it isn't but i'll try and take a look at this later today. On Fri, Dec 4, 2020 at 7:12 AM Tom Graves wrote: > thanks Shane and folks for great work. > > Not sure if this is at all related but I noticed the spark master deploy > job hasn't been running and the last one Dec 2nd failed: > > htt

Re: [build system] WE'RE LIVE!

2020-12-04 Thread Tom Graves
thanks Shane and folks for great work. Not sure if this is at all related but I noticed the spark master deploy job hasn't been running and the last one Dec 2nd failed:https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/3186/ Not sure if this is result

Re: Spark branch-3.1

2020-12-04 Thread Tom Graves
Can we update the version number on the master branch ? its still 3.1.0-SNAPSHOT Thanks,Tom  On Friday, December 4, 2020, 04:54:12 AM CST, Hyukjin Kwon wrote: Hi all, It’s 4th PDT and branch-3.1 is cut out now as planned. Mid Dec 2020 QA period. Focus on bug fixes, tests, stability

AWS Consistent S3 & Apache Hadoop's S3A connector

2020-12-04 Thread Steve Loughran
as sent to hadoop-general. TL;DR. S3 is consistent; S3A now works perfectly with S3Guard turned off, if not, file a JIRA. rename still isn't real, so don't rely on that or create(path, overwrite=false) for atomic operations --- If you've missed the announcement, AWS S3 storage is now strong

Spark branch-3.1

2020-12-04 Thread Hyukjin Kwon
Hi all, It’s 4th PDT and branch-3.1 is cut out now as planned. Mid Dec 2020 QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged Now we’re in the QA period. Please focus on testing, polishing, stability and docs for Spark 3.1.0, and hope we can have a nice