Re: [ANNOUNCE] Announcing Apache Spark 2.4.4

2019-09-01 Thread Hyukjin Kwon
YaY! 2019년 9월 2일 (월) 오후 1:27, Wenchen Fan 님이 작성: > Great! Thanks! > > On Mon, Sep 2, 2019 at 5:55 AM Dongjoon Hyun > wrote: > >> We are happy to announce the availability of Spark 2.4.4! >> >> Spark 2.4.4 is a maintenance release containing stability fixes. This >> release is based on the branch

Re: [ANNOUNCE] Announcing Apache Spark 2.4.4

2019-09-01 Thread Wenchen Fan
Great! Thanks! On Mon, Sep 2, 2019 at 5:55 AM Dongjoon Hyun wrote: > We are happy to announce the availability of Spark 2.4.4! > > Spark 2.4.4 is a maintenance release containing stability fixes. This > release is based on the branch-2.4 maintenance branch of Spark. We strongly > recommend all 2

Re: concurrent writes with dynamic partition overwrite mode

2019-09-01 Thread Koert Kuipers
https://issues.apache.org/jira/browse/SPARK-28945 On Sun, Sep 1, 2019 at 2:53 PM Koert Kuipers wrote: > hi, > i am struggling to understand if concurrent writes to same basedir but > different partitions are save with file sources such as parquet. > > i tested this in spark 2.4 and spark 3.0.0-S

Re: [DISCUSSION]JDK11 for Apache 2.x?

2019-09-01 Thread Sean Owen
That's just the Oracle 'commercial' JDK though. OpenJDK is still OSS and 8 is updated there too. Oracle JDK is still free too, just not for commercial use and commercial support for 8 (IIUC). I think that's the 'audience' here, and I don't know how large or small that is. That said a key first que

Re: [DISCUSSION]JDK11 for Apache 2.x?

2019-09-01 Thread Holden Karau
So I'm not a lawyer, but looking at the JDK8 License FAQ ( https://www.oracle.com/technetwork/java/javase/overview/oracle-jdk-faqs.html ) makes me feel like we should support folks who are on the 2.x branch and want to migrate away from JDK8. Although the Redhat post about OpenJDK8 is reassuring (

Re: Design review of SPARK-28594

2019-09-01 Thread Jungtaek Lim
Great, thanks for reviewing, Felix! On Mon, Sep 2, 2019 at 2:16 AM Felix Cheung wrote: > I did review it and solving this problem makes sense. I will comment in > the JIRA. > > -- > *From:* Jungtaek Lim > *Sent:* Sunday, August 25, 2019 3:34:22 PM > *To:* dev > *Sub

[ANNOUNCE] Announcing Apache Spark 2.4.4

2019-09-01 Thread Dongjoon Hyun
We are happy to announce the availability of Spark 2.4.4! Spark 2.4.4 is a maintenance release containing stability fixes. This release is based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4 users to upgrade to this stable release. To download Spark 2.4.4, head over

concurrent writes with dynamic partition overwrite mode

2019-09-01 Thread Koert Kuipers
hi, i am struggling to understand if concurrent writes to same basedir but different partitions are save with file sources such as parquet. i tested this in spark 2.4 and spark 3.0.0-SNAPSHOT with real concurrent jobs on hdfs and it seemed to work fine. admittedly this was a rather limited test. a

Re: Design review of SPARK-28594

2019-09-01 Thread Felix Cheung
I did review it and solving this problem makes sense. I will comment in the JIRA. From: Jungtaek Lim Sent: Sunday, August 25, 2019 3:34:22 PM To: dev Subject: Design review of SPARK-28594 Hi devs, I have been working on designing SPARK-28594 [1] (though I've s