Re: [Reminder] Spark 3.5 RC Cut

2023-08-04 Thread Dongjoon Hyun
Thank you again, Emil and Bjorn. FYI, SPARK-44678 landed at branch-3.5 like the following. https://github.com/apache/spark/pull/42345 [SPARK-44678][BUILD][3.5] Downgrade Hadoop to 3.3.4 Dongjoon. On 2023/08/02 18:58:51 Bjørn Jørgensen wrote: > @Dongjoon Hyun FYI > [image: image.png] > > We be

Re: [Reminder] Spark 3.5 RC Cut

2023-08-02 Thread Bjørn Jørgensen
@Dongjoon Hyun FYI [image: image.png] We better ask common-...@hadoop.apache.org. ons. 2. aug. 2023 kl. 18:03 skrev Dongjoon Hyun : > Oh, I got it, Emil and Bjorn. > > Dongjoon. > > On Wed, Aug 2, 2023 at 12:32 AM Bjørn Jørgensen > wrote: > >> "*As far as I can tell this makes both 3.3.5 and 3

Re: [Reminder] Spark 3.5 RC Cut

2023-08-02 Thread Dongjoon Hyun
Oh, I got it, Emil and Bjorn. Dongjoon. On Wed, Aug 2, 2023 at 12:32 AM Bjørn Jørgensen wrote: > "*As far as I can tell this makes both 3.3.5 and 3.3.6 unusable with s3 > without providing an alternative committer code.*" > > https://github.com/apache/hadoop/pull/5706#issuecomment-1619927992 >

Re: [Reminder] Spark 3.5 RC Cut

2023-08-02 Thread Bjørn Jørgensen
"*As far as I can tell this makes both 3.3.5 and 3.3.6 unusable with s3 without providing an alternative committer code.*" https://github.com/apache/hadoop/pull/5706#issuecomment-1619927992 ons. 2. aug. 2023 kl. 08:05 skrev Emil Ejbyfeldt : > > Apache Spark is not affected by HADOOP-18757 becau

Re: [Reminder] Spark 3.5 RC Cut

2023-08-01 Thread Emil Ejbyfeldt
> Apache Spark is not affected by HADOOP-18757 because it is not a part of > both Apache Hadoop 3.3.5 and 3.3.6. I am not sure I am following what you are trying to say here. Is that the jira is saying that only 3.3.5 is affected? Here I think the Jira is just incorrect. The jira was created (a

Re: [Reminder] Spark 3.5 RC Cut

2023-08-01 Thread Dongjoon Hyun
It's still invalid information, Emil. Apache Spark is not affected by HADOOP-18757 because it is not a part of both Apache Hadoop 3.3.5 and 3.3.6. HADOOP-18757 seems to be merged just two weeks ago and there is no Apache Hadoop release with it, isn't it? Could you check your local branch once mo

Re: [Reminder] Spark 3.5 RC Cut

2023-08-01 Thread Emil Ejbyfeldt
Hi, Yes, sorry about that seem to have messed up the link. Should have been https://issues.apache.org/jira/browse/HADOOP-18757 Best, Emil On 01/08/2023 19:08, Dongjoon Hyun wrote: Hi, Emil. HADOOP-18568 is still open and it seems to be never a part of the Hadoop trunk branch. Do you mean

Re: [Reminder] Spark 3.5 RC Cut

2023-08-01 Thread Dongjoon Hyun
Hi, Emil. HADOOP-18568 is still open and it seems to be never a part of the Hadoop trunk branch. Do you mean another JIRA? Dongjoon. On Tue, Aug 1, 2023 at 2:59 AM Emil Ejbyfeldt wrote: > Hi, > > We previously ran some experiments on builds from the 3.5 branch and > noticed that Hadoop had

Re: [Reminder] Spark 3.5 RC Cut

2023-08-01 Thread Emil Ejbyfeldt
Hi, We previously ran some experiments on builds from the 3.5 branch and noticed that Hadoop had a regression (https://issues.apache.org/jira/browse/HADOOP-18568) in their s3a committer affecting 3.3.5 and 3.3.6 (Spark 3.4 uses hadoop 3.3.4). This fix has been merged into Hadoop and will be p

[Reminder] Spark 3.5 RC Cut

2023-07-29 Thread Yuanjian Li
Hi everyone, Following the release timeline, I will cut the RC on* Tuesday, Aug 1st at 1 pm PST* as scheduled. DateEvent July 17th 2023 Late July 2023 Code freeze. Release branch cut. QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. August 2023 Releas