Re: [Reminder] Spark 3.5 RC Cut

pache Hadoop 3.3.5 and 3.3.6. HADOOP-18757 seems to be merged just two weeks ago and there is no Apache Hadoop release with it, isn't it? Could you check your local branch once more, please? Dongjoon. On Tue, Aug 1, 2023 at 9:46 PM Emil Ejbyfeldt <mailto:eejbyfe...@liveintent.c

Re: [Reminder] Spark 3.5 RC Cut

mean another JIRA? Dongjoon. On Tue, Aug 1, 2023 at 2:59 AM Emil Ejbyfeldt wrote: Hi, We previously ran some experiments on builds from the 3.5 branch and noticed that Hadoop had a regression (https://issues.apache.org/jira/browse/HADOOP-18568 <https://issues.apache.

Re: [Reminder] Spark 3.5 RC Cut

Hi, We previously ran some experiments on builds from the 3.5 branch and noticed that Hadoop had a regression (https://issues.apache.org/jira/browse/HADOOP-18568) in their s3a committer affecting 3.3.5 and 3.3.6 (Spark 3.4 uses hadoop 3.3.4). This fix has been merged into Hadoop and will be p

Re: Spark 3.4.0 with Hadoop2.7 cannot be downloaded

Hi, I think this is expected as it was dropped from the release process in https://issues.apache.org/jira/browse/SPARK-40651 Also I don't see a Hadoop2.7 option when selecting Spark 3.4.0 on https://spark.apache.org/downloads.html Not really sure why you could be seeing that. Best, Emil O

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 (non-binding) Ran some tests with the Scala 2.13 build using part of our internal spark workload. On 12/04/2023 19:52, Chris Nauroth wrote: +1 (non-binding) * Verified all checksums. * Verified all signatures. * Built from source, with multiple profiles, to full success:     * build/mvn

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

It might being caused by the v3.4.0-rc3 tag not being part of the 3.4 branch branch-3.4: $ git log --pretty='format:%d %h' --graph origin/branch-3.4 v3.4.0-rc3 | head -n 10 * (HEAD, origin/branch-3.4) e38e619946 * f3e69a1fe2 * 74cf1a32b0 * 0191a5bde0 * afced91348 | * (tag: v3.4.0-rc3) b

Re: Missing data in spark output

Hi, We have observed similar behavior in older versions of spark. But we were are currently using 3.3.0 where we have not seen such issues. Which version of Spark and Hadoop are you using? On 18/10/2022 19:48, Sandeep Vinayak wrote: Hello Everyone, We are recently observing an intermittent

Re: [VOTE] Release Spark 3.3.0 (RC2)

Hi, When testing out Spark 3.3.0 on our production spark workload it was noticed that https://issues.apache.org/jira/browse/SPARK-38681 is actually a regression from 3.2 (I did not know this a the time of creating the ticket) seem like the bug was introduced in https://github.com/apache/spark

MetadataFetchFailedException due to decommission block migrations

As noted in SPARK-34939 there is race when using broadcast for map output status. Explanation from SPARK-34939 > After map statuses are broadcasted and the executors obtain serialized broadcasted map statuses. If any fetch failure happens after, Spark scheduler invalidates cached map statuses

Re: [SPARK-20384][SQL] Support value class in schema of Dataset (third time's a charm)

ed in this feature to leave a review/comments on that PR. Also it would be great if some admin would be able to review it. Or any tips on steps that should be taken in order to have the PR reviewed would be appreciated. / Emil On 25/05/2021 16:33, Emil Ejbyfeldt wrote: Hi dev, I am inter

[SPARK-20384][SQL] Support value class in schema of Dataset (third time's a charm)

Hi dev, I am interested getting the support value classes in schemas of Dataset merged and I am willing to work on it. There are two previous PRs created for this JIRA (SPARK-20384) first https://github.com/apache/spark/pull/22309 and more recently https://github.com/apache/spark/pull/27153