Re: [VOTE] Release Spark 3.2.0 (RC4)

[Peter Toth]

Hi All,

Sorry, but I've just run into this issue:
https://issues.apache.org/jira/browse/SPARK-35672?focusedCommentId=17419285&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17419285
I think SPARK-35672 is a breaking change.

Peter


On Thu, Sep 23, 2021 at 5:32 PM Yi Wu  wrote:

> +1 (non-binding)
>
> Thanks for the work, Gengliang!
>
> Bests,
> Yi
>
> On Thu, Sep 23, 2021 at 10:03 PM Gengliang Wang  wrote:
>
>> Starting with my +1(non-binding)
>>
>> Thanks,
>> Gengliang
>>
>> On Thu, Sep 23, 2021 at 10:02 PM Gengliang Wang  wrote:
>>
>>> Please vote on releasing the following candidate as
>>> Apache Spark version 3.2.0.
>>>
>>> The vote is open until 11:59pm Pacific time September 27 and passes if a
>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 3.2.0
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v3.2.0-rc4 (commit
>>> b609f2fe0c1dd9a7e7b3aedd31ab81e6311b9b3f):
>>> https://github.com/apache/spark/tree/v3.2.0-rc4
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc4-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1391
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc4-docs/
>>>
>>> The list of bug fixes going into 3.2.0 can be found at the following URL:
>>> https://issues.apache.org/jira/projects/SPARK/versions/12349407
>>>
>>> This release is using the release script of the tag v3.2.0-rc4.
>>>
>>>
>>> FAQ
>>>
>>> =
>>> How can I help test this release?
>>> =
>>> If you are a Spark user, you can help us test this release by taking
>>> an existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>> the current RC and see if anything important breaks, in the Java/Scala
>>> you can add the staging repository to your projects resolvers and test
>>> with the RC (make sure to clean up the artifact cache before/after so
>>> you don't end up building with a out of date RC going forward).
>>>
>>> ===
>>> What should happen to JIRA tickets still targeting 3.2.0?
>>> ===
>>> The current list of open tickets targeted at 3.2.0 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.2.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>> fixes, documentation, and API tweaks that impact compatibility should
>>> be worked on immediately. Everything else please retarget to an
>>> appropriate release.
>>>
>>> ==
>>> But my bug isn't fixed?
>>> ==
>>> In order to make timely releases, we will typically not hold the
>>> release unless the bug in question is a regression from the previous
>>> release. That being said, if there is something which is a regression
>>> that has not been correctly targeted please ping me or a committer to
>>> help target the issue.
>>>
>>

Re: [VOTE] Release Spark 3.2.0 (RC7)

[Peter Toth]

+1 (non-binding).

Peter


On Fri, Oct 8, 2021 at 9:16 AM Cheng Su  wrote:

> +1 (non-binding).
>
>
>
> Thanks,
>
> Cheng Su
>
>
>
> *From: *Reynold Xin 
> *Date: *Thursday, October 7, 2021 at 11:57 PM
> *To: *Yuming Wang 
> *Cc: *Dongjoon Hyun , 郑瑞峰 ,
> Sean Owen , Gengliang Wang , dev <
> dev@spark.apache.org>
> *Subject: *Re: [VOTE] Release Spark 3.2.0 (RC7)
>
> +1
>
> [image: Image removed by sender.]
>
>
>
>
>
> On Thu, Oct 07, 2021 at 11:54 PM, Yuming Wang  wrote:
>
> +1 (non-binding).
>
>
>
> On Fri, Oct 8, 2021 at 1:02 PM Dongjoon Hyun 
> wrote:
>
> +1 for Apache Spark 3.2.0 RC7.
>
>
>
> It looks good to me. I tested with EKS 1.21 additionally.
>
>
>
> Cheers,
>
> Dongjoon.
>
>
>
>
>
> On Thu, Oct 7, 2021 at 7:46 PM 郑瑞峰  wrote:
>
> +1 (non-binding)
>
>
>
>
>
> -- 原始邮件 --
>
> *发件人**:* "Sean Owen" ;
>
> *发送时间**:* 2021年10月7日(星期四) 晚上10:23
>
> *收件人**:* "Gengliang Wang";
>
> *抄送**:* "dev";
>
> *主题**:* Re: [VOTE] Release Spark 3.2.0 (RC7)
>
>
>
> +1 again. Looks good in Scala 2.12, 2.13, and in Java 11.
>
> I note that the mem requirements for Java 11 tests seem to need to be
> increased but we're handling that separately. It doesn't really affect
> users.
>
>
>
> On Wed, Oct 6, 2021 at 11:49 AM Gengliang Wang  wrote:
>
> Please vote on releasing the following candidate as
> Apache Spark version 3.2.0.
>
>
>
> The vote is open until 11:59pm Pacific time October 11 and passes if a
> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
>
>
> [ ] +1 Release this package as Apache Spark 3.2.0
>
> [ ] -1 Do not release this package because ...
>
>
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
>
>
> The tag to be voted on is v3.2.0-rc7 (commit
> 5d45a415f3a29898d92380380cfd82bfc7f579ea):
>
> https://github.com/apache/spark/tree/v3.2.0-rc7
>
>
>
> The release files, including signatures, digests, etc. can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc7-bin/
>
>
>
> Signatures used for Spark RCs can be found in this file:
>
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
>
>
> The staging repository for this release can be found at:
>
> https://repository.apache.org/content/repositories/orgapachespark-1394
>
>
>
> The documentation corresponding to this release can be found at:
>
> https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc7-docs/
>
>
>
> The list of bug fixes going into 3.2.0 can be found at the following URL:
>
> https://issues.apache.org/jira/projects/SPARK/versions/12349407
>
>
>
> This release is using the release script of the tag v3.2.0-rc7.
>
>
>
>
>
> FAQ
>
>
>
> =
>
> How can I help test this release?
>
> =
>
> If you are a Spark user, you can help us test this release by taking
>
> an existing Spark workload and running on this release candidate, then
>
> reporting any regressions.
>
>
>
> If you're working in PySpark you can set up a virtual env and install
>
> the current RC and see if anything important breaks, in the Java/Scala
>
> you can add the staging repository to your projects resolvers and test
>
> with the RC (make sure to clean up the artifact cache before/after so
>
> you don't end up building with a out of date RC going forward).
>
>
>
> ===
>
> What should happen to JIRA tickets still targeting 3.2.0?
>
> ===
>
> The current list of open tickets targeted at 3.2.0 can be found at:
>
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.2.0
>
>
>
> Committers should look at those and triage. Extremely important bug
>
> fixes, documentation, and API tweaks that impact compatibility should
>
> be worked on immediately. Everything else please retarget to an
>
> appropriate release.
>
>
>
> ==
>
> But my bug isn't fixed?
>
> ==
>
> In order to make timely releases, we will typically not hold the
>
> release unless the bug in question is a regression from the previous
>
> release. That being said, if there is something which is a regression
>
> that has not been correctly targeted please ping me or a committer to
>
> help target the issue.
>
>
>

Re: [VOTE] Release Spark 3.3.0 (RC6)

[Peter Toth]

+1


Martin Grigorov  ezt írta (időpont: 2022. jún. 14.,
K, 13:23):

> Hi,
>
> [X] +1 Release this package as Apache Spark 3.3.0
>
> Tested:
> - make local distribution from sources (with ./dev/make-distribution.sh
> --tgz --name with-volcano -Pkubernetes,volcano,hadoop-3)
> - create a Docker image (with JDK 11)
> - run Pi example on
> -- local
> -- Kubernetes with default scheduler
> -- Kubernetes with Volcano scheduler
>
> On both Linux x86_64 and aarch64 !
>
> Regards,
> Martin
>
> On Fri, Jun 10, 2022 at 7:28 AM Maxim Gekk
>  wrote:
>
>> Please vote on releasing the following candidate as
>> Apache Spark version 3.3.0.
>>
>> The vote is open until 11:59pm Pacific time June 14th and passes if a
>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.3.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.3.0-rc6 (commit
>> f74867bddfbcdd4d08076db36851e88b15e66556):
>> https://github.com/apache/spark/tree/v3.3.0-rc6
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc6-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1407
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.3.0-rc6-docs/
>>
>> The list of bug fixes going into 3.3.0 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12350369
>>
>> This release is using the release script of the tag v3.3.0-rc6.
>>
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.3.0?
>> ===
>> The current list of open tickets targeted at 3.3.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.3.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>> Maxim Gekk
>>
>> Software Engineer
>>
>> Databricks, Inc.
>>
>

Re: Welcome Xinrong Meng as a Spark committer

[Peter Toth]

Congratulations!

Bjørn Jørgensen  ezt írta (időpont: 2022. aug.
10., Sze, 12:21):

> Congratulations :)
>
> tir. 9. aug. 2022 kl. 10:13 skrev Hyukjin Kwon :
>
>> Hi all,
>>
>> The Spark PMC recently added Xinrong Meng as a committer on the project.
>> Xinrong is the major contributor of PySpark especially Pandas API on Spark.
>> She has guided a lot of new contributors enthusiastically. Please join me
>> in welcoming Xinrong!
>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

[Peter Toth]

+1, thanks for the proposal!

416161...@qq.com  ezt írta (időpont: 2022. nov. 17.,
Cs, 6:22):

> +1
>
> --
> Ruifeng Zheng
> ruife...@foxmail.com
>
> 
>
>
>
> -- Original --
> *From:* "Wenchen Fan" ;
> *Date:* Thu, Nov 17, 2022 10:27 AM
> *To:* "Ye Zhou";
> *Cc:* "Jungtaek Lim";"Chris Nauroth"<
> cnaur...@apache.org>;"Maciej";"dev"<
> dev@spark.apache.org>;
> *Subject:* Re: [VOTE][SPIP] Better Spark UI scalability and Driver
> stability for large applications
>
> +1, I'm looking forward to it!
>
> On Thu, Nov 17, 2022 at 9:44 AM Ye Zhou  wrote:
>
>> +1 (non-binding)
>> Thanks for proposing this improvement to SHS, it resolves the main
>> performance issue within SHS.
>>
>> On Wed, Nov 16, 2022 at 1:15 PM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> +1
>>>
>>> Nice to see the chance for driver to reduce resource usage and increase
>>> stability, especially the fact that the driver is SPOF. It's even promising
>>> to have a future plan to pre-bake the kvstore for SHS from the driver.
>>>
>>> Thanks for driving the effort, Gengliang!
>>>
>>> On Thu, Nov 17, 2022 at 5:32 AM Chris Nauroth 
>>> wrote:
>>>
 +1 (non-binding)

 Gengliang, thank you for the SPIP.

 Chris Nauroth


 On Wed, Nov 16, 2022 at 4:27 AM Maciej  wrote:

> +1
>
> On 11/16/22 13:19, Yuming Wang wrote:
> > +1, non-binding
> >
> > On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF)  > > wrote:
> >
> > +1, non-binding
> >
> > __ __
> >
> > Yang Jie
> >
> > __ __
> >
> > *发件人**: *Mridul Muralidharan  > >
> > *日期**: *2022年11月16日星期三17:35
> > *收件人**: *Kent Yao mailto:y...@apache.org>>
> > *抄送**: *Gengliang Wang  > >, dev  > >
> > *主题**: *Re: [VOTE][SPIP] Better Spark UI scalability and Driver
> > stability for large applications
> >
> > __ __
> >
> > __ __
> >
> > +1
> >
> > __ __
> >
> > Would be great to see history server performance improvements and
> > lower resource utilization at driver !
> >
> > __ __
> >
> > Regards,
> >
> > Mridul 
> >
> > __ __
> >
> > On Wed, Nov 16, 2022 at 2:38 AM Kent Yao  > > wrote:
> >
> > +1, non-binding
> >
> > Gengliang Wang mailto:ltn...@gmail.com>>
> 于
> > 2022年11月16日周三16:36写道：
> > >
> > > Hi all,
> > >
> > > I’d like to start a vote for SPIP: "Better Spark UI
> scalability and Driver stability for large applications"
> > >
> > > The goal of the SPIP is to improve the Driver's stability
> by supporting storing Spark's UI data on RocksDB. Furthermore, to fasten
> the read and write operations on RocksDB, it introduces a new Protobuf
> serializer.
> > >
> > > Please also refer to the following:
> > >
> > > Previous discussion in the dev mailing list: [DISCUSS]
> SPIP: Better Spark UI scalability and Driver stability for large
> applications
> > > Design Doc: Better Spark UI scalability and Driver
> stability for large applications
> > > JIRA: SPARK-41053
> > >
> > >
> > > Please vote on the SPIP for the next 72 hours:
> > >
> > > [ ] +1: Accept the proposal as an official SPIP
> > > [ ] +0
> > > [ ] -1: I don’t think this is a good idea because …
> > >
> > > Kind Regards,
> > > Gengliang
> >
> >
>  -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > 
> >
>
> --
> Best regards,
> Maciej Szymkiewicz
>
> Web: https://zero323.net
> PGP: A30CEF0C31A501EC
>
>
>>
>> --
>>
>> *Zhou, Ye  **周晔*
>>
>

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

[Peter Toth]

Hi Xinrong,

I've opened a PR to fix a regression from 3.3 to 3.4:
https://github.com/apache/spark/pull/40364
Please wait with the RC4 cut if possible.

Thanks,
Peter

Xinrong Meng  ezt írta (időpont: 2023. márc. 10.,
P, 0:07):

> Thank you Hyukjin! :)
>
> I would prefer to cut v3.4.0-rc4 now if there are no objections.
>
> On Fri, Mar 10, 2023 at 7:01 AM Hyukjin Kwon  wrote:
>
>> BTW doing another RC isn't a very big deal (compared to what I did before
>> :-) ) since it's not a canonical release yet.
>>
>> On Fri, Mar 10, 2023 at 7:58 AM Hyukjin Kwon  wrote:
>>
>>> I guess directly tagging is fine too I guess.
>>> I don't mind cutting the RC4 right away either if that's what you prefer.
>>>
>>> On Fri, Mar 10, 2023 at 7:06 AM Xinrong Meng 
>>> wrote:
>>>
 Hi All,

 Thank you all for catching that. Unfortunately, the release script
 failed to push the release tag v3.4.0-rc3 to branch-3.4. Sorry about the
 issue.

 Shall we cut v3.4.0-rc4 immediately or wait until March 14th?

 On Fri, Mar 10, 2023 at 5:34 AM Sean Owen  wrote:

> If the issue were just tags, then you can simply delete the tag and
> re-tag the right commit. That doesn't change a commit log.
> But is the issue that the relevant commits aren't in branch-3.4? Like
> I don't see the usual release commits in
> https://github.com/apache/spark/commits/branch-3.4
> Yeah OK that needs a re-do.
>
> We can still test this release.
> It works for me, except that I still get the weird
> infinite-compile-loop issue that doesn't seem to be related to Spark. The
> Spark Connect parts seem to work.
>
> On Thu, Mar 9, 2023 at 3:25 PM Dongjoon Hyun 
> wrote:
>
>> No~ We cannot in the AS-IS commit log status because it's screwed
>> already as Emil wrote.
>> Did you check the branch-3.2 commit log, Sean?
>>
>> Dongjoon.
>>
>>
>> On Thu, Mar 9, 2023 at 11:42 AM Sean Owen  wrote:
>>
>>> We can just push the tags onto the branches as needed right? No need
>>> to roll a new release
>>>
>>> On Thu, Mar 9, 2023, 1:36 PM Dongjoon Hyun 
>>> wrote:
>>>
 Yes, I also confirmed that the v3.4.0-rc3 tag is invalid.

 I guess we need RC4.

 Dongjoon.

 On Thu, Mar 9, 2023 at 7:13 AM Emil Ejbyfeldt
  wrote:

> It might being caused by the v3.4.0-rc3 tag not being part of the
> 3.4
> branch branch-3.4:
>
> $ git log --pretty='format:%d %h' --graph origin/branch-3.4
> v3.4.0-rc3
> | head -n 10
> *  (HEAD, origin/branch-3.4) e38e619946
> *  f3e69a1fe2
> *  74cf1a32b0
> *  0191a5bde0
> *  afced91348
> | *  (tag: v3.4.0-rc3) b9be9ce15a
> |/
> *  006e838ede
> *  fc29b07a31
> *  8655dfe66d
>
>
> Best,
> Emil
>
> On 09/03/2023 15:50, yangjie01 wrote:
> > HI, all
> >
> > I can't git check out the tag of v3.4.0-rc3. At the same time,
> there is
> > the following information on the Github page.
> >
> > Does anyone else have the same problem?
> >
> > Yang Jie
> >
> > *发件人**: *Xinrong Meng 
> > *日期**: *2023年3月9日星期四20:05
> > *收件人**: *dev 
> > *主题**: *[VOTE] Release Apache Spark 3.4.0 (RC3)
> >
> > Please vote on releasing the following candidate(RC3) as Apache
> Spark
> > version 3.4.0.
> >
> > The vote is open until 11:59pm Pacific time *March 14th* and
> passes if a
> > majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 3.4.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> http://spark.apache.org/
> > <
> https://mailshield.baidu.com/check?q=eJcUboQ1HRRomPZKEwRzpl69wA8DbI%2fNIiRNsQ%3d%3d
> >
> >
> > The tag to be voted on is *v3.4.0-rc3* (commit
> > b9be9ce15a82b18cca080ee365d308c0820a29a9):
> > https://github.com/apache/spark/tree/v3.4.0-rc3
> > <
> https://mailshield.baidu.com/check?q=ScnsHLDD3dexVfW9cjs3GovMbG2LLAZqBLq9cA8V%2fTOpCQ1LdeNWoD0%2fy7eVo%2b3de8Rk%2bQ%3d%3d
> >
> >
> > The release files, including signatures, digests, etc. can be
> found at:
> > https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc3-bin/
> > <
> https://mailshield.baidu.com/check?q=U%2fLs35p0l%2bUUTclb%2blAPSYb%2bALxMfer1Jc%2b3i965Bjh2CxHpG45RFLW0NqSwMx00Ci3MRMz%2b7mTmcKUIa27Pww%3d%3d
> >
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

[Peter Toth]

+1

Jia Fan  ezt írta (időpont: 2023. ápr. 11., K, 9:09):

> +1
>
> Wenchen Fan  于2023年4月11日周二 14:32写道：
>
>> +1
>>
>> On Tue, Apr 11, 2023 at 9:57 AM Yuming Wang  wrote:
>>
>>> +1.
>>>
>>> On Tue, Apr 11, 2023 at 9:14 AM Yikun Jiang  wrote:
>>>
 +1 (non-binding)

 Also ran the docker image related test (signatures/standalone/k8s) with
 rc7: https://github.com/apache/spark-docker/pull/32

 Regards,
 Yikun


 On Tue, Apr 11, 2023 at 4:44 AM Jacek Laskowski 
 wrote:

> +1
>
> * Built fine with Scala 2.13
> and -Pkubernetes,hadoop-cloud,hive,hive-thriftserver,scala-2.13,volcano
> * Ran some demos on Java 17
> * Mac mini / Apple M2 Pro / Ventura 13.3.1
>
> Pozdrawiam,
> Jacek Laskowski
> 
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>
>
> On Sat, Apr 8, 2023 at 1:30 AM Xinrong Meng 
> wrote:
>
>> Please vote on releasing the following candidate(RC7) as Apache Spark
>> version 3.4.0.
>>
>> The vote is open until 11:59pm Pacific time *April 12th* and passes
>> if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.4.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.4.0-rc7 (commit
>> 87a5442f7ed96b11051d8a9333476d080054e5a0):
>> https://github.com/apache/spark/tree/v3.4.0-rc7
>>
>> The release files, including signatures, digests, etc. can be found
>> at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc7-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1441
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc7-docs/
>>
>> The list of bug fixes going into 3.4.0 can be found at the following
>> URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12351465
>>
>> This release is using the release script of the tag v3.4.0-rc7.
>>
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with an out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.4.0?
>> ===
>> The current list of open tickets targeted at 3.4.0 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.4.0
>>
>> Committers should look at those and triage. Extremely important bug
>> fixes, documentation, and API tweaks that impact compatibility should
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ==
>> But my bug isn't fixed?
>> ==
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>> Thanks,
>> Xinrong Meng
>>
>

Re: [VOTE] Release Spark 3.4.1 (RC1)

[Peter Toth]

+1

Ruifeng Zheng  ezt írta (időpont: 2023. jún. 21., Sze,
9:43):

> +1
>
> On Wed, Jun 21, 2023 at 2:26 PM huaxin gao  wrote:
>
>> +1
>>
>> On Tue, Jun 20, 2023 at 11:21 PM Hyukjin Kwon 
>> wrote:
>>
>>> +1
>>>
>>> On Wed, 21 Jun 2023 at 14:23, yangjie01  wrote:
>>>
 +1


 在 2023/6/21 13:20，“L. C. Hsieh”>>> vii...@gmail.com>> 写入:


 +1


 On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun >>> > wrote:
 >
 > +1
 >
 > Dongjoon
 >
 > On 2023/06/20 02:51:32 Jia Fan wrote:
 > > +1
 > >
 > > Dongjoon Hyun mailto:dongj...@apache.org>>
 于2023年6月20日周二 10:41写道：
 > >
 > > > Please vote on releasing the following candidate as Apache Spark
 version
 > > > 3.4.1.
 > > >
 > > > The vote is open until June 23rd 1AM (PST) and passes if a
 majority +1 PMC
 > > > votes are cast, with a minimum of 3 +1 votes.
 > > >
 > > > [ ] +1 Release this package as Apache Spark 3.4.1
 > > > [ ] -1 Do not release this package because ...
 > > >
 > > > To learn more about Apache Spark, please see
 https://spark.apache.org/ 
 > > >
 > > > The tag to be voted on is v3.4.1-rc1 (commit
 > > > 6b1ff22dde1ead51cbf370be6e48a802daae58b6)
 > > > https://github.com/apache/spark/tree/v3.4.1-rc1 <
 https://github.com/apache/spark/tree/v3.4.1-rc1>
 > > >
 > > > The release files, including signatures, digests, etc. can be
 found at:
 > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/ <
 https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-bin/>
 > > >
 > > > Signatures used for Spark RCs can be found in this file:
 > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
 https://dist.apache.org/repos/dist/dev/spark/KEYS>
 > > >
 > > > The staging repository for this release can be found at:
 > > >
 https://repository.apache.org/content/repositories/orgapachespark-1443/
 <
 https://repository.apache.org/content/repositories/orgapachespark-1443/
 >
 > > >
 > > > The documentation corresponding to this release can be found at:
 > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/ <
 https://dist.apache.org/repos/dist/dev/spark/v3.4.1-rc1-docs/>
 > > >
 > > > The list of bug fixes going into 3.4.1 can be found at the
 following URL:
 > > > https://issues.apache.org/jira/projects/SPARK/versions/12352874 <
 https://issues.apache.org/jira/projects/SPARK/versions/12352874>
 > > >
 > > > This release is using the release script of the tag v3.4.1-rc1.
 > > >
 > > > FAQ
 > > >
 > > > =
 > > > How can I help test this release?
 > > > =
 > > >
 > > > If you are a Spark user, you can help us test this release by
 taking
 > > > an existing Spark workload and running on this release candidate,
 then
 > > > reporting any regressions.
 > > >
 > > > If you're working in PySpark you can set up a virtual env and
 install
 > > > the current RC and see if anything important breaks, in the
 Java/Scala
 > > > you can add the staging repository to your projects resolvers and
 test
 > > > with the RC (make sure to clean up the artifact cache
 before/after so
 > > > you don't end up building with a out of date RC going forward).
 > > >
 > > > ===
 > > > What should happen to JIRA tickets still targeting 3.4.1?
 > > > ===
 > > >
 > > > The current list of open tickets targeted at 3.4.1 can be found
 at:
 > > > https://issues.apache.org/jira/projects/SPARK <
 https://issues.apache.org/jira/projects/SPARK> and search for "Target
 > > > Version/s" = 3.4.1
 > > >
 > > > Committers should look at those and triage. Extremely important
 bug
 > > > fixes, documentation, and API tweaks that impact compatibility
 should
 > > > be worked on immediately. Everything else please retarget to an
 > > > appropriate release.
 > > >
 > > > ==
 > > > But my bug isn't fixed?
 > > > ==
 > > >
 > > > In order to make timely releases, we will typically not hold the
 > > > release unless the bug in question is a regression from the
 previous
 > > > release. That being said, if there is something which is a
 regression
 > > > that has not been correctly targeted please ping me or a
 committer to
 > > > help target the issue.
 > > >
 > >
 >
 > -
 > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> dev-unsubscr...@spark.apache.org>
 >


 -

Re: Welcome two new Apache Spark committers

[Peter Toth]

Thank you all!

On Mon, Aug 7, 2023, 19:24 L. C. Hsieh  wrote:

> Congratulations!
>
> On Mon, Aug 7, 2023 at 9:44 AM huaxin gao  wrote:
> >
> > Congratulations! Peter and Xiduo!
> >
> > On Mon, Aug 7, 2023 at 9:40 AM Dongjoon Hyun 
> wrote:
> >>
> >> Congratulations, Peter and Xiduo. :)
> >>
> >> Dongjoon.
> >>
> >> On Sun, Aug 6, 2023 at 10:08 PM XiDuo You 
> wrote:
> >>>
> >>> Thank you all !
> >>>
> >>> Jia Fan  于2023年8月7日周一 11:31写道：
> >>> >
> >>> > Congratulations!
> >>> > 
> >>> >
> >>> > Jia Fan
> >>> >
> >>> >
> >>> > 2023年8月7日 11:28，Ye Xianjin  写道：
> >>> >
> >>> > Congratulations!
> >>> >
> >>> > Sent from my iPhone
> >>> >
> >>> > On Aug 7, 2023, at 11:16 AM, Yuming Wang  wrote:
> >>> >
> >>> > 
> >>> >
> >>> > Congratulations!
> >>> >
> >>> > On Mon, Aug 7, 2023 at 11:11 AM Kent Yao  wrote:
> >>> >>
> >>> >> Congrats! Peter and Xiduo!
> >>> >>
> >>> >> Cheng Pan  于2023年8月7日周一 11:01写道：
> >>> >> >
> >>> >> > Congratulations! Peter and Xiduo!
> >>> >> >
> >>> >> > Thanks,
> >>> >> > Cheng Pan
> >>> >> >
> >>> >> >
> >>> >> > > On Aug 7, 2023, at 10:58, Gengliang Wang 
> wrote:
> >>> >> > >
> >>> >> > > Congratulations! Peter and Xiduo!
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> -
> >>> >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>> >> >
> >>> >>
> >>> >>
> -
> >>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>> >>
> >>> >
> >>>
> >>> -
> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

[Peter Toth]

Hi Yuanjian,

This is a correctness issue that we should probably fix in 3.5:
https://issues.apache.org/jira/browse/SPARK-44871 /
https://github.com/apache/spark/pull/42559

Cheers,
Peter

yangjie01  ezt írta (időpont: 2023. aug. 12.,
Szo, 15:38):

> Hi, Yuanjian,
>
>
>
> Maybe there is another issue that needs to be fixed
>
>
>
> -[SPARK-44784] 
> Failure in testing `SparkSessionE2ESuite` using Maven
>
>
>
> Maven daily tests are still failing:
> https://github.com/apache/spark/actions/runs/5832898984/job/15819181762
>
>
>
> I think we should address this issue before the release of Apache Spark
> 3.5.0.
>
>
>
> Jie Yang
>
>
>
> *发件人**: *Yuanjian Li 
> *日期**: *2023年8月12日 星期六 15:20
> *收件人**: *Yuming Wang 
> *抄送**: *yangjie01 , Sean Owen <
> sro...@gmail.com>, Spark dev list 
> *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC1)
>
>
>
> Thanks for all updates!
>
> The vote has failed. Here is the status of known blockers:
>
>- [SPARK-44719]
>
> 
>  NoClassDefFoundError
>when using Hive UDF - *Resolved*
>- [SPARK-44653
>
> ]
>  non-trivial
>DataFrame unions should not break caching - *Resolved*
>- [SPARK-43646
>
> ]
>Test failure of Connect: from_protobuf_messageClassName - *WIP*
>
> I'll cut RC2 once all blockers are resolved.
>
>
>
>
>
> Yuming Wang  于2023年8月8日周二 05:29写道：
>
> -1. I found a NoClassDefFoundError bug:
> https://issues.apache.org/jira/browse/SPARK-44719
> 
> .
>
>
>
> On Mon, Aug 7, 2023 at 11:24 AM yangjie01 
> wrote:
>
>
>
> I submitted a PR last week to try and solve this issue:
> https://github.com/apache/spark/pull/42236
> 
> .
>
>
>
> *发件人**: *Sean Owen 
> *日期**: *2023年8月7日 星期一 11:05
> *收件人**: *Yuanjian Li 
> *抄送**: *Spark dev list 
> *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC1)
>
>
> --
>
> *【外部邮件】信息安全要牢记，账号密码不传递！*
> --
>
>
>
> Let's keep testing 3.5.0 of course while that change is going in. (See
> https://github.com/apache/spark/pull/42364#issuecomment-1666878287
> 
> )
>
>
>
> Otherwise testing is pretty much as usual, except I get this test failure
> in Connect, which is new. Anyone else? this is Java 8, Scala 2.13, Debian
> 12.
>
>
>
> - from_protobuf_messageClassName_options *** FAILED ***
>   org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_PROTOBUF_CLASS]
> Could not load Protobuf class with name
> org.apache.spark.connect.proto.StorageLevel.
> org.apache.spark.connect.proto.StorageLevel does not extend shaded Protobuf
> Message class org.sparkproject.spark_protobuf.protobuf.Message. The jar
> with Protobuf classes needs to be shaded (com.google.protobuf.* -->
> org.sparkproject.spark_protobuf.protobuf.*).
>   at
> org.apache.spark.sql.errors.QueryCompilationErrors$.protobufClassLoadError(QueryCompilationErrors.scala:3554)
>   at
> org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptorFromJavaClass(ProtobufUtils.scala:198)
>   at
> org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:156)
>   at
> org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor$lzycompute(ProtobufDataToCatalyst.scala:58)
>   at
> org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor(ProtobufDataToCatalyst.scala:57)
>   at
> org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType$lzycompute(ProtobufDataToCatalyst.scala:43)
>   at
> org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType(ProtobufDataToCatalyst.scala:42)
>   at
> org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:194)
>   at
> org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:73)
>   at scala.collection.immutable.List.map(List.scala:246)
>
>
>
> On Sat, Aug 5, 2023 at 5:42 PM Sean Owen  wrote:
>
> I'm still testing other combinations, but it looks like tests fail on Java
> 17 after building with Java 8, which should be a normal supported
> configuration.
>
> This is described at https://github.com/apache/spark/pull/41943
> 
> and looks like it is resolved by moving back to Scala 2.1

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

[Peter Toth]

Hi Yuanjian,

Sorry, -1 from me. Let's not introduce this bugs in 3.5:
https://issues.apache.org/jira/browse/SPARK-45109 /
https://github.com/apache/spark/pull/42863

Best,
Peter

Yuanjian Li  ezt írta (időpont: 2023. szept. 10.,
V, 10:39):

> Yes, SPARK-44805 has been included. For the commits from RC4 to RC5,
> please refer to https://github.com/apache/spark/commits/v3.5.0-rc5.
>
> Mich Talebzadeh  于2023年9月9日周六 08:09写道：
>
>> Apologies that should read ... release 3.5.0 (RC4) plus ..
>>
>> Mich Talebzadeh,
>> Distinguished Technologist, Solutions Architect & Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sat, 9 Sept 2023 at 15:58, Mich Talebzadeh 
>> wrote:
>>
>>> Hi,
>>>
>>> Can you please confirm that this cut is release 3.4.0 plus the resolved
>>> Jira  https://issues.apache.org/jira/browse/SPARK-44805 which was
>>> already fixed yesterday?
>>>
>>> Nothing else I believe?
>>>
>>> Thanks
>>>
>>> Mich
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sat, 9 Sept 2023 at 15:42, Yuanjian Li 
>>> wrote:
>>>
 Please vote on releasing the following candidate(RC5) as Apache Spark
 version 3.5.0.

 The vote is open until 11:59pm Pacific time Sep 11th and passes if a
 majority +1 PMC votes are cast, with a minimum of 3 +1 votes.

 [ ] +1 Release this package as Apache Spark 3.5.0

 [ ] -1 Do not release this package because ...

 To learn more about Apache Spark, please see http://spark.apache.org/

 The tag to be voted on is v3.5.0-rc5 (commit
 ce5ddad990373636e94071e7cef2f31021add07b):

 https://github.com/apache/spark/tree/v3.5.0-rc5

 The release files, including signatures, digests, etc. can be found at:

 https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc5-bin/

 Signatures used for Spark RCs can be found in this file:

 https://dist.apache.org/repos/dist/dev/spark/KEYS

 The staging repository for this release can be found at:

 https://repository.apache.org/content/repositories/orgapachespark-1449

 The documentation corresponding to this release can be found at:

 https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc5-docs/

 The list of bug fixes going into 3.5.0 can be found at the following
 URL:

 https://issues.apache.org/jira/projects/SPARK/versions/12352848

 This release is using the release script of the tag v3.5.0-rc5.


 FAQ

 =

 How can I help test this release?

 =

 If you are a Spark user, you can help us test this release by taking

 an existing Spark workload and running on this release candidate, then

 reporting any regressions.

 If you're working in PySpark you can set up a virtual env and install

 the current RC and see if anything important breaks, in the Java/Scala

 you can add the staging repository to your projects resolvers and test

 with the RC (make sure to clean up the artifact cache before/after so

 you don't end up building with an out of date RC going forward).

 ===

 What should happen to JIRA tickets still targeting 3.5.0?

 ===

 The current list of open tickets targeted at 3.5.0 can be found at:

 https://issues.apache.org/jira/projects/SPARK and search for "Target
 Version/s" = 3.5.0

 Committers should look at those and triage. Extremely important bug

 fixes, documentation, and API tweaks that impact compatibility should

 be worked on immediately. Everything else please retarget to an

 appropriate release.

 ==

 But my bug isn't fixed?

 ==

 In order to make timely releases, we will typically not hold the

 release unless the bug in qu

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

[Peter Toth]

Thanks Yuanjian. Please disregard my -1 then.

Yuanjian Li  ezt írta (időpont: 2023. szept. 11.,
H, 18:36):

> @Peter Toth  I've looked into the details of this
> issue, and it appears that it's neither a regression in version 3.5.0 nor a
> correctness issue. It's a bug related to a new feature. I think we can fix
> this in 3.5.1 and list it as a known issue of the Scala client of Spark
> Connect in 3.5.0.
>
> Mridul Muralidharan  于2023年9月10日周日 04:12写道：
>
>>
>> +1
>>
>> Signatures, digests, etc check out fine.
>> Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes
>>
>> Regards,
>> Mridul
>>
>> On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li 
>> wrote:
>>
>>> Please vote on releasing the following candidate(RC5) as Apache Spark
>>> version 3.5.0.
>>>
>>> The vote is open until 11:59pm Pacific time Sep 11th and passes if a
>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>
>>> [ ] +1 Release this package as Apache Spark 3.5.0
>>>
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>
>>> The tag to be voted on is v3.5.0-rc5 (commit
>>> ce5ddad990373636e94071e7cef2f31021add07b):
>>>
>>> https://github.com/apache/spark/tree/v3.5.0-rc5
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>>
>>> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc5-bin/
>>>
>>> Signatures used for Spark RCs can be found in this file:
>>>
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>
>>> The staging repository for this release can be found at:
>>>
>>> https://repository.apache.org/content/repositories/orgapachespark-1449
>>>
>>> The documentation corresponding to this release can be found at:
>>>
>>> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc5-docs/
>>>
>>> The list of bug fixes going into 3.5.0 can be found at the following URL:
>>>
>>> https://issues.apache.org/jira/projects/SPARK/versions/12352848
>>>
>>> This release is using the release script of the tag v3.5.0-rc5.
>>>
>>>
>>> FAQ
>>>
>>> =
>>>
>>> How can I help test this release?
>>>
>>> =
>>>
>>> If you are a Spark user, you can help us test this release by taking
>>>
>>> an existing Spark workload and running on this release candidate, then
>>>
>>> reporting any regressions.
>>>
>>> If you're working in PySpark you can set up a virtual env and install
>>>
>>> the current RC and see if anything important breaks, in the Java/Scala
>>>
>>> you can add the staging repository to your projects resolvers and test
>>>
>>> with the RC (make sure to clean up the artifact cache before/after so
>>>
>>> you don't end up building with an out of date RC going forward).
>>>
>>> ===
>>>
>>> What should happen to JIRA tickets still targeting 3.5.0?
>>>
>>> ===
>>>
>>> The current list of open tickets targeted at 3.5.0 can be found at:
>>>
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.5.0
>>>
>>> Committers should look at those and triage. Extremely important bug
>>>
>>> fixes, documentation, and API tweaks that impact compatibility should
>>>
>>> be worked on immediately. Everything else please retarget to an
>>>
>>> appropriate release.
>>>
>>> ==
>>>
>>> But my bug isn't fixed?
>>>
>>> ==
>>>
>>> In order to make timely releases, we will typically not hold the
>>>
>>> release unless the bug in question is a regression from the previous
>>>
>>> release. That being said, if there is something which is a regression
>>>
>>> that has not been correctly targeted please ping me or a committer to
>>>
>>> help target the issue.
>>>
>>> Thanks,
>>>
>>> Yuanjian Li
>>>
>>

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

[Peter Toth]

+1

On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen 
wrote:

> +1
>
> fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu :
>
>> just curious what happened on google’s spark operator?
>>
>> On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko  wrote:
>>
>>> +1
>>>
>>> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue  wrote:
>>>
 +1

 On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala  wrote:

> +1 for creating an official Kubernetes operator for Apache Spark
>
> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao 
> wrote:
>
>> +1
>>
>
>> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai  wrote:
>>
>>> +1
>>>
>>> To be completely transparent, I am employed in the same department
>>> as Zhou at Apple.
>>>
>>> I support this proposal, provided that we witness community adoption
>>> following the release of the Flink Kubernetes operator, streamlining 
>>> Flink
>>> deployment on Kubernetes.
>>>
>>> A well-maintained official Spark Kubernetes operator is essential
>>> for our Spark community as well.
>>>
>>> DB Tsai  |  https://www.dbtsai.com/
>>> 
>>>  |  PGP 42E5B25A8F7A82C1
>>>
>>> On Nov 9, 2023, at 12:05 PM, Zhou Jiang 
>>> wrote:
>>>
>>> Hi Spark community,
>>> I'm reaching out to initiate a conversation about the possibility of
>>> developing a Java-based Kubernetes operator for Apache Spark. Following 
>>> the
>>> operator pattern (
>>> https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
>>> ),
>>> Spark users may manage applications and related components seamlessly 
>>> using
>>> native tools like kubectl. The primary goal is to simplify the Spark 
>>> user
>>> experience on Kubernetes, minimizing the learning curve and operational
>>> complexities and therefore enable users to focus on the Spark 
>>> application
>>> development.
>>> Although there are several open-source Spark on Kubernetes operators
>>> available, none of them are officially integrated into the Apache Spark
>>> project. As a result, these operators may lack active support and
>>> development for new features. Within this proposal, our aim is to 
>>> introduce
>>> a Java-based Spark operator as an integral component of the Apache Spark
>>> project. This solution has been employed internally at Apple for 
>>> multiple
>>> years, operating millions of executors in real production environments. 
>>> The
>>> use of Java in this solution is intended to accommodate a wider user and
>>> contributor audience, especially those who are familiar with Scala.
>>> Ideally, this operator should have its dedicated repository, similar
>>> to Spark Connect Golang or Spark Docker, allowing it to maintain a loose
>>> connection with the Spark release cycle. This model is also followed by 
>>> the
>>> Apache Flink Kubernetes operator.
>>> We believe that this project holds the potential to evolve into a
>>> thriving community project over the long run. A comparison can be drawn
>>> with the Flink Kubernetes Operator: Apple has open-sourced internal 
>>> Flink
>>> Kubernetes operator, making it a part of the Apache Flink project (
>>> https://github.com/apache/flink-kubernetes-operator
>>> ).
>>> This move has gained wide industry adoption and contributions from the
>>> community. In a mere year, the Flink operator has garnered more than 600
>>> stars and has attracted contributions from over 80 contributors. This
>>> showcases the level of community interest and collaborative momentum 
>>> that
>>> can be achieved in similar scenarios.
>>> More de

Re: [ANNOUNCE] Apache Spark 3.5.1 released

[Peter Toth]

Congratulations and thanks Jungtaek for driving this!

Xinrong Meng  ezt írta (időpont: 2024. márc. 1.,
P, 5:24):

> Congratulations!
>
> Thanks,
> Xinrong
>
> On Thu, Feb 29, 2024 at 11:16 AM Dongjoon Hyun 
> wrote:
>
>> Congratulations!
>>
>> Bests,
>> Dongjoon.
>>
>> On Wed, Feb 28, 2024 at 11:43 AM beliefer  wrote:
>>
>>> Congratulations!
>>>
>>>
>>>
>>> At 2024-02-28 17:43:25, "Jungtaek Lim" 
>>> wrote:
>>>
>>> Hi everyone,
>>>
>>> We are happy to announce the availability of Spark 3.5.1!
>>>
>>> Spark 3.5.1 is a maintenance release containing stability fixes. This
>>> release is based on the branch-3.5 maintenance branch of Spark. We
>>> strongly
>>> recommend all 3.5 users to upgrade to this stable release.
>>>
>>> To download Spark 3.5.1, head over to the download page:
>>> https://spark.apache.org/downloads.html
>>>
>>> To view the release notes:
>>> https://spark.apache.org/releases/spark-release-3-5-1.html
>>>
>>> We would like to acknowledge all community members for contributing to
>>> this
>>> release. This release would not have been possible without you.
>>>
>>> Jungtaek Lim
>>>
>>> ps. Yikun is helping us through releasing the official docker image for
>>> Spark 3.5.1 (Thanks Yikun!) It may take some time to be generally available.
>>>
>>>

Re: [VOTE] Release Spark 3.1.1 (RC1)

[Peter Toth]

Hey,

Sorry for chiming in a bit late, but I would like to suggest my PR (
https://github.com/apache/spark/pull/28885) for review and inclusion into
3.1.1.

Currently, invalid reuse reference nodes appear in many queries, causing
performance issues and incorrect explain plans. Now that
https://github.com/apache/spark/pull/31243 got merged these invalid
references can be easily found in many of our golden files on master:
https://github.com/apache/spark/pull/28885#issuecomment-767530441.
But the issue isn't master (3.2) specific, actually it has been there since
3.0 when Dynamic Partition Pruning was added.
So it is not a regression from 3.0 to 3.1.1, but in some cases (like TPCDS
q23b) it is causing performance regression from 2.4 to 3.x.

Thanks,
Peter

On Tue, Jan 26, 2021 at 6:30 AM Hyukjin Kwon  wrote:

> Guys, I plan to make an RC as soon as we have no visible issues. I have
> merged a few correctness issues. There look:
> - https://github.com/apache/spark/pull/31319 waiting for a review (I will
> do it too soon).
> - https://github.com/apache/spark/pull/31336
> - I know Max's investigating the perf regression one which hopefully will
> be fixed soon.
>
> Are there any more blockers or correctness issues? Please ping me or say
> it out here.
> I would like to avoid making an RC when there are clearly some issues to
> be fixed.
> If you're investigating something suspicious, that's fine too. It's better
> to make sure we're safe instead of rushing an RC without finishing the
> investigation.
>
> Thanks all.
>
>
> 2021년 1월 22일 (금) 오후 6:19, Hyukjin Kwon 님이 작성:
>
>> Sure, thanks guys. I'll start another RC after the fixes. Looks like
>> we're almost there.
>>
>> On Fri, 22 Jan 2021, 17:47 Wenchen Fan,  wrote:
>>
>>> BTW, there is a correctness bug being fixed at
>>> https://github.com/apache/spark/pull/30788 . It's not a regression, but
>>> the fix is very simple and it would be better to start the next RC after
>>> merging that fix.
>>>
>>> On Fri, Jan 22, 2021 at 3:54 PM Maxim Gekk 
>>> wrote:
>>>
 Also I am investigating a performance regression in some TPC-DS queries
 (q88 for instance) that is caused by a recent commit in 3.1, highly likely
 in the period from 19th November, 2020 to 18th December, 2020.

 Maxim Gekk

 Software Engineer

 Databricks, Inc.


 On Fri, Jan 22, 2021 at 10:45 AM Wenchen Fan 
 wrote:

> -1 as I just found a regression in 3.1. A self-join query works well
> in 3.0 but fails in 3.1. It's being fixed at
> https://github.com/apache/spark/pull/31287
>
> On Fri, Jan 22, 2021 at 4:34 AM Tom Graves
>  wrote:
>
>> +1
>>
>> built from tarball, verified sha and regular CI and tests all pass.
>>
>> Tom
>>
>> On Monday, January 18, 2021, 06:06:42 AM CST, Hyukjin Kwon <
>> gurwls...@gmail.com> wrote:
>>
>>
>> Please vote on releasing the following candidate as Apache Spark
>> version 3.1.1.
>>
>> The vote is open until January 22nd 4PM PST and passes if a majority
>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.1.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.1.1-rc1 (commit
>> 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):
>> https://github.com/apache/spark/tree/v3.1.1-rc1
>>
>> The release files, including signatures, digests, etc. can be found
>> at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1364
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-docs/
>>
>> The list of bug fixes going into 3.1.1 can be found at the following
>> URL:
>> https://s.apache.org/41kf2
>>
>> This release is using the release script of the tag v3.1.1-rc1.
>>
>> FAQ
>>
>> ===
>> What happened to 3.1.0?
>> ===
>>
>> There was a technical issue during Apache Spark 3.1.0 preparation,
>> and it was discussed and decided to skip 3.1.0.
>> Please see
>> https://spark.apache.org/news/next-official-release-spark-3.1.1.html
>> for more details.
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're workin

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

[Peter Toth]

+1

Wenchen Fan  ezt írta (időpont: 2024. ápr. 15., H,
9:08):

> +1
>
> On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun  wrote:
>
>> I'll start from my +1.
>>
>> Dongjoon.
>>
>> On 2024/04/13 22:22:05 Dongjoon Hyun wrote:
>> > Please vote on SPARK-4 to use ANSI SQL mode by default.
>> > The technical scope is defined in the following PR which is
>> > one line of code change and one line of migration guide.
>> >
>> > - DISCUSSION:
>> > https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz
>> > - JIRA: https://issues.apache.org/jira/browse/SPARK-4
>> > - PR: https://github.com/apache/spark/pull/46013
>> >
>> > The vote is open until April 17th 1AM (PST) and passes
>> > if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Use ANSI SQL mode by default
>> > [ ] -1 Do not use ANSI SQL mode by default because ...
>> >
>> > Thank you in advance.
>> >
>> > Dongjoon
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

[Peter Toth]

+1

John Zhuge  ezt írta (időpont: 2024. júl. 4., Cs, 5:38):

> +1
>
>
> John Zhuge
>
>
> On Wed, Jul 3, 2024 at 7:41 PM Gengliang Wang  wrote:
>
>> +1
>>
>> On Wed, Jul 3, 2024 at 4:48 PM Reynold Xin 
>> wrote:
>>
>>> +1
>>>
>>> On Wed, Jul 3, 2024 at 4:45 PM L. C. Hsieh  wrote:
>>>
 +1

 On Wed, Jul 3, 2024 at 3:54 PM Dongjoon Hyun 
 wrote:
 >
 > +1
 >
 > Dongjoon
 >
 > On Wed, Jul 3, 2024 at 10:58 Xinrong Meng  wrote:
 >>
 >> +1
 >>
 >> Thank you @Hyukjin Kwon !
 >>
 >> On Wed, Jul 3, 2024 at 8:55 AM bo yang  wrote:
 >>>
 >>> +1 (non-binding)
 >>>
 >>>
 >>> On Tue, Jul 2, 2024 at 11:22 PM Cheng Pan 
 wrote:
 
  +1 (non-binding)
 
  Thanks,
  Cheng Pan
 
 
  On Jul 3, 2024, at 08:59, Hyukjin Kwon 
 wrote:
 
  Hi all,
 
  I’d like to start a vote for moving Spark Connect server to
 builtin package (Client API layer stays external).
 
  Please also refer to:
 
 - Discussion thread:
 https://lists.apache.org/thread/odlx9b552dp8yllhrdlp24pf9m9s4tmx
 - JIRA ticket:
 https://issues.apache.org/jira/browse/SPARK-48763
 
  Please vote on the SPIP for the next 72 hours:
 
  [ ] +1: Accept the proposal
  [ ] +0
  [ ] -1: I don’t think this is a good idea because …
 
  Thank you!
 
 

 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

[Peter Toth]

+1

huaxin gao  ezt írta (időpont: 2024. júl. 24., Sze,
11:14):

> +1
>
> On Tue, Jul 23, 2024 at 9:18 PM XiDuo You  wrote:
>
>> +1 (non-binding)
>>
>> L. C. Hsieh  于2024年7月24日周三 11:40写道：
>> >
>> > +1
>> >
>> > Thanks.
>> >
>> > On Tue, Jul 23, 2024 at 8:35 PM Dongjoon Hyun 
>> wrote:
>> > >
>> > > +1
>> > >
>> > > Dongjoon.
>> > >
>> > > On 2024/07/24 03:28:58 Wenchen Fan wrote:
>> > > > +1
>> > > >
>> > > > On Wed, Jul 24, 2024 at 10:51 AM Kent Yao  wrote:
>> > > >
>> > > > > +1(non-binding), I have checked:
>> > > > >
>> > > > > - Download links are OK
>> > > > > - Signatures, Checksums, and the KEYS file are OK
>> > > > > - LICENSE and NOTICE are present
>> > > > > - No unexpected binary files in source releases
>> > > > > - Successfully built from source
>> > > > >
>> > > > > Thanks,
>> > > > > Kent Yao
>> > > > >
>> > > > > On 2024/07/23 06:55:28 yangjie01 wrote:
>> > > > > > +1, Thanks Kent Yao ~
>> > > > > >
>> > > > > > 在 2024/7/22 17:01，“Kent Yao”> y...@apache.org>>
>> > > > > 写入:
>> > > > > >
>> > > > > >
>> > > > > > Hi dev,
>> > > > > >
>> > > > > >
>> > > > > > Please vote on releasing the following candidate as Apache
>> Spark version
>> > > > > 3.5.2.
>> > > > > >
>> > > > > >
>> > > > > > The vote is open until Jul 25, 09:00:00 AM UTC, and passes if a
>> majority
>> > > > > +1
>> > > > > > PMC votes are cast, with
>> > > > > > a minimum of 3 +1 votes.
>> > > > > >
>> > > > > >
>> > > > > > [ ] +1 Release this package as Apache Spark 3.5.2
>> > > > > > [ ] -1 Do not release this package because ...
>> > > > > >
>> > > > > >
>> > > > > > To learn more about Apache Spark, please see
>> https://spark.apache.org/ <
>> > > > > https://spark.apache.org/>
>> > > > > >
>> > > > > >
>> > > > > > The tag to be voted on is v3.5.2-rc2 (commit
>> > > > > > 6d8f511430881fa7a3203405260da174df424103):
>> > > > > > https://github.com/apache/spark/tree/v3.5.2-rc2 <
>> > > > > https://github.com/apache/spark/tree/v3.5.2-rc2>
>> > > > > >
>> > > > > >
>> > > > > > The release files, including signatures, digests, etc. can be
>> found at:
>> > > > > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-bin/ <
>> > > > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-bin/>
>> > > > > >
>> > > > > >
>> > > > > > Signatures used for Spark RCs can be found in this file:
>> > > > > > https://dist.apache.org/repos/dist/dev/spark/KEYS <
>> > > > > https://dist.apache.org/repos/dist/dev/spark/KEYS>
>> > > > > >
>> > > > > >
>> > > > > > The staging repository for this release can be found at:
>> > > > > >
>> https://repository.apache.org/content/repositories/orgapachespark-1458/
>> > > > > <
>> https://repository.apache.org/content/repositories/orgapachespark-1458/>
>> > > > > >
>> > > > > >
>> > > > > > The documentation corresponding to this release can be found at:
>> > > > > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-docs/ <
>> > > > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-docs/>
>> > > > > >
>> > > > > >
>> > > > > > The list of bug fixes going into 3.5.2 can be found at the
>> following URL:
>> > > > > > https://issues.apache.org/jira/projects/SPARK/versions/12353980
>> <
>> > > > > https://issues.apache.org/jira/projects/SPARK/versions/12353980>
>> > > > > >
>> > > > > >
>> > > > > > FAQ
>> > > > > >
>> > > > > >
>> > > > > > =
>> > > > > > How can I help test this release?
>> > > > > > =
>> > > > > >
>> > > > > >
>> > > > > > If you are a Spark user, you can help us test this release by
>> taking
>> > > > > > an existing Spark workload and running on this release
>> candidate, then
>> > > > > > reporting any regressions.
>> > > > > >
>> > > > > >
>> > > > > > If you're working in PySpark you can set up a virtual env and
>> install
>> > > > > > the current RC via "pip install
>> > > > > >
>> > > > >
>> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-bin/pyspark-3.5.2.tar.gz
>> "
>> > > > > <
>> > > > >
>> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc2-bin/pyspark-3.5.2.tar.gz"
>> > > > > ;>
>> > > > > > and see if anything important breaks.
>> > > > > > In the Java/Scala, you can add the staging repository to your
>> projects
>> > > > > > resolvers and test
>> > > > > > with the RC (make sure to clean up the artifact cache
>> before/after so
>> > > > > > you don't end up building with an out of date RC going forward).
>> > > > > >
>> > > > > >
>> > > > > > ===
>> > > > > > What should happen to JIRA tickets still targeting 3.5.2?
>> > > > > > ===
>> > > > > >
>> > > > > >
>> > > > > > The current list of open tickets targeted at 3.5.2 can be found
>> at:
>> > > > > > https://issues.apache.org/jira/projects/SPARK <
>> > > > > https://issues.apache.org/jira/projects/SPARK> and search for
>> > > > > > "Target Version/s" = 3.5.2
>> > > > > >
>> > > > > >
>> > > > > > Committers should look at those and triage. Extremely importa

Re: [VOTE] Release Spark 3.5.2 (RC4)

[Peter Toth]

+1

XiDuo You  ezt írta (időpont: 2024. júl. 30., K,
7:56):

> +1
>
> Zhou Jiang  于2024年7月30日周二 02:08写道：
> >
> > +1 (non-binding)
> >
> > Zhou JIANG
> >
> >
> >
> > On Mon, Jul 29, 2024 at 11:06 L. C. Hsieh  wrote:
> >>
> >> +1
> >>
> >> On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan 
> wrote:
> >> >
> >> > +1
> >> >
> >> > On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun <
> dongjoon.h...@gmail.com> wrote:
> >> >>
> >> >> +1
> >> >>
> >> >> Thank you, Kent.
> >> >>
> >> >> Dongjoon.
> >> >>
> >> >> On Fri, Jul 26, 2024 at 6:37 AM Kent Yao  wrote:
> >> >>>
> >> >>> Hi dev,
> >> >>>
> >> >>> Please vote on releasing the following candidate as Apache Spark
> version 3.5.2.
> >> >>>
> >> >>> The vote is open until Jul 29, 14:00:00 UTC, and passes if a
> majority +1
> >> >>> PMC votes are cast, with a minimum of 3 +1 votes.
> >> >>>
> >> >>> [ ] +1 Release this package as Apache Spark 3.5.2
> >> >>> [ ] -1 Do not release this package because ...
> >> >>>
> >> >>> To learn more about Apache Spark, please see
> https://spark.apache.org/
> >> >>>
> >> >>> The tag to be voted on is v3.5.2-rc4 (commit
> >> >>> 1edbddfadeb46581134fa477d35399ddc63b7163):
> >> >>> https://github.com/apache/spark/tree/v3.5.2-rc4
> >> >>>
> >> >>> The release files, including signatures, digests, etc. can be found
> at:
> >> >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc4-bin/
> >> >>>
> >> >>> Signatures used for Spark RCs can be found in this file:
> >> >>> https://dist.apache.org/repos/dist/dev/spark/KEYS
> >> >>>
> >> >>> The staging repository for this release can be found at:
> >> >>>
> https://repository.apache.org/content/repositories/orgapachespark-1460/
> >> >>>
> >> >>> The documentation corresponding to this release can be found at:
> >> >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc4-docs/
> >> >>>
> >> >>> The list of bug fixes going into 3.5.2 can be found at the
> following URL:
> >> >>> https://issues.apache.org/jira/projects/SPARK/versions/12353980
> >> >>>
> >> >>> FAQ
> >> >>>
> >> >>> =
> >> >>> How can I help test this release?
> >> >>> =
> >> >>>
> >> >>> If you are a Spark user, you can help us test this release by taking
> >> >>> an existing Spark workload and running on this release candidate,
> then
> >> >>> reporting any regressions.
> >> >>>
> >> >>> If you're working in PySpark you can set up a virtual env and
> install
> >> >>> the current RC via "pip install
> >> >>>
> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc4-bin/pyspark-3.5.2.tar.gz
> "
> >> >>> and see if anything important breaks.
> >> >>> In the Java/Scala, you can add the staging repository to your
> projects
> >> >>> resolvers and test
> >> >>> with the RC (make sure to clean up the artifact cache before/after
> so
> >> >>> you don't end up building with an out of date RC going forward).
> >> >>>
> >> >>> ===
> >> >>> What should happen to JIRA tickets still targeting 3.5.2?
> >> >>> ===
> >> >>>
> >> >>> The current list of open tickets targeted at 3.5.2 can be found at:
> >> >>> https://issues.apache.org/jira/projects/SPARK and search for
> >> >>> "Target Version/s" = 3.5.2
> >> >>>
> >> >>> Committers should look at those and triage. Extremely important bug
> >> >>> fixes, documentation, and API tweaks that impact compatibility
> should
> >> >>> be worked on immediately. Everything else please retarget to an
> >> >>> appropriate release.
> >> >>>
> >> >>> ==
> >> >>> But my bug isn't fixed?
> >> >>> ==
> >> >>>
> >> >>> In order to make timely releases, we will typically not hold the
> >> >>> release unless the bug in question is a regression from the previous
> >> >>> release. That being said, if there is something which is a
> regression
> >> >>> that has not been correctly targeted please ping me or a committer
> to
> >> >>> help target the issue.
> >> >>>
> >> >>> Thanks,
> >> >>> Kent Yao
> >> >>>
> >> >>>
> -
> >> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >> >>>
> >>
> >> -
> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 3.5.2 (RC5)

[Peter Toth]

+1

huaxin gao  ezt írta (időpont: 2024. aug. 8., Cs,
21:19):

> +1
>
> On Thu, Aug 8, 2024 at 11:41 AM L. C. Hsieh  wrote:
>
>> Then,
>>
>> +1 again
>>
>> On Thu, Aug 8, 2024 at 11:38 AM Dongjoon Hyun 
>> wrote:
>> >
>> > +1
>> >
>> > I'm resending my vote.
>> >
>> > Dongjoon.
>> >
>> > On 2024/08/06 16:06:00 Kent Yao wrote:
>> > > Hi dev,
>> > >
>> > > Please vote on releasing the following candidate as Apache Spark
>> version 3.5.2.
>> > >
>> > > The vote is open until Aug 9, 17:00:00 UTC, and passes if a majority
>> +1
>> > > PMC votes are cast, with a minimum of 3 +1 votes.
>> > >
>> > > [ ] +1 Release this package as Apache Spark 3.5.2
>> > > [ ] -1 Do not release this package because ...
>> > >
>> > > To learn more about Apache Spark, please see
>> https://spark.apache.org/
>> > >
>> > > The tag to be voted on is v3.5.2-rc5 (commit
>> > > bb7846dd487f259994fdc69e18e03382e3f64f42):
>> > > https://github.com/apache/spark/tree/v3.5.2-rc5
>> > >
>> > > The release files, including signatures, digests, etc. can be found
>> at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc5-bin/
>> > >
>> > > Signatures used for Spark RCs can be found in this file:
>> > > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> > >
>> > > The staging repository for this release can be found at:
>> > >
>> https://repository.apache.org/content/repositories/orgapachespark-1462/
>> > >
>> > > The documentation corresponding to this release can be found at:
>> > > https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc5-docs/
>> > >
>> > > The list of bug fixes going into 3.5.2 can be found at the following
>> URL:
>> > > https://issues.apache.org/jira/projects/SPARK/versions/12353980
>> > >
>> > > FAQ
>> > >
>> > > =
>> > > How can I help test this release?
>> > > =
>> > >
>> > > If you are a Spark user, you can help us test this release by taking
>> > > an existing Spark workload and running on this release candidate, then
>> > > reporting any regressions.
>> > >
>> > > If you're working in PySpark you can set up a virtual env and install
>> > > the current RC via "pip install
>> > >
>> https://dist.apache.org/repos/dist/dev/spark/v3.5.2-rc5-bin/pyspark-3.5.2.tar.gz
>> "
>> > > and see if anything important breaks.
>> > > In the Java/Scala, you can add the staging repository to your projects
>> > > resolvers and test
>> > > with the RC (make sure to clean up the artifact cache before/after so
>> > > you don't end up building with an out of date RC going forward).
>> > >
>> > > ===
>> > > What should happen to JIRA tickets still targeting 3.5.2?
>> > > ===
>> > >
>> > > The current list of open tickets targeted at 3.5.2 can be found at:
>> > > https://issues.apache.org/jira/projects/SPARK and search for
>> > > "Target Version/s" = 3.5.2
>> > >
>> > > Committers should look at those and triage. Extremely important bug
>> > > fixes, documentation, and API tweaks that impact compatibility should
>> > > be worked on immediately. Everything else please retarget to an
>> > > appropriate release.
>> > >
>> > > ==
>> > > But my bug isn't fixed?
>> > > ==
>> > >
>> > > In order to make timely releases, we will typically not hold the
>> > > release unless the bug in question is a regression from the previous
>> > > release. That being said, if there is something which is a regression
>> > > that has not been correctly targeted please ping me or a committer to
>> > > help target the issue.
>> > >
>> > > Thanks,
>> > > Kent Yao
>> > >
>> > > -
>> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> > >
>> > >
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: Welcoming a new PMC member

[Peter Toth]

Congratulations!

Mridul Muralidharan  ezt írta (időpont: 2024. aug. 13.,
K, 8:46):

>
> Congratulations Kent !
>
> Regards,
> Mridul
>
> On Mon, Aug 12, 2024 at 8:46 PM Dongjoon Hyun 
> wrote:
>
>> Congratulations, Kent.
>>
>> Dongjoon.
>>
>> On Mon, Aug 12, 2024 at 5:22 PM Xiao Li  wrote:
>>
>>> Congratulations !
>>>
>>> Hyukjin Kwon  于2024年8月12日周一 17:20写道：
>>>
 Hi all,

 The Spark PMC recently voted to add a new PMC member, Kent Yao. Join me
 in welcoming him to his new role!

Re: Welcome new Apache Spark committers

[Peter Toth]

Congratulations!

Gengliang Wang  ezt írta (időpont: 2024. aug. 13., K,
6:15):

> Congratulations, everyone!
>
> On Mon, Aug 12, 2024 at 7:10 PM Denny Lee  wrote:
>
>> Congrats Allison, Martin, and Haejoon!
>>
>> On Tue, Aug 13, 2024 at 9:59 AM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> Congrats everyone!
>>>
>>> On Tue, Aug 13, 2024 at 9:21 AM Xiao Li  wrote:
>>>
 Congratulations!

 Hyukjin Kwon  于2024年8月12日周一 17:19写道：

> Hi all,
>
> The Spark PMC recently voted to add three new committers. Please join
> me in welcoming them to their new role!
>
> - Martin Grund
> - Haejoon Lee
> - Allison Wang
>
> They consistently made contributions to the project and clearly showed
> their expertise. We are very excited to have them join as committers!
>
>

Re: [VOTE] Release Spark 3.4.4 (RC1)

[Peter Toth]

+1, thanks Dongjoon

On Tue, Oct 22, 2024, 07:58 Yang Jie  wrote:

> +1, thank you Dongjoon ~
>
> Jie Yang
>
> On 2024/10/22 03:10:46 Kent Yao wrote:
> > +1, thank you Dongjoon!
> >
> > Kent Yao
> >
> > L. C. Hsieh  于2024年10月22日周二 10:21写道：
> > >
> > > +1
> > >
> > > Thanks Dongjoon!
> > >
> > > On Mon, Oct 21, 2024 at 5:45 PM Xinrong Meng 
> wrote:
> > > >
> > > > +1
> > > >
> > > > Thank you Dongjoon!
> > > >
> > > > On Tue, Oct 22, 2024 at 8:35 AM Ruifeng Zheng 
> wrote:
> > > >>
> > > >> +1
> > > >>
> > > >> Thank you Dongjoon for driving this release!
> > > >>
> > > >> On Tue, Oct 22, 2024 at 6:39 AM huaxin gao 
> wrote:
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> On Mon, Oct 21, 2024 at 1:00 PM Zhou Jiang 
> wrote:
> > > 
> > >  +1
> > > 
> > >  Zhou JIANG
> > > 
> > > 
> > > 
> > >  On Mon, Oct 21, 2024 at 11:04 Dongjoon Hyun 
> wrote:
> > > >
> > > > +1
> > > >
> > > > Dongjoon Hyun.
> > > >
> > > > On 2024/10/21 06:58:17 Dongjoon Hyun wrote:
> > > > > Please vote on releasing the following candidate as Apache
> Spark version
> > > > > 3.4.4.
> > > > >
> > > > > The vote is open until October 25th 1AM (PDT) and passes if a
> majority +1
> > > > > PMC
> > > > > votes are cast, with a minimum of 3 +1 votes.
> > > > >
> > > > > [ ] +1 Release this package as Apache Spark 3.4.4
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > To learn more about Apache Spark, please see
> https://spark.apache.org/
> > > > >
> > > > > The tag to be voted on is v3.4.4-rc1 (commit
> > > > > 6729992c76fc59ab07f63f97a9858691274447d0)
> > > > > https://github.com/apache/spark/tree/v3.4.4-rc1
> > > > >
> > > > > The release files, including signatures, digests, etc. can be
> found at:
> > > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.4-rc1-bin/
> > > > >
> > > > > Signatures used for Spark RCs can be found in this file:
> > > > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > > > >
> > > > > The staging repository for this release can be found at:
> > > > >
> https://repository.apache.org/content/repositories/orgapachespark-1470/
> > > > >
> > > > > The documentation corresponding to this release can be found
> at:
> > > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.4-rc1-docs/
> > > > >
> > > > > The list of bug fixes going into 3.4.4 can be found at the
> following URL:
> > > > >
> https://issues.apache.org/jira/projects/SPARK/versions/12354565
> > > > >
> > > > > This release is using the release script of the tag v3.4.4-rc1.
> > > > >
> > > > > FAQ
> > > > >
> > > > > =
> > > > > How can I help test this release?
> > > > > =
> > > > >
> > > > > If you are a Spark user, you can help us test this release by
> taking
> > > > > an existing Spark workload and running on this release
> candidate, then
> > > > > reporting any regressions.
> > > > >
> > > > > If you're working in PySpark you can set up a virtual env and
> install
> > > > > the current RC and see if anything important breaks, in the
> Java/Scala
> > > > > you can add the staging repository to your projects resolvers
> and test
> > > > > with the RC (make sure to clean up the artifact cache
> before/after so
> > > > > you don't end up building with a out of date RC going forward).
> > > > >
> > > > > ===
> > > > > What should happen to JIRA tickets still targeting 3.4.4?
> > > > > ===
> > > > >
> > > > > The current list of open tickets targeted at 3.4.4 can be
> found at:
> > > > > https://issues.apache.org/jira/projects/SPARK and search for
> "Target
> > > > > Version/s" = 3.4.4
> > > > >
> > > > > Committers should look at those and triage. Extremely
> important bug
> > > > > fixes, documentation, and API tweaks that impact compatibility
> should
> > > > > be worked on immediately. Everything else please retarget to an
> > > > > appropriate release.
> > > > >
> > > > > ==
> > > > > But my bug isn't fixed?
> > > > > ==
> > > > >
> > > > > In order to make timely releases, we will typically not hold
> the
> > > > > release unless the bug in question is a regression from the
> previous
> > > > > release. That being said, if there is something which is a
> regression
> > > > > that has not been correctly targeted please ping me or a
> committer to
> > > > > help target the issue.
> > > > >
> > > >
> > > >
> -
> > > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > > >
> > >
> > > 

Re: [VOTE] Single-pass Analyzer for Catalyst

[Peter Toth]

+1

On Tue, Oct 1, 2024, 08:33 Yang Jie  wrote:

> +1, Thanks
>
> Jie Yang
>
> On 2024/10/01 03:26:40 John Zhuge wrote:
> > +1 (non-binding)
> >
> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang
> >  wrote:
> >
> > > +1
> > >
> > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <
> kabhwan.opensou...@gmail.com>
> > > wrote:
> > >
> > >> +1 (non-binding), promising proposal!
> > >>
> > >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun 님이 작성:
> > >>
> > >>> Thank you for the swift clarification, Reynold and Xiao.
> > >>>
> > >>> It seems that the Target Version was set mistakenly initially.
> > >>>
> > >>> I removed the `Target Version` from the SPIP JIRA.
> > >>>
> > >>> https://issues.apache.org/jira/browse/SPARK-49834
> > >>>
> > >>> I'm switching my cast to +1 for this SPIP vote.
> > >>>
> > >>> Thanks,
> > >>> Dongjoon.
> > >>>
> > >>> On 2024/09/30 22:55:41 Xiao Li wrote:
> > >>> > +1 in support of the direction of the Single-pass Analyzer for
> > >>> Catalyst.
> > >>> >
> > >>> > I think we should not have a target version for the new Catalyst
> > >>> SPARK-49834
> > >>> > . It should
> not be
> > >>> a
> > >>> > blocker for Spark 4.0. When implementing the new analyzer, the code
> > >>> changes
> > >>> > must not affect users of the existing analyzer to avoid any
> user-facing
> > >>> > impacts.
> > >>> >
> > >>> > Reynold Xin  于2024年9月30日周一 15:39写道：
> > >>> >
> > >>> > > I don't actually "lead" this. But I don't think this needs to
> target
> > >>> a
> > >>> > > specific Spark version given it should not have any user facing
> > >>> > > consequences?
> > >>> > >
> > >>> > >
> > >>> > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun <
> dongj...@apache.org>
> > >>> wrote:
> > >>> > >
> > >>> > >> Thank you for leading this, Vladimir, Reynold, Herman.
> > >>> > >>
> > >>> > >> I'm wondering if this is really achievable goal for Apache Spark
> > >>> 4.0.0.
> > >>> > >>
> > >>> > >> If it's expected that we are unable to deliver it, shall we
> > >>> postpone this
> > >>> > >> vote until 4.1.0 planning?
> > >>> > >>
> > >>> > >> Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,
> > >>> > >>
> > >>> > >> -1 from my side.
> > >>> > >>
> > >>> > >> Thanks,
> > >>> > >> Dongjoon.
> > >>> > >>
> > >>> > >>
> > >>> > >> On 2024/09/30 17:51:24 Herman van Hovell wrote:
> > >>> > >> > +1
> > >>> > >> >
> > >>> > >> > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin
> > >>>  > >>> > >> >
> > >>> > >> > wrote:
> > >>> > >> >
> > >>> > >> > > +1
> > >>> > >> > >
> > >>> > >> > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev <
> > >>> vvdr@gmail.com>
> > >>> > >> > > wrote:
> > >>> > >> > >
> > >>> > >> > >> Hi all,
> > >>> > >> > >>
> > >>> > >> > >> I’d like to start a vote for a single-pass Analyzer for the
> > >>> Catalyst
> > >>> > >> > >> project. This project will introduce a new analysis
> framework
> > >>> to the
> > >>> > >> > >> Catalyst, which will eventually replace the fixed-point
> one.
> > >>> > >> > >>
> > >>> > >> > >> Please refer to the SPIP jira:
> > >>> > >> > >> https://issues.apache.org/jira/browse/SPARK-49834
> > >>> > >> > >>
> > >>> > >> > >> [ ] +1: Accept the proposal
> > >>> > >> > >> [ ] +0
> > >>> > >> > >> [ ] -1: I don’t think this is a good idea because …
> > >>> > >> > >>
> > >>> > >> > >> Thanks!
> > >>> > >> > >>
> > >>> > >> > >> Vladimir
> > >>> > >> > >>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> > >>
> > >>> -
> > >>> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >>> > >>
> > >>> > >>
> > >>> >
> > >>>
> > >>> -
> > >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >>>
> > >>>
> >
> > --
> > John Zhuge
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [外部邮件] Re: New committer: Bingkun Pan

[Peter Toth]

Congratulations!

On Wed, Nov 20, 2024 at 7:16 AM roryqi  wrote:

> Congrats!
>
> Xinrong Meng  于2024年11月20日周三 09:58写道：
> >
> > Congratulations Bingkun, well deserved!
> >
> > On Tue, Nov 19, 2024 at 10:30 PM Wenchen Fan 
> wrote:
> >>
> >> Congrats!
> >>
> >> On Sun, Nov 17, 2024 at 8:12 PM Mridul Muralidharan 
> wrote:
> >>>
> >>> Congratulations Bingkun !
> >>>
> >>> Regards,
> >>> Mridul
> >>>
> >>> On Sun, Nov 17, 2024 at 12:17 AM Yang,Jie(INF)
>  wrote:
> 
>  Congratulations, Bingkun ~
> 
> 
> 
>  发件人: Hyukjin Kwon 
>  日期: 2024年11月17日 星期日 14:05
>  收件人: Jungtaek Lim 
>  抄送: Ruifeng Zheng , Dongjoon Hyun <
> dongjoon.h...@gmail.com>, dev 
>  主题: [外部邮件] Re: New committer: Bingkun Pan
> 
> 
> 
>  Congrats!!
> 
> 
> 
>  On Sat, Nov 16, 2024 at 11:46 AM Jungtaek Lim <
> kabhwan.opensou...@gmail.com> wrote:
> 
>  Congrats Bingkun!
> 
> 
> 
>  2024년 11월 16일 (토) 오전 11:56, Ruifeng Zheng 님이 작성:
> 
>  Congratulations,Bingkun!
> 
> 
> 
>  On Sat, Nov 16, 2024 at 12:11 AM Dongjoon Hyun <
> dongjoon.h...@gmail.com> wrote:
> 
>  The Project Management Committee (PMC) for Apache Spark
>  has invited Bingkun Pan to become a committer and we are pleased
>  to announce that he has accepted.
> 
>  Please join us in welcoming Bingkun to his new role and
>  responsibility in our project community.
> 
>  Dongjoon Hyun
>  On behalf of the Spark PMC
> 
> 
> 
>  --
> 
>  Ruifeng Zheng
>  E-mail: zrfli...@gmail.com
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Use plain text logs by default

[Peter Toth]

+1

On Thu, Jan 9, 2025 at 8:37 AM Allison Wang  wrote:

> +1
>
> On Thu, Jan 9, 2025 at 2:45 PM L. C. Hsieh  wrote:
>
>> +1
>>
>> On Wed, Jan 8, 2025 at 10:27 PM Gengliang Wang  wrote:
>> >
>> > +1
>> >
>> > On Wed, Jan 8, 2025 at 10:17 PM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>> >>
>> >> +1 (non-binding)
>> >>
>> >> 2025년 1월 9일 (목) 오후 3:06, Cheng Pan 님이 작성:
>> >>>
>> >>> +1
>> >>>
>> >>> Thanks,
>> >>> Cheng Pan
>> >>>
>> >>>
>> >>>
>> >>> On Jan 9, 2025, at 12:28, Wenchen Fan  wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Following the discussion[1], I'd like to start the vote for 'Use
>> plain text logs by default'.
>> >>>
>> >>> Note: This is not to overthrow the previous vote that adds the
>> structured logging framework. The framework is still there, but users need
>> to explicitly turn it on.
>> >>>
>> >>> Please vote for the next 72 hours:
>> >>>
>> >>>  [ ] +1: Accept the proposal
>> >>>  [ ] +0
>> >>>  [ ]- 1: I don’t think this is a good idea because …
>> >>>
>> >>>
>> >>>
>> >>> Bests,
>> >>> Wenchen Fan
>> >>>
>> >>> [1] https://lists.apache.org/thread/4fnlnvhsqym72k53jw8cjhcdjbhpmm95
>> >>>
>> >>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] SPIP: Constraints in DSv2

[Peter Toth]

+1

On Fri, Mar 21, 2025 at 10:24 PM Szehon Ho  wrote:

> +1 (non binding)
>
> Agree with Anton, data sources like the open table formats define the
> requirement, and definitely need engines to write to it accordingly.
>
> Thanks,
> Szehon
>
> On Fri, Mar 21, 2025 at 1:31 PM Anton Okolnychyi 
> wrote:
>
>> -1 (non-binding): Breaks the Chain of Responsibility. Constraints should
>>> be defined and enforced by the data sources themselves, not Spark. Spark is
>>> a processing engine, and enforcing constraints at this level blurs
>>> architectural boundaries, making Spark responsible for something it does
>>> not control.
>>>
>>
>> I disagree that this breaks the chain of responsibility. It may be quite
>> the opposite, in fact. Spark is already responsible for enforcing NOT NULL
>> constraints by adding AssertNotNull for required columns today. Connectors
>> like Iceberg and Delta store constraint definitions but rely on engines
>> like Spark to enforce them during INSERT, DELETE, UPDATE, and MERGE
>> operations. Without this API, each connector would need to reimplement the
>> same logic, creating duplication.
>>
>> The proposal is aligned with the SQL standard and other relational
>> databases. In my view, it simply makes Spark a better engine, facilitates
>> data accuracy and consistency, and enables performance optimizations.
>>
>> - Anton
>>
>> пт, 21 бер. 2025 р. о 12:59 Ángel Álvarez Pascua <
>> angel.alvarez.pas...@gmail.com> пише:
>>
>>> -1 (non-binding): Breaks the Chain of Responsibility. Constraints should
>>> be defined and enforced by the data sources themselves, not Spark. Spark is
>>> a processing engine, and enforcing constraints at this level blurs
>>> architectural boundaries, making Spark responsible for something it does
>>> not control.
>>>
>>> El vie, 21 mar 2025 a las 20:18, L. C. Hsieh ()
>>> escribió:
>>>
 +1

 On Fri, Mar 21, 2025 at 12:13 PM huaxin gao 
 wrote:
 >
 > +1
 >
 > On Fri, Mar 21, 2025 at 12:08 PM Denny Lee 
 wrote:
 >>
 >> +1 (non-binding)
 >>
 >> On Fri, Mar 21, 2025 at 11:52 Gengliang Wang 
 wrote:
 >>>
 >>> +1
 >>>
 >>> On Fri, Mar 21, 2025 at 11:46 AM Anton Okolnychyi <
 aokolnyc...@gmail.com> wrote:
 
  Hi all,
 
  I would like to start a vote on adding support for constraints to
 DSv2.
 
  Discussion thread:
 https://lists.apache.org/thread/njqjcryq0lot9rkbf10mtvf7d1t602bj
  SPIP:
 https://docs.google.com/document/d/1EHjB4W1LjiXxsK_G7067j9pPX0y15LUF1Z5DlUPoPIo
  PR with the API changes:
 https://github.com/apache/spark/pull/50253
  JIRA: https://issues.apache.org/jira/browse/SPARK-51207
 
  Please vote on the SPIP for the next 72 hours:
 
  [ ] +1: Accept the proposal as an official SPIP
  [ ] +0
  [ ] -1: I don’t think this is a good idea because …
 
  - Anton

 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPIP: Declarative Pipelines

[Peter Toth]

+1

On Wed, Apr 9, 2025 at 7:40 PM DB Tsai  wrote:

> +1.
>
> Really exciting to see the community actively investing on decorative
> pipeline APIs and materialized views. Thanks.
>
> DB Tsai  |  https://www.dbtsai.com/  |  PGP 42E5B25A8F7A82C1
>
> On Apr 9, 2025, at 9:43 AM, Jules Damji  wrote:
>
> +1 (non-binding)
>
> Excuse the thumb typos
>
>
> On Wed, 09 Apr 2025 at 7:22 AM, Sandy Ryza  wrote:
>
>> We started to get some votes on the discussion thread, so I'd like to
>> move to a formal vote on adding support for declarative pipelines.
>>
>> *Discussion thread: *
>> https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m574ly
>> *SPIP:*
>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4
>> *JIRA:* https://issues.apache.org/jira/browse/SPARK-51727
>>
>> Please vote on the SPIP for the next 72 hours:
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> -Sandy
>>
>>
>

Re: [DISCUSS] SPIP: Declarative Pipelines

[Peter Toth]

+1

On Wed, Apr 9, 2025 at 8:51 AM Cheng Pan  wrote:

> +1 (non-binding)
>
> Glad to see Spark SQL extended to streaming use cases.
>
> Thanks,
> Cheng Pan
>
>
>
> On Apr 9, 2025, at 14:43, Anton Okolnychyi  wrote:
>
> +1
>
> вт, 8 квіт. 2025 р. о 23:36 Jacky Lee  пише:
>
>> +1 I'm delighted that it will be open-sourced, enabling greater
>> integration with Iceberg/Delta to unlock more value.
>>
>> Jungtaek Lim  于2025年4月9日周三 10:47写道：
>> >
>> > +1 looking forward to seeing this make progress!
>> >
>> > On Wed, Apr 9, 2025 at 11:32 AM Yang Jie  wrote:
>> >>
>> >> +1
>> >>
>> >> On 2025/04/09 01:07:57 Hyukjin Kwon wrote:
>> >> > +1
>> >> >
>> >> > I am actually pretty excited to have this. Happy to see this being
>> proposed.
>> >> >
>> >> > On Wed, 9 Apr 2025 at 01:55, Chao Sun  wrote:
>> >> >
>> >> > > +1. Super excited about this effort!
>> >> > >
>> >> > > On Tue, Apr 8, 2025 at 9:47 AM huaxin gao 
>> wrote:
>> >> > >
>> >> > >> +1 I support this SPIP because it simplifies data pipeline
>> management and
>> >> > >> enhances error detection.
>> >> > >>
>> >> > >>
>> >> > >> On Tue, Apr 8, 2025 at 9:33 AM Dilip Biswal 
>> wrote:
>> >> > >>
>> >> > >>> Excited to see this heading toward open source — materialized
>> views and
>> >> > >>> other features will bring a lot of value.
>> >> > >>> +1 (non-binding)
>> >> > >>>
>> >> > >>> On Mon, Apr 7, 2025 at 10:37 AM Sandy Ryza 
>> wrote:
>> >> > >>>
>> >> >  Hi Khalid – the CLI in the current proposal will need to be
>> built on
>> >> >  top of internal APIs for constructing and launching pipeline
>> executions.
>> >> >  We'll have the option to expose these in the future.
>> >> > 
>> >> >  It would be worthwhile to understand the use cases in more
>> depth before
>> >> >  exposing these, because APIs are one-way doors and can be
>> costly to
>> >> >  maintain.
>> >> > 
>> >> >  On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <
>> >> >  khalidmammad...@gmail.com> wrote:
>> >> > 
>> >> > > Looks great!
>> >> > > QQ: will user able to run this pipeline from normal code? I.e.
>> can I
>> >> > > trigger a pipeline from *driver* code based on some condition
>> etc. or
>> >> > > it must be executed via separate shell command ?
>> >> > > As a background Databricks imposes similar limitation where as
>> you
>> >> > > cannot run normal Spark code and DLT on the same cluster for
>> some reason
>> >> > > and forces to use two clusters increasing the cost and latency.
>> >> > >
>> >> > > On Sat, 5 Apr 2025 at 23:03, Sandy Ryza 
>> wrote:
>> >> > >
>> >> > >> Hi all – starting a discussion thread for a SPIP that I've
>> been
>> >> > >> working on with Chao Sun, Kent Yao, Yuming Wang, and Jie
>> Yang: [JIRA
>> >> > >> ] [Doc
>> >> > >> <
>> https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
>> >
>> >> > >> ].
>> >> > >>
>> >> > >> The SPIP proposes extending Spark's lazy, declarative
>> execution model
>> >> > >> beyond single queries, to pipelines that keep multiple
>> datasets up to date.
>> >> > >> It introduces the ability to compose multiple transformations
>> into a single
>> >> > >> declarative dataflow graph.
>> >> > >>
>> >> > >> Declarative pipelines aim to simplify the development and
>> management
>> >> > >> of data pipelines, by  removing the need for manual
>> orchestration of
>> >> > >> dependencies and making it possible to catch many errors
>> before any
>> >> > >> execution steps are launched.
>> >> > >>
>> >> > >> Declarative pipelines can include both batch and streaming
>> >> > >> computations, leveraging Structured Streaming for stream
>> processing and new
>> >> > >> materialized view syntax for batch processing. Tight
>> integration with Spark
>> >> > >> SQL's analyzer enables deeper analysis and earlier error
>> detection than is
>> >> > >> achievable with more generic frameworks.
>> >> > >>
>> >> > >> Let us know what you think!
>> >> > >>
>> >> > >>
>> >> >
>> >>
>> >> -
>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

[Peter Toth]

+1

On Tue, May 6, 2025 at 9:59 AM Yang Jie  wrote:

> +1, A big thank you to Dongjoon for all the hard work you've put into this!
>
> On 2025/05/05 18:19:33 DB Tsai wrote:
> > +1, it’s exciting to see Spark Connect Swift client, showcasing Spark
> Connect
> > as a truly language-agnostic protocol, and also powering Swift users to
> use
> > Spark!
> >
> >
> >
> > Sent from my iPhone
> >
> >
> >
> > > On May 5, 2025, at 1:11 AM, Gabor Somogyi 
> wrote:
> > >
> > >
> >
> > > 
> > >
> > > +1 (non-binding)
> > >
> > >
> > >
> > >
> > >
> > > G
> > >
> > >
> > >
> > >
> > > On Mon, May 5, 2025 at 8:35 AM huaxin gao
> > > <[huaxin.ga...@gmail.com](mailto:huaxin.ga...@gmail.com)> wrote:
> > >
> > >
> >
> > >> +1 Thanks Dongjoon.
> >
> > >>
> >
> > >>
> > >
> > >>
> >
> > >> On Sun, May 4, 2025 at 5:21 PM Dongjoon Hyun
> > <[dongj...@apache.org](mailto:dongj...@apache.org)> wrote:
> > >
> > >>
> >
> > >>> +1
> > >
> > >  I checked the checksum and signatures, and tested with Apache Spark
> 4.0.0
> > > RC4 on Swift 6.1.
> > >
> > >  This is the initial release (v0.1) with 105 patches to provide a
> tangible
> > > release to the users.
> > >
> > >  v0.2 is under planning in SPARK-51999.
> > >
> > >  Dongjoon.
> > >
> > >  On 2025/05/04 22:14:54 Dongjoon Hyun wrote:
> > >  > Please vote on releasing the following candidate as Apache Spark
> Connect
> > >  > Swift Client 0.1.0. This vote is open for the next 72 hours and
> passes if
> > > a
> > >  > majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
> > >  >
> > >  > [ ] +1 Release this package as Apache Spark Connect Swift Client
> 0.1.0
> > >  > [ ] -1 Do not release this package because ...
> > >  >
> > >  > TAG:
> > >  > <
> https://github.com/apache/spark-connect-swift/releases/tag/v0.1.0-rc1>
> > >  > (commit: e1e8a327ce13d53ecaa4b7c2a5e8d69ffb822940)
> > >  >
> > >  > RELEASE FILES:
> > >  >  > > swift-v0.1.0-rc1/>
> > >  >
> > >  > LIST OF ISSUES:
> > >  > 
> > >  >
> > >  > Thanks,
> > >  > Dongjoon.
> > >  >
> > >
> > >
> \-
> > >  To unsubscribe e-mail: [dev-unsubscr...@spark.apache.org](mailto:dev-
> > > unsubscr...@spark.apache.org)
> > >
> > >
> >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration

[Peter Toth]

+1

On Wed, Feb 19, 2025 at 10:20 AM Max Gekk  wrote:

> +1
>
> On Wed, Feb 19, 2025 at 9:15 AM L. C. Hsieh  wrote:
>
>> +1
>>
>> On Tue, Feb 18, 2025 at 9:46 PM dongjoon.hyun 
>> wrote:
>> >
>> > Please vote to deprecate `spark.databricks.*` configuration at Apache
>> Spark 3.5.5.
>> > This is a part of the following on-going discussion.
>> >
>> > - DISCUSSION:
>> https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
>> >   (Deprecating and banning `spark.databricks.*` config from Apache
>> Spark repository)
>> >
>> > The vote is open for the next 72 hours and passes
>> > if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release Apache Spark 3.5.5 deprecating `spark.databricks.*`
>> configuration
>> > [ ] -1 Do not deprecate `spark.databricks.*` configuration at Apache
>> Spark 3.5.5 because ...
>> >
>> > Thank you in advance.
>> >
>> > Dongjoon
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [DISCUSS] New Spark Connect Client repository for Swift language

[Peter Toth]

+1

On Mon, Mar 10, 2025 at 5:39 PM Kent Yao  wrote:

> +1
>
> Kent
>
> 在 2025年3月10日星期一，Max Gekk  写道：
>
>> +1
>>
>> On Mon, Mar 10, 2025 at 6:17 PM Wenchen Fan  wrote:
>> >
>> > +1
>> >
>> > On Mon, Mar 10, 2025 at 8:02 AM L. C. Hsieh  wrote:
>> >>
>> >> +1
>> >>
>> >> Thanks Dongjoon for contributing to Swift implementation.
>> >>
>> >> On Mon, Mar 10, 2025 at 7:18 AM Hyukjin Kwon 
>> wrote:
>> >> >
>> >> > +1
>> >> > On Mon, Mar 10, 2025 at 6:48 AM Yang Jie 
>> wrote:
>> >> >>
>> >> >> Great! Really happy to see that spark-connect supports more
>> programming languages.
>> >> >>
>> >> >>
>> >> >> On 2025/03/10 07:00:32 Martin Grund wrote:
>> >> >> > Great work and proposal!
>> >> >> >
>> >> >> > I'm supportive.
>> >> >> >
>> >> >> > On Sun, Mar 9, 2025 at 23:31 Dongjoon Hyun 
>> wrote:
>> >> >> >
>> >> >> > > Hi, All.
>> >> >> > >
>> >> >> > > I'd like to propose to add a new Apache Spark repository for
>> `Spark
>> >> >> > > Connect Client for Swift` in Apache Spark 4.1.0 timeframe.
>> >> >> > >
>> >> >> > > https://github.com/apache/spark-connect-swift
>> >> >> > >
>> >> >> > > To do this, I created an umbrella JIRA issue and initial
>> implementation
>> >> >> > > here.
>> >> >> > >
>> >> >> > > - SPARK-51448 Support Swift language in Spark Connect
>> >> >> > > - https://github.com/dongjoon-hyun/spark-connect-swift
>> >> >> > >
>> >> >> > > This new repository creation follows our previous community
>> decision for
>> >> >> > > `Go` language.
>> >> >> > >
>> >> >> > > -
>> https://lists.apache.org/thread/qhnvtzmq64wbc0glrg2j89lslqb70q67
>> >> >> > >   New Clients for Go and Rust
>> >> >> > > - https://github.com/apache/spark-connect-go
>> >> >> > >
>> >> >> > > I hope this official support helps the adoption of `Spark
>> Connect` in
>> >> >> > > Apache Spark 4+ more.
>> >> >> > >
>> >> >> > > WDYT about `spark-connect-swift` repository?
>> >> >> > >
>> >> >> > > Thanks,
>> >> >> > > Dongjoon.
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> -
>> >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >> >>
>> >>
>> >> -
>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Release Spark 4.1.0-preview1 (RC1)

[Peter Toth]

+1

On Thu, Jul 10, 2025 at 9:12 AM Kent Yao  wrote:

> Thank you all for the verification, +1
>
> Kent
>
> 在 2025年7月10日星期四，Hyukjin Kwon  写道：
>
>> It was a mistake in the email. The artifact shouldn't have a problem.
>>
>> On Thu, 10 Jul 2025 at 16:00, Hyukjin Kwon  wrote:
>>
>>> oh yeah. I think I should change the email contents.
>>>
>>> On Thu, 10 Jul 2025 at 15:02, Saruta, Kousuke
>>>  wrote:
>>>
 Using dev1 rather than preview1 seems intended.


 https://github.com/apache/spark/blob/v4.1.0-preview1-rc1/dev/create-release/release-build.sh#L127



 *送信元**: *Jungtaek Lim 
 *日付**: *2025年7月10日 木曜日 14:30
 *宛先**: *Kent Yao 
 *Cc: *Anton Okolnychyi , Max Gekk <
 max.g...@gmail.com>, Sandy Ryza ,
 Wenchen Fan , Kousuke Saruta ,
 "dev@spark.apache.org" 
 *件名**: *RE: [EXTERNAL] [VOTE] Release Spark 4.1.0-preview1 (RC1)



 *CAUTION*: This email originated from outside of the organization. Do
 not click links or open attachments unless you can confirm the sender and
 know the content is safe.



 I think we used "dev1" for 4.0.0 "preview1" as well. I guess this is
 based on the name convention on python?



 On Thu, Jul 10, 2025 at 1:07 PM Kent Yao  wrote:

 -1,



 There is a 404 for
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz



 pip install
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz
 Collecting
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz
   ERROR: HTTP error 404 while getting
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz
 ERROR: Could not install requirement
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz
 because of HTTP error 404 Client Error: Not Found for url:
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz
 for URL
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz



 Can you check?





 Kent





 Jungtaek Lim  于2025年7月10日周四 07:58写道：

 +1 (non-binding) Let's give it a try!



 On Thu, Jul 10, 2025 at 12:24 AM Anton Okolnychyi <
 aokolnyc...@gmail.com> wrote:

 +1 (non-binding)



 On Wed, Jul 9, 2025 at 8:07 AM Max Gekk  wrote:

 +1



 On Wed, Jul 9, 2025 at 4:04 PM Sandy Ryza 
 wrote:

 +1 (non-binding)



 On Wed, Jul 9, 2025 at 6:57 AM Wenchen Fan  wrote:

 +1



 On Wed, Jul 9, 2025 at 1:16 AM Kousuke Saruta 
 wrote:

 +1



 2025年7月9日(水) 2:12 Rozov, Vlad :

 +1 (non-binding)



 Thank you,



 Vlad



 *From: *Dongjoon Hyun 
 *Date: *Tuesday, July 8, 2025 at 8:09 AM
 *To: *Hyukjin Kwon 
 *Cc: *"dev@spark.apache.org" 
 *Subject: *RE: [EXTERNAL] [VOTE] Release Spark 4.1.0-preview1 (RC1)



 +1



 Dongjoon



 On Tue, Jul 8, 2025 at 05:41 Hyukjin Kwon  wrote:

 Alright. +1 from myself :-).



 On Tue, Jul 8, 2025 at 9:39 PM  wrote:

 Please vote on releasing the following candidate as Apache Spark
 version 4.1.0-preview1.

 The vote is open until Sat, 12 Jul 2025 05:38:35 PDT and passes if a
 majority +1 PMC votes are cast, with
 a minimum of 3 +1 votes.

 [ ] +1 Release this package as Apache Spark 4.1.0-preview1
 [ ] -1 Do not release this package because ...

 To learn more about Apache Spark, please see https://spark.apache.org/

 The tag to be voted on is v4.1.0-preview1-rc1 (commit f3ac67ee9b3):
 https://github.com/apache/spark/tree/v4.1.0-preview1-rc1

 The release files, including signatures, digests, etc. can be found at:
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/

 Signatures used for Spark RCs can be found in this file:
 https://downloads.apache.org/spark/KEYS

 The staging repository for this release can be found at:
 https://repository.apache.org/content/repositories/orgapachespark-1500/

 The documentation corresponding to this release can be found at:
 https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-docs/

 The list of bug fixes going into 4.1.0-preview1 can be found at the
 following URL:
 https://issues.apache.org/jira/projects/SPARK/versions/12355581

 FAQ

 =
 How can I help test this release?
 =

 If you 

Re: [VOTE] SPIP: Monthly preview release

[Peter Toth]

+1

On Fri, Jul 4, 2025 at 6:30 AM Ruifeng Zheng  wrote:

> +1
>
> On Fri, Jul 4, 2025 at 10:17 AM John Zhuge  wrote:
>
>> +1 (non-binding)
>>
>> John Zhuge
>>
>>
>> On Thu, Jul 3, 2025 at 1:47 PM Jungtaek Lim 
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> It would be great outcome regardless of whether this effort would be
>>> successful or not. We may even find the way to do official release more
>>> often, which would be a huge win.
>>>
>>> 2025년 7월 4일 (금) 오전 3:13, Szehon Ho 님이 작성:
>>>
 +1 (non-binding)

 Thanks for the proposal, hope one day to get faster releases in Spark.

 Thanks
 Szehon

 On Thu, Jul 3, 2025 at 6:58 AM Sandy Ryza 
 wrote:

> +1 (non-binding)
>
> On Thu, Jul 3, 2025 at 6:47 AM Jules Damji 
> wrote:
>
>> +1 (non-binding)
>> —
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> > On Jul 2, 2025, at 11:44 PM, L. C. Hsieh  wrote:
>> >
>> > +1
>> >
>> >> On Wed, Jul 2, 2025 at 9:38 PM Hyukjin Kwon 
>> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I would like to start a vote on the monthly preview releases.
>> >>
>> >> Discussion thread:
>> https://lists.apache.org/thread/1hmsb3g7lm5k2f9xnp6x2hmss8yrd5h8
>> >> SPIP:
>> https://docs.google.com/document/d/1ysJ16z_NUfIdsYqq1Qq7k8htmMWFpo8kXqX-8lGzCGc/edit?tab=t.0#heading=h.89yty49abp67
>> >> JIRA: https://issues.apache.org/jira/browse/SPARK-52625
>> >>
>> >> Please vote on the SPIP for the next 72 hours:
>> >>
>> >> [ ] +1: Accept the proposal as an official SPIP
>> >> [ ] +0
>> >> [ ] -1: I don’t think this is a good idea because …
>> >>
>> >>
>> >> Starting with my own +1.
>> >
>> >
>> -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Release Apache Spark K8s Operator 0.4.0 (RC1)

[Peter Toth]

+1

Checked checksum, signature.
Thanks Dongjoon.


On Mon, Jun 30, 2025 at 5:51 PM Dongjoon Hyun 
wrote:

> Please vote on releasing the following candidate as Apache Spark K8s
> Operator 0.4.0. This vote is open for the next 72 hours and passes if a
> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark K8s Operator 0.4.0
> [ ] -1 Do not release this package because ...
>
> TAG:
> https://github.com/apache/spark-kubernetes-operator/releases/tag/0.4.0-rc.1
> (commit: 38794598af051e5d1ca4d2a7ef453cac562fe021)
>
> RELEASE FILES:
>
> https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.4.0-rc.1/
>
> LIST OF ISSUES:
> https://issues.apache.org/jira/projects/SPARK/versions/12355993
>
> Thanks,
> Dongjoon.
>

Re: [VOTE] Release Apache Spark Connect Swift Client 0.3.0 (RC1)

[Peter Toth]

+1

On Sun, Jun 1, 2025 at 9:00 AM L. C. Hsieh  wrote:

> +1
>
> On Sat, May 31, 2025 at 9:18 PM Dongjoon Hyun 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark Connect
> Swift Client 0.3.0. This is the first release tested with the official
> Apache Spark 4.0.0 release. This vote is open for the next 72 hours and
> passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark Connect Swift Client 0.3.0
> > [ ] -1 Do not release this package because ...
> >
> > TAG:
> > https://github.com/apache/spark-connect-swift/releases/tag/0.3.0-rc.1
> > (commit: 60fa9c40174c134c07b1eec8678b7ed4c588e22a)
> >
> > RELEASE FILES:
> >
> https://dist.apache.org/repos/dist/dev/spark/spark-connect-swift-0.3.0-rc.1/
> >
> > LIST OF ISSUES:
> > https://issues.apache.org/jira/projects/SPARK/versions/12355948
> >
> > Thanks,
> > Dongjoon.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark K8s Operator 0.3.0 (RC1)

[Peter Toth]

+1

On Sun, Jun 1, 2025 at 9:01 AM L. C. Hsieh  wrote:

> +1
>
> On Sat, May 31, 2025 at 10:38 PM Dongjoon Hyun 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark K8s
> Operator 0.3.0. This is the first release built on Apache Spark 4.0,
> running on Java 24, and promoting CRDs to v1beta1 from v1alpha1. This vote
> is open for the next 72 hours and passes if a majority +1 PMC votes are
> cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark K8s Operator 0.3.0
> > [ ] -1 Do not release this package because ...
> >
> > TAG:
> >
> https://github.com/apache/spark-kubernetes-operator/releases/tag/0.3.0-rc.1
> > (commit: d9ba7eb888af7751259cfa44e8db3115e6a3c05c)
> >
> > RELEASE FILES:
> >
> https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.3.0-rc.1/
> >
> > LIST OF ISSUES:
> > https://issues.apache.org/jira/projects/SPARK/versions/12355949
> >
> > Thanks,
> > Dongjoon.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Re: [VOTE] Release Spark 4.0.0 (RC5)

[Peter Toth]

+1

On Tue, May 13, 2025 at 12:24 PM beliefer  wrote:

> +1
>
>
> 在 2025-05-13 16:26:32，"Yuanjian Li"  写道：
>
> +1
>
> Szehon Ho  于2025年5月12日周一 22:32写道：
>
>> +1 (non binding)
>>
>> Checked license, signature, checksum, ran basic test on
>> spark-4.0.0-bin-hadoop3.
>>
>> Thanks
>> Szehon
>>
>> On Mon, May 12, 2025 at 9:02 PM Sakthi  wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Mon, May 12, 2025 at 7:38 PM Jungtaek Lim <
>>> kabhwan.opensou...@gmail.com> wrote:
>>>
 +1 (non-binding)

 Thanks Wenchen for driving the release!

 On Tue, May 13, 2025 at 11:35 AM Yang Jie  wrote:

> +1, thank you Wenchen
>
> On 2025/05/13 02:11:02 "Rozov, Vlad" wrote:
> > +1 (non-binding)
> >
> > Thank you,
> >
> > Vlad
> >
> > On May 12, 2025, at 5:44 PM, huaxin gao 
> wrote:
> >
> > +1
> >
> > On Mon, May 12, 2025 at 5:34 PM Hyukjin Kwon  > wrote:
> > +1
> >
> > On Tue, 13 May 2025 at 03:24, Xinrong Meng  > wrote:
> > +1
> >
> > Thank you Wenchen!
> >
> > On Mon, May 12, 2025 at 10:03 AM Yuming Wang  > wrote:
> > +1
> >
> > On Tue, May 13, 2025 at 12:07 AM Gengliang Wang  > wrote:
> > +1
> >
> > On Mon, May 12, 2025 at 6:52 AM Wenchen Fan  > wrote:
> > I'll start with my own +1.
> >
> > All the known blockers are fixed, and I verified that the new Spark
> Connect distribution works as expected.
> >
> > On Fri, May 9, 2025 at 8:16 PM Wenchen Fan  > wrote:
> > Please vote on releasing the following candidate as Apache Spark
> version 4.0.0.
> >
> > The vote is open until May 15 (PST) and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 4.0.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> https://spark.apache.org/
> >
> > The tag to be voted on is v4.0.0-rc5 (commit
> f35a2ee6dc7833ea0cff757147132c9fdc26c113)
> > https://github.com/apache/spark/tree/v4.0.0-rc5
> >
> > The release files, including signatures, digests, etc. can be found
> at:
> > https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc5-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> >
> https://repository.apache.org/content/repositories/orgapachespark-1483/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc5-docs/
> >
> > The list of bug fixes going into 4.0.0 can be found at the following
> URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12353359
> >
> > This release is using the release script of the tag v4.0.0-rc5.
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate,
> then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the
> Java/Scala
> > you can add the staging repository to your projects resolvers and
> test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 4.0.0 (RC6)

[Peter Toth]

+1

On Wed, May 14, 2025 at 12:07 PM Cheng Pan  wrote:

> +1 (non-binding)
>
> Deployed on a YARN cluster, run some TPC-H queries.
> Passed Apache Kyuubi integration test.
>
> Thanks,
> Cheng Pan
>
>
>
> On May 14, 2025, at 06:28, Wenchen Fan  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 4.0.0.
>
> The vote is open until May 16 (PST) and passes if a majority +1 PMC votes
> are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 4.0.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see https://spark.apache.org/
>
> The tag to be voted on is v4.0.0-rc6 (commit
> 9a99ecb03a2d35f5f38decd686b55511a5c7c535)
> https://github.com/apache/spark/tree/v4.0.0-rc6
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc6-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1484/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc6-docs/
>
> The list of bug fixes going into 4.0.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12353359
>
> This release is using the release script of the tag v4.0.0-rc6.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
>
>

Re: [VOTE] Release Apache Spark Connect Swift Client 0.2.0 (RC1)

[Peter Toth]

+1

Checked checksum, signature and ran some examples.

On Sat, May 17, 2025 at 4:30 PM Dongjoon Hyun  wrote:

> +1.
>
> I checked the checksum and signature and tested with Spark 4.0.0 RC6.
>
> Thanks,
> Dongjoon.
>
> On 2025/05/17 12:39:39 Dongjoon Hyun wrote:
> > Please vote on releasing the following candidate as Apache Spark Connect
> > Swift Client 0.2.0. This vote is open for the next 72 hours and passes
> if a
> > majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark Connect Swift Client 0.2.0
> > [ ] -1 Do not release this package because ...
> >
> > TAG:
> > https://github.com/apache/spark-connect-swift/releases/tag/0.2.0-rc.1
> > (commit: a87fc42fcc30262d7bd34f52ed423b26e23cceaf)
> >
> > RELEASE FILES:
> >
> https://dist.apache.org/repos/dist/dev/spark/spark-connect-swift-0.2.0-rc.1/
> >
> > LIST OF ISSUES:
> > https://issues.apache.org/jira/projects/SPARK/versions/12355907
> >
> > Thanks,
> > Dongjoon.
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 4.0.0 (RC7)

[Peter Toth]

+1

On Tue, May 20, 2025, 07:57 Yang Jie  wrote:

> +1
>
> On 2025/05/20 04:54:50 Yuming Wang wrote:
> > +1
> >
> > On Tue, May 20, 2025 at 12:19 PM Szehon Ho 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Checked signature, checksum, ran basic tests on spark-4.0.0-bin-hadoop3
> > > Thanks
> > > Szehon
> > >
> > >
> > >
> > > On Mon, May 19, 2025 at 9:07 PM Denny Lee 
> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> On Mon, May 19, 2025 at 9:02 PM Rozov, Vlad  >
> > >> wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> Vlad
> > >>>
> > >>> On May 19, 2025, at 8:56 PM, Jules Damji 
> wrote:
> > >>>
> > >>> + 1 (non-binding)
> > >>> —
> > >>> Sent from my iPhone
> > >>> Pardon the dumb thumb typos :)
> > >>>
> > >>> On May 19, 2025, at 5:26 PM, Gengliang Wang 
> wrote:
> > >>>
> > >>> 
> > >>> +1
> > >>>
> > >>> On Mon, May 19, 2025 at 5:21 PM Jungtaek Lim <
> > >>> kabhwan.opensou...@gmail.com> wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  On Tue, May 20, 2025 at 8:47 AM Ruifeng Zheng 
> >  wrote:
> > 
> > > +1
> > >
> > > On Tue, May 20, 2025 at 7:04 AM Hyukjin Kwon  >
> > > wrote:
> > >
> > >> +1
> > >>
> > >> On Mon, 19 May 2025 at 21:27, Wenchen Fan 
> > >> wrote:
> > >>
> > >>> Same as before, I'll start with my own +1.
> > >>>
> > >>> On Mon, May 19, 2025 at 8:25 PM Wenchen Fan  >
> > >>> wrote:
> > >>>
> >  Please vote on releasing the following candidate as Apache Spark
> >  version 4.0.0.
> > 
> >  The vote is open until May 22 (PST) and passes if a majority +1
> PMC
> >  votes are cast, with a minimum of 3 +1 votes.
> > 
> >  [ ] +1 Release this package as Apache Spark 4.0.0
> >  [ ] -1 Do not release this package because ...
> > 
> >  To learn more about Apache Spark, please see
> >  https://spark.apache.org/
> > 
> >  The tag to be voted on is v4.0.0-rc7 (commit
> >  fa33ea000a0bda9e5a3fa1af98e8e85b8cc5e4d4)
> >  https://github.com/apache/spark/tree/v4.0.0-rc7
> > 
> >  The release files, including signatures, digests, etc. can be
> found
> >  at:
> >  https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc7-bin/
> > 
> >  Signatures used for Spark RCs can be found in this file:
> >  https://dist.apache.org/repos/dist/dev/spark/KEYS
> > 
> >  The staging repository for this release can be found at:
> > 
> > 
> https://repository.apache.org/content/repositories/orgapachespark-1485/
> > 
> >  The documentation corresponding to this release can be found at:
> >  https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc7-docs/
> > 
> >  The list of bug fixes going into 4.0.0 can be found at the
> >  following URL:
> >  https://issues.apache.org/jira/projects/SPARK/versions/12353359
> > 
> >  This release is using the release script of the tag v4.0.0-rc7.
> > 
> >  FAQ
> > 
> >  =
> >  How can I help test this release?
> >  =
> > 
> >  If you are a Spark user, you can help us test this release by
> taking
> >  an existing Spark workload and running on this release
> candidate,
> >  then
> >  reporting any regressions.
> > 
> >  If you're working in PySpark you can set up a virtual env and
> >  install
> >  the current RC and see if anything important breaks, in the
> >  Java/Scala
> >  you can add the staging repository to your projects resolvers
> and
> >  test
> >  with the RC (make sure to clean up the artifact cache
> before/after
> >  so
> >  you don't end up building with a out of date RC going forward).
> > 
> > >>>
> > >>>
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Apache Spark K8s Operator 0.2.0 (RC1)

[Peter Toth]

+1

On Sun, May 18, 2025 at 1:29 AM Hyukjin Kwon  wrote:

> +1
>
> On Sun, 18 May 2025 at 07:37, huaxin gao  wrote:
>
>> +1
>> Thanks Dongjoon!
>>
>> On Sat, May 17, 2025 at 2:44 PM L. C. Hsieh  wrote:
>>
>>> +1
>>>
>>> Thanks Dongjoon.
>>>
>>> On Sat, May 17, 2025 at 8:22 AM Dongjoon Hyun 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark K8s
>>> Operator 0.2.0. This vote is open for the next 72 hours and passes if a
>>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark K8s Operator 0.2.0
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > TAG:
>>> >
>>> https://github.com/apache/spark-kubernetes-operator/releases/tag/0.2.0-rc.1
>>> > (commit: 85f17fbe5290db5fda90d7d4324381833d0eefa8)
>>> >
>>> > RELEASE FILES:
>>> >
>>> https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.2.0-rc.1/
>>> >
>>> > LIST OF ISSUES:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12355908
>>> >
>>> > Thanks,
>>> > Dongjoon.
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>

Re: [DISCUSS] New Spark Connect Client repository for Rust language

[Peter Toth]

+1

On Sun, May 18, 2025, 23:27 huaxin gao  wrote:

> +1
>
> On Sun, May 18, 2025 at 1:48 PM Jules Damji  wrote:
>
>> +1 (non-binding)
>> —
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> On May 16, 2025, at 10:22 AM, Yuanjian Li  wrote:
>>
>> 
>>
>> Happy to see the Rust client proposal and glad to help with the follow-up
>> repo management work.
>>
>> Let’s use this thread for voting—posting my +1 here.
>>
>> Jules Damji  ：
>>
>>> I’ll ask them.
>>>
>>> Cheers
>>> Jules
>>>
>>> Excuse the thumb typos
>>>
>>>
>>> On Fri, 16 May 2025 at 2:38 AM, Renjie Liu 
>>> wrote:
>>>
 Thanks Denny and Jules for the response.

 It seems there is no objection about this proposal, would some
 committer/PMC member help to create the repo?

 On Wed, May 14, 2025 at 3:45 AM Jules Damji 
 wrote:

> +1 in this effort.
> —
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On May 9, 2025, at 1:53 AM, Renjie Liu 
> wrote:
>
> 
> Hi, All:
>
> I'd like to propose to add a new Apache Spark repository for `Spark
> Connect Client for Rust`.
>
> https://github.com/apache/spark-connect-
> rust
>
> There are already some efforts for building spark-connect client in
> rust: https://github.com/sjrusso8/spark-connect-rs
>
> We have contacted the author and he's positive in donating this to the
> apache community, see this discussion issue:
> https://github.com/sjrusso8/spark-connect-rs
>
> If the community agrees with creating this new repo, we could build
> the spark-connect-rs client based on existing work. (I don't mean to copy
> codes to the new repo, but it will be based on the existing effort).
>
>

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

[Peter Toth]

+1

On Mon, Jun 2, 2025 at 2:33 PM xianjin  wrote:

> +1.
> Sent from my iPhone
>
> On Jun 2, 2025, at 12:50 PM, DB Tsai  wrote:
>
> +1 looking forward to seeing real-time mode.
> Sent from my iPhone
>
> On Jun 1, 2025, at 9:47 PM, Xiao Li  wrote:
>
> 
> +1
>
> huaxin gao  于2025年6月1日周日 20:00写道：
>
>> +1
>>
>> On Sun, Jun 1, 2025 at 7:50 PM Tathagata Das 
>> wrote:
>>
>>> +1 (binding)
>>> super excited about this!
>>>
>>> On Sun, Jun 1, 2025 at 10:45 PM Yuanjian Li 
>>> wrote:
>>>
 +1

 On Sun, Jun 1, 2025 at 19:00 Dongjoon Hyun 
 wrote:

> +1
>
> Dongjoon
>
>
> On Sun, Jun 1, 2025 at 12:02 L. C. Hsieh  wrote:
>
>> Hi all,
>>
>> I would like to start a vote on the new real-time mode in Apache Spark
>> Structured Streaming.
>>
>> Discussion thread:
>> https://lists.apache.org/thread/ovmfbzfkc3t9odvv5gs75fhpvdckn90f
>> SPIP:
>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?tab=t.0#heading=h.ulas5788cm9t
>> JIRA: https://issues.apache.org/jira/browse/SPARK-52330
>>
>> Please vote on the SPIP for the next 72 hours:
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Release Spark 4.0.1 (RC1)

[Peter Toth]

+1

On Tue, Sep 2, 2025 at 11:49 AM Yang Jie  wrote:

> +1
>
> On 2025/09/02 08:17:17 Max Gekk wrote:
> > +1
> >
> > On Tue, Sep 2, 2025 at 7:48 AM  wrote:
> >
> > > Please vote on releasing the following candidate as Apache Spark
> version
> > > 4.0.1.
> > >
> > > The vote is open until Fri, 05 Sep 2025 22:47:52 PDT and passes if a
> > > majority +1 PMC votes are cast, with
> > > a minimum of 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark 4.0.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see https://spark.apache.org/
> > >
> > > The tag to be voted on is v4.0.1-rc1 (commit 29434ea766b):
> > > https://github.com/apache/spark/tree/v4.0.1-rc1
> > >
> > > The release files, including signatures, digests, etc. can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v4.0.1-rc1-bin/
> > >
> > > Signatures used for Spark RCs can be found in this file:
> > > https://downloads.apache.org/spark/KEYS
> > >
> > > The staging repository for this release can be found at:
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1501/
> > >
> > > The documentation corresponding to this release can be found at:
> > > https://dist.apache.org/repos/dist/dev/spark/v4.0.1-rc1-docs/
> > >
> > > The list of bug fixes going into 4.0.1 can be found at the following
> URL:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12355941
> > >
> > > FAQ
> > >
> > > =
> > > How can I help test this release?
> > > =
> > >
> > > If you are a Spark user, you can help us test this release by taking
> > > an existing Spark workload and running on this release candidate, then
> > > reporting any regressions.
> > >
> > > If you're working in PySpark you can set up a virtual env and install
> > > the current RC via "pip install
> > >
> https://dist.apache.org/repos/dist/dev/spark/v4.0.1-rc1-bin/pyspark-4.0.1.tar.gz
> > > "
> > > and see if anything important breaks.
> > > In the Java/Scala, you can add the staging repository to your project's
> > > resolvers and test
> > > with the RC (make sure to clean up the artifact cache before/after so
> > > you don't end up building with an out of date RC going forward).
> > >
> > > -
> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >
> > >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [DISCUSS] Release Apache Spark 3.5.7

[Peter Toth]

Thank you all for the positive feedback!

On Fri, Sep 12, 2025 at 7:26 AM Jungtaek Lim 
wrote:

> +1 sounds like a plan.
>
> On Wed, Sep 10, 2025 at 6:02 PM Kousuke Saruta  wrote:
>
>> +1
>>
>> 2025年9月10日(水) 17:44 Max Gekk :
>>
>>> +1
>>>
>>> On Wed, Sep 10, 2025 at 8:13 AM Shaoyun Chen  wrote:
>>>
>>>> +1
>>>>
>>>> SPARK-46941[1] also fixed an issue with incorrect results.
>>>>
>>>> 1. https://issues.apache.org/jira/browse/SPARK-46941
>>>>
>>>> Yang Jie  于2025年9月10日周三 11:49写道：
>>>> >
>>>> > +1
>>>> >
>>>> > On 2025/09/10 02:32:29 Wenchen Fan wrote:
>>>> > > +1
>>>> > >
>>>> > > On Wed, Sep 10, 2025 at 4:13 AM Mich Talebzadeh <
>>>> mich.talebza...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Agreed +1
>>>> > > > Dr Mich Talebzadeh,
>>>> > > > Architect | Data Science | Financial Crime | Forensic Analysis |
>>>> GDPR
>>>> > > >
>>>> > > >view my Linkedin profile
>>>> > > > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > On Tue, 9 Sept 2025 at 16:51, Peter Toth 
>>>> wrote:
>>>> > > >
>>>> > > >> Hi dev list,
>>>> > > >>
>>>> > > >> Apache Spark 3.5.6 was released on May 29, 2025, so it's been
>>>> more than 3
>>>> > > >> months.
>>>> > > >> As far as I can see, we have ~40 unreleased commits on the
>>>> branch and 34
>>>> > > >> resolved tickets:
>>>> > > >> https://issues.apache.org/jira/projects/SPARK/versions/12355975
>>>> > > >> including 3 correctness ones:
>>>> > > >>
>>>> https://issues.apache.org/jira/browse/SPARK-52873?jql=fixVersion%20%3D%203.5.7%20and%20project%20%3D%20Spark%20and%20labels%20%3D%20correctness
>>>> > > >>
>>>> > > >> I have an open ticket that I would like to fix this week and
>>>> include in
>>>> > > >> the next 3.5 release:
>>>> https://issues.apache.org/jira/browse/SPARK-53399
>>>> > > >>
>>>> > > >> But my question is, do you think it's time to release 3.5.7? I'm
>>>> happy to
>>>> > > >> volunteer to take the role of release manager if it is.
>>>> > > >>
>>>> > > >> Best,
>>>> > > >> Peter
>>>> > > >>
>>>> > > >>
>>>> > >
>>>> >
>>>> > -
>>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>> >
>>>>
>>>> -
>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>>
>>>>

Re: SPARK-51166 Prepare Apache Spark 4.1.0 for November 2025

[Peter Toth]

Hi,

Yeah, as we will have 3 preview releases out before the first RC, hopefully
the RC period won't take that long.

Best,
Peter

On Tue, Sep 9, 2025 at 7:35 AM Dongjoon Hyun  wrote:

> Hi, Xiao.
>
> Apache Spark project has a world-wide community which is working on
> November and the community already decided to put more efforts via the
> monthly releases. Let me rephrase the community schedule.
>
> Apache Spark 4.1.0-preview1 (2025-09-02)
> Apache Spark 4.1.0-preview2 (2025-10-02)
> Apache Spark 4.1.0-preview3 (2025-11-02)
> Apache Spark 4.1.0 Feature Freeze (2025-11-02)
> Apache Spark 4.1.0 RC1 (2025-11-23)
>
> I'm sure that we have a sufficient plan to handle Spark 4.1.0 for
> November. So, please don't worry.
>
> Dongjoon.
>
> On 2025/09/09 05:23:21 Xiao Li wrote:
> > I have the same concerns as Holden regarding the release timeline. Would
> it
> > make sense to shift our RC to January?
> >
> > Just to clarify, this isn’t an issue with the release manager. The
> > challenge is more about the level of community involvement during the RC
> > stage, and we’ll need stronger engagement from the community to properly
> > test it in different environment and production for higher quality.
> >
> > On Mon, Sep 8, 2025 at 11:32 AM Dongjoon Hyun 
> > wrote:
> >
> > > Thank you, Holden. Yes, it's true and I agree with all your comments.
> > >
> > > At this time, we are in a much better situation because we have Apache
> > > Spark 4.1.0-preview1 already.
> > >
> > > In addition, I expect Apache Spark 4.1.0-preview2 in October. So, the
> > > 4.1.0 release will be smoother than ever.
> > >
> > > I will volunteer as the release manager of Apache Spark 4.1.0 to
> finish it
> > > on time.
> > >
> > > Dongjoon.
> > >
> > >
> > > On Mon, Sep 8, 2025 at 11:25 AM Holden Karau 
> > > wrote:
> > >
> > >> I like that, I think it’s a little tight but if we scope down it
> should
> > >> be fine. Our history of trying to do releases in December is not
> great so
> > >> getting the first RC out the door before then makes sense to me.
> > >>
> > >> Twitter: https://twitter.com/holdenkarau
> > >> Fight Health Insurance: https://www.fighthealthinsurance.com/
> > >> 
> > >> Books (Learning Spark, High Performance Spark, etc.):
> > >> https://amzn.to/2MaRAG9  
> > >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> > >> Pronouns: she/her
> > >>
> > >>
> > >> On Mon, Sep 8, 2025 at 11:21 AM Dongjoon Hyun <
> dongjoon.h...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi, All.
> > >>>
> > >>> As of now, the Apache Spark Versioning Policy page is a little
> outdated
> > >>> because it still shows only the delivered Spark 4.0.0 release window.
> > >>>
> > >>> https://spark.apache.org/versioning-policy.html
> > >>>
> > >>> Since Apache Spark 4.0.0 was announced on May 23rd, I believe we can
> > >>> release 4.1.0 after 6 months from that, which means November 23,
> 2025.
> > >>>
> > >>> So, I'd like to propose to update the Apache Spark Versioning Policy
> > >>> page to show 4.1.0 release windows like the following PR.
> > >>>
> > >>> https://github.com/apache/spark-website/pull/627
> > >>>
> > >>> 2025-11-01 Code Freeze. Release branch cut.
> > >>> 2025-11-15 QA Period.
> > >>> 2025-11-23 Release candidates (RC)
> > >>>
> > >>> For Apache Spark 4.1.0 progress, you can refer to the umbrella JIRA
> > >>> issue and its subtasks. In addition, you can add your on-going JIRA
> issues
> > >>> there too.
> > >>>
> > >>> https://issues.apache.org/jira/browse/SPARK-51166
> > >>> (SPARK-51166 Prepare Apache Spark 4.1.0)
> > >>>
> > >>> WDYT?
> > >>>
> > >>> Thanks,
> > >>> Dongjoon.
> > >>>
> > >>
> >
> > --
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Apache Spark 4.0.1 ?

[Peter Toth]

+1

On Tue, Aug 26, 2025 at 5:10 AM Yang Jie  wrote:

> +1, thank you Dongjoon
>
> Thanks
> Jie Yang
>
> On 2025/08/26 02:52:35 Kent Yao wrote:
> > +1, thank you Dongjoon
> >
> > Cheng Pan  于2025年8月26日周二 10:15写道：
> >
> > > +1, thank you for driving this.
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > >
> > >
> > > On Aug 26, 2025, at 00:31, Dongjoon Hyun 
> wrote:
> > >
> > > Hi, All.
> > >
> > > Since the Apache Spark 4.0.0 tag was created in May, more than three
> > > months have passed.
> > >
> > > https://github.com/apache/spark/releases/tag/v4.0.0 (2025-05-19)
> > >
> > > So far, 124 commits (mostly bug fixes) have been merged into the
> > > branch-4.0 branch.
> > >
> > > $ git log --oneline v4.0.0...HEAD | wc -l
> > > 124
> > >
> > > I think it's time to release v4.0.1 to deliver these bug fixes to
> Apache
> > > Spark 4 users. What do you think?
> > >
> > > I'm also volunteering to serve as the release manager for Apache Spark
> > > 4.0.1. Let me know if you need any additional backport patches. I plan
> to
> > > start the RC vote on 2025-09-02.
> > >
> > > Best regards,
> > > Dongjoon Hyun.
> > >
> > > PS. I don't think any PyPi issues need to block the ASF release
> processes.
> > > So, the official ASF distribution channels and Maven Central are the
> main
> > > target for this release.
> > >
> > >
> > >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

[DISCUSS] Release Apache Spark 3.5.7

[Peter Toth]

Hi dev list,

Apache Spark 3.5.6 was released on May 29, 2025, so it's been more than 3
months.
As far as I can see, we have ~40 unreleased commits on the branch and 34
resolved tickets:
https://issues.apache.org/jira/projects/SPARK/versions/12355975 including 3
correctness ones:
https://issues.apache.org/jira/browse/SPARK-52873?jql=fixVersion%20%3D%203.5.7%20and%20project%20%3D%20Spark%20and%20labels%20%3D%20correctness

I have an open ticket that I would like to fix this week and include in the
next 3.5 release: https://issues.apache.org/jira/browse/SPARK-53399

But my question is, do you think it's time to release 3.5.7? I'm happy to
volunteer to take the role of release manager if it is.

Best,
Peter

[VOTE][RESULT] Release Spark 3.5.7 (RC1)

[Peter Toth]

The vote passes with 11 +1s (5 binding +1s).
Thanks to all who helped with the release!

(* = binding)
+1:
- Zhou Jiang
- Dongjoon Hyun *
- Kazuyuki Tanimura
- Huaxin Gao *
- Kousuke Saruta *
- Wenchen Fan *
- Max Gekk *
- Jungtaek Lim
- Cheng Pan
- Vlad Rozov
- Peter Toth

+0: None

-0: None

Best,
Peter

Re: [VOTE] SPIP: JDBC Driver for Spark Connect

[Peter Toth]

+1 (non-binding)

On Wed, Sep 24, 2025 at 8:04 AM Cheng Pan  wrote:

> +1 (non-binding)
>
> To Nimrod and Mich,
>
> From my understanding, the JDBC API is already a good and de facto
> abstraction layer for user to connect to the different database services
> to execute SQL, the Hive JDBC driver and the proposed Connect JDBC driver
> will use a different package name and can coexist under user's application
> classpath, the right driver will be selected automatically according to
> user provided JDBC URL, given that, I don't see much value to create an
> unified JDBC dirver that support both, this has a few benefits (stripe
> one dependency from user classpath?) but involves much efforts.
>
> Thanks,
> Cheng Pan
>
>
>
> On Sep 24, 2025, at 12:12, Jules Damji  wrote:
>
> +1 (non-binding)
> —
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Sep 23, 2025, at 3:03 AM, Ruifeng Zheng  wrote:
>
> 
> +1
>
> On Tue, Sep 23, 2025 at 5:13 PM Yang Jie  wrote:
>
>> +1
>>
>> Jie Yang
>>
>> On 2025/09/23 05:04:14 Fu Chen wrote:
>> > +1
>> >
>> > On 2025/09/22 13:04:37 杨杰 wrote:
>> > > Hi Spark devs,
>> > >
>> > > I would like to start a vote on the SPIP: JDBC Driver for Spark
>> Connect
>> > >
>> > > Discussion thread:
>> > > https://lists.apache.org/thread/rx5pqh01c86slpqv9161hqwgm5lwxxzq
>> > > SPIP:
>> > >
>> https://docs.google.com/document/d/1Ahk4C16o1Jj1TbLg5ylzgHjvu2Ic2zTrcMuvLjqSoAQ/edit?tab=t.0#heading=h.1gf0bimgty0t
>> > > JIRA: https://issues.apache.org/jira/browse/SPARK-53484
>> > >
>> > > Please vote on the SPIP for the next 72 hours:
>> > >
>> > > [ ] +1: Accept the proposal as an official SPIP
>> > > [ ] +0
>> > > [ ] -1: I don’t think this is a good idea because
>> > >
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>

[ANNOUNCE] Apache Spark 3.5.7 released

[Peter Toth]

We are happy to announce the availability of Apache Spark 3.5.7!

Spark 3.5.7 is the seventh maintenance release containing security and
correctness fixes. This release is based on the branch-3.5 maintenance
branch of Spark. We strongly recommend all 3.5 users to upgrade to this
stable release.

To download Spark 3.5.7, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-5-7.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Peter

Re: [VOTE] Release Apache Spark K8s Operator 0.5.0 (RC1)

[Peter Toth]

+1 (non-binding), thanks Dongjoon!

On Sun, Sep 28, 2025 at 7:44 AM huaxin gao  wrote:

> +1
>
> On Sat, Sep 27, 2025 at 8:28 PM L. C. Hsieh  wrote:
>
>> +1
>>
>> Thanks Dongjoon!
>>
>> On Sat, Sep 27, 2025 at 7:48 PM Augusto Vivaldelli
>>  wrote:
>> >
>> > +1
>> >
>> > El sáb, 27 de sept de 2025, 23:39, Zhou Jiang 
>> escribió:
>> >>
>> >> +1 thanks Dongjoon for driving this release!
>> >>
>> >> Zhou JIANG
>> >>
>> >>
>> >>
>> >> On Sat, Sep 27, 2025 at 19:37 Yang Jie  wrote:
>> >>>
>> >>> +1
>> >>>
>> >>> On 2025/09/27 22:21:14 Dongjoon Hyun wrote:
>> >>> > I'll start with my +1.
>> >>> >
>> >>> > This is tested with the following additionally.
>> >>> > - Java 25
>> >>> > - K8s 1.34
>> >>> > - Apache Spark 4.1.0-preview2 (RC1)
>> >>> >
>> >>> > Dongjoon
>> >>> >
>> >>> > On 2025/09/27 20:44:00 Dongjoon Hyun wrote:
>> >>> > > Please vote on releasing the following candidate as Apache Spark
>> K8s
>> >>> > > Operator 0.5.0. This vote is open for the next 72 hours and
>> passes if a
>> >>> > > majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>> >>> > >
>> >>> > > [ ] +1 Release this package as Apache Spark K8s Operator 0.5.0
>> >>> > > [ ] -1 Do not release this package because ...
>> >>> > >
>> >>> > > TAG:
>> >>> > >
>> https://github.com/apache/spark-kubernetes-operator/releases/tag/0.5.0-rc.1
>> >>> > > (commit: 0d47cfec54457e65aab3b5bcee862d0912ac48ba)
>> >>> > >
>> >>> > > RELEASE FILES:
>> >>> > >
>> https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.5.0-rc.1/
>> >>> > >
>> >>> > > LIST OF ISSUES:
>> >>> > > https://issues.apache.org/jira/projects/SPARK/versions/12356036
>> >>> > >
>> >>> > > Thanks,
>> >>> > > Dongjoon.
>> >>> > >
>> >>> >
>> >>> >
>> -
>> >>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>> >
>> >>> >
>> >>>
>> >>> -
>> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Release Apache Spark Connect Swift Client 0.4.0 (RC1)

[Peter Toth]

+1 (non-binding)

On Sun, Sep 28, 2025 at 5:29 AM L. C. Hsieh  wrote:

> +1
>
> Thanks Dongjoon!
>
> On Sat, Sep 27, 2025 at 2:39 PM Dongjoon Hyun  wrote:
> >
> > I'll start with my +1.
> >
> > This is tested with the following additionally.
> > - Swift 6.2
> > - MacOS 26
> > - Apache Spark 4.1.0-preview2 (RC1)
> >
> > Dongjoon
> >
> > On 2025/09/27 19:40:18 Dongjoon Hyun wrote:
> > > Please vote on releasing the following candidate as Apache Spark
> Connect
> > > Swift Client 0.4.0. This vote is open for the next 72 hours and passes
> if a
> > > majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
> > >
> > > [ ] +1 Release this package as Apache Spark Connect Swift Client 0.4.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > TAG:
> > > https://github.com/apache/spark-connect-swift/releases/tag/0.4.0-rc.1
> > > (commit: 07a276e0b77135894509d5637d8e5447345afe26)
> > >
> > > RELEASE FILES:
> > >
> https://dist.apache.org/repos/dist/dev/spark/spark-connect-swift-0.4.0-rc.1/
> > >
> > > LIST OF ISSUES:
> > > https://issues.apache.org/jira/projects/SPARK/versions/12355992
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [ANNOUNCE] Apache Spark Kubernetes Operator 0.5.0 released

[Peter Toth]

Thank you Dongjoon for driving this!

On Thu, Oct 2, 2025 at 6:28 PM Dongjoon Hyun 
wrote:

> Hi All.
>
> We are happy to announce the availability of Apache Spark Kubernetes
> Operator 0.5.0!
>
> - Website
>   * https://s.apache.org/spark-kubernetes-operator/
>
> - Artifact Hub
>   *
> https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
>
> - Release Note
>   * https://github.com/apache/spark-kubernetes-operator/releases/tag/0.5.0
>   * https://s.apache.org/spark-kubernetes-operator-0.5.0
>
> - Published Docker Image
>   * apache/spark-kubernetes-operator:0.5.0
>
> Cheers,
> Dongjoon.
>

Re: [ANNOUNCE] Apache Spark 3.5.7 released

[Peter Toth]

Hi Kevin,

Publishing to conda forge doesn't seem to be part of our release process:
https://spark.apache.org/release-process.html, so I have no idea how
previous releases ended up there.
Dongjoon, as the release manager or 3.5.5 and 4.0.1, did you manually
publish the files there?

Peter


On Mon, Sep 29, 2025 at 8:04 PM Appel, Kevin  wrote:

> Thank you Peter, is this also going to get published to conda forge?
>
> The 3.5.6 and 3.5.7 isn’t published there, the last was 3.5.5 and then
> 4.0.1 is there
>
> https://anaconda.org/conda-forge/pyspark/files
>
>
>
>
>
> *From:* Peter Toth 
> *Sent:* Thursday, September 25, 2025 2:37 PM
> *To:* dev ; user 
> *Subject:* [ANNOUNCE] Apache Spark 3.5.7 released
>
>
>
> We are happy to announce the availability of Apache Spark 3.5.7!
>
> Spark 3.5.7 is the seventh maintenance release containing security and
> correctness fixes. This release is based on the branch-3.5 maintenance
> branch of Spark. We strongly recommend all 3.5 users to upgrade to this
> stable release.
>
>
> To download Spark 3.5.7, head over to the download page:
> https://spark.apache.org/downloads.html
> <https://urldefense.com/v3/__https:/spark.apache.org/downloads.html__;!!I2XIyG2ANlwasLbx!WpuHLRaC3Dc5vbDq7QzxTFeFNVNgj8sCIop-_R0UhTUEk6lIT2pCli3QBP2Anc0_vZRca3WdlvkbbfQ4MX5r$>
>
> To view the release notes:
> https://spark.apache.org/releases/spark-release-3-5-7.html
> <https://urldefense.com/v3/__https:/spark.apache.org/releases/spark-release-3-5-7.html__;!!I2XIyG2ANlwasLbx!WpuHLRaC3Dc5vbDq7QzxTFeFNVNgj8sCIop-_R0UhTUEk6lIT2pCli3QBP2Anc0_vZRca3WdlvkbbRqzVEFe$>
>
> We would like to acknowledge all community members for contributing to
> this release. This release would not have been possible without you.
>
>
>
> Peter
>
>
> --
> This message, and any attachment(s), is for the intended recipient(s)
> only, may contain information that is privileged, confidential and/or
> proprietary and subject to important terms and conditions available at
> http://www.bankofamerica.com/electronic-disclaimer. If you are not the
> intended recipient, please delete this message. For more information about
> how Bank of America protects your privacy, including specific rights that
> may apply, please visit the following pages:
> https://business.bofa.com/en-us/content/global-privacy-notices.html
> (which includes global privacy notices) and
> https://www.bankofamerica.com/security-center/privacy-overview/ (which
> includes US State specific privacy notices such as the
> http://www.bankofamerica.com/ccpa-notice).
>