Re: [DISCUSS] Automation of RC email

We can always invalidate the vote with -1 in case it is found to be sent incorrectly ... As long as the automation does not end up generating a tonne of mails, that is, it should be fairly manageable :) I am in favor of automating it with option 1. Thanks for driving this Hyukjin ! Regards, Mri

Re: [VOTE] Release Spark 3.5.6 (RC1)

know if you have any > concern! > > On Fri, May 23, 2025 at 11:25 PM Mridul Muralidharan > wrote: > >> Hi Hyukjin, >> >> This thread is to test the automated release, right ? >> Not to actually release it ? >> >> Regards, >> Mridul >>

Re: [VOTE] Release Spark 3.5.6 (RC1)

Hi Hyukjin, This thread is to test the automated release, right ? Not to actually release it ? Regards, Mridul On Fri, May 23, 2025 at 8:26 AM Ruifeng Zheng wrote: > +1 > > On Fri, May 23, 2025 at 5:27 PM Hyukjin Kwon wrote: > >> Please vote on releasing the following candidate as Apache Sp

Re: [DISCUSS] New Spark Connect Client repository for Rust language

Please start an explicit vote thread - so that it is not confusing. This shows us as a DISCUSS thread currently. Regards, Mridul On Mon, May 19, 2025 at 6:44 AM Niranjan Jayakar wrote: > Thank you Renji and Fokko for your responses. Let's move forward with a > separate repo then. > > +1. > > O

Re: [ANNOUNCE] Announcing Apache Spark Kubernetes Operator 0.1.0

I had not checked the release. The release notes mention that Apache Spark 4.0 is supported - which has not yet been released. While I don’t expect drastic changes - and most likely the support which will continue to work, the messaging is not accurate - Mridul On Wed, May 7, 2025 at 8:54 PM Do

Re: [VOTE] SPIP: Add geospatial types to Spark

s in >> Spark, which allows third-party data sources and user-defined functions to >> support geo data types natively when integrating with Spark. The majority >> of geo processing functions will still be in Apache Sedona (or other Spark >> plugin projects) instead of

Re: [VOTE] SPIP: Add geospatial types to Spark

Hi, What was the conclusion of the discussion with Apache Sedona project ? Are they aligned with leveraging the support being built here ? Regards, Mridul On Mon, May 5, 2025 at 10:22 PM Gengliang Wang wrote: > +1 > > On Mon, May 5, 2025 at 6:54 PM Ruifeng Zheng wrote: > >> +1 >> >> On Tue,

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

FWIW, I am +1 on the proposal (though I missed the vote on this !) Regards, Mridul On Fri, Mar 14, 2025 at 1:31 AM Mridul Muralidharan wrote: > > I agree with Mark, imo this is a qualified veto. > We should give Dongjoon the opportunity to give his clarification, if any. > &g

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

I agree with Mark, imo this is a qualified veto. We should give Dongjoon the opportunity to give his clarification, if any. I do realize this delays the RC process, but this deserves to be looked into carefully. Thanks, Mridul On Thu, Mar 13, 2025 at 9:35 PM Mark Hamstra wrote: > Absolutely

Re: Seek for consensus on landing Spark Connect implementation for transformWithState in Spark 4.0.0

features in the > RC phase. (If this change were later than the proposed RC date, I should > have posted to ask for postponing RC a bit.) It unfortunately took time to > review them. > > I hope this could influence your thoughts about this. > > Thanks, > Jungtaek Lim (HeartSaVioR) &

Re: Seek for consensus on landing Spark Connect implementation for transformWithState in Spark 4.0.0

Hi Jungtaek, We are already in RC2 for 4.0, right ? A bit too late for this IMO - we can always introduce it in 4.1 Regards, Mridul On Tue, Mar 4, 2025 at 7:22 AM Herman van Hovell wrote: > +1 > > On Tue, Mar 4, 2025 at 2:07 AM Anish Shrigondekar > wrote: > >> +1 - Would be great to get t

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 Regards, Mridul On Wed, Feb 5, 2025 at 1:06 AM Wenchen Fan wrote: > Hi all, > > Given the positive feedback in the previous DISCUSS email > , I'd > like to start the vote for the proposal "Publish additional Spark > distribut

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 to new distribution mechanisms which will increase Spark adoption ! I do agree with Dongjoon’s concerns that this should not result in slipping the schedule; something to watch out for. Regards, Mridul On Tue, Feb 4, 2025 at 8:07 PM Hyukjin Kwon wrote: > I am fine with providing another o

Re: [VOTE] Use plain text logs by default

+1 Regards, Mridul On Thu, Jan 9, 2025 at 12:05 AM Cheng Pan wrote: > +1 > > Thanks, > Cheng Pan > > > > On Jan 9, 2025, at 12:28, Wenchen Fan wrote: > > Hi all, > > Following the discussion[1], I'd like to start the vote for 'Use plain > text logs by default'. > > Note: This is not to overth

Re: [DISCUSS] Release Apache Spark 3.5.4

+1 for 3.5.4 Regards, Mridul On Wed, Dec 4, 2024 at 1:12 AM Dongjoon Hyun wrote: > +1 for releasing 3.5.4. > > Thank you so much for the head-ups and volunteering as the release > manager, Jie. > > I can help you with the preparation and the steps requiring PMC permission. > > Please feel free

Re: [外部邮件] Re: Spark Connect the default API in Spark 4.0

Hi, I agree with Holden, I am leaning -1 on the proposal as well. Unlike removal of deprecated features, which we align on a major version boundary, changing the default is something we can do in a minor version as well - once there is api parity. Irrespective of which major/minor version we ma

Re: [DISCUSS] Use plain text logs by default

+1 to defaulting to text logs ! Regards, Mridul On Fri, Nov 22, 2024 at 6:21 PM Gengliang Wang wrote: > Hi all, > > Earlier this year, we introduced JSON logging as the default in Spark with > the aim of enhancing log structure and facilitating better analysis. While > this change was made with

Re: [外部邮件] Re: New committer: Bingkun Pan

Congratulations Bingkun ! Regards, Mridul On Sun, Nov 17, 2024 at 12:17 AM Yang,Jie(INF) wrote: > Congratulations, Bingkun ~ > > > > *发件人**: *Hyukjin Kwon > *日期**: *2024年11月17日 星期日 14:05 > *收件人**: *Jungtaek Lim > *抄送**: *Ruifeng Zheng , Dongjoon Hyun < > dongjoon.h...@gmail.com>, dev > *主题**

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1. Regards, Mridul PS: In the past, I did have fun with graphx ... unfortunate that it has come to this :-( On Mon, Sep 30, 2024 at 6:23 PM Sean Owen wrote: > For reasons in the previous thread, yes +1 to deprecation > > On Mon, Sep 30, 2024 at 1:02 PM Holden Karau > wrote: > >> I think it

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 Regards, Mridul On Mon, Sep 30, 2024 at 5:39 PM Reynold Xin wrote: > I don't actually "lead" this. But I don't think this needs to target a > specific Spark version given it should not have any user facing > consequences? > > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun wrote: > >> Thank

Re: [VOTE] Move Variant to Parquet

Hi, What was the conclusions of discussions with Parquet and Iceberg communities on this ? Thanks, Mridul On Mon, Sep 2, 2024 at 12:48 PM Gene Pang wrote: > Hi all, > > I’d like to start a vote for moving the Variant specification and library > to the Parquet project. This allows the Variant

Re: [VOTE] Deprecate SparkR

+1 Regards, Mridul On Wed, Aug 21, 2024 at 11:46 AM Reynold Xin wrote: > +1 > > On Wed, Aug 21, 2024 at 6:42 PM Shivaram Venkataraman < > shivaram.venkatara...@gmail.com> wrote: > >> Hi all >> >> Based on the previous discussion thread [1], I hereby call a vote to >> deprecate the SparkR modu

Re: Welcome new Apache Spark committers

Congratulations ! Regards, Mridul On Mon, Aug 12, 2024 at 7:27 PM Dongjoon Hyun wrote: > Congratulations, Martin, Haejoon, Allison. :) > > Dongjoon > > On Mon, Aug 12, 2024 at 5:19 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new committers. Please join m

Re: Welcoming a new PMC member

Congratulations Kent ! Regards, Mridul On Mon, Aug 12, 2024 at 8:46 PM Dongjoon Hyun wrote: > Congratulations, Kent. > > Dongjoon. > > On Mon, Aug 12, 2024 at 5:22 PM Xiao Li wrote: > >> Congratulations ! >> >> Hyukjin Kwon 于2024年8月12日周一 17:20写道： >> >>> Hi all, >>> >>> The Spark PMC recently

Re: [DISCUSS] Deprecating SparkR

+1 Regards, Mridul On Mon, Aug 12, 2024 at 6:18 PM Hyukjin Kwon wrote: > +1 > > On Tue, Aug 13, 2024 at 7:04 AM Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> And just for the record, the stats that I screenshotted >>

Re: ASF board report for August 2024

Looks good to me, thanks Matei. Regards, Mridul On Sat, Aug 10, 2024 at 7:02 PM Matei Zaharia wrote: > Hi all, > > It’s time to submit our board report soon on August 14th. Here’s a draft, > feel free to leave any comments: > > = > > Description: > > Apache Spark is a fa

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pkubernetes Regards, Mridul On Tue, Jul 23, 2024 at 9:51 PM Kent Yao wrote: > +1(non-binding), I have checked: > > - Download links are OK > - Signatures, Checksums, and the KEYS file are OK > - L

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

It is not necessary for users to create a SparkSession Martin - they can initialize a SparkContext and work with RDD api: which would be what Dongjoon is referring to IMO. Even after Spark Connect GA, I am not in favor of deprecating RDD Api at least until we have parity between both (which we don

Re: [DISCUSS] Release Apache Spark 3.5.2

+1 Thanks for volunteering ! Regards, Mridul On Thu, Jul 11, 2024 at 4:03 AM Kent Yao wrote: > Hi dev, > > It's been approximately 5 months since Feb 23, 2024, when > we released version 3.5.1 for branch-3.5. The patchset differing > from 3.5.1 has grown significantly, now consisting of over 16

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 Regards, Mridul On Tue, Jul 9, 2024 at 10:19 AM Xianjin YE wrote: > +1 > > > On Jul 9, 2024, at 22:41, L. C. Hsieh wrote: > > > > +1 > > > > On Tue, Jul 9, 2024 at 1:13 AM Wenchen Fan wrote: > >> > >> +1 > >> > >> On Tue, Jul 9, 2024 at 10:47 AM Reynold Xin > wrote: > >>> > >>> +1 > >>>

Re: [DISCUSS] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 Regards, Mridul On Mon, Jul 8, 2024 at 8:04 PM Ruifeng Zheng wrote: > +1 > > On Sat, Jul 6, 2024 at 4:45 AM bo yang wrote: > >> +1 This is a great suggestion, thanks Hyukjin! >> >> >> On Thu, Jul 4, 2024 at 4:11 AM Hyukjin Kwon wrote: >> >>> Alright! let me start the vote! >>> >>> On Thu,

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

Hi, Versioned documentation has the benefit that users can have reasonable confidence that features, functionality and examples mentioned will work with that released Spark version. A versionless guide runs into potential issues with deprecation, behavioral changes and new features. My concern

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pkubernetes Regards, Mridul On Sun, Apr 14, 2024 at 11:31 PM Dongjoon Hyun wrote: > I'll start with my +1. > > - Checked checksum and signature > - Checked Scala/Java/R/Python/SQL Document's Spark

Re: Versioning of Spark Operator

I am trying to understand if we can simply align with Spark's version for this ? Makes the release and jira management much more simpler for developers and intuitive for users. Regards, Mridul On Tue, Apr 9, 2024 at 10:09 AM Dongjoon Hyun wrote: > Hi, Liang-Chi. > > Thank you for leading Apa

Re: Apache Spark 3.4.3 (?)

Hi Dongjoon, Thanks for volunteering ! I would suggest to wait for SPARK-47318 to be merged as well for 3.4 Regards, Mridul On Sat, Apr 6, 2024 at 6:49 PM Dongjoon Hyun wrote: > Hi, All. > > Apache Spark 3.4.2 tag was created on Nov 24th and `branch-3.4` has 85 > commits including important

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

--- >>> *From:* Denny Lee >>> *Sent:* Monday, April 1, 2024 10:06:14 AM >>> *To:* Hussein Awala >>> *Cc:* Chao Sun ; Hyukjin Kwon ; >>> Mridul Muralidharan ; dev >>> *Subject:* Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect) >&

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

Can you point me to the SPIP’s discussion thread please ? I was not able to find it, but I was on vacation, and so might have missed this … Regards, Mridul On Sun, Mar 31, 2024 at 9:08 PM Haejoon Lee wrote: > +1 > > On Mon, Apr 1, 2024 at 10:15 AM Hyukjin Kwon wrote: > >> Hi all, >> >> I'd li

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

are some > open discussion threads on the doc you shared. > > @Mridul Muralidharan In what state are your efforts > along this? Is it something that your team is actively pursuing/ building > or are mostly planning right now? Asking so that we can align efforts on > this. >

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

I am supportive of the proposal - this is a step in the right direction ! Additional metadata (explicit and inferred) for log records, and exposing them for indexing is extremely useful. The specifics of the API still need some work IMO and does not need to be this disruptive, but I consider tha

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Gengling, Thanks for sharing this ! I added a few queries to the proposal doc, and we can continue discussing there, but overall I am in favor of this. Regards, Mridul On Fri, Mar 1, 2024 at 1:35 AM Gengliang Wang wrote: > Hi All, > > I propose to enhance our logging system by transition

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

Hi, We are internally exploring adding support for dynamically changing the resource profile of a stage based on runtime characteristics. This includes failures due to OOM and the like, slowness due to excessive GC, resource wastage due to excessive overprovisioning, etc. Essentially handles sca

Re: [VOTE] Release Spark 3.3.4 (RC1)

I am seeing a bunch of python related (43) failures in the sql module (for example [1]) ... I am currently on Python 3.11.6, java 8. Not sure if ubuntu modified anything from under me, thoughts ? I am currently testing this against an older branch to make sure it is not an issue with my desktop.

Re: Apache Spark 3.3.4 EOL Release?

+1 Regards, Mridul On Mon, Dec 4, 2023 at 11:40 AM L. C. Hsieh wrote: > +1 > > Thanks Dongjoon! > > On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: > > > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > > > Jie Yang > > > > On 2023/12/04 15:08:25 Tom Graves wrote: > > > +1 for a 3.3.4 EOL Re

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: > +1(non-binding) > > Jie Yang > > On 2023/11/29 02:08:04 Kent Yao wrote: > > +1(non-binding) > > > > Kent Yao >

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Regards, Mridul On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: > Hi Spark Dev, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > > The SPIP aims to improve the test coverage and develop experience for > Spark UI-related javascript codes. > > This thread will b

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

This should be a very good addition ! Regards, Mridul On Tue, Nov 21, 2023 at 7:46 PM Dongjoon Hyun wrote: > Thank you for proposing a new UI test framework for Apache Spark 4.0. > > It looks very useful. > > Thanks, > Dongjoon. > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > >> Hi Spa

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 Regards, Mridul On Tue, Nov 14, 2023 at 12:45 PM Holden Karau wrote: > +1 > > On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: > >> +1 >> >> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> >> On Nov 14, 2023, at 10:14 AM, Vakaris Baškirov < >> vakaris.bashki...@gmail.com> wro

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations ! Looking forward to more exciting contributions :-) Regards, Mridul On Tue, Oct 3, 2023 at 2:51 AM Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at 1

Re: Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

+1 for moving to a newer version. Thanks for driving this Jie Yang ! Regards, Mridul On Mon, Sep 25, 2023 at 10:15 AM 杨杰 wrote: > Hi all, > > In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit > test framework used in Spark 4.0 from Junit4 to Junit5. > > > Although this i

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC5) as Apache Spark > version 3.5.0. > > The vote is open

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Aug 30, 2023 at 6:10 AM yangjie01 wrote: > Hi, Sean > > > > I have performed testing with Java 17 and Scala 2.13 using maven (`mvn > clean install` and

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Aug 11, 2023 at 2:00 AM Cheng Pan wrote: > +1 (non-binding) > > Passed integration test with Apache Kyuubi. > > Thanks for driving this release. > > Tha

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ! Regards, Mridul On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: > We are happy to announce the availability of Apache Spark 3.4.1! > > Spark 3.4.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.4 maintenance branch of Spark. W

Re: [VOTE][RESULT] Release Spark 3.4.1 (RC1)

A late +1 from me too … forgot to send this yesterday :-) Regards, Mridul On Fri, Jun 23, 2023 at 3:20 AM Dongjoon Hyun wrote: > The vote passes with 15 +1s (10 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Jia Fan > - Dongjoon Hyun * > - Liang-Chi Hsieh

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

I agree with Holden, we should have some understanding of what we are targeting for 4.0, given it is a major ver bump - and work from there on the release date. Regards, Mridul On Mon, Jun 12, 2023 at 8:53 PM Jia Fan wrote: > By the way, like Holden said, what's big feature for 4.0.0? I think v

Re: Apache Spark 3.4.1 Release?

+1, thanks Dongjoon ! Regards, Mridul On Thu, Jun 8, 2023 at 7:16 PM Jia Fan wrote: > +1 > > > > > Jia Fan > > > > 2023年6月9日 08:00，Yuming Wang 写道： > > +1. > > On Fri, Jun 9, 2023 at 7:14 AM Chao Sun wrote: > >> +1 too >> >> On Thu, Jun 8, 2023 at 2:34 PM kazuyuki tani

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Mon, Apr 10, 2023 at 10:34 AM huaxin gao wrote: > +1 > > On Mon, Apr 10, 2023 at 8:17 AM Chao Sun wrote: > >> +1 (non-binding) >> >> On Mon, Apr 10, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Apr 8, 2023 at 12:13 PM L. C. Hsieh wrote: > +1 > > Thanks Xinrong. > > On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > > > +1 > > > > > > > > 发件人:

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good to me. Thanks, Mridul On Tue, Apr 4, 2023 at 1:39 PM huaxin gao wrote: > +1 > > On Tue, Apr 4, 2023 at 11:17 AM Chao Sun wrote: > >> +1 >> >> On Tue, Apr 4, 2023 at 11:12 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Apr 4, 2023 at 11:04 AM L. C. Hsieh wrote: >>> +

Re: Slack for PySpark users

Thanks for flagging the concern Dongjoon, I was not aware of the discussion - but I can understand the concern. Would be great if you or Matei could update the thread on the result of deliberations, once it reaches a logical consensus: before we set up official policy around it. Regards, Mridul

Re: Ammonite as REPL for Spark Connect

getting started > with connect, and/or doing debugging. > > On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan > wrote: > >> >> What is unclear to me is why we are introducing this integration, how >> users will leverage it. >> >> * Are we replacing spark-s

Re: Ammonite as REPL for Spark Connect

pache Spark. > > On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan > wrote: > >> >> Will this be maintained externally or included into Apache Spark ? >> >> Regards , >> Mridul >> >> >> >> On Wed, Mar 22, 2023 at 6:50 PM Herman van H

Re: Ammonite as REPL for Spark Connect

Will this be maintained externally or included into Apache Spark ? Regards , Mridul On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell wrote: > Hi All, > > For Spark Connect Scala Client we are working on making the REPL > experience a bit nicer . In

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Other than the tag issue, the sigs/artifacts/build/etc worked for me. So the next RC candidate looks promising ! Regards, Mridul On Thu, Mar 9, 2023 at 5:07 PM Xinrong Meng wrote: > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if there are no objections. > > On Fri, Mar 10, 2

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ... On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan wrote: > > Thanks Xinrong ! > The signature verifications are fine now ... will continue with testing > the release. > > > Regards, > Mridul > &

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Thanks Xinrong ! The signature verifications are fine now ... will continue with testing the release. Regards, Mridul On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng wrote: > Hi Mridul, > > Would you please try that again? It should work now. > > On Wed, Feb 22, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Hi Xinrong, Was it signed with the same key as present in KEYS [1] ? I am seeing errors with gpg when validating. For example: $ gpg --verify pyspark-3.4.0.tar.gz.asc gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST gpg:

Re: [VOTE] Release Spark 3.3.2 (RC1)

ct in > > https://repository.apache.org/content/repositories/orgapachespark-1433/org/apache/spark/spark-mllib-local_2.13/3.3.2/ > . > Did I miss something? > > Liang-Chi > > On Sat, Feb 11, 2023 at 10:08 AM Mridul Muralidharan > wrote: > > > > > > Hi, > > > &

Re: [VOTE] Release Spark 3.3.2 (RC1)

Hi, The following file is missing in the staging repository - there is a corresponding asc sig file, without the artifact. * org/apache/spark/spark-mllib-local_2.13/3.3.2/spark-mllib-local_2.13-3.3.2-test-sources.jar Can we have this fixed please ? Rest of the signatures, digests, etc check out f

Re: Time for Spark 3.4.0 release?

+1, Thanks ! Regards, Mridul On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: > +1, thanks for driving the release! > > > Gengliang > > On Tue, Jan 3, 2023 at 10:55 PM Dongjoon Hyun > wrote: > >> +1 >> >> Thank you! >> >> Dongjoon >> >> On Tue, Jan 3, 2023 at 9:44 PM Rui Wang wrote: >> >>

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 Regards, Mridul On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang wrote: > +1 > > On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim > wrote: > >> Starting with +1 from me. >> >> On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to s

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

e is not a goal of this project. If that >> happens eventually, that would be a side-effect. Someone may have concerns >> that we have two different projects aiming for similar thing, but I'd >> rather see both projects having competition. If anyone willing to improve >&g

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Hi Jungtaek, Given the goal of the SPIP is reducing latency for stateless apps, and should reasonably fit continuous mode design goals, it feels odd to not support it fin the proposal. I know you have raised concerns about continuous mode in past as well in dev@ list, and we are further ignorin

Re: [VOTE][RESULT] Release Spark 3.2.3, RC1

. Hsieh (*) > - Huaxin Gao (*) > - Kazuyuki Tanimura > - Mridul Muralidharan (*) > - Yuming Wang > - Chris Nauroth > - Yang Jie > - Wenche Fan (*) > - Ruifeng Zheng > - Chao Sun > > +0: None > > -1: None > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 Would be great to see history server performance improvements and lower resource utilization at driver ! Regards, Mridul On Wed, Nov 16, 2022 at 2:38 AM Kent Yao wrote: > +1, non-binding > > Gengliang Wang 于2022年11月16日周三 16:36写道： > > > > Hi all, > > > > I’d like to start a vote for SPIP: "

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Tue, Nov 15, 2022 at 1:00 PM kazuyuki tanimura wrote: > +1 (non-binding) > > Thank you Chao > > Kazu > > >  | Kazuyuki Tanimura | ktanim...@apple.com | +1-408-207-

Re: [VOTE] Release Spark 3.3.1 (RC4)

GitHub Action: > https://github.com/apache/spark/actions?query=branch%3Abranch-3.3 > Apple Silicon Jenkins Farm: > https://apache-spark.s3.fr-par.scw.cloud/BRANCH-3.3.html > > Dongjoon. > > > On Fri, Oct 21, 2022 at 8:48 AM Mridul Muralidharan > wrote: > >> Hi, &

Re: [VOTE] Release Spark 3.3.1 (RC4)

Hi, I saw a couple of test failures I have not observed before: a) FsHistoryProviderSuite - "SPARK-33146: don't let one bad rolling log folder prevent loading other applications" b) MesosClusterSchedulerSuite - "accept/decline offers with driver constraints" I ended up 'ignore''ing them to ma

Re: Welcome Yikun Jiang as a Spark committer

Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM Yuming Wang wrote: > Congratulations Yikun! > > On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently added Yikun Jiang as a committer on the project. >> Yikun is the major contributor of

Re: [VOTE] Release Spark 3.3.1 (RC2)

+1 from me, with a few comments. I saw the following failures, are these known issues/flakey tests ? * PersistenceEngineSuite.ZooKeeperPersistenceEngine Looks like a port conflict issue from a quick look into logs (conflict with starting admin port at 8080) - is this expected behavior for the tes

Re: How to set platform-level defaults for array-like configs?

Hi, Wenchen, would be great if you could chime in with your thoughts - given the feedback you originally had on the PR. It would be great to hear feedback from others on this, particularly folks managing spark deployments - how this is mitigated/avoided in your case, any other pain points with c

Re: Welcoming three new PMC members

Congratulations ! Great to have you join the PMC !! Regards, Mridul On Tue, Aug 9, 2022 at 11:57 AM vaquar khan wrote: > Congratulations > > On Tue, Aug 9, 2022, 11:40 AM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new PMC members. Join me in >> welcoming them t

Re: Welcome Xinrong Meng as a Spark committer

Congratulations Xinrong ! Regards, Mridul On Tue, Aug 9, 2022 at 3:13 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Xinrong Meng as a committer on the project. > Xinrong is the major contributor of PySpark especially Pandas API on Spark. > She has guided a lot of new contrib

Re: [VOTE] Release Spark 3.2.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes" As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails in my env; but other than that, the rest looks good. Regards, Mridul On Tue, Jul 12, 2022

Re: Apache Spark 3.2.2 Release?

+1 Thanks for driving this Dongjoon ! Regards, Mridul On Thu, Jul 7, 2022 at 12:36 AM Gengliang Wang wrote: > +1. > Thank you, Dongjoon. > > On Wed, Jul 6, 2022 at 10:21 PM Wenchen Fan wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 10:41 AM Xinrong Meng >> wrote: >> >>> +1 >>> >>> Thanks! >>> >

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes The test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails; but other than that, rest looks good. Regards, Mridul On Mon, Jun 13, 2022 at 4:25 PM Tom Graves wr

Re: [VOTE] Release Spark 3.3.0 (RC1)

I will also try to get a PR out to fix the first test failure that Sean reported. I will have a PR ready by EOD. Regards, Mridul On Fri, May 6, 2022 at 10:31 AM Gengliang Wang wrote: > Hi Maxim, > > Thanks for the work! > There is a bug fix from Bruce merged on branch-3.3 right after the RC1 i

Re: Apache Spark 3.3 Release

Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two weeks > and try to close off key remaining issues. Final release pending on how > those go, and testing, b

Re: [VOTE] Spark 3.1.3 RC4

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Feb 16, 2022 at 8:32 AM Thomas graves wrote: > +1 > > Tom > > On Mon, Feb 14, 2022 at 2:55 PM Holden Karau wrote: > > > > Please vote on releasing the followi

Re: [VOTE] Spark 3.1.3 RC3

ds, Mridul [1] "The tag to be voted on is v3.2.1-rc1" - the commit hash and git url are correct. On Wed, Feb 2, 2022 at 9:30 AM Mridul Muralidharan wrote: > > Thanks Tom ! > I missed [1] (or probably forgot) the 3.1 part of the discussion given it > centered around 3.2 ..

Re: [VOTE] Spark 3.1.3 RC3

nce lines back at beginning of > December (Dec 6) when we were talking about release 3.2.1. > > Tom > > On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan > wrote: > > > > Hi Holden, > > > > Not that I am against releasing 3.1.3 (given the fixes tha

Re: [VOTE] Spark 3.1.3 RC3

Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache S

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Jan 21, 2022 at 9:01 PM Sean Owen wrote: > +1 with same result as last time. > > On Thu, Jan 20, 2022 at 9:59 PM huaxin gao wrote: > >> Please vote on releasin

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please make

Re: Time for Spark 3.2.1?

+1 for maintenance release, and also +1 for doing this in Jan ! Thanks, Mridul On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang wrote: > +1 for new maintenance releases for all 3.x branches as well. > > On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon wrote: > >> SGTM! >> >> On Wed, 8 Dec 2021 at 09:

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Nice job ! There are some nice API's which should be interesting to explore with JDK 17 :-) Regards. Mridul On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > Cool, thank you Dongjoon. > > On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > >> woot! nice work everyone! :) >> >> On Fri,

Re: Update Spark 3.3 release window?

+1 to EOL 2.x Mid march sounds like a good placeholder for 3.3. Regards, Mridul On Wed, Oct 27, 2021 at 10:38 PM Sean Owen wrote: > Seems fine to me - as good a placeholder as anything. > Would that be about time to call 2.x end-of-life? > > On Wed, Oct 27, 2021 at 9:36 PM Hyukjin Kwon wrote:

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phadoop-2.7 -Pyarn -Pmesos -Pkubernetes. Regards, Mridul On Wed, Oct 6, 2021 at 12:55 PM Michael Heuer wrote: > +1 (non-binding) > >michael > > > On Oct 6, 2021, at 11:49 AM, Gengliang Wang wrote: > > Start

Re: [VOTE] Release Spark 3.2.0 (RC6)

Yi Wu helped identify an issue which causes correctness (duplication) and hangs - waiting for validation to complete before submitting a patch. Regards, Mridul On Wed, Sep 29, 2021 at 11:34 AM Holden Karau wrote: > PySpark smoke tests pass, I'

  1   2   3   >