Re: [VOTE] Release Spark 3.1.1 (RC2)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite: removing this suite gets the build through though - does any

Re: [DISCUSS] assignee practice on committers+ (possible issue on preemption)

I agree, Assignee has been used primarily to give recognition to the contributor who ended up submitting the patch which got merged. Typically jira's remain unassigned - even if it were to be assigned, it conveys no meaning or ownership or ongoing work : IMO it is equivalent to an unassigned jira

Re: [VOTE] Release Spark 3.1.1 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with * org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite * org.apache.spark.sql.kafka010.KafkaDelegationTokenSuit

Re: [VOTE] Release Spark 3.1.1 (RC3)

rk/commit/0d5d248bdc4cdc71627162a3d20c42ad19f24ef4 > and .. KafkaDelegationTokenSuite is flaky ( > https://issues.apache.org/jira/browse/SPARK-31250). > > 2021년 2월 24일 (수) 오후 5:19, Mridul Muralidharan 님이 작성: > >> >> Signatures, digests, etc check out fine. >> Checked out tag and build/tested

Re: [VOTE] Release Spark 3.1.1 (RC3)

different > results between Spark 3.0 and Spark 3.1. We need a few more days to > understand whether these changes are expected. > > Xiao > > > Mridul Muralidharan 于2021年2月24日周三 上午10:41写道： > >> >> Sounds good, thanks for clarifying Hyukjin ! >> +1 on release. >

Re: Apache Spark 3.2 Expectation

Nit: Java 17 -> should be available by Sept 2021 :-) Adoption would also depend on some of our nontrivial dependencies supporting it - it might be a stretch to get it in for Apache Spark 3.2 ? Features: Push based shuffle and disaggregated shuffle should also be in 3.2 Regards, Mridul On T

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Thanks Hyukjin and congratulations everyone on the release ! Regards, Mridul On Tue, Mar 2, 2021 at 8:54 PM Yuming Wang wrote: > Great work, Hyukjin! > > On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote: > >> We are excited to announce Spark 3.1.1 today. >> >> Apache Spark 3.1.1 is the second

Re: Welcoming six new Apache Spark committers

Congratulations, looking forward to more exciting contributions ! Regards, Mridul On Fri, Mar 26, 2021 at 8:21 PM Dongjoon Hyun wrote: > > Congratulations! :) > > Bests, > Dongjoon. > > On Fri, Mar 26, 2021 at 5:55 PM angers zhu wrote: > >> Congratulations >> >> Prashant Sharma 于2021年3月27

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1 Regards, Mridul On Sat, Mar 27, 2021 at 6:09 PM Xiao Li wrote: > +1 > > Xiao > > Takeshi Yamamuro 于2021年3月26日周五 下午4:14写道： > >> +1 (non-binding) >> >> On Sat, Mar 27, 2021 at 4:53 AM Liang-Chi Hsieh wrote: >> >>> +1 (non-binding) >>> >>> >>> rxin wrote >>> > +1. Would open up a huge persona

Re: [VOTE] Release Spark 2.4.8 (RC1)

Do we have a fix for this in 3.x/master which can be backported without too much surrounding change ? Given we are expecting 2.4.7 to probably be the last release for 2.4, if we can fix it, that would be great. Regards, Mridul On Wed, Apr 7, 2021 at 9:31 PM Liang-Chi Hsieh wrote: > Thanks for v

Re: [VOTE] Release Spark 2.4.8 (RC4)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested. Regards, Mridul On Sun, May 9, 2021 at 4:22 PM Liang-Chi Hsieh wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.8. > > The vote is open until May 14th at 9AM PST and passes if a

Re: Resolves too old JIRAs as incomplete

+1, thanks Takeshi ! Regards, Mridul On Wed, May 19, 2021 at 8:48 PM Takeshi Yamamuro wrote: > Hi, dev, > > As you know, we have too many open JIRAs now: > # of open JIRAs=2698: JQL='project = SPARK AND status in (Open, "In > Progress", Reopened)' > > We've recently released v2.4.8(EOL), so I'd

Re: Apache Spark 3.0.3 Release?

+1 Regards, Mridul On Tue, Jun 8, 2021 at 10:11 PM Hyukjin Kwon wrote: > Yeah, +1 > > 2021년 6월 9일 (수) 오후 12:06, Yi Wu 님이 작성: > >> Hi, All. >> >> Since Apache Spark 3.0.2 tag creation (Feb 16), >> new 119 patches (92 issues >> >>

Re: [VOTE] Release Spark 3.0.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes Regards, Mridul PS: Might be related to some quirk of my local env - the first test run (after clean + package) usually fails for me (typically for hive tests) - with a seco

Re: ASF board report draft for August

Hi Matei, 3.2 will also include support for pushed based shuffle (spip SPARK-30602). Regards, Mridul On Mon, Aug 9, 2021 at 9:26 PM Hyukjin Kwon wrote: > > Are you referring to what version of Koala project? 1.8.1? > > Yes, the latest version 1.8.1. > > 2021년 8월 10일 (화) 오전 11:07, Igor Costa

Re: -1s on committed but not released code?

Hi Holden, In the past, I have seen discussions on the merged pr to thrash out the details. Usually it would be clear whether to revert and reformulate the change or concerns get addressed and possibly result in follow up work. This is usually helped by the fact that we typically are conservati

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes I am seeing test failures which are addressed by #33790 - this is in branch-3.2, but after the RC tag. After updating to the h

Re: [VOTE] Release Spark 3.2.0 (RC2)

I have filed a blocker, SPARK-36705 which will need to be addressed. Regards, Mridul On Sun, Sep 5, 2021 at 8:47 AM Gengliang Wang wrote: > Hi all, > > the voting fails. > Liang-Chi reported a new block SPARK-36669 >

Re: [VOTE] Release Spark 3.2.0 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes, this worked fine. I found that including "-Phadoop-2.7" failed on lz4 tests ("native lz4 library not available"). Regards, Mridul On Tue, Sep 21, 2021 at 10:18 AM Gengliang Wang wrote: >

Re: [VOTE] Release Spark 3.2.0 (RC3)

;> On Tue, Sep 21, 2021 at 2:05 PM Chao Sun wrote: >>> >>>> Mridul, is the LZ4 failure about Parquet? I think Parquet currently >>>> uses Hadoop compression codec while Hadoop 2.7 still depends on native lib >>>> for the LZ4. Maybe we should run the t

Re: [VOTE] Release Spark 3.2.0 (RC6)

Yi Wu helped identify an issue which causes correctness (duplication) and hangs - waiting for validation to complete before submitting a patch. Regards, Mridul On Wed, Sep 29, 2021 at 11:34 AM Holden Karau wrote: > PySpark smoke tests pass, I'

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phadoop-2.7 -Pyarn -Pmesos -Pkubernetes. Regards, Mridul On Wed, Oct 6, 2021 at 12:55 PM Michael Heuer wrote: > +1 (non-binding) > >michael > > > On Oct 6, 2021, at 11:49 AM, Gengliang Wang wrote: > > Start

Re: [ANNOUNCE] Apache Spark 3.2.0

Congratulations everyone ! And thanks Gengliang for sheparding the release out :-) Regards, Mridul On Tue, Oct 19, 2021 at 9:25 AM Yuming Wang wrote: > Congrats and thanks! > > On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of

Re: Update Spark 3.3 release window?

+1 to EOL 2.x Mid march sounds like a good placeholder for 3.3. Regards, Mridul On Wed, Oct 27, 2021 at 10:38 PM Sean Owen wrote: > Seems fine to me - as good a placeholder as anything. > Would that be about time to call 2.x end-of-life? > > On Wed, Oct 27, 2021 at 9:36 PM Hyukjin Kwon wrote:

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Nice job ! There are some nice API's which should be interesting to explore with JDK 17 :-) Regards. Mridul On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > Cool, thank you Dongjoon. > > On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > >> woot! nice work everyone! :) >> >> On Fri,

Re: Time for Spark 3.2.1?

+1 for maintenance release, and also +1 for doing this in Jan ! Thanks, Mridul On Tue, Dec 7, 2021 at 11:41 PM Gengliang Wang wrote: > +1 for new maintenance releases for all 3.x branches as well. > > On Wed, Dec 8, 2021 at 8:19 AM Hyukjin Kwon wrote: > >> SGTM! >> >> On Wed, 8 Dec 2021 at 09:

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (binding) This should be a great improvement ! Regards, Mridul On Wed, Jan 12, 2022 at 4:04 AM Kent Yao wrote: > +1 (non-binding) > > Thomas Graves 于2022年1月12日周三 11:52写道： > >> +1 (binding). >> >> One minor note since I haven't had time to look at the implementation >> details is please make

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Jan 21, 2022 at 9:01 PM Sean Owen wrote: > +1 with same result as last time. > > On Thu, Jan 20, 2022 at 9:59 PM huaxin gao wrote: > >> Please vote on releasin

Re: [VOTE] Spark 3.1.3 RC3

Hi Holden, Not that I am against releasing 3.1.3 (given the fixes that have already gone in), but did we discuss releasing it ? I might have missed the thread ... Regards, Mridul On Tue, Feb 1, 2022 at 7:12 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache S

Re: [VOTE] Spark 3.1.3 RC3

nce lines back at beginning of > December (Dec 6) when we were talking about release 3.2.1. > > Tom > > On Wed, Feb 2, 2022 at 2:07 AM Mridul Muralidharan > wrote: > > > > Hi Holden, > > > > Not that I am against releasing 3.1.3 (given the fixes tha

Re: [VOTE] Spark 3.1.3 RC3

ds, Mridul [1] "The tag to be voted on is v3.2.1-rc1" - the commit hash and git url are correct. On Wed, Feb 2, 2022 at 9:30 AM Mridul Muralidharan wrote: > > Thanks Tom ! > I missed [1] (or probably forgot) the 3.1 part of the discussion given it > centered around 3.2 ..

Re: [VOTE] Spark 3.1.3 RC4

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Feb 16, 2022 at 8:32 AM Thomas graves wrote: > +1 > > Tom > > On Mon, Feb 14, 2022 at 2:55 PM Holden Karau wrote: > > > > Please vote on releasing the followi

Re: Apache Spark 3.3 Release

Agree with Sean, code freeze by mid March sounds good. Regards, Mridul On Thu, Mar 3, 2022 at 12:47 PM Sean Owen wrote: > I think it's fine to pursue the existing plan - code freeze in two weeks > and try to close off key remaining issues. Final release pending on how > those go, and testing, b

Re: [VOTE] Release Spark 3.3.0 (RC1)

I will also try to get a PR out to fix the first test failure that Sean reported. I will have a PR ready by EOD. Regards, Mridul On Fri, May 6, 2022 at 10:31 AM Gengliang Wang wrote: > Hi Maxim, > > Thanks for the work! > There is a bug fix from Bruce merged on branch-3.3 right after the RC1 i

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes The test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails; but other than that, rest looks good. Regards, Mridul On Mon, Jun 13, 2022 at 4:25 PM Tom Graves wr

Re: Apache Spark 3.2.2 Release?

+1 Thanks for driving this Dongjoon ! Regards, Mridul On Thu, Jul 7, 2022 at 12:36 AM Gengliang Wang wrote: > +1. > Thank you, Dongjoon. > > On Wed, Jul 6, 2022 at 10:21 PM Wenchen Fan wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 10:41 AM Xinrong Meng >> wrote: >> >>> +1 >>> >>> Thanks! >>> >

Re: [VOTE] Release Spark 3.2.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes" As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in sql.SQLQuerySuite fails in my env; but other than that, the rest looks good. Regards, Mridul On Tue, Jul 12, 2022

Re: Welcome Xinrong Meng as a Spark committer

Congratulations Xinrong ! Regards, Mridul On Tue, Aug 9, 2022 at 3:13 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Xinrong Meng as a committer on the project. > Xinrong is the major contributor of PySpark especially Pandas API on Spark. > She has guided a lot of new contrib

Re: Welcoming three new PMC members

Congratulations ! Great to have you join the PMC !! Regards, Mridul On Tue, Aug 9, 2022 at 11:57 AM vaquar khan wrote: > Congratulations > > On Tue, Aug 9, 2022, 11:40 AM Xiao Li wrote: > >> Hi all, >> >> The Spark PMC recently voted to add three new PMC members. Join me in >> welcoming them t

Re: How to set platform-level defaults for array-like configs?

Hi, Wenchen, would be great if you could chime in with your thoughts - given the feedback you originally had on the PR. It would be great to hear feedback from others on this, particularly folks managing spark deployments - how this is mitigated/avoided in your case, any other pain points with c

Re: [VOTE] Release Spark 3.3.1 (RC2)

+1 from me, with a few comments. I saw the following failures, are these known issues/flakey tests ? * PersistenceEngineSuite.ZooKeeperPersistenceEngine Looks like a port conflict issue from a quick look into logs (conflict with starting admin port at 8080) - is this expected behavior for the tes

Re: Welcome Yikun Jiang as a Spark committer

Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM Yuming Wang wrote: > Congratulations Yikun! > > On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > >> Hi all, >> >> The Spark PMC recently added Yikun Jiang as a committer on the project. >> Yikun is the major contributor of

Re: [VOTE] Release Spark 3.3.1 (RC4)

Hi, I saw a couple of test failures I have not observed before: a) FsHistoryProviderSuite - "SPARK-33146: don't let one bad rolling log folder prevent loading other applications" b) MesosClusterSchedulerSuite - "accept/decline offers with driver constraints" I ended up 'ignore''ing them to ma

Re: [VOTE] Release Spark 3.3.1 (RC4)

GitHub Action: > https://github.com/apache/spark/actions?query=branch%3Abranch-3.3 > Apple Silicon Jenkins Farm: > https://apache-spark.s3.fr-par.scw.cloud/BRANCH-3.3.html > > Dongjoon. > > > On Fri, Oct 21, 2022 at 8:48 AM Mridul Muralidharan > wrote: > >> Hi, &

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Tue, Nov 15, 2022 at 1:00 PM kazuyuki tanimura wrote: > +1 (non-binding) > > Thank you Chao > > Kazu > > >  | Kazuyuki Tanimura | ktanim...@apple.com | +1-408-207-

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 Would be great to see history server performance improvements and lower resource utilization at driver ! Regards, Mridul On Wed, Nov 16, 2022 at 2:38 AM Kent Yao wrote: > +1, non-binding > > Gengliang Wang 于2022年11月16日周三 16:36写道： > > > > Hi all, > > > > I’d like to start a vote for SPIP: "

Re: [VOTE][RESULT] Release Spark 3.2.3, RC1

. Hsieh (*) > - Huaxin Gao (*) > - Kazuyuki Tanimura > - Mridul Muralidharan (*) > - Yuming Wang > - Chris Nauroth > - Yang Jie > - Wenche Fan (*) > - Ruifeng Zheng > - Chao Sun > > +0: None > > -1: None > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

Hi Jungtaek, Given the goal of the SPIP is reducing latency for stateless apps, and should reasonably fit continuous mode design goals, it feels odd to not support it fin the proposal. I know you have raised concerns about continuous mode in past as well in dev@ list, and we are further ignorin

Re: [DISCUSSION] SPIP: Asynchronous Offset Management in Structured Streaming

e is not a goal of this project. If that >> happens eventually, that would be a side-effect. Someone may have concerns >> that we have two different projects aiming for similar thing, but I'd >> rather see both projects having competition. If anyone willing to improve >&g

Re: [VOTE][SPIP] Asynchronous Offset Management in Structured Streaming

+1 Regards, Mridul On Wed, Nov 30, 2022 at 8:55 PM Xingbo Jiang wrote: > +1 > > On Wed, Nov 30, 2022 at 5:59 PM Jungtaek Lim > wrote: > >> Starting with +1 from me. >> >> On Thu, Dec 1, 2022 at 10:54 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to s

Re: Time for Spark 3.4.0 release?

+1, Thanks ! Regards, Mridul On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: > +1, thanks for driving the release! > > > Gengliang > > On Tue, Jan 3, 2023 at 10:55 PM Dongjoon Hyun > wrote: > >> +1 >> >> Thank you! >> >> Dongjoon >> >> On Tue, Jan 3, 2023 at 9:44 PM Rui Wang wrote: >> >>

Re: [VOTE] Release Spark 3.3.2 (RC1)

Hi, The following file is missing in the staging repository - there is a corresponding asc sig file, without the artifact. * org/apache/spark/spark-mllib-local_2.13/3.3.2/spark-mllib-local_2.13-3.3.2-test-sources.jar Can we have this fixed please ? Rest of the signatures, digests, etc check out f

Re: [VOTE] Release Spark 3.3.2 (RC1)

ct in > > https://repository.apache.org/content/repositories/orgapachespark-1433/org/apache/spark/spark-mllib-local_2.13/3.3.2/ > . > Did I miss something? > > Liang-Chi > > On Sat, Feb 11, 2023 at 10:08 AM Mridul Muralidharan > wrote: > > > > > > Hi, > > > &

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Hi Xinrong, Was it signed with the same key as present in KEYS [1] ? I am seeing errors with gpg when validating. For example: $ gpg --verify pyspark-3.4.0.tar.gz.asc gpg: assuming signed data in 'pyspark-3.4.0.tar.gz' gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST gpg:

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

Thanks Xinrong ! The signature verifications are fine now ... will continue with testing the release. Regards, Mridul On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng wrote: > Hi Mridul, > > Would you please try that again? It should work now. > > On Wed, Feb 22, 2023 at

Re: [VOTE] Release Apache Spark 3.4.0 (RC1)

scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ... On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan wrote: > > Thanks Xinrong ! > The signature verifications are fine now ... will continue with testing > the release. > > > Regards, > Mridul > &

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Other than the tag issue, the sigs/artifacts/build/etc worked for me. So the next RC candidate looks promising ! Regards, Mridul On Thu, Mar 9, 2023 at 5:07 PM Xinrong Meng wrote: > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if there are no objections. > > On Fri, Mar 10, 2

Re: Ammonite as REPL for Spark Connect

Will this be maintained externally or included into Apache Spark ? Regards , Mridul On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell wrote: > Hi All, > > For Spark Connect Scala Client we are working on making the REPL > experience a bit nicer . In

Re: Ammonite as REPL for Spark Connect

pache Spark. > > On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan > wrote: > >> >> Will this be maintained externally or included into Apache Spark ? >> >> Regards , >> Mridul >> >> >> >> On Wed, Mar 22, 2023 at 6:50 PM Herman van H

Re: Ammonite as REPL for Spark Connect

getting started > with connect, and/or doing debugging. > > On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan > wrote: > >> >> What is unclear to me is why we are introducing this integration, how >> users will leverage it. >> >> * Are we replacing spark-s

Re: Slack for PySpark users

Thanks for flagging the concern Dongjoon, I was not aware of the discussion - but I can understand the concern. Would be great if you or Matei could update the thread on the result of deliberations, once it reaches a logical consensus: before we set up official policy around it. Regards, Mridul

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good to me. Thanks, Mridul On Tue, Apr 4, 2023 at 1:39 PM huaxin gao wrote: > +1 > > On Tue, Apr 4, 2023 at 11:17 AM Chao Sun wrote: > >> +1 >> >> On Tue, Apr 4, 2023 at 11:12 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Apr 4, 2023 at 11:04 AM L. C. Hsieh wrote: >>> +

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Apr 8, 2023 at 12:13 PM L. C. Hsieh wrote: > +1 > > Thanks Xinrong. > > On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > > > +1 > > > > > > > > 发件人:

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Mon, Apr 10, 2023 at 10:34 AM huaxin gao wrote: > +1 > > On Mon, Apr 10, 2023 at 8:17 AM Chao Sun wrote: > >> +1 (non-binding) >> >> On Mon, Apr 10, 2023 at

Re: Apache Spark 3.4.1 Release?

+1, thanks Dongjoon ! Regards, Mridul On Thu, Jun 8, 2023 at 7:16 PM Jia Fan wrote: > +1 > > > > > Jia Fan > > > > 2023年6月9日 08:00，Yuming Wang 写道： > > +1. > > On Fri, Jun 9, 2023 at 7:14 AM Chao Sun wrote: > >> +1 too >> >> On Thu, Jun 8, 2023 at 2:34 PM kazuyuki tani

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

I agree with Holden, we should have some understanding of what we are targeting for 4.0, given it is a major ver bump - and work from there on the release date. Regards, Mridul On Mon, Jun 12, 2023 at 8:53 PM Jia Fan wrote: > By the way, like Holden said, what's big feature for 4.0.0? I think v

Re: [VOTE][RESULT] Release Spark 3.4.1 (RC1)

A late +1 from me too … forgot to send this yesterday :-) Regards, Mridul On Fri, Jun 23, 2023 at 3:20 AM Dongjoon Hyun wrote: > The vote passes with 15 +1s (10 binding +1s). > Thanks to all who helped with the release! > > (* = binding) > +1: > - Jia Fan > - Dongjoon Hyun * > - Liang-Chi Hsieh

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ! Regards, Mridul On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: > We are happy to announce the availability of Apache Spark 3.4.1! > > Spark 3.4.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.4 maintenance branch of Spark. W

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Fri, Aug 11, 2023 at 2:00 AM Cheng Pan wrote: > +1 (non-binding) > > Passed integration test with Apache Kyuubi. > > Thanks for driving this release. > > Tha

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Aug 30, 2023 at 6:10 AM yangjie01 wrote: > Hi, Sean > > > > I have performed testing with Java 17 and Scala 2.13 using maven (`mvn > clean install` and

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC5) as Apache Spark > version 3.5.0. > > The vote is open

Re: Migrating the Junit framework used in Apache Spark 4.0 from 4.x to 5.x

+1 for moving to a newer version. Thanks for driving this Jie Yang ! Regards, Mridul On Mon, Sep 25, 2023 at 10:15 AM 杨杰 wrote: > Hi all, > > In SPARK-44170 (apache/spark#43074 [1]), I’m trying to migrate the Junit > test framework used in Spark 4.0 from Junit4 to Junit5. > > > Although this i

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations ! Looking forward to more exciting contributions :-) Regards, Mridul On Tue, Oct 3, 2023 at 2:51 AM Hussein Awala wrote: > Congrats to all of you! > > On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: > >> Congratulations! Well deserved! >> >> -Rui >> >> >> On Mon, Oct 2, 2023 at 1

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 Regards, Mridul On Tue, Nov 14, 2023 at 12:45 PM Holden Karau wrote: > +1 > > On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: > >> +1 >> >> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> >> On Nov 14, 2023, at 10:14 AM, Vakaris Baškirov < >> vakaris.bashki...@gmail.com> wro

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

This should be a very good addition ! Regards, Mridul On Tue, Nov 21, 2023 at 7:46 PM Dongjoon Hyun wrote: > Thank you for proposing a new UI test framework for Apache Spark 4.0. > > It looks very useful. > > Thanks, > Dongjoon. > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > >> Hi Spa

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Regards, Mridul On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: > Hi Spark Dev, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > > The SPIP aims to improve the test coverage and develop experience for > Spark UI-related javascript codes. > > This thread will b

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Signatures, digests, etc check out fine. Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes Regards, Mridul On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: > +1(non-binding) > > Jie Yang > > On 2023/11/29 02:08:04 Kent Yao wrote: > > +1(non-binding) > > > > Kent Yao >

Re: Apache Spark 3.3.4 EOL Release?

+1 Regards, Mridul On Mon, Dec 4, 2023 at 11:40 AM L. C. Hsieh wrote: > +1 > > Thanks Dongjoon! > > On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: > > > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > > > Jie Yang > > > > On 2023/12/04 15:08:25 Tom Graves wrote: > > > +1 for a 3.3.4 EOL Re

Re: [VOTE] Release Spark 3.3.4 (RC1)

I am seeing a bunch of python related (43) failures in the sql module (for example [1]) ... I am currently on Python 3.11.6, java 8. Not sure if ubuntu modified anything from under me, thoughts ? I am currently testing this against an older branch to make sure it is not an issue with my desktop.

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

Hi, We are internally exploring adding support for dynamically changing the resource profile of a stage based on runtime characteristics. This includes failures due to OOM and the like, slowness due to excessive GC, resource wastage due to excessive overprovisioning, etc. Essentially handles sca

Re: [DISCUSS] SPIP: Structured Spark Logging

Hi Gengling, Thanks for sharing this ! I added a few queries to the proposal doc, and we can continue discussing there, but overall I am in favor of this. Regards, Mridul On Fri, Mar 1, 2024 at 1:35 AM Gengliang Wang wrote: > Hi All, > > I propose to enhance our logging system by transition

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

Unfortunately I do not have bandwidth to do a detailed review, but a few things come to mind after a quick read: - While it might be tactically beneficial to align with existing implementation, a clean design which does not tie into existing shuffle implementation would be preferable (if it can be

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

Add a +1 from me as well. Just managed to finish going over it. Thanks Bobby for leading this effort ! Regards, Mridul On Wed, May 29, 2019 at 2:51 PM Tom Graves wrote: > > Ok, I'm going to call this vote and send the result email. We had 9 +1's (4 > binding) and 1 +0 and no -1's. > > Tom > >

Re: [DISCUSS] Preferred approach on dealing with SPARK-29322

Makes more sense to drop support for zstd assuming the fix is not something at spark end (configuration, etc). Does not make sense to try to detect deadlock in codec. Regards, Mridul On Tue, Oct 1, 2019 at 8:39 PM Jungtaek Lim wrote: > > Hi devs, > > I've discovered an issue with event logger, s

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Just for completeness sake, spark is not version neutral to hadoop; particularly in yarn mode, there is a minimum version requirement (though fairly generous I believe). I agree with Steve, it is a long standing pain that we are bundling a positively ancient version of hive. Having said that, we s

Re: Is RDD thread safe?

Very well put Imran. This is a variant of executor failure after an RDD has been computed (including caching). In general, non determinism in spark is going to lead to inconsistency. The only reasonable solution for us, at that time, was to make pseudo-randomness repeatable and checkpoint after so

Re: [VOTE] Amend Spark's Semantic Versioning Policy

I am in broad agreement with the prposal, as any developer, I prefer stable well designed API's :-) Can we tie the proposal to stability guarantees given by spark and reasonable expectation from users ? In my opinion, an unstable or evolving could change - while an experimental api which has been

Re: [DISCUSS] filling affected versions on JIRA issue

I agree with what Sean detailed. The only place where I can see some amount of investigation being required would be for security issues or correctness issues. Knowing the affected versions, particularly if an earlier supported version does not have the bug, will help users understand the broken/in

Re: [VOTE] Release Spark 2.4.6 (RC8)

+1 (binding) Thanks, Mridul On Sun, May 31, 2020 at 4:47 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache Spark > version 2.4.6. > > The vote is open until June 5th at 9AM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. >

Re: [VOTE] Release Spark 2.4.6 (RC8)

Is this a behavior change in 2.4.x from earlier version ? Or are we proposing to introduce a functionality to help with adoption ? Regards, Mridul On Wed, Jun 3, 2020 at 10:32 AM Xiao Li wrote: > Yes. Spark 3.0 RC2 works well. > > I think the current behavior in Spark 2.4 affects the adopti

Re: [vote] Apache Spark 3.0 RC3

+1 Regards, Mridul On Sat, Jun 6, 2020 at 1:20 PM Reynold Xin wrote: > Apologies for the mistake. The vote is open till 11:59pm Pacific time on > Mon June 9th. > > On Sat, Jun 6, 2020 at 1:08 PM Reynold Xin wrote: > >> Please vote on releasing the following candidate as Apache Spark version >>

Re: [ANNOUNCE] Apache Spark 3.0.0

Great job everyone ! Congratulations :-) Regards, Mridul On Thu, Jun 18, 2020 at 10:21 AM Reynold Xin wrote: > Hi all, > > Apache Spark 3.0.0 is the first release of the 3.x line. It builds on many > of the innovations from Spark 2.x, bringing new ideas as well as continuing > long-term project

Re: [DISCUSS][SPIP] Graceful Decommissioning

Thanks for shepherding this Holden ! I left a few comments, but overall it looks good to me. Regards, Mridul On Sat, Jun 27, 2020 at 9:34 PM Holden Karau wrote: > There’s been some comments & a few additions in the doc, but it seems like > the folks taking a look generally agree on the desig

Re: [VOTE] Decommissioning SPIP

+1 Thanks, Mridul On Wed, Jul 1, 2020 at 6:36 PM Hyukjin Kwon wrote: > +1 > > 2020년 7월 2일 (목) 오전 10:08, Marcelo Vanzin 님이 작성: > >> I reviewed the docs and PRs from way before an SPIP was explicitly >> asked, so I'm comfortable with giving a +1 even if I haven't really >> fully read the new docu

Re: Welcoming some new Apache Spark committers

Congratulations ! Regards, Mridul On Tue, Jul 14, 2020 at 12:37 PM Matei Zaharia wrote: > Hi all, > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > - Huaxin Gao > - Jungtaek Lim > - Dilip Biswal > >

Re: [DISCUSS] Amend the commiter guidelines on the subject of -1s & how we expect PR discussion to be treated.

Thanks Holden, this version looks good to me. +1 Regards, Mridul On Thu, Jul 23, 2020 at 3:56 PM Imran Rashid wrote: > Sure, that sounds good to me. +1 > > On Wed, Jul 22, 2020 at 1:50 PM Holden Karau wrote: > >> >> >> On Wed, Jul 22, 2020 at 7:39 AM Imran Rashid < iras...@apache.org > >> wr

Re: [DISCUSS] Apache Spark 3.0.1 Release

I agree, that would be a new feature; and unless compelling reason (like security concerns) would not qualify. Regards, Mridul On Wed, Jul 15, 2020 at 11:46 AM Wenchen Fan wrote: > Supporting Python 3.8.0 sounds like a new feature, and doesn't qualify a > backport. But I'm open to other opinion

Re: [VOTE] Update the committer guidelines to clarify when to commit changes.

+1 Thanks, Mridul On Thu, Jul 30, 2020 at 4:49 PM Holden Karau wrote: > Hi Spark Developers, > > After the discussion of the proposal to amend Spark committer guidelines, > it appears folks are generally in agreement on policy clarifications. (See > https://lists.apache.org/thread.html/r6706e97

Re: Push-based shuffle SPIP

Hi, Thanks for sending out the proposal Min ! For the SPIP requirements, I am willing to act as the shepherd for this proposal. The jira + paper + proposal provides the high level design and implementation details. The vldb paper discusses the performance gains in detail for the inhouse deploym

Re: [VOTE] Release Spark 2.4.7 (RC3)

+1 Signatures, digests, etc check out fine. Checked out tag and built/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes Thanks, Mridul On Tue, Sep 8, 2020 at 8:55 AM Prashant Sharma wrote: > Please vote on releasing the following candidate as Apache Spark > versi

  1   2   3   >