Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1, A big thank you to Dongjoon for all the hard work you've put into this! On 2025/05/05 18:19:33 DB Tsai wrote: > +1, it’s exciting to see Spark Connect Swift client, showcasing Spark Connect > as a truly language-agnostic protocol, and also powering Swift users to use > Spark! > > > > Sent

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1, Thanks Dongjoon On 2025/05/06 02:45:40 Kent Yao wrote: > +1, Thank you Dongjoon. > > Kent Yao > > Hyukjin Kwon 于2025年5月6日周二 10:31写道： > > > +1 > > > > On Tue, 6 May 2025 at 10:55, Ruifeng Zheng wrote: > > > >> +1 > >> > >> On Tue, May 6, 2025 at 9:00 AM Wenchen Fan wrote: > >> > >>> +1, t

Re: [VOTE] SPIP: Add geospatial types to Spark

Hi, What was the conclusion of the discussion with Apache Sedona project ? Are they aligned with leveraging the support being built here ? Regards, Mridul On Mon, May 5, 2025 at 10:22 PM Gengliang Wang wrote: > +1 > > On Mon, May 5, 2025 at 6:54 PM Ruifeng Zheng wrote: > >> +1 >> >> On Tue,

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 8:22 PM Gengliang Wang wrote: > > +1 > > On Mon, May 5, 2025 at 6:54 PM Ruifeng Zheng wrote: >> >> +1 >> >> On Tue, May 6, 2025 at 9:51 AM Xiao Li wrote: >>> >>> +1 >>> >>> On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: +1 On Tue, May 6, 2025 at 9

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 6:54 PM Ruifeng Zheng wrote: > +1 > > On Tue, May 6, 2025 at 9:51 AM Xiao Li wrote: > >> +1 >> >> On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: >> >>> +1 >>> >>> On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: >>> +1 (non-binding) On Mon, May 5, 2025

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1, Thank you Dongjoon. Kent Yao Hyukjin Kwon 于2025年5月6日周二 10:31写道： > +1 > > On Tue, 6 May 2025 at 10:55, Ruifeng Zheng wrote: > >> +1 >> >> On Tue, May 6, 2025 at 9:00 AM Wenchen Fan wrote: >> >>> +1, thanks! >>> >>> On Tue, May 6, 2025 at 2:21 AM DB Tsai wrote: >>> +1 On May

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 On Tue, 6 May 2025 at 10:55, Ruifeng Zheng wrote: > +1 > > On Tue, May 6, 2025 at 9:00 AM Wenchen Fan wrote: > >> +1, thanks! >> >> On Tue, May 6, 2025 at 2:21 AM DB Tsai wrote: >> >>> +1 >>> >>> On May 5, 2025, at 1:10 AM, Gabor Somogyi >>> wrote: >>> >>>  >>> +1 (non-binding) >>> >>> G

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 On Tue, May 6, 2025 at 9:00 AM Wenchen Fan wrote: > +1, thanks! > > On Tue, May 6, 2025 at 2:21 AM DB Tsai wrote: > >> +1 >> >> On May 5, 2025, at 1:10 AM, Gabor Somogyi >> wrote: >> >>  >> +1 (non-binding) >> >> G >> >> >> On Mon, May 5, 2025 at 8:59 AM huaxin gao wrote: >> >>> +1 Thanks

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Tue, May 6, 2025 at 9:51 AM Xiao Li wrote: > +1 > > On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: > >> +1 >> >> On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: >> >>> +1 (non-binding) >>> >>> On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: >>> +1 On Tue, May 6, 2025 at

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (Non-biding? Excuse the thumb typos On Mon, 05 May 2025 at 6:50 PM, Xiao Li wrote: > +1 > > On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: > >> +1 >> >> On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: >> >>> +1 (non-binding) >>> >>> On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: >>> >>

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: > +1 > > On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: > >> +1 (non-binding) >> >> On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: >> >>> +1 >>> >>> On Tue, May 6, 2025 at 3:40 AM Reynold Xin >>> wrote: >>> +1 On Mon, Ma

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: > +1 (non-binding) > > On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: > >> +1 >> >> On Tue, May 6, 2025 at 3:40 AM Reynold Xin >> wrote: >> >>> +1 >>> >>> >>> On Mon, May 5, 2025 at 12:37 PM Bjørn Jørgensen < >>> bjornjorgen...@gmail.com> wro

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (non-binding) On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: > +1 > > On Tue, May 6, 2025 at 3:40 AM Reynold Xin > wrote: > >> +1 >> >> >> On Mon, May 5, 2025 at 12:37 PM Bjørn Jørgensen >> wrote: >> >>> +1 >>> >>> man. 5. mai 2025 kl. 21:28 skrev Milan Stefanovic < >>> stefanovic.mila...@

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Tue, May 6, 2025 at 3:40 AM Reynold Xin wrote: > +1 > > > On Mon, May 5, 2025 at 12:37 PM Bjørn Jørgensen > wrote: > >> +1 >> >> man. 5. mai 2025 kl. 21:28 skrev Milan Stefanovic < >> stefanovic.mila...@gmail.com>: >> >>> +1 (non-binding) >>> >>> Thanks, >>> Milan >>> >>> On Mon, 5 May 20

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1, thanks! On Tue, May 6, 2025 at 2:21 AM DB Tsai wrote: > +1 > > On May 5, 2025, at 1:10 AM, Gabor Somogyi > wrote: > >  > +1 (non-binding) > > G > > > On Mon, May 5, 2025 at 8:59 AM huaxin gao wrote: > >> +1 Thanks Dongjoon. >> >> On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad >> wrote: >> >>

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 12:37 PM Bjørn Jørgensen wrote: > +1 > > man. 5. mai 2025 kl. 21:28 skrev Milan Stefanovic < > stefanovic.mila...@gmail.com>: > >> +1 (non-binding) >> >> Thanks, >> Milan >> >> On Mon, 5 May 2025 at 21:25, Jia Yu wrote: >> >>> Thanks for putting this together. >>> >>>

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 man. 5. mai 2025 kl. 21:28 skrev Milan Stefanovic < stefanovic.mila...@gmail.com>: > +1 (non-binding) > > Thanks, > Milan > > On Mon, 5 May 2025 at 21:25, Jia Yu wrote: > >> Thanks for putting this together. >> >> +0 (non-binding) from my side. Happy to see geospatial data is getting >> atten

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (non-binding) Thanks, Milan On Mon, 5 May 2025 at 21:25, Jia Yu wrote: > Thanks for putting this together. > > +0 (non-binding) from my side. Happy to see geospatial data is getting > attention but we need to make it right. > > > Jia Yu > > > > On Mon, May 5, 2025 at 12:15 PM Szehon Ho wrot

Re: [VOTE] SPIP: Add geospatial types to Spark

Thanks for putting this together. +0 (non-binding) from my side. Happy to see geospatial data is getting attention but we need to make it right. Jia Yu On Mon, May 5, 2025 at 12:15 PM Szehon Ho wrote: > +1 (non binding) > > Thanks > Szehon > > On Mon, May 5, 2025 at 11:17 AM DB Tsai wrote:

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (non binding) Thanks Szehon On Mon, May 5, 2025 at 11:17 AM DB Tsai wrote: > +1, geospatial types will be a great feature for Spark. Thanks for working > on it. > > On May 5, 2025, at 11:04 AM, Menelaos Karavelas < > menelaos.karave...@gmail.com> wrote: > > I started the discussion on addin

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1On May 5, 2025, at 1:10 AM, Gabor Somogyi wrote:+1 (non-binding)GOn Mon, May 5, 2025 at 8:59 AM huaxin gao wrote:+1 Thanks Dongjoon.On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad wrote:+1 (not binding) Checked checksum and signatures, build,  and confirmed that binary fil

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1, it’s exciting to see Spark Connect Swift client, showcasing Spark Connect as a truly language-agnostic protocol, and also powering Swift users to use Spark!Sent from my iPhoneOn May 5, 2025, at 1:11 AM, Gabor Somogyi wrote:+1 (non-binding)GOn Mon, May 5, 2025 at 8:35 AM huaxin gao

Re: [VOTE] SPIP: Add geospatial types to Spark

+1, geospatial types will be a great feature for Spark. Thanks for working on it. > On May 5, 2025, at 11:04 AM, Menelaos Karavelas > wrote: > > I started the discussion on adding geospatial types to Spark on March 28th. > Since then there has been some discussion in the dev mailing list, as

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

Not sure this counts as -1, but by cursory checking the code, I found that the way the TLS connection is set up is not always working: https://github.com/apache/spark-connect-swift/blob/v0.1.0-rc1/Sources/SparkConnect/DataFrame.swift#L276-L288 Shows that DataFrame operations explicitly set plaint

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 (non-binding) G On Mon, May 5, 2025 at 8:35 AM huaxin gao wrote: > +1 Thanks Dongjoon. > > On Sun, May 4, 2025 at 5:21 PM Dongjoon Hyun wrote: > >> +1 >> >> I checked the checksum and signatures, and tested with Apache Spark 4.0.0 >> RC4 on Swift 6.1. >> >> This is the initial release (v0.1

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 (non-binding) G On Mon, May 5, 2025 at 8:59 AM huaxin gao wrote: > +1 Thanks Dongjoon. > > On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad > wrote: > >> +1 (not binding) >> >> Checked checksum and signatures, build, and confirmed that binary files >> are not included into the release. >> >> Is

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 (non-binding) Kazu > On May 4, 2025, at 11:57 PM, huaxin gao wrote: > > +1 Thanks Dongjoon. > > On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad wrote: >> +1 (not binding) >> >> Checked checksum and signatures, build, and confirmed that binary files are >> not included into the release. >> >

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 (non-binding) Kazu > On May 4, 2025, at 11:31 PM, huaxin gao wrote: > > +1 Thanks Dongjoon. > > On Sun, May 4, 2025 at 5:21 PM Dongjoon Hyun > wrote: >> +1 >> >> I checked the checksum and signatures, and tested with Apache Spark 4.0.0 >> RC4 on Swift 6.1. >>

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 Thanks Dongjoon. On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad wrote: > +1 (not binding) > > Checked checksum and signatures, build, and confirmed that binary files > are not included into the release. > > Is Apache RAT part of the gradle build? If not, how headers are validated > to include co

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 Thanks Dongjoon. On Sun, May 4, 2025 at 5:21 PM Dongjoon Hyun wrote: > +1 > > I checked the checksum and signatures, and tested with Apache Spark 4.0.0 > RC4 on Swift 6.1. > > This is the initial release (v0.1) with 105 patches to provide a tangible > release to the users. > > v0.2 is under p

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 On Sun, May 4, 2025 at 3:15 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark Connect > Swift Client 0.1.0. This vote is open for the next 72 hours and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release th

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 On Sun, May 4, 2025 at 4:58 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark K8s Operator > 0.1.0. This vote is open for the next 72 hours and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this packa

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 (not binding) Checked checksum and signatures, build, and confirmed that binary files are not included into the release. Is Apache RAT part of the gradle build? If not, how headers are validated to include correct license? Thank you, Vlad > On May 4, 2025, at 5:38 PM, Dongjoon Hyun wr

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 I checked the checksum and signatures, and tested with K8s v1.32. Dongjoon. On 2025/05/04 23:58:54 Zhou Jiang wrote: > +1 , thanks for driving this release! > > *Zhou JIANG* > > > > On Sun, May 4, 2025 at 16:58 Dongjoon Hyun wrote: > > > Please vote on releasing the following candidate

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 I checked the checksum and signatures, and tested with Apache Spark 4.0.0 RC4 on Swift 6.1. This is the initial release (v0.1) with 105 patches to provide a tangible release to the users. v0.2 is under planning in SPARK-51999. Dongjoon. On 2025/05/04 22:14:54 Dongjoon Hyun wrote: > Please

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 , thanks for driving this release! *Zhou JIANG* On Sun, May 4, 2025 at 16:58 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark K8s > Operator 0.1.0. This vote is open for the next 72 hours and passes if a > majority +1 PMC votes are cast, with a minimu

Re: Issue with Spark 4.0.0rc4 and ~/.ivy2.5.2

Does the following options works for you? ./bin/spark-shell --conf spark.jars.ivy=${HOME}/.ivy2 ./bin/spark-shell --conf spark.jars.ivy=/Users/yourname/.ivy2 I think the issue is that ~ is not interpreted by shell and just passthrough to the Ivy lib. Thanks, Cheng Pan > On Apr 29, 2025, at 0

Re: Issue with Spark 4.0.0rc4 and ~/.ivy2.5.2

Hi Jacek, Thanks for the confirmation! Let's change the wording first, and open a JIRA ticket for the relative path support. Wenchen On Tue, Apr 29, 2025 at 2:41 AM Jacek Laskowski wrote: > Hi Wenchen, > > Looks like it didn't work in 3.5 either. > > ❯ ./bin/spark-shell --version > 25/04/28 20

Re: Issue with Spark 4.0.0rc4 and ~/.ivy2.5.2

Hi Wenchen, Looks like it didn't work in 3.5 either. ❯ ./bin/spark-shell --version 25/04/28 20:37:48 WARN Utils: Your hostname, Jaceks-Mac-mini.local resolves to a loopback address: 127.0.0.1; using 192.168.68.100 instead (on interface en1) 25/04/28 20:37:48 WARN Utils: Set SPARK_LOCAL_IP if you

Re: Issue with Spark 4.0.0rc4 and ~/.ivy2.5.2

Hi Jacek, Thanks for reporting the issue! Did you hit the same problem when you set the `spark.jars.ivy` config with Spark 3.5? If this config never worked with a relative path, we should change the wording in the migration guide. Thanks, Wenchen On Sun, Apr 27, 2025 at 10:27 PM Jacek Laskowski

Re: [VOTE] Release Spark 4.0.0 (RC4)

One more small fix (on another topic) for the next RC: https://github.com/apache/spark/pull/50685 Thanks! Szehon On Tue, Apr 22, 2025 at 10:07 AM Rozov, Vlad wrote: > Correct, to me it looks like a Spark bug > https://issues.apache.org/jira/browse/SPARK-51821 that may be hard to > trigger and i

Re: [VOTE] Release Spark 4.0.0 (RC4)

Correct, to me it looks like a Spark bug https://issues.apache.org/jira/browse/SPARK-51821 that may be hard to trigger and is reproduce using the test case provided in https://github.com/apache/spark/pull/50594: 1. Spark UninterruptibleThread “task” is interrupted by “test” thread while “task”

Re: [VOTE] Release Spark 4.0.0 (RC4)

Correct me if I'm wrong: this is a long-standing Spark bug that is very hard to trigger, but the new Parquet version happens to hit the trigger condition and exposes the bug. If this is the case, I'm +1 to fix the Spark bug instead of downgrading the Parquet version. Let's move the technical discu

Re: [VOTE] Release Spark 4.0.0 (RC4)

I don't think PARQUET-2432 has any issue itself. It looks to have triggered a deadlock case like https://github.com/apache/spark/pull/50594. I'd suggest that we fix forward if possible. Thanks, Manu On Mon, Apr 21, 2025 at 11:19 PM Rozov, Vlad wrote: > The deadlock is reproducible without Parqu

Re: [VOTE] Release Spark 4.0.0 (RC4)

The deadlock is reproducible without Parquet. Please see https://github.com/apache/spark/pull/50594. Thank you, Vlad On Apr 21, 2025, at 1:59 AM, Cheng Pan wrote: The deadlock is introduced by PARQUET-2432(1.14.0), if we decide downgrade, the latest workable version is Parquet 1.13.1. Thank

Re: [VOTE] Release Spark 4.0.0 (RC4)

The deadlock is introduced by PARQUET-2432(1.14.0), if we decide downgrade, the latest workable version is Parquet 1.13.1. Thanks, Cheng Pan > On Apr 21, 2025, at 16:53, Wenchen Fan wrote: > > +1 to downgrade to Parquet 1.15.0 for Spark 4.0. According to > https://github.com/apache/spark/pu

Re: [VOTE] Release Spark 4.0.0 (RC4)

+1 to downgrade to Parquet 1.15.0 for Spark 4.0. According to https://github.com/apache/spark/pull/50583#issuecomment-2815243571 , the Parquet CVE does not affect Spark. On Mon, Apr 21, 2025 at 2:45 PM Hyukjin Kwon wrote: > That's nice but we need to wait for them to release, and upgrade right?

Re: [VOTE] Release Spark 4.0.0 (RC4)

It seems this patch(https://github.com/apache/parquet-java/pull/3196) can avoid deadlock issue if using Parquet 1.15.1. On Wed, Apr 16, 2025 at 5:39 PM Niranjan Jayakar wrote: > I found another bug introduced in 4.0 that breaks Spark connect client x > server compatibility: https://github.com/ap

Re: [VOTE] Release Spark 4.0.0 (RC4)

That's nice but we need to wait for them to release, and upgrade right? Let's revert the parquet upgrade out of 4.0 branch since we're not directly affected by the CVE anyway. On Mon, 21 Apr 2025 at 15:42, Yuming Wang wrote: > It seems this patch(https://github.com/apache/parquet-java/pull/3196)

Re: [VOTE] Release Spark 4.0.0 (RC4)

I found another bug introduced in 4.0 that breaks Spark connect client x server compatibility: https://github.com/apache/spark/pull/50604. Once merged, this should be included in the next RC. On Thu, Apr 10, 2025 at 5:21 PM Wenchen Fan wrote: > Please vote on releasing the following candidate a

Re: [VOTE] Release Spark 4.0.0 (RC4)

It may not be the Parquet introduced issue. It looks like a race condition between Spark UninterruptibleThread and Hadoop/HDFS DFSOutputStream. I tried to resolve the deadlock in https://github.com/apache/spark/pull/50594. Can you give it a try? I will see if I can reproduce the deadlock in a un

Re: [VOTE] Release Spark 4.0.0 (RC4)

This release uses Parquet 1.15.1. It seems Parquet 1.15.1 may cause deadlock. Found one Java-level deadlock: = "Executor 566 task launch worker for task 202024534, task 19644.1 in stage 13967543.0 of app application_1736396393732_100191": waiting to lock monitor 0

Re: [VOTE] Release Spark 4.0.0 (RC4)

I have reported this issue to the Parquet community: https://github.com/apache/parquet-java/issues/3193 On Tue, Apr 15, 2025 at 9:47 AM Wenchen Fan wrote: > Hi Yuming, > > 1.51.1 is the latest release of Apache Parquet for the 1.x line. Is it a > known issue the Parquet community is working on,

Re: [VOTE] Release Spark 4.0.0 (RC4)

Hi Yuming, 1.51.1 is the latest release of Apache Parquet for the 1.x line. Is it a known issue the Parquet community is working on, or are you still investigating it? If the issue is confirmed by the Parquet community, we can probably roll back to the previous Parquet version for Spark 4.0. Than

Re: [VOTE] SPIP: Declarative Pipelines

Yuming, you replied to a wrong thread mistakenly I suspect :-). On Tue, 15 Apr 2025 at 08:13, Yuming Wang wrote: > This release uses Parquet 1.15.1. It seems Parquet 1.15.1 may cause > deadlocks. > > > Found one Java-level deadlock: > > = > > "Executor 566 task launch

Re: [VOTE] SPIP: Declarative Pipelines

This release uses Parquet 1.15.1. It seems Parquet 1.15.1 may cause deadlocks. Found one Java-level deadlock: = "Executor 566 task launch worker for task 202024534, task 19644.1 in stage 13967543.0 of app application_1736396393732_100191": waiting to lock monitor

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

Thanks for noticing, Jungtaek. No worries, Jacky. Amended total: 30 +1s (14 binding +1s) and no -1s. On Sun, Apr 13, 2025 at 4:37 AM Jacky Lee wrote: > I'm just interested in this, but I'm not a PMC member of Apache Spark, > sorry for mistake. > > Jungtaek Lim 于2025年4月13日周日 15:05写道： > > > > C

Re: [VOTE] Release Spark 4.0.0 (RC4)

Made a fix at https://github.com/apache/spark/pull/50575 👍 On Mon, 14 Apr 2025 at 11:42, Wenchen Fan wrote: > I'm testing the new spark-connect distribution and here is the result: > > 4 packages are tested: pip install pyspark, pip install pyspark_connect (I > installed them with the RC4 pyspar

Re: [VOTE] Release Spark 4.0.0 (RC4)

I'm testing the new spark-connect distribution and here is the result: 4 packages are tested: pip install pyspark, pip install pyspark_connect (I installed them with the RC4 pyspark tarballs), the classic tarball (spark-4.0.0-bin-hadoop3.tgz), the connect tarball (spark-4.0.0-bin-hadoop3-spark-con

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

Sorry, just a mistake. Jungtaek Lim 于2025年4月13日周日 15:05写道： > > Congrats! > > Btw, I'm not 100% sure the vote from Jacky Lee is properly counted. I see > you've just counted from Jacky Lee's content, but I'm not sure Jacky Lee is > listed under PMC membership. > > https://people.apache.org/phone

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

I'm just interested in this, but I'm not a PMC member of Apache Spark, sorry for mistake. Jungtaek Lim 于2025年4月13日周日 15:05写道： > > Congrats! > > Btw, I'm not 100% sure the vote from Jacky Lee is properly counted. I see > you've just counted from Jacky Lee's content, but I'm not sure Jacky Lee is

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

Congrats! Btw, I'm not 100% sure the vote from Jacky Lee is properly counted. I see you've just counted from Jacky Lee's content, but I'm not sure Jacky Lee is listed under PMC membership. https://people.apache.org/phonebook.html?pmc=spark @Jacky Lee Could you please clarify whether you have a

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Fri, Apr 11, 2025 at 3:47 AM Ruifeng Zheng wrote: > +1 > > On Fri, Apr 11, 2025 at 12:37 PM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> +1 (non-binding) >> >> On Thu, Apr 10, 2025 at 6:52 PM Liu Cao wrote: >> >>> +1 (non-binding) >>> >>> On Thu, Apr 10, 2025

Re: SPARK ON KUBERNETS IS SLOW

{ "emoji": "👍", "version": 1 }

Re: SPARK ON KUBERNETS IS SLOW

Thanks Karan for input but found  the issue of slowness in kubernetes while doing broadcast it takes 2 to 3 times than YARN . need to check why such a big difference. In our case we are doing around 40 tables manual broadcast as the size of tables is more than 10 mb. What can be done in kubernetes

Re: SPARK ON KUBERNETS IS SLOW

Pls check if there are resource constraints on the pods/nodes especially if they are shared. MinIO connectivity performance needs to be checked. With YARN and External Spark Shuffle, the sparkshuffle is a lot more optimized, so we can experience slowness with spark on k8s, especially if there is a

Re: SPARK ON KUBERNETS IS SLOW

Hello Karan,I am using Spark open source in kubernetes and Spark mapr bundle in YARN.For launching job in both approach it takes same 10 secs .For shuffle I am using local in both yarn and kubernetes.Sent from my iPhoneOn Apr 11, 2025, at 11:24 AM, karan alang wrote:Hi Prem,Which distribution of

Re: SPARK ON KUBERNETS IS SLOW

Hi Prem, Which distribution of Spark are you using ? how long does it take to launch the job ? wrt Spark Shuffle, what is the approach you are using - storing shuffle data in MinIO or using host path ? regds, Karan On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo wrote: > Hello Team, > I have a pecu

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Fri, Apr 11, 2025 at 12:37 PM Walaa Eldin Moustafa wrote: > +1 (non-binding) > > On Thu, Apr 10, 2025 at 6:52 PM Liu Cao wrote: > >> +1 (non-binding) >> >> On Thu, Apr 10, 2025 at 9:51 AM Prashant Singh >> wrote: >> >>> +1 (non-binding) >>> >>> On Thu, Apr 10, 2025 at 9:46 AM Xiao Li wr

Re: [DISCUSS] SPIP: Declarative Pipelines

This sounds quite interesting. +1 to What Szheon said about excitement around MVs. Happy to collaborate. On Wed, Apr 9, 2025 at 5:29 PM Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com> wrote: > +1 (non-binding) > > El jue, 10 abr 2025, 1:50, Burak Yavuz escribió: > >> +1 >> >> On Wed, Apr

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Thu, Apr 10, 2025 at 6:52 PM Liu Cao wrote: > +1 (non-binding) > > On Thu, Apr 10, 2025 at 9:51 AM Prashant Singh > wrote: > >> +1 (non-binding) >> >> On Thu, Apr 10, 2025 at 9:46 AM Xiao Li wrote: >> >>> +1 (binding) >>> >>> On Thu, Apr 10, 2025 at 00:57 John Zhuge wrote:

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Thu, Apr 10, 2025 at 9:51 AM Prashant Singh wrote: > +1 (non-binding) > > On Thu, Apr 10, 2025 at 9:46 AM Xiao Li wrote: > >> +1 (binding) >> >> On Thu, Apr 10, 2025 at 00:57 John Zhuge wrote: >> >>> +1 (non-binding) >>> >>> On Wed, Apr 9, 2025 at 9:11 PM Jacky Lee wrote:

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) On April 9, 2025 7:29:40 AM GMT+02:00, Rishab Joshi wrote: >+1 Exciting. >Rishab Joshi > >On Tue, Apr 8, 2025, 10:04 PM Ruifeng Zheng wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 12:57 PM Denny Lee wrote: >> >>> +1 (non-binding) >>> >>> On Tue, Apr 8, 2025 at 9:53 PM Yuming Wan

Re: [DISCUSS] SPIP: Declarative Pipelines

Hi Khalid – the CLI in the current proposal will need to be built on top of internal APIs for constructing and launching pipeline executions. We'll have the option to expose these in the future. It would be worthwhile to understand the use cases in more depth before exposing these, because APIs ar

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) Kent Yao Yang Jie 于2025年4月10日周四 10:27写道： > +1 (binding) > > On 2025/04/10 02:20:02 Cheng Pan wrote: > > +1 (non-binding) > > > > Thanks, > > Cheng Pan > > > > > > > > > On Apr 9, 2025, at 22:22, Sandy Ryza wrote: > > > > > > We started to get some votes on the discussion thread, s

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) On Tue, Apr 8, 2025 at 9:53 PM Yuming Wang wrote: > +1 > > On Wed, Apr 9, 2025 at 10:47 AM Jungtaek Lim > wrote: > >> +1 looking forward to seeing this make progress! >> >> On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: >> >>> +1 >>> >>> On 2025/04/09 01:07:57 Hyukjin Kwon wr

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 looking forward to seeing this make progress! On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: > +1 > > On 2025/04/09 01:07:57 Hyukjin Kwon wrote: > > +1 > > > > I am actually pretty excited to have this. Happy to see this being > proposed. > > > > On Wed, 9 Apr 2025 at 01:55, Chao Sun wrote:

Re: Security model update

Sure, how about here though? https://github.com/apache/spark-website/pull/602 On Mon, Apr 7, 2025 at 9:30 AM Arnout Engelen wrote: > On Mon, Apr 7, 2025 at 4:16 PM Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> But I will note that that person’s reply to the ASF Security Team’s >>

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Thu, Apr 10, 2025 at 9:46 AM Xiao Li wrote: > +1 (binding) > > On Thu, Apr 10, 2025 at 00:57 John Zhuge wrote: > >> +1 (non-binding) >> >> On Wed, Apr 9, 2025 at 9:11 PM Jacky Lee wrote: >> >>> +1 (binding) >>> >>> Kent Yao 于2025年4月10日周四 12:00写道： >>> > >>> > +1 (binding) >

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) On Thu, Apr 10, 2025 at 00:57 John Zhuge wrote: > +1 (non-binding) > > On Wed, Apr 9, 2025 at 9:11 PM Jacky Lee wrote: > >> +1 (binding) >> >> Kent Yao 于2025年4月10日周四 12:00写道： >> > >> > +1 (binding) >> > >> > Kent Yao >> > >> > Yang Jie 于2025年4月10日周四 10:27写道： >> >> >> >> +1 (bindi

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Wed, Apr 9, 2025 at 9:11 PM Jacky Lee wrote: > +1 (binding) > > Kent Yao 于2025年4月10日周四 12:00写道： > > > > +1 (binding) > > > > Kent Yao > > > > Yang Jie 于2025年4月10日周四 10:27写道： > >> > >> +1 (binding) > >> > >> On 2025/04/10 02:20:02 Cheng Pan wrote: > >> > +1 (non-binding) > >

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho wrote: > +1 really excited to finally see Materialized View finally make its way to > Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) > already supporting it. > > Thanks > Szehon > > On Wed, Apr 9, 2025 at 2:33 AM Martin Grund

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) Kent Yao 于2025年4月10日周四 12:00写道： > > +1 (binding) > > Kent Yao > > Yang Jie 于2025年4月10日周四 10:27写道： >> >> +1 (binding) >> >> On 2025/04/10 02:20:02 Cheng Pan wrote: >> > +1 (non-binding) >> > >> > Thanks, >> > Cheng Pan >> > >> > >> > >> > > On Apr 9, 2025, at 22:22, Sandy Ryza wrote

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) On 2025/04/10 02:20:02 Cheng Pan wrote: > +1 (non-binding) > > Thanks, > Cheng Pan > > > > > On Apr 9, 2025, at 22:22, Sandy Ryza wrote: > > > > We started to get some votes on the discussion thread, so I'd like to move > > to a formal vote on adding support for declarative pip

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) Thanks, Cheng Pan > On Apr 9, 2025, at 22:22, Sandy Ryza wrote: > > We started to get some votes on the discussion thread, so I'd like to move to > a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv8f8

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) On Thu, Apr 10, 2025 at 7:30 AM Szehon Ho wrote: > +1 (non-binding) > > Thanks > Szehon > > On Wed, Apr 9, 2025 at 3:42 PM Hyukjin Kwon wrote: > >> I will shephard. >> >> On Thu, 10 Apr 2025 at 07:28, Anton Okolnychyi >> wrote: >> >>> +1 (non-binding) >>> >>> - Anton >>> >>> ср, 9

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 (non-binding) El jue, 10 abr 2025, 1:50, Burak Yavuz escribió: > +1 > > On Wed, Apr 9, 2025 at 4:33 PM Szehon Ho wrote: > >> +1 really excited to finally see Materialized View finally make its way >> to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) >> already suppo

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 really excited to finally see Materialized View finally make its way to Spark, as many other ecosystem projects (Trino, Starrocks, soon Iceberg) already supporting it. Thanks Szehon On Wed, Apr 9, 2025 at 2:33 AM Martin Grund wrote: > +1 > > On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh >

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) Thanks Szehon On Wed, Apr 9, 2025 at 3:42 PM Hyukjin Kwon wrote: > I will shephard. > > On Thu, 10 Apr 2025 at 07:28, Anton Okolnychyi > wrote: > >> +1 (non-binding) >> >> - Anton >> >> ср, 9 квіт. 2025 р. о 15:01 Jungtaek Lim >> пише: >> >>> Btw who is going to shephard this

Re: [VOTE] SPIP: Declarative Pipelines

I will shephard. On Thu, 10 Apr 2025 at 07:28, Anton Okolnychyi wrote: > +1 (non-binding) > > - Anton > > ср, 9 квіт. 2025 р. о 15:01 Jungtaek Lim > пише: > >> Btw who is going to shephard this SPIP? I don't see this in the >> doc/JIRA/discussion thread. I understand there are PMC members in th

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) - Anton ср, 9 квіт. 2025 р. о 15:01 Jungtaek Lim пише: > Btw who is going to shephard this SPIP? I don't see this in the > doc/JIRA/discussion thread. I understand there are PMC members in the > author list, but probably good to be explicit about "who" is > shepherding this SPI

Re: [VOTE] SPIP: Declarative Pipelines

Btw who is going to shephard this SPIP? I don't see this in the doc/JIRA/discussion thread. I understand there are PMC members in the author list, but probably good to be explicit about "who" is shepherding this SPIP. On Wed, Apr 9, 2025 at 11:22 PM Sandy Ryza wrote: > We started to get some vot

Re: [VOTE] SPIP: Declarative Pipelines

+1!!! On Thu, Apr 10, 2025 at 6:03 AM Jungtaek Lim wrote: > +1 (non-binding) > > On Wed, Apr 9, 2025 at 11:22 PM Sandy Ryza wrote: > >> We started to get some votes on the discussion thread, so I'd like to >> move to a formal vote on adding support for declarative pipelines. >> >> *Discussion t

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Wed, Apr 9, 2025 at 11:22 PM Sandy Ryza wrote: > We started to get some votes on the discussion thread, so I'd like to move > to a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m5

Re: [VOTE] SPIP: Declarative Pipelines

+1 Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 YouTube Live Streams: https://www.yo

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 7:22 AM Sandy Ryza wrote: > > We started to get some votes on the discussion thread, so I'd like to move to > a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m574ly > *SPIP

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 8:43 PM Denny Lee wrote: > +1 (non-binding) > > On Wed, Apr 9, 2025 at 11:03 Chao Sun wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 10:55 AM L. C. Hsieh wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 7:22 AM Sandy Ryza wrote: >>> > >>> > We started to get some votes o

Re: [VOTE] SPIP: Declarative Pipelines

+1 Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile On Wed, 9 Apr 2025 at 20:05, Gengliang Wang wrote: > +1 > > On Wed, Apr 9, 2025 at 11:57 AM Martin Grund > wr

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 11:57 AM Martin Grund wrote: > +1 > > On Wed, Apr 9, 2025 at 8:43 PM Denny Lee wrote: > >> +1 (non-binding) >> >> On Wed, Apr 9, 2025 at 11:03 Chao Sun wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 10:55 AM L. C. Hsieh wrote: >>> +1 On Wed, Apr 9, 20

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Wed, Apr 9, 2025 at 11:03 Chao Sun wrote: > +1 > > On Wed, Apr 9, 2025 at 10:55 AM L. C. Hsieh wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 7:22 AM Sandy Ryza wrote: >> > >> > We started to get some votes on the discussion thread, so I'd like to >> move to a formal vote on a

  1   2   3   4   5   6   7   8   9   10   >