Re: [VOTE] Release Spark 3.5.6 (RC1)

+1 On Mon, May 26, 2025 at 6:51 PM Wenchen Fan wrote: > > +1. When this release is out, let's also update the release process document > to introduce the new way of making releases with GitHub Action jobs. > > On Tue, May 27, 2025 at 6:22 AM Dongjoon Hyun wrote: >> >> +1 from my side. >> >> Tha

Re: [VOTE] Release Spark 4.0.0 (RC7)

+1 On Mon, May 19, 2025 at 5:27 AM Wenchen Fan wrote: > > Please vote on releasing the following candidate as Apache Spark version > 4.0.0. > > The vote is open until May 22 (PST) and passes if a majority +1 PMC votes are > cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this package as

Re: [VOTE] Release Apache Spark Connect Swift Client 0.2.0 (RC1)

+1 Thanks Dongjoon. On Sat, May 17, 2025 at 5:40 AM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark Connect > Swift Client 0.2.0. This vote is open for the next 72 hours and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >

Re: [VOTE] Release Apache Spark K8s Operator 0.2.0 (RC1)

+1 Thanks Dongjoon. On Sat, May 17, 2025 at 8:22 AM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark K8s Operator > 0.2.0. This vote is open for the next 72 hours and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1

Re: [VOTE] Release Spark 4.0.0 (RC6)

+1 On Tue, May 13, 2025 at 3:28 PM Wenchen Fan wrote: > > Please vote on releasing the following candidate as Apache Spark version > 4.0.0. > > The vote is open until May 16 (PST) and passes if a majority +1 PMC votes are > cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this package as

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 8:22 PM Gengliang Wang wrote: > > +1 > > On Mon, May 5, 2025 at 6:54 PM Ruifeng Zheng wrote: >> >> +1 >> >> On Tue, May 6, 2025 at 9:51 AM Xiao Li wrote: >>> >>> +1 >>> >>> On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: +1 On Tue, May 6, 2025 at 9

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 On Sun, May 4, 2025 at 3:15 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark Connect > Swift Client 0.1.0. This vote is open for the next 72 hours and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release th

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1 On Sun, May 4, 2025 at 4:58 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark K8s Operator > 0.1.0. This vote is open for the next 72 hours and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this packa

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 7:22 AM Sandy Ryza wrote: > > We started to get some votes on the discussion thread, so I'd like to move to > a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m574ly > *SPIP

Re: [VOTE] SPIP: Constraints in DSv2

+1 On Fri, Mar 21, 2025 at 12:13 PM huaxin gao wrote: > > +1 > > On Fri, Mar 21, 2025 at 12:08 PM Denny Lee wrote: >> >> +1 (non-binding) >> >> On Fri, Mar 21, 2025 at 11:52 Gengliang Wang wrote: >>> >>> +1 >>> >>> On Fri, Mar 21, 2025 at 11:46 AM Anton Okolnychyi >>> wrote: Hi all,

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 Thanks Dongjoon for contributing to Swift implementation. On Mon, Mar 10, 2025 at 7:18 AM Hyukjin Kwon wrote: > > +1 > On Mon, Mar 10, 2025 at 6:48 AM Yang Jie wrote: >> >> Great! Really happy to see that spark-connect supports more programming >> languages. >> >> >> On 2025/03/10 07:00:32

Re: [VOTE] SPIP: Add the TIME data type

+1 On Sun, Feb 23, 2025 at 7:51 AM Max Gekk wrote: > > Hi Spark devs, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > The SPIP aims to add a new data type TIME to Spark SQL types. New type should > conform to TIME(n) WITHOUT TIME ZONE as defined by the SQL stand

Re: [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration

+1 On Tue, Feb 18, 2025 at 9:46 PM dongjoon.hyun wrote: > > Please vote to deprecate `spark.databricks.*` configuration at Apache Spark > 3.5.5. > This is a part of the following on-going discussion. > > - DISCUSSION: https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd > (Deprecat

Re: [DISCUSS] SPIP: Constraints in DSv2

+1 On Fri, Feb 14, 2025 at 10:56 AM DB Tsai wrote: > > +1 > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > On Feb 13, 2025, at 10:21 PM, Gengliang Wang wrote: > > +1, the proposal will unify constraint management in DSv2 and reduce > redundant work across connectors > > On T

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 On Tue, Feb 4, 2025 at 11:56 PM Gengliang Wang wrote: > > +1 > > On Tue, Feb 4, 2025 at 11:38 PM Sakthi wrote: >> >> +1 (non-binding) >> >> On Tue, Feb 4, 2025 at 11:25 PM DB Tsai wrote: >>> >>> +1 >>> >>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>> >>> On Feb 4, 2025, a

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 for the additional option. Agreed that we should keep on track with the schedule. If as mentioned earlier that there are no critical blockers, it should be fine. On Tue, Feb 4, 2025 at 8:05 PM Denny Lee wrote: > > +1 (non-binding) on this proposal. Just as long as there are no schedule > co

Re: [VOTE] Use plain text logs by default

+1 On Wed, Jan 8, 2025 at 10:27 PM Gengliang Wang wrote: > > +1 > > On Wed, Jan 8, 2025 at 10:17 PM Jungtaek Lim > wrote: >> >> +1 (non-binding) >> >> 2025년 1월 9일 (목) 오후 3:06, Cheng Pan 님이 작성: >>> >>> +1 >>> >>> Thanks, >>> Cheng Pan >>> >>> >>> >>> On Jan 9, 2025, at 12:28, Wenchen Fan wrote:

Re: [VOTE] Release Spark 3.4.4 (RC1)

+1 Thanks Dongjoon! On Mon, Oct 21, 2024 at 5:45 PM Xinrong Meng wrote: > > +1 > > Thank you Dongjoon! > > On Tue, Oct 22, 2024 at 8:35 AM Ruifeng Zheng wrote: >> >> +1 >> >> Thank you Dongjoon for driving this release! >> >> On Tue, Oct 22, 2024 at 6:39 AM huaxin gao wrote: >>> >>> +1 >>> >>>

Re: Apache Spark 3.4.4 EOL Release?

+1 Thanks Dongjoon. On Wed, Oct 16, 2024 at 11:41 AM Holden Karau wrote: > > +1 on a 3.4.4 EOL release > > On Wed, Oct 16, 2024 at 9:37 AM Dongjoon Hyun wrote: >> >> Hi, All. >> >> Since the Apache Spark 3.4.0 RC7 vote passed on Apr 6, 2023, branch-3.4 has >> been maintained and served well u

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Thu, Oct 3, 2024 at 7:31 AM Wenchen Fan wrote: > > +1 > > On Wed, Oct 2, 2024 at 7:50 AM Peter Toth wrote: >> >> +1 >> >> >> On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: >>> >>> +1, Thanks >>> >>> Jie Yang >>> >>> On 2024/10/01 03:26:40 John Zhuge wrote: >>> > +1 (non-binding) >>> > >>> > O

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 On Mon, Sep 30, 2024 at 1:25 PM Herman van Hovell wrote: > > +1 > > On Mon, Sep 30, 2024 at 12:21 PM Dongjoon Hyun wrote: >> >> +1 >> >> Thank you, Holden. >> >> Dongjoon. >> >> On 2024/09/30 18:01:17 Holden Karau wrote: >> > I think it has been de-facto deprecated, we haven’t updated it mean

Re: [VOTE] Release Spark 4.0.0-preview2 (RC1)

+1 On Mon, Sep 16, 2024 at 5:56 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On Mon, Sep 16, 2024 at 10:57 AM Holden Karau wrote: >> >> +1 >> >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 >> YouTube Live Streams: h

Re: [VOTE] Release Apache Spark 3.5.3 (RC3)

+1 Thanks. On Wed, Sep 11, 2024 at 10:41 AM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On 2024/09/11 13:51:23 Herman van Hovell wrote: > > +1 > > > > On Wed, Sep 11, 2024 at 3:30 AM Kent Yao wrote: > > > > > +1, thank you, Haejoon > > > Kent > > > > > > On 2024/09/11 06:12:19 Gengliang Wang w

Re: Apache Spark 4.0.0-preview2 (?)

+1 Thanks Dongjoon. On Fri, Sep 6, 2024 at 12:18 PM Dongjoon Hyun wrote: > > Hi, All. > > Since the Apache Spark 4.0.0-preview1 tag was created in May, it's been over > 3 months. > > https://github.com/apache/spark/releases/tag/v4.0.0-preview1 (2024-05-28) > > Almost 1k commits including improv

Re: [DISCUSS] Deprecating SparkR

+1 On Tue, Aug 13, 2024 at 2:54 AM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On Mon, Aug 12, 2024 at 17:52 Holden Karau wrote: >> >> +1 >> >> Are the sparklyr folks on this list? >> >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): https://am

Re: [VOTE] Release Spark 3.5.2 (RC5)

Then, +1 again On Thu, Aug 8, 2024 at 11:38 AM Dongjoon Hyun wrote: > > +1 > > I'm resending my vote. > > Dongjoon. > > On 2024/08/06 16:06:00 Kent Yao wrote: > > Hi dev, > > > > Please vote on releasing the following candidate as Apache Spark version > > 3.5.2. > > > > The vote is open until A

Re: [VOTE] Release Spark 3.5.2 (RC5)

+1 Thanks Kent. On Wed, Aug 7, 2024 at 8:31 AM Dongjoon Hyun wrote: > > +1 > > Thank you, Kent. > > Dongjoon. > > On 2024/08/06 16:06:00 Kent Yao wrote: > > Hi dev, > > > > Please vote on releasing the following candidate as Apache Spark version > > 3.5.2. > > > > The vote is open until Aug 9,

Re: [VOTE] Release Spark 3.5.2 (RC4)

+1 On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan wrote: > > +1 > > On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun > wrote: >> >> +1 >> >> Thank you, Kent. >> >> Dongjoon. >> >> On Fri, Jul 26, 2024 at 6:37 AM Kent Yao wrote: >>> >>> Hi dev, >>> >>> Please vote on releasing the following candidat

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

+1 Thanks. On Tue, Jul 23, 2024 at 8:35 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon. > > On 2024/07/24 03:28:58 Wenchen Fan wrote: > > +1 > > > > On Wed, Jul 24, 2024 at 10:51 AM Kent Yao wrote: > > > > > +1(non-binding), I have checked: > > > > > > - Download links are OK > > > - Signatures, C

Re: [VOTE] Release Spark 3.5.2 (RC1)

I also support -1 to include the fix. On Thu, Jul 18, 2024 at 8:46 PM huaxin gao wrote: > > -1 because we need to include this fix > https://github.com/apache/spark/pull/47406 > > On Thu, Jul 18, 2024 at 4:01 AM Kent Yao wrote: >> >> Thank you Wenchen. >> >> The vote is open until Jul 21, 11 AM

Re: [DISCUSS] Release Apache Spark 3.5.2

+1 On Thu, Jul 11, 2024 at 3:22 PM Zhou Jiang wrote: > > +1 for releasing 3.5.2, which would also benefit the Spark Operator > multi-version support. > > On Thu, Jul 11, 2024 at 7:56 AM Dongjoon Hyun wrote: >> >> Thank you for the head-up and volunteering, Kent. >> >> +1 for 3.5.2 release. >> >

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 On Tue, Jul 9, 2024 at 1:13 AM Wenchen Fan wrote: > > +1 > > On Tue, Jul 9, 2024 at 10:47 AM Reynold Xin > wrote: >> >> +1 >> >> On Mon, Jul 8, 2024 at 7:44 PM haydn wrote: >>> >>> +1 >>> >>> On Mon, Jul 8, 2024 at 7:41 PM haydn wrote: +1 On Mon, Jul 8, 2024 at 19:41 Ta

Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

+1 On Wed, Jul 3, 2024 at 3:54 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On Wed, Jul 3, 2024 at 10:58 Xinrong Meng wrote: >> >> +1 >> >> Thank you @Hyukjin Kwon ! >> >> On Wed, Jul 3, 2024 at 8:55 AM bo yang wrote: >>> >>> +1 (non-binding) >>> >>> >>> On Tue, Jul 2, 2024 at 11:22 PM Cheng

Re: Deploying Spark on Kubernetes Operator

Thanks for being interested in the Spark Kubernetes Operator. Because the initial PR is large so it is split into several PRs which are good to review and merge. And seems the initial series of PRs to merge the codes into the repo is not done yet. For example, you can see there is PR to add the op

[VOTE][RESULT] SPIP: Stored Procedures API for Catalogs

The vote passes with 13+1s (8 binding +1s) and 1+0. (* = binding) +1: Chao Sun (*) Liang-Chi Hsieh (*) Huaxin Gao (*) Bo Yang Dongjoon Hyun (*) Kent Yao Wenchen Fan (*) Ryan Blue Anton Okolnychyi Zhou Jiang Gengliang Wang (*) Xiao Li (*) Hyukjin Kwon (*) +0: None Mich Talebzadeh -1: None Thank

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

> On Tue, May 14, 2024 at 8:19 AM Zhou Jiang wrote: >>> >>> +1 (non-binding) >>> >>> On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: >>>> >>>> Hi all, >>>> >>>> I’d like to start a vote for SPIP: Stored Proced

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 On Sat, May 11, 2024 at 3:11 PM Chao Sun wrote: > > +1 > > On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: >> >> Hi all, >> >> I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. >> >> Please also refer to: >>

[VOTE] SPIP: Stored Procedures API for Catalogs

Hi all, I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. Please also refer to: - Discussion thread: https://lists.apache.org/thread/7r04pz544c9qs3gc8q2nyj3fpzfnv8oo - JIRA ticket: https://issues.apache.org/jira/browse/SPARK-44167 - SPIP doc: https://docs.google.co

Re: [DISCUSS] SPIP: Stored Procedures API for Catalogs

Thanks Anton. Thank you, Wenchen, Dongjoon, Ryan, Serge, Allison and others if I miss those who are participating in the discussion. I suppose we have reached a consensus or close to being in the design. If you have some more comments, please let us know. If not, I will go to start a vote soon a

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Fri, Apr 26, 2024 at 10:01 AM Dongjoon Hyun wrote: > > I'll start with my +1. > > Dongjoon. > > On 2024/04/26 16:45:51 Dongjoon Hyun wrote: > > Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault > > to `false` by default. The technical scope is defined in the follow

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Thu, Apr 25, 2024 at 8:16 PM Yuming Wang wrote: > +1 > > On Fri, Apr 26, 2024 at 8:25 AM Nimrod Ofek wrote: > >> Of course, I can't think of a scenario of thousands of tables with single >> in memory Spark cluster with in memory catalog. >> Thanks for the help! >> >> בתאריך יום ה׳, 25 באפ

Re: [FYI] SPARK-47993: Drop Python 3.8

+1 On Thu, Apr 25, 2024 at 11:19 AM Maciej wrote: > > +1 > > Best regards, > Maciej Szymkiewicz > > Web: https://zero323.net > PGP: A30CEF0C31A501EC > > On 4/25/24 6:21 PM, Reynold Xin wrote: > > +1 > > On Thu, Apr 25, 2024 at 9:01 AM Santosh Pingale > wrote: >> >> +1 >> >> On Thu, Apr 25, 2024

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 On Tue, Apr 16, 2024 at 4:08 AM Wenchen Fan wrote: > > +1 > > On Mon, Apr 15, 2024 at 12:31 PM Dongjoon Hyun wrote: >> >> I'll start with my +1. >> >> - Checked checksum and signature >> - Checked Scala/Java/R/Python/SQL Document's Spark version >> - Checked published Maven artifacts >> - All

[VOTE][RESULT] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

Hi all, The vote passes with 7+1s (5 binding +1s). (* = binding) +1: Dongjoon Hyun(*) Liang-Chi Hsieh(*) Huaxin Gao(*) Bo Yang Xiao Li(*) Chao Sun(*) Hussein Awala +0: None -1: None Thanks. - To unsubscribe e-mail: dev-unsubs

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Sat, Apr 13, 2024 at 4:12 PM Hyukjin Kwon wrote: > > +1 > > On Sun, Apr 14, 2024 at 7:46 AM Chao Sun wrote: >> >> +1. >> >> This feature is very helpful for guarding against correctness issues, such >> as null results due to invalid input or math overflows. It’s been there for >> a while

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

> Dongjoon. > > On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > > Hi all, > > > > Thanks for all discussions in the thread of "Versioning of Spark > > Operator": https://lists.apache.org/thread/zhc7nb2sxm8jjxdppq8qjcmlf4rcsthh > > > &

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

+1 I believe ANSI mode is well developed after many releases. No doubt it could be used. Since it is very easy to disable it to restore to current behavior, I guess the impact could be limited. Do we have known the possible impacts such as what are the major changes (e.g., what kind of queries/exp

[VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

Hi all, Thanks for all discussions in the thread of "Versioning of Spark Operator": https://lists.apache.org/thread/zhc7nb2sxm8jjxdppq8qjcmlf4rcsthh I would like to create this vote to get the consensus for versioning of the Spark Kubernetes Operator. The proposal is to use an independent versio

Re: SPIP: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

+1 for Wenchen's point. I don't see a strong reason to pull these transformations into Spark instead of keeping them in third party packages/projects. On Wed, Apr 10, 2024 at 5:32 AM Wenchen Fan wrote: > > It's good to reduce duplication between different native accelerators of > Spark, and AFA

Re: Versioning of Spark Operator

This approach makes sense to me. If Spark K8s operator is aligned with Spark versions, for example, it uses 4.0.0 now. Because these JIRA tickets are not actually targeting Spark 4.0.0, it will cause confusion and more questions, like when we are going to cut Spark release, should we include Spark

Re: Versioning of Spark Operator

> >> Sadly, there is no release at all and no activity since last 6 months. > > >> It seems to be the first time for Apache Spark community to consider > > >> these sister repositories (Go and K8s Operator). > > >> > > >> https://github.com/apa

Re: Versioning of Spark Operator

gt; > On Tue, Apr 9, 2024 at 10:09 AM Dongjoon Hyun > > <mailto:dongj...@apache.org>> wrote: > > > > >> Hi, Liang-Chi. > > > > >> > > > > >> Thank you for leading Apache Spark K8s operator as a shepherd. > > > &g

Versioning of Spark Operator

Hi all, We've opened the dedicated repository of Spark Kubernetes Operator, and the first PR is created. Thank you for the review from the community so far. About the versioning of Spark Operator, there are questions. As we are using Spark JIRA, when we are going to merge PRs, we need to choose

Re: Apache Spark 3.4.3 (?)

+1 Thanks Dongjoon! On Sun, Apr 7, 2024 at 1:56 AM Kent Yao wrote: > > +1, thank you, Dongjoon > > > Kent > > Holden Karau 于2024年4月7日周日 14:54写道： > > > > Sounds good to me :) > > > > Twitter: https://twitter.com/holdenkarau > > Books (Learning Spark, High Performance Spark, etc.): > > https://a

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Thanks Hyukjin. On Sun, Mar 31, 2024 at 10:52 PM Dongjoon Hyun wrote: > > +1 > > Thank you, Hyukjin. > > Dongjoon > > On Sun, Mar 31, 2024 at 19:07 Haejoon Lee > wrote: >> >> +1 >> >> On Mon, Apr 1, 2024 at 10:15 AM Hyukjin Kwon wrote: >>> >>> Hi all, >>> >>> I'd like to start the vote for

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

if we can make it >>>> compatible or even merge the two projects to make it the new official >>>> operator in spark project, it would be the best. >>>> 3. The new Spark Operator should continue being spark agnostic and >>>> continue having this lightweight/s

The dedicated repository for Kubernetes Operator for Apache Spark

Hi all, For the passed SPIP: An Official Kubernetes Operator for Apache Spark, the developers have been working on code cleaning and refactoring for open source in the last few months. They are ready to contribute the code to Spark now. As we discussed, I will go to create a dedicated repository

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 On Tue, Mar 12, 2024 at 8:20 AM Chao Sun wrote: > +1 > > On Tue, Mar 12, 2024 at 8:03 AM Xiao Li > wrote: > >> +1 >> >> On Tue, Mar 12, 2024 at 6:09 AM Holden Karau >> wrote: >> >>> +1 >>> >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.)

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

+1 On Wed, Jan 10, 2024 at 9:06 AM Bhuwan Sahni wrote: > +1. This is a good addition. > > > *Bhuwan Sahni* > Staff Software Engineer > > bhuwan.sa...@databricks.com > 500 108th Ave. NE > Bellevue, WA 98004 > USA > > > On Wed, Jan 10, 2024 at 9:00 AM Burak Yavuz wrote

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

+1 I left some comments in the SPIP doc and got replies quickly. The new API looks good and more comprehensive. I think it will help Spark Structured Streaming to be more useful in more complicated streaming use cases. On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz wrote: > > I'm also a +1 on the ne

Re: [VOTE] Release Spark 3.3.4 (RC1)

+1 On Sun, Dec 10, 2023 at 6:15 PM Kent Yao wrote: > > +1(non-binding > > Kent Yao > > Yuming Wang 于2023年12月11日周一 09:33写道： > > > > +1 > > > > On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun wrote: > >> > >> +1 > >> > >> Dongjoon > >> > >> On 2023/12/08 21:41:00 Dongjoon Hyun wrote: > >> > Please

Re: Apache Spark 3.3.4 EOL Release?

+1 Thanks Dongjoon! On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > Jie Yang > > On 2023/12/04 15:08:25 Tom Graves wrote: > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > Tom > > On Friday, December 1, 2023 at 02:48:22 PM CST, Dongjoon

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Thanks Dongjoon! On Wed, Nov 29, 2023 at 7:53 PM Mridul Muralidharan wrote: > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: >> >> +1(non-bi

[VOTE][RESULT] SPIP: An Official Kubernetes Operator for Apache Spark

Hi all, The vote passes with 19 +1s (11 binding +1s). Thanks to all who reviews the SPIP doc and votes! (* = binding) +1: - Ye Zhou - L. C. Hsieh (*) - Chao Sun (*) - Vakaris Baškirov - DB Tsai (*) - Holden Karau (*) - Lucian Neghina - Mridul Muralidharan (*) - Huaxin Gao (*) - Cheng Pan

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Tue, Nov 14, 2023 at 9:46 AM Ye Zhou wrote: > > +1(Non-binding) > > On Tue, Nov 14, 2023 at 9:42 AM L. C. Hsieh wrote: >> >> Hi all, >> >> I’d like to start a vote for SPIP: An Official Kubernetes Operator for >> Apache Spark. >> >

[VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

Hi all, I’d like to start a vote for SPIP: An Official Kubernetes Operator for Apache Spark. The proposal is to develop an official Java-based Kubernetes operator for Apache Spark to automate the deployment and simplify the lifecycle management and orchestration of Spark applications and Spark cl

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks for all the support from the community for the SPIP proposal. Since all questions/discussion are settled down (if I didn't miss any major ones), if no more questions or concerns, I'll be the shepherd for this SPIP proposal and call for a vote tomorrow. Thank you all! On Mon, Nov 13, 2023

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 7:57 PM Chao Sun wrote: > > +1 > > > On Thu, Nov 9, 2023 at 6:36 PM Xiao Li wrote: > > > > +1 > > > > huaxin gao 于2023年11月9日周四 16:53写道： > >> > >> +1 > >> > >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >>> > >>> +1 > >>> > >>> To be completely transparent, I am e

Re: Apache Spark 3.4.2 (?)

+1 On Tue, Nov 7, 2023 at 4:56 PM Dongjoon Hyun wrote: > > Thank you all! > > Dongjoon > > On Mon, Nov 6, 2023 at 6:03 PM Holden Karau wrote: >> >> +1 >> >> On Mon, Nov 6, 2023 at 4:30 PM yangjie01 wrote: >>> >>> +1 >>> >>> >>> >>> 发件人: Yuming Wang >>> 日期: 2023年11月7日 星期二 07:00 >>> 收件人: Santosh

Re: [VOTE] SPIP: State Data Source - Reader

+1 On Mon, Oct 23, 2023 at 6:31 PM Anish Shrigondekar wrote: > > +1 (non-binding) > > Thanks, > Anish > > On Mon, Oct 23, 2023 at 5:01 PM Wenchen Fan wrote: >> >> +1 >> >> On Mon, Oct 23, 2023 at 4:03 PM Jungtaek Lim >> wrote: >>> >>> Starting with my +1 (non-binding). Thanks! >>> >>> On Mon,

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Thanks Yuming. On Thu, Aug 10, 2023 at 3:24 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On 2023/08/10 07:14:07 yangjie01 wrote: > > +1 > > Thanks, Jie Yang > > > > > > 发件人: Yuming Wang > > 日期: 2023年8月10日 星期四 13:33 > > 收件人: Dongjoon Hyun > > 抄送: dev > > 主题: Re: [VOTE] Release Apache Spa

Re: Welcome two new Apache Spark committers

Congratulations! On Mon, Aug 7, 2023 at 9:44 AM huaxin gao wrote: > > Congratulations! Peter and Xiduo! > > On Mon, Aug 7, 2023 at 9:40 AM Dongjoon Hyun wrote: >> >> Congratulations, Peter and Xiduo. :) >> >> Dongjoon. >> >> On Sun, Aug 6, 2023 at 10:08 PM XiDuo You wrote: >>> >>> Thank you all

Re: Time for Spark v3.5.0 release

+1 Thanks Yuanjian. On Tue, Jul 4, 2023 at 7:45 AM yangjie01 wrote: > > +1 > > > > 发件人: Maxim Gekk > 日期: 2023年7月4日 星期二 17:24 > 收件人: Kent Yao > 抄送: "dev@spark.apache.org" > 主题: Re: Time for Spark v3.5.0 release > > > > +1 > > On Tue, Jul 4, 2023 at 11:55 AM Kent Yao wrote: > > +1， thank you >

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon! On Fri, Jun 23, 2023 at 7:10 PM Hyukjin Kwon wrote: > > Thanks! > > On Sat, Jun 24, 2023 at 11:01 AM Mridul Muralidharan wrote: >> >> >> Thanks Dongjoon ! >> >> Regards, >> Mridul >> >> On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: >>> >>> We are happy to announce the av

Re: [VOTE][SPIP] PySpark Test Framework

+1 On Thu, Jun 22, 2023 at 3:10 PM Xinrong Meng wrote: > > +1 > > Thanks for driving that! > > On Wed, Jun 21, 2023 at 10:25 PM Ruifeng Zheng wrote: >> >> +1 >> >> On Thu, Jun 22, 2023 at 1:11 PM Dongjoon Hyun >> wrote: >>> >>> +1 >>> >>> Dongjoon >>> >>> On Wed, Jun 21, 2023 at 8:56 PM Hyukji

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1 On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On 2023/06/20 02:51:32 Jia Fan wrote: > > +1 > > > > Dongjoon Hyun 于2023年6月20日周二 10:41写道： > > > > > Please vote on releasing the following candidate as Apache Spark version > > > 3.4.1. > > > > > > The vote is open u

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

+1 On Mon, Jun 12, 2023 at 11:06 AM huaxin gao wrote: > > +1 > > On Mon, Jun 12, 2023 at 11:05 AM Dongjoon Hyun wrote: >> >> +1 >> >> Dongjoon >> >> On 2023/06/12 18:00:38 Dongjoon Hyun wrote: >> > Please vote on the release plan for Apache Spark 4.0.0. >> > >> > The vote is open until June 16th

Re: Apache Spark 3.4.1 Release?

+1 Thanks Dongjoon for driving this. On Thu, Jun 8, 2023 at 2:25 PM Dongjoon Hyun wrote: > > Hi, All. > > `branch-3.4` already has 77 commits since v3.4.0 tag. > > https://github.com/apache/spark/releases/v3.4.0 (Tagged on April 6th) > > $ git log --oneline v3.4.0..HEAD | wc -l >

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Thanks Dongjoon On Sun, Apr 9, 2023 at 5:20 PM Dongjoon Hyun wrote: > > I'll start with my +1. > > I verified the checksum, signatures of the artifacts, and documentations. > Also, ran the tests with YARN and K8s modules. > > Dongjoon. > > On 2023/04/09 23:46:10 Dongjoon Hyun wrote: > > Pleas

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Thanks Xinrong. On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > +1 > > > > 发件人: Sean Owen > 日期: 2023年4月8日 星期六 20:27 > 收件人: Xinrong Meng > 抄送: dev > 主题: Re: [VOTE] Release Apache Spark 3.4.0 (RC7) > > > > +1 form me, same result as last time. > > > > On Fri, Apr 7, 2023 at 6:30 PM Xinro

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good and thanks Dongjoon for driving this. On 2023/04/04 17:24:54 Dongjoon Hyun wrote: > Hi, All. > > Since Apache Spark 3.2.0 passed RC7 vote on October 12, 2021, branch-3.2 > has been maintained and served well until now. > > - https://github.com/apache/spark/releases/tag/v3.2.0 (ta

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

+1 Thanks Xinrong. On Mon, Apr 3, 2023 at 12:35 PM Dongjoon Hyun wrote: > > +1 > > I also verified that RC5 has SBOM artifacts. > > https://repository.apache.org/content/repositories/orgapachespark-1439/org/apache/spark/spark-core_2.12/3.4.0/spark-core_2.12-3.4.0-cyclonedx.json > https://reposit

[VOTE][RESULT][SPIP] Lazy Materialization for Parquet Read Performance Improvement

The vote passes with 9 +1s (4 binding +1s). Thanks to all who reviews the SPIP doc and votes! (* = binding) +1: - Dongjoon Hyun (*) - Huaxin Gao (*) - Mich Talebzadeh - L. C. Hsieh (*) - Prem Sahoo - Yuming Wang - Guo Weijie - DB Tsai (*) - Kazuyuki Tanimura +0: None -1: None Thanks

[ANNOUNCE] Apache Spark 3.3.2 released

We are happy to announce the availability of Apache Spark 3.3.2! Spark 3.3.2 is a maintenance release containing stability fixes. This release is based on the branch-3.3 maintenance branch of Spark. We strongly recommend all 3.3 users to upgrade to this stable release. To download Spark 3.3.2, he

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

t;> +1 >> >> Yuming Wang 于2023年2月14日周二 15:58写道： >>> >>> +1 >>> >>> On Tue, Feb 14, 2023 at 11:27 AM Prem Sahoo wrote: >>>> >>>> +1 >>>> >>>> On Mon, Feb 13, 2023 at 8:13 PM L

[VOTE][RESULT] Release Spark 3.3.2 (RC1)

The vote passes with 12 +1s (4 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Dongjoon Hyun (*) - Sean Owen (*) - Enrico Minack - Bjørn Jørgensen - Yikun Jiang - Yang Jie - Yuming Wang - John Zhuge - William Hyun - Chao Sun - L. C. Hsieh

Re: [VOTE] Release Spark 3.3.2 (RC1)

want it in branch-3.3. >> >> We need to talk. :) >> >> Bests, >> Dongjoon. >> >> >> On Mon, Feb 13, 2023 at 9:31 AM Chao Sun wrote: >>> >>> +1 >>> >>> On Mon, Feb 13, 2023 at 9:20 AM L. C. Hsieh wrote: >>>

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

uch loss, damage or destruction. > > > > > On Mon, 13 Feb 2023 at 23:18, huaxin gao wrote: > >> +1 >> >> On Mon, Feb 13, 2023 at 3:09 PM Dongjoon Hyun >> wrote: >> >>> +1 >>> >>> Dongjoon >>> >>> On 2023

[VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

Hi all, I'd like to start the vote for SPIP: Lazy Materialization for Parquet Read Performance Improvement. The high summary of the SPIP is that it proposes an improvement to the Parquet reader with lazy materialization which only materializes (i.e. decompress, de-code, etc...) necessary values.

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

> > On Mon, 13 Feb 2023 at 20:41, kazuyuki tanimura > wrote: > >> Thank you Liang-Chi! >> >> Kazu >> >> On Feb 11, 2023, at 7:12 PM, L. C. Hsieh wrote: >> >> Thanks all for your feedback. >> >> Given this positive feedback, if there

Re: [VOTE] Release Spark 3.3.2 (RC1)

kl. 03:24 skrev yangjie01 : >>>> >>>> Which Python version do you use for testing? When I use the latest Python >>>> 3.11, I can reproduce similar test failures (43 tests of sql module fail), >>>> but when I use python 3.10, they will succeed >>

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

Thanks all for your feedback. Given this positive feedback, if there is no other comments/discussion, I will go to start a vote in the next few days. Thank you again! On Thu, Feb 2, 2023 at 10:12 AM kazuyuki tanimura wrote: > Thank you all for +1s and reviewing the SPIP doc. > > Kazu > > On Fe

Re: [VOTE] Release Spark 3.3.2 (RC1)

- > [INFO] BUILD FAILURE > [INFO] > ---- > [INFO] Total time: 02:30 h > [INFO] Finished at: 2023-02-11T17:32:45+01:00 > > lør. 11. feb. 2023 kl. 06:01 skrev L. C. Hsieh : >

Re: [VOTE] Release Spark 3.3.2 (RC1)

ignatures, digests, etc check out fine. > > Built and tested with "-Phive -Pyarn -Pmesos -Pkubernetes". > > Regards, > Mridul > > > > > On Fri, Feb 10, 2023 at 11:01 PM L. C. Hsieh wrote: >> >> Please vote on releasing the following candidate a

[VOTE] Release Spark 3.3.2 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.3.2. The vote is open until Feb 15th 9AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.2 [ ] -1 Do not release this package because ...

Time for release v3.3.2

Hi Spark devs, As you know, it has been 4 months since Spark 3.3.1 was released on 2022/10, it seems a good time to think about next maintenance release, i.e. Spark 3.3.2. I'm thinking of the release of Spark 3.3.2 this Feb (2023/02). What do you think? I am willing to volunteer for Spark 3.3.2

Re: [DISCUSS] Deprecate DStream in 3.4

+1 On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim wrote: > > Yes, exactly. I'm sorry to bring confusion - should have clarified action > items on the proposal. > > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: >> >> Then, could you elaborate `the proposed code change` specifically? >> May

Re: Time for Spark 3.4.0 release?

+1 Thank you! On Wed, Jan 4, 2023 at 9:13 AM Chao Sun wrote: > +1, thanks! > > Chao > > On Wed, Jan 4, 2023 at 1:56 AM Mridul Muralidharan > wrote: > >> >> +1, Thanks ! >> >> Regards, >> Mridul >> >> On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: >> >>> +1, thanks for driving the releas

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks, Chao! On Wed, Nov 30, 2022 at 9:58 AM huaxin gao wrote: > > Thanks Chao for driving the release! > > On Wed, Nov 30, 2022 at 9:24 AM Dongjoon Hyun wrote: >> >> Thank you, Chao! >> >> On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote: >>> >>> Thanks, Chao! >>> >>> >>> >>> 发件人: Maxim Ge

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 Thanks Chao. On Mon, Nov 14, 2022 at 6:55 PM Dongjoon Hyun wrote: > > +1 > > Thank you, Chao. > > On Mon, Nov 14, 2022 at 4:12 PM Chao Sun wrote: >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.2.3. >> >> The vote is open until 11:59pm Pacific time Nov 17

  1   2   >