Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 huaxin gao 于2025年6月1日周日 20:00写道： > +1 > > On Sun, Jun 1, 2025 at 7:50 PM Tathagata Das > wrote: > >> +1 (binding) >> super excited about this! >> >> On Sun, Jun 1, 2025 at 10:45 PM Yuanjian Li >> wrote: >> >>> +1 >>> >>> On Sun, Jun 1, 2025 at 19:00 Dongjoon Hyun >>> wrote: >>> +1 >>>

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 Yuming Wang 于2025年5月29日周四 02:22写道： > +1. > > On Thu, May 29, 2025 at 3:36 PM DB Tsai wrote: > >> +1 >> Sent from my iPhone >> >> On May 29, 2025, at 12:15 AM, John Zhuge wrote: >> >>  >> +1 Nice feature >> >> On Wed, May 28, 2025 at 9:53 PM Yuanjian Li >> wrote: >> >>> +1 >>> >>> Kent Yao

Re: [VOTE] Release Spark 4.0.0 (RC7)

+1 L. C. Hsieh 于2025年5月20日周二 08:52写道： > +1 > > On Mon, May 19, 2025 at 5:27 AM Wenchen Fan wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > 4.0.0. > > > > The vote is open until May 22 (PST) and passes if a majority +1 PMC > votes are cast, with a minimu

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: > +1 > > On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: > >> +1 (non-binding) >> >> On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: >> >>> +1 >>> >>> On Tue, May 6, 2025 at 3:40 AM Reynold Xin >>> wrote: >>> +1 On Mon, Ma

Re: [VOTE] SPIP: Declarative Pipelines

+1 (binding) On Thu, Apr 10, 2025 at 00:57 John Zhuge wrote: > +1 (non-binding) > > On Wed, Apr 9, 2025 at 9:11 PM Jacky Lee wrote: > >> +1 (binding) >> >> Kent Yao 于2025年4月10日周四 12:00写道： >> > >> > +1 (binding) >> > >> > Kent Yao >> > >> > Yang Jie 于2025年4月10日周四 10:27写道： >> >> >> >> +1 (bindi

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

> > this vote is to allow streaming queries which had been ever run in Spark > 3.5.4 to be upgraded with Spark 4.0.x, "without having to be upgraded with > Spark 3.5.5+ in prior". In the history of Apache Spark, have we ever required users to upgrade to the next maintenance release before moving

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 Xiao Jungtaek Lim 于2025年2月4日周二 23:11写道： > +1 (non-binding) > > Sounds like a great alternative! > > On Wed, Feb 5, 2025 at 4:10 PM Hyukjin Kwon wrote: > >> +1 >> >> On Wed, 5 Feb 2025 at 16:06, Wenchen Fan wrote: >> >>> Hi all, >>> >>> Given the positive feedback in the previous DISCUSS em

Re: A documentation change is a user-facing change

Thank you for pointing it out! Let’s update the template to exclude documentation changes from the behavior change question. At the same time, I strongly believe that all behavior changes should be clearly documented in the PR description. For instance, in the first example PR https://github.com/a

Re: [VOTE] Use plain text logs by default

+1 Dongjoon Hyun 于2025年1月11日周六 12:47写道： > +1 > > Dongjoon > > On 2025/01/10 00:57:25 Ye Zhou wrote: > > +1 (non-binding) > > > > On Thu, Jan 9, 2025 at 3:02 PM Sakthi wrote: > > > > > +1 (non-binding) > > > > > > Sakthi > > > > > > On Thu, Jan 9, 2025 at 9:16 AM Rozov, Vlad > > > wrote: > > >

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Hi, Russell, After reviewing the JIRAs, it seems that only SPARK-42856 is directly relevant to GraphX. While the other three JIRAs mention GraphX in their descriptions, they appear to be more related to the build or the REPL rather than GraphX itself. Thanks, Xiao On Nov 16, 2024 at 5:39:

Re: [DISCUSS] Support spark.ml on Spark Connect

Thank you for working on this! Xiao Martin Grund 于2024年10月10日周四 03:01写道： > > Hi Bobby, > > Awesome to see the proposal! I'm very much looking forward to the > contributions! > > Martin > > On Thu, Oct 10, 2024 at 4:16 AM Ángel > wrote: > >> You have my vote (btw, great idea, ML is so sexy now

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 in support of the direction of the Single-pass Analyzer for Catalyst. I think we should not have a target version for the new Catalyst SPARK-49834 . It should not be a blocker for Spark 4.0. When implementing the new analyzer, the code changes

Re: [VOTE] Release Spark 4.0.0-preview2 (RC1)

+1 On Wed, Sep 18, 2024 at 18:22 Yuming Wang wrote: > +1 > > On Wed, Sep 18, 2024 at 6:07 PM Cheng Pan wrote: > >> +1 (non-binding) >> >> I checked >> - Signatures and checksums are good. >> - Build success from source code. >> - Pass integration test with Apache Kyuubi [1] >> >> [1] https://gi

Re: [VOTE] Document and Feature Preview via GitHub Pages

+1 Hyukjin Kwon 于2024年9月11日周三 15:49写道： > +1 > > On Thu, Sep 12, 2024 at 5:00 AM Gengliang Wang wrote: > >> +1 >> >> On Wed, Sep 11, 2024 at 6:30 AM Wenchen Fan wrote: >> >>> +1 >>> >>> On Wed, Sep 11, 2024 at 5:15 PM Martin Grund >>> wrote: >>> +1 On Wed, Sep 11, 2024 at 9:39 A

Re: [VOTE] Move Variant to Parquet

+1 On Mon, Sep 2, 2024 at 10:19 Reynold Xin wrote: > +1 > > On Mon, Sep 2, 2024 at 9:30 AM Gene Pang wrote: > >> Hi all, >> >> I’d like to start a vote for moving the Variant specification and library >> to the Parquet project. This allows the Variant binary format and shredding >> format to be

Re: [DISCUSS] release Spark 3.5.3?

+1 Yuming Wang 于2024年8月30日周五 02:34写道： > +1, Could we include two additional issues: > https://issues.apache.org/jira/browse/SPARK-49472 > https://issues.apache.org/jira/browse/SPARK-49349 > > On Wed, Aug 28, 2024 at 7:01 PM Wenchen Fan wrote: > >> Hi all, >> >> It's unfortunate that we missed m

Re: [DISCUSS] Move Variant to Parquet?

Parquet is the right project for standardizing the new data type Variant huaxin gao 于2024年8月21日周三 15:26写道： > +1 for moving variant type to Parquet, as it promotes standardization and > interoperability across numerous projects. > > Huaxin > > On Wed, Aug 21, 2024 at 1:28 PM Yufei Gu wrote: > >>

Re: [VOTE] Deprecate SparkR

+1 Hyukjin Kwon 于2024年8月21日周三 16:46写道： > +1 > > On Thu, 22 Aug 2024 at 05:37, Dongjoon Hyun wrote: > >> +1 >> >> Dongjoon >> >> On 2024/08/21 19:00:46 Holden Karau wrote: >> > +1 >> > >> > Twitter: https://twitter.com/holdenkarau >> > Books (Learning Spark, High Performance Spark, etc.): >> > h

[DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

This sounds like a good idea! The Analyzer is complex. The changes in the new Analyzer should not affect the existing one. The users could add the QO rules and rely on the existing structures and patterns of the logical plan trees generated by the current one. The new Analyzer needs to generate t

Re: [ANNOUNCE] Apache Spark 3.5.2 released

Thank you, Kent! Kent Yao 于2024年8月12日周一 08:03写道： > We are happy to announce the availability of Apache Spark 3.5.2! > > Spark 3.5.2 is the second maintenance release containing security > and correctness fixes. This release is based on the branch-3.5 > maintenance branch of Spark. We strongly re

Re: Welcoming a new PMC member

Congratulations ! Hyukjin Kwon 于2024年8月12日周一 17:20写道： > Hi all, > > The Spark PMC recently voted to add a new PMC member, Kent Yao. Join me in > welcoming him to his new role! > >

Re: [DISCUSS] Deprecating SparkR

+1 Hyukjin Kwon 于2024年8月12日周一 16:18写道： > +1 > > On Tue, Aug 13, 2024 at 7:04 AM Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> And just for the record, the stats that I screenshotted >>

Re: Welcome new Apache Spark committers

Congratulations! Hyukjin Kwon 于2024年8月12日周一 17:19写道： > Hi all, > > The Spark PMC recently voted to add three new committers. Please join me > in welcoming them to their new role! > > - Martin Grund > - Haejoon Lee > - Allison Wang > > They consistently made contributions to the project and clear

Re: [VOTE] Using Github Issues for Spark-Connect-Go _only_ issues.

+1 Matthew Powers 于2024年8月12日周一 09:14写道： > +1 (non-binding) > > On Mon, Aug 12, 2024 at 12:11 PM Denny Lee wrote: > >> +1 (non-binding) >> >> On Mon, Aug 12, 2024 at 16:43 Reynold Xin >> wrote: >> >>> +1 >>> >>> On Mon, Aug 12, 2024 at 10:28 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com>

Re: [VOTE] Archive Spark Documentations in Apache Archives

+1 Mich Talebzadeh 于2024年8月12日周一 13:11写道： > Hi Kent, > > Can you if possible please provide a heuristic estimate of storage > reduction that will be achieved through this approach? > > Thanks > > Mich Talebzadeh, > > Architect | Data Engineer | Data Science | Financial Crime > PhD

Re: [DISCUSS] Differentiate Spark without Spark Connect from Spark Connect

Classic is much better than Legacy. : ) Hyukjin Kwon 于2024年7月18日周四 16:58写道： > Hi all, > > I noticed that we need to standardize our terminology before moving > forward. For instance, when documenting, 'Spark without Spark Connect' is > too long and verbose. Additionally, I've observed that we us

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

+1 Cheng Pan 于2024年5月30日周四 09:48写道： > +1 (non-binding) > > - All links are valid > - Run some basic quires using YARN client mode with Apache Hadoop v3.3.6, > HMS 2.3.9 > - Pass integration tests with Apache Kyuubi v1.9.1 RC0 > > Thanks, > Cheng Pan > > > On May 29, 2024, at 02:48, Wenchen Fan

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 Gengliang Wang 于2024年5月13日周一 16:24写道： > +1 > > On Mon, May 13, 2024 at 12:30 PM Zhou Jiang > wrote: > >> +1 (non-binding) >> >> On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: >> >>> Hi all, >>> >>> I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. >>> >>> Please als

Re: [DISCUSS] Spark 4.0.0 release

+1 for next Monday. We can do more previews when the other features are ready for preview. Tathagata Das 于2024年5月1日周三 08:46写道： > Next week sounds great! Thank you Wenchen! > > On Wed, May 1, 2024 at 11:16 AM Wenchen Fan wrote: > >> Yea I think a preview release won't hurt (without a branch cut

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Sat, Apr 13, 2024 at 17:21 huaxin gao wrote: > +1 > > On Sat, Apr 13, 2024 at 4:36 PM L. C. Hsieh wrote: > >> +1 >> >> On Sat, Apr 13, 2024 at 4:12 PM Hyukjin Kwon >> wrote: >> > >> > +1 >> > >> > On Sun, Apr 14, 2024 at 7:46 AM Chao Sun wrote: >> >> >> >> +1. >> >> >> >> This feature i

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

+1 On Fri, Apr 12, 2024 at 14:30 bo yang wrote: > +1 > > On Fri, Apr 12, 2024 at 12:34 PM huaxin gao > wrote: > >> +1 >> >> On Fri, Apr 12, 2024 at 9:07 AM Dongjoon Hyun >> wrote: >> >>> +1 >>> >>> Thank you! >>> >>> I hope we can customize `dev/merge_spark_pr.py` script per repository >>>

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Hussein Awala 于2024年4月1日周一 08:07写道： > +1(non-binding) I add to the difference will it make that it will also > simplify package maintenance and easily release a bug fix/new feature > without needing to wait for Pyspark to release. > > On Mon, Apr 1, 2024 at 4:56 PM Chao Sun wrote: > >> +1 >>

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 On Tue, Mar 12, 2024 at 6:09 AM Holden Karau wrote: > +1 > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > > On Mon, Mar

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

+1 Xiao Cheng Pan 于2024年2月20日周二 04:59写道： > +1 (non-binding) > > - Build successfully from source code. > - Pass integration tests with Spark ClickHouse Connector[1] > > [1] https://github.com/housepower/spark-clickhouse-connector/pull/299 > > Thanks, > Cheng Pan > > > > On Feb 20, 2024, at 10:5

Re: Re: [DISCUSS] Release Spark 3.5.1?

+1 On Sun, Feb 4, 2024 at 6:07 AM beliefer wrote: > +1 > > > > 在 2024-02-04 15:26:13，"Dongjoon Hyun" 写道： > > +1 > > On Sat, Feb 3, 2024 at 9:18 PM yangjie01 > wrote: > >> +1 >> >> 在 2024/2/4 13:13，“Kent Yao”mailto:y...@apache.org>> 写入: >> >> >> +1 >> >> >> Jungtaek Lim > kabhwan.opensou...@gm

Re: Remove HiveContext from Apache Spark 4.0

Thank you for raising it in the dev list. I do not think we should remove HiveContext based on the cost of break and maintenance. FYI, when releasing Spark 3.0, we had a lot of discussions about the related topics https://lists.apache.org/thread/mrx0y078cf3ozs7czykvv864y6dr55xq Dongjoon Hyun 于2

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 bo yang 于2023年11月15日周三 05:55写道： > +1 > > On Tue, Nov 14, 2023 at 7:18 PM huaxin gao wrote: > >> +1 >> >> On Tue, Nov 14, 2023 at 10:45 AM Holden Karau >> wrote: >> >>> +1 >>> >>> On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: >>> +1 DB Tsai | https://www.dbtsai.com/ | P

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 huaxin gao 于2023年11月9日周四 16:53写道： > +1 > > On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >> +1 >> >> To be completely transparent, I am employed in the same department as >> Zhou at Apple. >> >> I support this proposal, provided that we witness community adoption >> following the release o

Welcome to Our New Apache Spark Committer and PMCs

Hi all, The Spark PMC is delighted to announce that we have voted to add one new committer and two new PMC members. These individuals have consistently contributed to the project and have clearly demonstrated their expertise. New Committer: - Jiaan Geng (focusing on Spark Connect and Spark SQL)

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 Xiao Yuanjian Li 于2023年9月11日周一 10:53写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > this in 3.5.1 and list it as a known issue

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 Xiao Herman van Hovell 于2023年9月6日周三 22:08写道： > Tested connect, and everything looks good. > > +1 > > On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li wrote: > >> Please vote on releasing the following candidate(RC4) as Apache Spark >> version 3.5.0. >> >> The vote is open until 11:59pm Pacific ti

Re: Welcome two new Apache Spark committers

Congratulations, Peter and Xiduo! Debasish Das 于2023年8月6日周日 19:08写道： > Congratulations Peter and Xidou. > > On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan wrote: > >> Hi all, >> >> The Spark PMC recently voted to add two new committers. Please join me in >> welcoming them to their new role! >> >> -

Re: [VOTE] SPIP: XML data source support

+1 On Fri, Jul 28, 2023 at 15:54 Sean Owen wrote: > +1 I think that porting the package 'as is' into Spark is probably > worthwhile. > That's relatively easy; the code is already pretty battle-tested and not > that big and even originally came from Spark code, so is more or less > similar alread

Re: Spark Docker Official Image is now available

Thank you, Yikun! This is great! On Wed, Jul 19, 2023 at 7:55 PM Ruifeng Zheng wrote: > Awesome, thank you YiKun for driving this! > > On Thu, Jul 20, 2023 at 9:12 AM Hyukjin Kwon wrote: > >> This is amazing, finally! >> >> On Thu, 20 Jul 2023 at 10:10, Yikun Jiang wrote: >> >>> The spark Dock

Re: [VOTE][SPIP] Python Data Source API

+1 Xiao Hyukjin Kwon 于2023年7月5日周三 17:28写道： > +1. > > See https://youtu.be/yj7XlTB1Jvc?t=604 :-). > > On Thu, 6 Jul 2023 at 09:15, Allison Wang > wrote: > >> Hi all, >> >> I'd like to start the vote for SPIP: Python Data Source API. >> >> The high-level summary for the SPIP is that it aims to i

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

;>>> >>>> +1 >>>> >>>> On Mon, Jun 12, 2023 at 12:50 PM kazuyuki tanimura >>>> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> Thank you! >>>>> Kazu >>&

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

Thanks for starting the vote. I do have a concern about the target release date of Spark 4.0. L. C. Hsieh 于2023年6月12日周一 11:09写道： > +1 > > On Mon, Jun 12, 2023 at 11:06 AM huaxin gao > wrote: > > > > +1 > > > > On Mon, Jun 12, 2023 at 11:05 AM Dongjoon Hyun > wrote: > >> > >> +1 > >> > >> Dong

Re: Apache Spark 3.4.1 Release?

+1 On Fri, Jun 9, 2023 at 08:30 Wenchen Fan wrote: > +1 > > On Fri, Jun 9, 2023 at 8:52 PM Xinrong Meng wrote: > >> +1. Thank you Doonjoon! >> >> Thanks, >> >> Xinrong Meng >> >> Mridul Muralidharan 于2023年6月9日 周五上午5:22写道： >> >>> >>> +1, thanks Dongjoon ! >>> >>> Regards, >>> Mridul >>> >>> On T

Re: [ANNOUNCE] Apache Spark 3.4.0 released

gt; docker pull apache/spark-r:v3.4.0 > > Thanks, > Dongjoon > > > On Fri, Apr 14, 2023 at 2:56 PM Dongjoon Hyun > wrote: > >> Thank you, Xinrong! >> >> Dongjoon. >> >> >> On Fri, Apr 14, 2023 at 1:37 PM Xiao Li wrote: >> >>>

Re: [ANNOUNCE] Apache Spark 3.4.0 released

Thank you Xinrong! Congratulations everyone! This is a great release with tons of new features! Gengliang Wang 于2023年4月14日周五 13:04写道： > Congratulations everyone! > Thank you Xinrong for driving the release! > > On Fri, Apr 14, 2023 at 12:47 PM Xinrong Meng > wrote: > >> Hi All, >> >> We are

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Xiao Li Emil Ejbyfeldt 于2023年4月12日周三 12:39写道： > +1 (non-binding) > > Ran some tests with the Scala 2.13 build using part of our internal > spark workload. > > On 12/04/2023 19:52, Chris Nauroth wrote: > > +1 (non-binding) > > > > * Verified all chec

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

Thanks for testing it in your environment! > This is a minor issue itself, and only impacts the metrics for push-based > shuffle, but it will essentially completely eliminate the effort > in SPARK-36620. Based on my understanding, this is not a regression. It only affects the new enhancements h

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

Hi, Anton, Could you please provide a complete list of exceptions that are being used in the public connector API? Thanks, Xiao Xinrong Meng 于2023年4月5日周三 12:06写道： > Thank you! > > I created a blocker Jira for that for easier tracking: > https://issues.apache.org/jira/browse/SPARK-43041. > > >

Re: Slack for PySpark users

g official. > > Bests, > Dongjoon. > > > > On Wed, Mar 29, 2023 at 11:32 PM Xiao Li wrote: > >> +1 >> >> + @dev@spark.apache.org >> >> This is a good idea. The other Apache projects (e.g., Pinot, Druid, >> Flink) have created their own dedicate

Re: Slack for PySpark users

+1 + @dev@spark.apache.org This is a good idea. The other Apache projects (e.g., Pinot, Druid, Flink) have created their own dedicated Slack workspaces for faster communication. We can do the same in Apache Spark. The Slack workspace will be maintained by the Apache Spark PMC. I propose to initi

Re: [ANNOUNCE] Apache Spark 3.3.2 released

Thank you, Liang-Chi! Xiao On Sat, Feb 18, 2023 at 1:07 AM beliefer wrote: > Congratulations ! > > > > At 2023-02-17 16:58:22, "L. C. Hsieh" wrote: > >We are happy to announce the availability of Apache Spark 3.3.2! > > > >Spark 3.3.2 is a maintenance release containing stability fixes. This

Re: Welcome Yikun Jiang as a Spark committer

Congratulations, Yikun! Xiao Yikun Jiang 于2022年10月9日周日 19:34写道： > Thank you all! > > Regards, > Yikun > > > On Mon, Oct 10, 2022 at 3:18 AM Chao Sun wrote: > >> Congratulations Yikun! >> >> On Sun, Oct 9, 2022 at 11:14 AM vaquar khan >> wrote: >> >>> Congratulations. >>> >>> Regards, >>> Vaqu

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

+1. Xiao On Wed, Oct 5, 2022 at 12:49 PM Sean Owen wrote: > I'm OK with this. It simplifies maintenance a bit, and specifically may > allow us to finally move off of the ancient version of Guava (?) > > On Mon, Oct 3, 2022 at 10:16 PM Dongjoon Hyun > wrote: > >> Hi, All. >> >> I'm wondering if

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

+1 Yikun Jiang 于2022年9月21日周三 07:22写道： > Thanks for all your inputs! BTW, I also create a JIRA to track related > work: https://issues.apache.org/jira/browse/SPARK-40513 > > > can I be involved in this work? > > @qian Of course! Thanks! > > Regards, > Yikun > > On Wed, Sep 21, 2022 at 7:31 PM Xin

Welcoming three new PMC members

Hi all, The Spark PMC recently voted to add three new PMC members. Join me in welcoming them to their new roles! New PMC members: Huaxin Gao, Gengliang Wang and Maxim Gekk The Spark PMC

Re: Apache Spark 3.2.2 Release?

+1 Xiao Cheng Su 于2022年7月6日周三 19:16写道： > +1 (non-binding) > > Thanks, > Cheng Su > > On Wed, Jul 6, 2022 at 6:01 PM Yuming Wang wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 5:53 AM Maxim Gekk >> wrote: >> >>> +1 >>> >>> On Thu, Jul 7, 2022 at 12:26 AM John Zhuge wrote: >>> +1 Thanks for

Re: [PSA] Please rebase and sync your master branch in your forked repository

Thank you, Hyukjin! Xiao On Mon, Jun 20, 2022 at 7:01 PM Yi Wu wrote: > Thanks for the work, Hyukjin. > > On Tue, Jun 21, 2022 at 7:59 AM Yuming Wang wrote: > >> Thank you Hyukjin. >> >> On Tue, Jun 21, 2022 at 7:46 AM Hyukjin Kwon wrote: >> >>> After https://github.com/apache/spark/pull/3692

Re: Re: [VOTE][SPIP] Spark Connect

+1 Xiao beliefer 于2022年6月14日周二 03:35写道： > +1 > Yeah, I tried to use Apache Livy, so as we can runing interactive query. > But the Spark Driver in Livy looks heavy. > > The SPIP may resolve the issue. > > > > At 2022-06-14 18:11:21, "Wenchen Fan" wrote: > > +1 > > On Tue, Jun 14, 2022 at 9:38 A

Stickers and Swag

Hi, all, The ASF has an official store at RedBubble that Apache Community Development (ComDev) runs. If you are interested in buying Spark Swag, 70 products featuring the Spark logo are available: https://www.redbubble.com/shop/ap/113203780 Go Spark!

Re: 回复： [VOTE] Release Spark 3.3.0 (RC6)

+1 Xiao beliefer 于2022年6月13日周一 20:04写道： > +1 AFAIK, no blocking issues now. > Glad to hear to release 3.3.0 ! > > > 在 2022-06-14 09:38:35，"Ruifeng Zheng" 写道： > > +1 (non-binding) > > Maxim, thank you for driving this release! > > thanks, > ruifeng > > > > -- 原始邮件 --

Re: SIGMOD System Award for Apache Spark

Congratulations to everyone! Xiao On Fri, May 13, 2022 at 9:34 AM Dongjoon Hyun wrote: > Ya, it's really great!. Congratulations to the whole community! > > Dongjoon. > > On Fri, May 13, 2022 at 8:12 AM Chao Sun wrote: > >> Huge congrats to the whole community! >> >> On Fri, May 13, 2022 at 1:

Re: Apache Spark 3.3 Release

ot blocking what you want to > do. > > Please let the community start to ramp down as we agreed before. > > Dongjoon > > > > On Tue, Mar 15, 2022 at 3:07 PM Xiao Li wrote: > >> Please do not get me wrong. If we don't cut a branch, we are allowing all >&g

Re: Apache Spark 3.3 Release

we need to avoid backporting the feature work that are not being well > discussed. > > > > On Tue, Mar 15, 2022 at 12:12 PM Xiao Li wrote: > >> Cutting the branch is simple, but we need to avoid backporting the >> feature work that are not being well discussed. Not

Re: Apache Spark 3.3 Release

don't cut a branch. > > [SPARK-38335][SQL] Implement parser support for DEFAULT column values > > Let's cut `branch-3.3` Today for Apache Spark 3.3.0 preparation. > > Best, > Dongjoon. > > > On Tue, Mar 15, 2022 at 10:17 AM Chao Sun wrote: > >> Cool, th

Re: Apache Spark 3.3 Release

ew minutes ago. So, we can remove > it from the list. > > > > #35819 [SPARK-38524][SPARK-38553][K8S] Bump Volcano to v1.5.1 > > > > Thanks, > > Dongjoon. > > > > On Tue, Mar 15, 2022 at 9:48 AM Xiao Li wrote: > >> > >> Let me clarify my

Re: Apache Spark 3.3 Release

Let me clarify my above suggestion. Maybe we can wait 3 more days to collect the list of actively developed PRs that we want to merge to 3.3 after the branch cut? Please do not rush to merge the PRs that are not fully reviewed. We can cut the branch this Friday and continue merging the PRs that ha

Re: Apache Spark 3.3 Release

https://github.com/apache/spark/pull/35395 > - https://github.com/apache/spark/pull/35657 > > are actively being reviewed. It seems there are ongoing PRs for other > SPIPs as well but I'm not involved in those so not quite sure whether > they are intended for 3.3 release. > >

Re: Apache Spark 3.3 Release

Could you please list which features we want to finish before the branch cut? How long will they take? Xiao Chao Sun 于2022年3月14日周一 13:30写道： > Hi Max, > > As there are still some ongoing work for the above listed SPIPs, can we > still merge them after the branch cut? > > Thanks, > Chao > > On Mo

Re: [VOTE] SPIP: Catalog API for view metadata

Can we extend the voting window to next Wednesday? This week is a holiday week for the lunar new year. AFAIK, many members in Asia are taking the whole week off. They might not regularly check the emails. Also how about starting a separate email thread starting with [VOTE] ? Happy Lunar New Year!

Re: [Apache Spark Jenkins] build system shutting down Dec 23th, 2021

Hi, Shane, Thank you for your work on it! Xiao On Mon, Dec 6, 2021 at 6:20 PM L. C. Hsieh wrote: > Thank you, Shane. > > On Mon, Dec 6, 2021 at 4:27 PM Holden Karau wrote: > > > > Shane you kick ass thank you for everything you’ve done for us :) Keep > on rocking :) > > > > On Mon, Dec 6,

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Thank you! Great job! Xiao On Fri, Nov 12, 2021 at 7:02 PM Mridul Muralidharan wrote: > > Nice job ! > There are some nice API's which should be interesting to explore with JDK > 17 :-) > > Regards. > Mridul > > On Fri, Nov 12, 2021 at 7:08 PM Yuming Wang wrote: > >> Cool, thank you Dongjoon.

Re: [ANNOUNCE] Apache Spark 3.2.0

Thank you, Gengliang! Congrats to our community and all the contributors! Xiao Henrik Peng 于2021年10月19日周二 上午8:26写道： > Congrats and thanks! > > > Gengliang Wang 于2021年10月19日 周二下午10:16写道： > >> Hi all, >> >> Apache Spark 3.2.0 is the third release of the 3.x line. With tremendous >> contribution

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Xiao Li Yi Wu 于2021年10月11日周一 上午12:08写道： > +1 (non-binding) > > On Mon, Oct 11, 2021 at 1:57 PM Holden Karau wrote: > >> +1 >> >> On Sun, Oct 10, 2021 at 10:46 PM Wenchen Fan wrote: >> >>> +1 >>> >>> On Sat, Oct

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Chao, How long will it take? Normally, in the RC stage, we always revert the upgrade made in the current release. We did the parquet upgrade multiple times in the previous releases for avoiding the major delay in our Spark release Thanks， Xiao On Tue, Aug 31, 2021 at 11:03 AM Chao Sun wro

Re: [build system] half of the jenkins workers are down

Thank you, Shane! Xiao shane knapp ☠ 于2021年8月9日周一 下午1:26写道： > turns out that minikube/k8s and friends were being oom-killed and this was > causing all sorts of weirdnesses. > > i've upped the ram limits on all of the k8s jobs to 8G (from 6G), and > we'll keep an eye on things and see how they g

Re: Flaky build in GitHub Actions

Thank you, Liang-chi and Hyukjin! On Sun, Jul 25, 2021 at 6:25 PM Hyukjin Kwon wrote: > This is fixed up via Laingchi's PR: > https://github.com/apache/spark/pull/33447. The issue is almost fixed now > and less flaky. > I'm still interacting w/ GitHub Actions: they are still investigating the >

Re: Apache Spark 3.2 Expectation

> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft > cut and the committers still are able to commit to `branch-3.3` according > to their decisions. First, I think you are saying "branch-3.2"; Second, the "so cut" means no "code freeze", although we cut the branch. To

Re: [ANNOUNCE] Apache Spark 3.1.2 released

Thank you! Xiao On Tue, Jun 1, 2021 at 9:29 PM Hyukjin Kwon wrote: > awesome! > > 2021년 6월 2일 (수) 오전 9:59, Dongjoon Hyun 님이 작성: > >> We are happy to announce the availability of Spark 3.1.2! >> >> Spark 3.1.2 is a maintenance release containing stability fixes. This >> release is based on the b

Re: Apache Spark 3.1.2 Release?

+1 Thanks, Dongjoon! Xiao On Mon, May 17, 2021 at 8:45 PM Kent Yao wrote: > +1. thanks Dongjoon > > *Kent Yao * > @ Data Science Center, Hangzhou Research Institute, NetEase Corp. > *a spark enthusiast* > *kyuubi is a unified multi-tenant JDBC > interface f

Re: Welcoming six new Apache Spark committers

Congratulations, everyone! Xiao Chao Sun 于2021年3月26日周五 下午6:30写道： > Congrats everyone! > > On Fri, Mar 26, 2021 at 6:23 PM Mridul Muralidharan > wrote: > >> >> Congratulations, looking forward to more exciting contributions ! >> >> Regards, >> Mridul >> >> On Fri, Mar 26, 2021 at 8:21 PM Dongjo

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1 Xiao Takeshi Yamamuro 于2021年3月26日周五 下午4:14写道： > +1 (non-binding) > > On Sat, Mar 27, 2021 at 4:53 AM Liang-Chi Hsieh wrote: > >> +1 (non-binding) >> >> >> rxin wrote >> > +1. Would open up a huge persona for Spark. >> > >> > On Fri, Mar 26 2021 at 11:30 AM, Bryan Cutler < >> >> > cutlerb@ >

Re: Apache Spark 3.2 Expectation

Below are some nice-to-have features we can work on in Spark 3.2: Lateral Join support , interval data type, timestamp without time zone, un-nesting arbitrary queries, the returned metrics of DSV2, and error message standardization. Spark 3.2 will

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Thank you, Liang-Chi! Xiao On Thu, Mar 4, 2021 at 6:25 PM Hyukjin Kwon wrote: > Thanks @Liang-Chi Hsieh for driving this. > > 2021년 3월 5일 (금) 오전 5:21, Liang-Chi Hsieh 님이 작성: > >> >> Thanks all for the input. >> >> If there is no objection, I am going to cut the branch next Monday. >> >> Thanks

Re: Apache Spark 3.2 Expectation

Thank you, Dongjoon, for initiating this discussion. Let us keep it open. It might take 1-2 weeks to collect from the community all the features we plan to build and ship in 3.2 since we just finished the 3.1 voting. > 3. +100 for Apache Spark 3.2.0 in July 2021. Maybe, we need `branch-cut` > in

Re: [VOTE] Release Spark 3.1.1 (RC3)

indeed cause for concern. >>> +1 on extending the voting deadline until we finish investigation of >>> this. >>> >>> Regards, >>> Mridul >>> >>> >>> On Wed, Feb 24, 2021 at 12:55 PM Xiao Li wrote: >>> >>>>

Re: [VOTE] Release Spark 3.1.1 (RC3)

-1 Could we extend the voting deadline? A few TPC-DS queries (q17, q18, q39a, q39b) are returning different results between Spark 3.0 and Spark 3.1. We need a few more days to understand whether these changes are expected. Xiao Mridul Muralidharan 于2021年2月24日周三 上午10:41写道： > > Sounds good, tha

Re: Apache Spark 3.0.2 Release ?

+1 Happy Lunar New Year! Xiao On Fri, Feb 12, 2021 at 5:33 PM Hyukjin Kwon wrote: > Yeah, +1 too > > 2021년 2월 13일 (토) 오전 4:49, Dongjoon Hyun 님이 작성: > >> Thank you, Sean! >> >> On Fri, Feb 12, 2021 at 11:41 AM Sean Owen wrote: >> >>> Sounds like a fine time to me, sure. >>> >>> On Fri, Feb 12,

Re: [VOTE] Release Spark 3.1.0 (RC1)

> I will prepare to upload news in spark-website to explain that 3.1.0 is incompletely published because there was something wrong during the release process, and we go to 3.1.1 right away. +1 Sean Owen 于2021年1月7日周四 上午6:44写道： > While we can delete the tag, maybe just leave it. As a general rule

Re: [build system] WE'RE LIVE!

Thank you, Shane! Xiao On Tue, Dec 1, 2020 at 5:34 PM Dongjoon Hyun wrote: > Yay! Thanks! > > Bests, > Dongjoon > > On Tue, Dec 1, 2020 at 5:31 PM Takeshi Yamamuro > wrote: > >> Many thanks, guys! >> I've checked I can re-trigger Jenkins tests. >> >> Bests, >> Takeshi >> >> On Wed, Dec 2, 2020

Re: Seeking committers' help to review on SS PR

Just want to say thank you to all the active SS contributors. I saw many great features/improvements in Streaming have been merged and will be available in the upcoming 3.1 release. - Cache fetched list of files beyond maxFilesPerTrigger as unread file (SPARK-32568) - Streamline the logic

Re: jenkins downtime tomorrow evening/weekend

Thank you, Shane! On Mon, Nov 23, 2020 at 2:12 PM shane knapp ☠ wrote: > the third most terrifying event in the world, a massive jenkins plugin > update is happening in a couple of hours. i'm going to restart jenkins and > start working out any bugs/issues that pop up. > > this could be short,

Re: Spark 3.1 branch cut 4th Dec?

has been decided, it's just a matter of normal >> review comments. >> >> On Fri, Nov 20, 2020 at 9:05 AM Dongjoon Hyun >> wrote: >> >>> Thank you for sharing, Xiao. >>> >>> I hope we are able to make some agreement for CREATE TABLE DD

Re: Spark 3.1 branch cut 4th Dec?

our best to address it in 3.1. Thanks, Xiao Xiao Li 于2020年11月20日周五 上午8:52写道： > Hi, Dongjoon, > > Thank you for your feedback. I think *Early December* does not mean we > will cut the branch on Dec 1st. I do not think Dec 1st and Dec 4th are a > big deal. Normally, it would

Re: Spark 3.1 branch cut 4th Dec?

us if you make it clear what > features you are waiting for now. > > We are creating Apache Spark together. > > Bests, > Dongjoon. > > > On Thu, Nov 19, 2020 at 11:38 PM Xiao Li wrote: > >> Correction: >> >> Merging the feature work after the branch cut

Re: Spark 3.1 branch cut 4th Dec?

have two weeks ahead of the proposed branch cut date. I hope each feature owner might hurry up and try to finish it before the branch cut. Xiao Xiao Li 于2020年11月19日周四 下午11:36写道： > We should try to merge the feature work after the branch cut. This should > not be encouraged in general, al

  1   2   3   4   >