Re: [VOTE][RESULT] New Spark Connect Client Repository for Rust

FYI, new repo is created at the proposed location with the ASF license: https://github.com/apache/spark-connect-rust.git Yuanjian Li 于2025年5月26日周一 00:31写道： > The vote passes with 11 +1s (4 binding +1s) and no -1s. > Thank you to all who participated in the discussion/vote and for a

[VOTE][RESULT] New Spark Connect Client Repository for Rust

The vote passes with 11 +1s (4 binding +1s) and no -1s. Thank you to all who participated in the discussion/vote and for all the feedback! (*** = binding) *+1:* Jules Damji Yuanjian Li * Huaxin Gao * Peter Toth Renjie Liu Denny Lee Niranjan Jayakar Andy Grove Jungtaek Lim Wenchen Fan*

Re: [DISCUSS] Apache Spark 3.5.6 (automated release)

+1 — super excited to have this, Hyukjin! Yang Jie 于2025年5月19日周一 22:55写道： > +1, It would be truly awesome if we could use GitHub Actions for release. > > On 2025/05/20 05:08:16 Wenchen Fan wrote: > > +1, look forward to Spark's first automated release! > > > > On Tue, May 20, 2025 at 12:55 PM Yu

Re: [VOTE] New Spark Connect Client Repository for Rust

t who will author the > code and who will review? Also how many active committers + PMC members are > familiar with Rust language? > > On Tue, May 20, 2025 at 4:15 AM Yuanjian Li > wrote: > >> For reference, we are carrying forward the votes from the discussion >> threa

Re: [VOTE] New Spark Connect Client Repository for Rust

For reference, we are carrying forward the votes from the discussion thread: (*** = binding) *+1:* Jules Damji Yuanjian Li * Huaxin Gao * Peter Toth Renjie Liu Denny Lee Niranjan Jayakar *+0: *None *-1:*None Yuanjian Li ： > Please vote on the proposal to introduce a new Apa

[VOTE] New Spark Connect Client Repository for Rust

Please vote on the proposal to introduce a new Apache Spark repository for a Spark Connect Client in Rust. This vote will remain open for the next 72 hours and will pass if it receives a majority of +1 PMC votes, with a minimum of 3 binding +1 votes. As discussed in the previous thread: https://li

Re: [DISCUSS] New Spark Connect Client repository for Rust language

his approach helped to lower contribution bar as it would be easier to set >>>> up dev environment for one language. >>>> >>>> On Mon, May 19, 2025 at 1:50 PM Peter Toth >>>> wrote: >>>> >>>>> +1 >>

Re: [DISCUSS] New Spark Connect Client repository for Rust language

Happy to see the Rust client proposal and glad to help with the follow-up repo management work. Let’s use this thread for voting—posting my +1 here. Jules Damji ： > I’ll ask them. > > Cheers > Jules > > Excuse the thumb typos > > > On Fri, 16 May 2025 at 2:38 AM, Renjie Liu > wrote: > >> Thank

Re: [VOTE] Release Spark 4.0.0 (RC6)

+1 On Wed, May 14, 2025 at 07:27 Rozov, Vlad wrote: > +1 to wait for the fix. > > Thank you, > > Vlad > > > On May 14, 2025, at 5:30 AM, Weichen Xu > wrote: > > Hi folks, > > We have a RCE fix https://github.com/apache/spark/pull/50889 pending > merging, and it needs to be backported to 4.0.0.

Re: [VOTE] Release Spark 4.0.0 (RC5)

+1 Szehon Ho 于2025年5月12日周一 22:32写道： > +1 (non binding) > > Checked license, signature, checksum, ran basic test on > spark-4.0.0-bin-hadoop3. > > Thanks > Szehon > > On Mon, May 12, 2025 at 9:02 PM Sakthi wrote: > >> +1 (non-binding) >> >> On Mon, May 12, 2025 at 7:38 PM Jungtaek Lim < >> kabhw

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

+1 I would like to add my perspective to this discussion. It's important that we maintain a fair and objective technical evaluation process, especially when a veto is cast. A veto should be backed by concrete technical reasoning. >From my reading of the discussion, the key concern is whether the

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 Hyukjin Kwon 于2024年7月4日周四 16:54写道： > (I will leave this vote open till 10th July, considering that its holiday > season in US) > > On Fri, 5 Jul 2024 at 06:12, Martin Grund wrote: > >> +1 (non-binding) >> >> On Thu, Jul 4, 2024 at 7:15 PM Holden Karau >> wrote: >> >>> +1 >>> >>> Although gi

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Chao Sun 于2024年4月1日周一 07:56写道： > +1 > > On Sun, Mar 31, 2024 at 10:31 PM Hyukjin Kwon > wrote: > >> Oh I didn't send the discussion thread out as it's pretty simple, >> non-invasive and the discussion was sort of done as part of the Spark >> Connect initial discussion .. >> >> On Mon, Apr 1,

Re: [VOTE] SPIP: State Data Source - Reader

+1 Jungtaek Lim 于2023年10月25日周三 01:06写道： > Friendly reminder: the VOTE thread got 2 binding votes and needs 1 more > binding vote to pass. > > On Wed, Oct 25, 2023 at 1:21 AM Bartosz Konieczny > wrote: > >> +1 >> >> On Tuesday, October 24, 2023, Jia Fan wrote: >> >>> +1 >>> >>> L. C. Hsieh 于20

Re: [DISCUSS] SPIP: State Data Source - Reader

+1, I have no issues with the practicality and value of this feature itself. I've left some comments concerning ongoing maintenance and compatibility-related matters, which we can continue to discuss. Jungtaek Lim 于2023年10月17日周二 05:23写道： > Thanks Bartosz and Anish for your support! > > I'll wait

Re: [ANNOUNCE] Apache Spark 3.5.0 released

s release, Congratulations! > > On Mon, Sep 18, 2023 at 2:16 PM Maxim Gekk > wrote: > >> Thank you for the work, Yuanjian! >> >> On Mon, Sep 18, 2023 at 6:28 AM beliefer wrote: >> >>> Congratulations! Apache Spark. >>> >>> >>> >>

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 Denny Lee 于2023年9月26日周二 12:07写道： > +1 > > On Tue, Sep 26, 2023 at 10:52 Maciej wrote: > >> +1 >> >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >> On 9/26/23 17:12, Michel Miotto Barbosa wrote: >> >> +1 >> >> A disposição | At your disposal

[ANNOUNCE] Apache Spark 3.5.0 released

Hi All, We are happy to announce the availability of *Apache Spark 3.5.0*! Apache Spark 3.5.0 is the sixth release of the 3.x line. To download Spark 3.5.0, head over to the download page: https://spark.apache.org/downloads.html (Please note: the PyPi upload is pending due to a size limit reques

[VOTE][RESULT] Release Apache Spark 3.5.0 (RC5)

The vote passes with 13 +1s (8 binding +1s). Thank you all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Yuanjian Li - Xiao Li (*) - Gengliang Wang (*) - Hyukjin Kwon (*) - Ruifeng Zheng (*) - Jungtaek Lim - Wenchen Fan (*) - Jia Fan - Jie Yang - Yuming Wang

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding) Yuanjian Li 于2023年9月11日周一 09:36写道： > @Peter Toth I've looked into the details of this > issue, and it appears that it's neither a regression in version 3.5.0 nor a > correctness issue. It's a bug related to a new feature. I think we can fix > thi

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

3.5.0. Mridul Muralidharan 于2023年9月10日周日 04:12写道： > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > On Sat, Sep 9, 2023 at 10:02 AM Yuanjian Li > wrote: >

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

>> >> - https://github.com/apache/spark/pull/42850 >> - >> https://github.com/apache/spark/commit/b2b2ba97d3003d25d159943ab8a4bf50e421fdab >> (branch-3.5) >> >> Dongjoon. >> >> >>>>>> >>>>>> On Wed, Sep 6, 2023 a

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

om/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> discl

[VOTE] Release Apache Spark 3.5.0 (RC5)

t been correctly targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

d25d159943ab8a4bf50e421fdab > (branch-3.5) > > Dongjoon. > > >>>>> >>>>> On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li >>>>> wrote: >>>>> >>>>> Please vote on releasing the following candidate(RC4) as Apache Spark >&g

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 (non-binding) Xiao Li 于2023年9月6日周三 15:27写道： > +1 > > Xiao > > Herman van Hovell 于2023年9月6日周三 22:08写道： > >> Tested connect, and everything looks good. >> >> +1 >> >> On Wed, Sep 6, 2023 at 8:11 AM Yuanjian Li >> wrote: >> >>

Release Note of Apache Spark 3.5.0

draft release note of Apache Spark 3.5.0 and feel free to add your comments if any. Thanks, Yuanjian Li

[VOTE] Release Apache Spark 3.5.0 (RC4)

t been correctly targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

Sure, no problem. Holden Karau 于2023年9月2日周六 22:10写道： > Can we delay the next RC cut until after Labor Day? > > On Sat, Sep 2, 2023 at 9:59 PM Yuanjian Li wrote: > >> Thank you for all the reports! >> The vote has failed. I plan to cut RC4 in two days. >> >

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

42393 . I'm -1 and let's include >> this fix in 3.5. >> >> Thanks, >> Wenchen >> >> On Thu, Aug 31, 2023 at 9:09 PM Ian Manning >> wrote: >> >>> +1 (non-binding) >>> >>> Using Spark Core, Spark SQL, Structured Streamin

[VOTE] Release Apache Spark 3.5.0 (RC3)

t been correctly targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC2)

due to SPARK-43646 <https://issues.apache.org/jira/browse/SPARK-43646> > and SPARK-44784 <https://issues.apache.org/jira/browse/SPARK-44784> not > yet being fixed. > > > > Jie Yang > > > > *发件人**: *Sean Owen > *日期**: *2023年8月20日 星期日 04:43 > *收件人**: *

[VOTE] Release Apache Spark 3.5.0 (RC2)

t been correctly targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

://github.com/apache/spark/actions/runs/5832898984/job/15819181762 >> >> >> >> I think we should address this issue before the release of Apache Spark >> 3.5.0. >> >> >> >> Jie Yang >> >> >> >> *发件人**: *Yuanjian Li >

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

l/42236. >> >> >> >> *发件人**: *Sean Owen >> *日期**: *2023年8月7日 星期一 11:05 >> *收件人**: *Yuanjian Li >> *抄送**: *Spark dev list >> *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC1) >> >> >> -- >> >>

[VOTE] Release Apache Spark 3.5.0 (RC1)

t been correctly targeted please ping me or a committer to help target the issue. Thanks, Yuanjian Li

[Reminder] Spark 3.5 RC Cut

Hi everyone, Following the release timeline, I will cut the RC on* Tuesday, Aug 1st at 1 pm PST* as scheduled. DateEvent July 17th 2023 Late July 2023 Code freeze. Release branch cut. QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. August 2023 Releas

Re: Spark 3.5 Branch Cut

begin your QA against branch-3.5 now. Thank you! Raghu Angadi 于2023年7月17日周一 13:29写道： > Thanks Yuanjian for accepting these for warmfix. > > Raghu. > > On Mon, Jul 17, 2023 at 1:04 PM Yuanjian Li > wrote: > >> Hi, all >> >> FYI, I cut branch-3.5 as https://

Spark 3.5 Branch Cut

Hi, all FYI, I cut branch-3.5 as https://github.com/apache/spark/tree/branch-3.5 Here is the complete list of exception merge requests received before the cut: - SPARK-44421: Reattach to existing execute in Spark Connect (server mechanism) - SPARK-44423: Reattach to existing ex

Re: Time for Spark v3.5.0 release

gt;>>>> > >>>>> > >>>>> > 发件人: Maxim Gekk >>>>> > 日期: 2023年7月4日 星期二 17:24 >>>>> > 收件人: Kent Yao >>>>> > 抄送: "dev@spark.apache.org" >>>>> > 主题: Re: Time for Spark v3.5.0

[Reminder] Spark 3.5 Branch Cut

Hi everyone, As discussed earlier in "Time for Spark v3.5.0 release", I will cut branch-3.5 on *Monday, July 17th at 1 pm PST* as scheduled. Please plan your PR merge accordingly with the given timeline. Currently, we have received the following exception merge requests: - SPARK-44421: Reattac

Time for Spark v3.5.0 release

Hi All, According to the Spark versioning policy at https://spark.apache.org/versioning-policy.html, should we cut *branch-3.5* on *July 17th, 2023*? (We initially proposed January 16th, but since it's a Sunday, I suggest we postpone it by one day). I would like to volunteer as the release manage

Re: Welcoming three new PMC members

Congrats everyone! L. C. Hsieh 于2022年8月9日 周二19:01写道： > Congrats! > > On Tue, Aug 9, 2022 at 5:38 PM Chao Sun wrote: > > > > Congrats everyone! > > > > On Tue, Aug 9, 2022 at 5:36 PM Dongjoon Hyun > wrote: > > > > > > Congrat to all! > > > > > > Dongjoon. > > > > > > On Tue, Aug 9, 2022 at 5:13

Re: Welcome Xinrong Meng as a Spark committer

Congratulations, Xinrong! XiDuo You 于2022年8月9日 周二19:18写道： > Congratulations! > > Haejoon Lee 于2022年8月10日周三 09:30写道： > > > > Congrats, Xinrong!! > > > > On Tue, Aug 9, 2022 at 5:12 PM Hyukjin Kwon wrote: > >> > >> Hi all, > >> > >> The Spark PMC recently added Xinrong Meng as a committer on the

Re: [DISCUSS] Add RocksDB StateStore

Hi all, Following the latest comments in SPARK-34198 , Databricks decided to donate the commercial implementation of the RocksDBStateStore. Compared with the original decision, there’s only one topic we want to raise again for discussion: can we d

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Awesome! Thanks for making this happen, Hyukjin! Yi Wu 于2021年4月14日周三 下午2:51写道： > Thanks for the great work, Hyukjin! > > On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: > >> Thanks for the amazing work, Hyukjin! >> I created a PR for trial and it looks well so far: >> https://github.com/a

Re: Welcoming six new Apache Spark committers

Congrats all! Well deserved!! Yi Wu 于2021年3月29日周一 上午10:01写道： > Thank you, everyone! Thanks for all the help! > > Yi > > On Sun, Mar 28, 2021 at 4:53 PM Gengliang Wang wrote: > >> Congrats all! >> >> On Sun, Mar 28, 2021 at 7:09 AM Xiao Li wrote: >> >>> Congratulations, everyone! >>> >>> Xiao >

Re: What's the root cause of not supporting multiple aggregations in structured streaming?

gt; Regarding this subject I wrote a blog article that gives details about the > watermark architecture proposal that was discussed in the design doc and in > the PR: > > > https://echauchot.blogspot.com/2020/11/watermark-architecture-proposal-for.html > > Best > > Etienne >

Re: [DISCUSS] Disable streaming query with possible correctness issue by default

Already +1 in the PR. It would be great to mention the new config in the SS migration guide. Ryan Blue 于2020年11月11日周三 上午7:48写道： > +1, I agree with Tom. > > On Tue, Nov 10, 2020 at 3:00 PM Dongjoon Hyun > wrote: > >> +1 for Apache Spark 3.1.0. >> >> Bests, >> Dongjoon. >> >> On Tue, Nov 10, 2020

Re: What's the root cause of not supporting multiple aggregations in structured streaming?

Thanks for the great discussion! Also interested in this feature and did some investigation before. As Arun mentioned, similar to the "update" mode, the "complete" mode also needs more design. We might need an operation level output mode for the complete mode support. That is to say, if we use "co

Re: Welcoming some new Apache Spark committers

Congratulations!! huaxin gao 于2020年7月16日周四 上午6:24写道： > Thanks everyone! I am looking forward to working with you all in the > future. > > On Tue, Jul 14, 2020 at 5:02 PM Hyukjin Kwon wrote: > >> Congrats! >> >> 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: >> >>> Congrats, all! >>> >>> On W

Re: [DISCUSS] Drop Python 2, 3.4 and 3.5

+1, especially Python 2 Holden Karau 于2020年7月2日周四 上午10:20写道： > I’m ok with us dropping Python 2, 3.4, and 3.5 in Spark 3.1 forward. It > will be exciting to get to use more recent Python features. The most recent > Ubuntu LTS ships with 3.7, and while the previous LTS ships with 3.5, if > folks

[DISCUSS] Apache Spark 3.0.1 Release

Hi dev-list, I’m writing this to raise the discussion about Spark 3.0.1 feasibility since 4 blocker issues were found after Spark 3.0.0: 1. [SPARK-31990] The state store compatibility broken will cause a correctness issue when Stre

Re: [VOTE] Release Spark 2.4.6 (RC1)

pache Spark 2.4.0. >>> >>> Bests, >>> Dongjoon. >>> >>> On Fri, May 8, 2020 at 10:26 AM Holden Karau >>> wrote: >>> >>>> Can you provide a bit more context (is it a regression?) >>>> >>>> On Fri, May

Re: [VOTE] Release Spark 2.4.6 (RC1)

Hi Holden, I'm working on the bugfix of SPARK-31663 , let me post it here since it's a correctness bug and also affects 2.4.6. Best, Yuanjian Sean Owen 于2020年5月8日周五 下午11:42写道： > +1 from me. The usual: sigs OK, license looks as intended, tests

Re: [DISCUSS] PostgreSQL dialect

Thanks all of you for joining the discussion. The PR is given in https://github.com/apache/spark/pull/26763, all the PostgreSQL dialect related PRs are linked in the description. Hoping the authors could help in reviewing. Best, Yuanjian Driesprong, Fokko 于2019年12月1日周日 下午7:24写道： > +1 (non-bindi

Re: Welcoming some new committers and PMC members

Congratulations! sujith chacko 于2019年9月10日周二 上午10:15写道： > Congratulations all. > > On Tue, 10 Sep 2019 at 7:27 AM, Haibo wrote: > >> congratulations~ >> >> >> >> 在2019年09月10日 09:30，Joseph Torres >> 写道： >> >> congratulations! >> >> On Mon, Sep 9, 2019 at 6:27 PM 王 斐 wrote: >> >>> congratulatio

Re: [SPARK-23207] Repro

Hi Tyson, Thanks for the reporting! I reproduced this locally based on your code with some changes, which only keep the wrong answer job. The code as below: import scala.sys.process._ import org.apache.spark.TaskContext val res = spark.range(0, 1 * 1, 1).map{ x => (x % 1000, x)} // kill

Re: Welcome Jose Torres as a Spark committer

Congrats Jose! Best, Yuanjian Takeshi Yamamuro 于2019年1月30日周三 上午8:21写道： > Congrats, Jose! > > Best, > Takeshi > > On Wed, Jan 30, 2019 at 6:10 AM Jungtaek Lim wrote: > >> Congrats Jose! Well deserved. >> >> - Jungtaek Lim (HeartSaVioR) >> >> 2019년 1월 30일 (수) 오전 5:19, Dongjoon Hyun 님이 작성: >> >>>

Re: Continuous task retry support

> > *I found that task retries are currently not supported > > in > continuous processing mode. Is there an

Re: [DISCUSS] SPIP: Native support of session window

Cool, thanks! Sorry for the late reply, we'll check out the UT and your design doc ASAP when we back from National Day holiday. Thanks, Yuanjian Li Jungtaek Lim 于2018年9月29日周六 上午5:21写道： > Btw, just wrote up detailed design doc on existing patch: > > https://docs.google.

Re: welcome a new batch of committers

Congratulations to all and thanks for all your help!! Bhupendra Mishra 于2018年10月6日周六 上午11:38写道： > Congratulations to all of you > Good Luck > Regards > > On Wed, Oct 3, 2018 at 2:29 PM Reynold Xin wrote: > >> Hi all, >> >> The Apache Spark PMC has recently voted to add several new committers to

Re: [DISCUSS] SPIP: Native support of session window

<https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing> Thanks, Yuanjian Li > 在 2018年9月28日，06:22，Jungtaek Lim <mailto:kabh...@gmail.com>> 写道： > > Hi all, > > I would like to initiate discussion thread to discuss "Native support

Re: Something wrong of Jenkins proxy

un, Sep 23, 2018 at 8:37 PM, shane knapp wrote: >> >>> i just noticed this... taking a look now. >>> >>> On Sun, Sep 23, 2018 at 4:38 AM, Yuanjian Li >>> wrote: >>> >>>> Hi devs, >>>> Is there something wrong of Jenkins

Something wrong of Jenkins proxy

Hi devs, Is there something wrong of Jenkins proxy? [image: image.png] I got this proxy 500 whole days. Thanks, Yuanjian Li

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

will be done at October as expect. We'll post more benchmark and detailed work at that time. I'm still reading your discussion document and happy to give more feedback in the doc. Thanks, Yuanjian Li Matt Cheah 于2018年9月1日周六 上午8:42写道： > Hi everyone, > > > > I

Re: [DISCUSS] Adaptive execution in Spark SQL

componentsalgorithm in GraphFrame. With enabling AE, the duration of app reduce from 58min to 32min, almost 100% boosting on performance improvement. The detailed screenshot and config in the JIRA SPARK-23128 <https://issues.apache.org/jira/browse/SPARK-23128> attached pdf. Thanks, Yuanjian Li Wang, Car

Re: Design for continuous processing shuffle

Hi Joseph and devs, Happy to see the discussion of CP shuffle, as comment in https://issues.apache.org/jira/browse/SPARK-20928?focusedCommentId=16245556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16245556

Re: Welcome Zhenhua Wang as a Spark committer

Congratulations Zhenhua!! 2018-04-02 13:28 GMT+08:00 Wenchen Fan : > Hi all, > > The Spark PMC recently added Zhenhua Wang as a committer on the project. > Zhenhua is the major contributor of the CBO project, and has been > contributing across several areas of Spark for a while, focusing especial