[DISCUSS] SPIP: Python Data Source API

Hi everyone, I would like to start a discussion on “Python Data Source API”. This proposal aims to introduce a simple API in Python for Data Sources. The idea is to enable Python developers to create data sources without having to learn Scala or deal with the complexities of the current data sour

[VOTE][SPIP] Python Data Source API

Hi all, I'd like to start the vote for SPIP: Python Data Source API. The high-level summary for the SPIP is that it aims to introduce a simple API in Python for Data Sources. The idea is to enable Python developers to create data sources without learning Scala or dealing with the complexities of

[VOTE][RESULT] Python Data Source API

The vote passes with 12 +1s (8 binding +1s) and one +0 (binding). (* = binding) +1: - Hyukjin Kwon * - Xiao Li * - Denny Lee - Martin Grund - Mich Talebzadeh - Huaxin Gao * - Holden Karau * - Reynold Xin * - Jungtaek Lim - Ruifeng Zheng * - Takuya Ueshin * - Matei Zaharia * +0: Maciej Szymkiewicz

[DISCUSS] SPIP: Python Stored Procedures

Hi all, I would like to start a discussion on “Python Stored Procedures". This proposal aims to extend Spark SQL by introducing support for stored procedures, starting with Python as the procedural language. This will enable users to run complex logic using Python within their SQL workflows and s

Re: [DISCUSS] SPIP: Python Stored Procedures

isclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages aris

Re: [DISCUSS] SPIP: Python Stored Procedures

isclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages aris

Re: [DISCUSS] SPIP: Python Stored Procedures

Hi Mich, Thank you for your comments! I've left some comments on the SPIP, but let's continue the discussion here. You've highlighted the potential advantages of Python stored procedures, and I'd like to emphasize two important aspects: 1. *Versatility*: Integrating Python into SQL provides r

Re: [DISCUSS][SPIP] Standardize Spark Exception Messages

Hi Mridul, Thanks for reviewing the SPIP. As you correctly pointed out, this proposal is an internal, Spark engine developer-facing improvement to make error messages more consistent and user-friendly across the Spark codebase, as opposed to establishing a public contract with end-users. Thanks,

Re: [VOTE] Standardize Spark Exception Messages SPIP

Thanks everyone for voting! With 11 +1s and no -1s, this vote passes. +1s: Mridul Muralidharan Angers Zhu Chandni Singh Eve Liao Matei Zaharia Kalyan Wenchen Fan Gengliang Wang Xiao Li Takeshi Yamamuro Herman van Hovell Thanks, Allison -- Sent from: http://apache-spark-developers-list.1001551.

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

I'm a big +1 on this proposal. We should be able to continue improving the programming guides to enhance their quality and make this process easier. > Move the programming guide to the spark-website repo, to allow faster iterations and releases This is a great idea. It should work for structured

Re: [VOTE] Deprecate SparkR

+1 On Thu, Aug 22, 2024 at 10:10 AM Zhou Jiang wrote: > +1 (non-binding) > > On Wed, Aug 21, 2024 at 7:23 PM Xiao Li wrote: > >> +1 >> >> Hyukjin Kwon 于2024年8月21日周三 16:46写道： >> >>> +1 >>> >>> On Thu, 22 Aug 2024 at 05:37, Dongjoon Hyun wrote: >>> +1 Dongjoon On 2024/0

Re: [VOTE] Move Variant to Parquet

+1 On Tue, Sep 3, 2024 at 7:57 PM Wenchen Fan wrote: > +1 > > On Wed, Sep 4, 2024 at 10:26 AM Yufei Gu wrote: > >> +1 >> Yufei >> >> >> On Tue, Sep 3, 2024 at 4:35 PM Gengliang Wang wrote: >> >>> +1 >>> >>> On Tue, Sep 3, 2024 at 4:29 PM rdb...@gmail.com >>> wrote: >>> +1 On Tu

Re: [VOTE] Release Apache Spark 3.5.3 (RC1)

+1 On Tue, Sep 3, 2024 at 4:46 PM Hyukjin Kwon wrote: > +1 > > > On Wed, 4 Sept 2024 at 08:43, Haejoon Lee > wrote: > >> Hi, dev! >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.5.3. >> >> The vote is open until Sep 6, 11 PM UTC, and passes if a majority +1 P

Re: [外部邮件] Re: New committer: Bingkun Pan

Congrats Bingkun! On Wed, Nov 20, 2024 at 3:42 PM Peter Toth wrote: > Congratulations! > > On Wed, Nov 20, 2024 at 7:16 AM roryqi wrote: > >> Congrats! >> >> Xinrong Meng 于2024年11月20日周三 09:58写道： >> > >> > Congratulations Bingkun, well deserved! >> > >> > On Tue, Nov 19, 2024 at 10:30 PM Wenche

Re: [DISCUSS] Use plain text logs by default

+1. Text logs are much more human-readable. On Wed, Dec 11, 2024 at 12:12 PM Igor Dvorzhak wrote: > +1 > > > On Tue, Dec 10, 2024 at 7:48 PM Yang Jie wrote: > >> +1 >> >> On 2024/12/11 02:34:02 Kent Yao wrote: >> > +1 >> > >> > On 2024/11/23 02:50:36 Wenchen Fan wrote: >> > > Hi Martin, >> > >

Re: [VOTE] Use plain text logs by default

+1 On Thu, Jan 9, 2025 at 2:45 PM L. C. Hsieh wrote: > +1 > > On Wed, Jan 8, 2025 at 10:27 PM Gengliang Wang wrote: > > > > +1 > > > > On Wed, Jan 8, 2025 at 10:17 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> > >> +1 (non-binding) > >> > >> 2025년 1월 9일 (목) 오후 3:06, Cheng Pan 님이

Re: [DISCUSS] Ongoing projects for Spark 4.0

I am working on SQL user-defined functions (SPARK-46057 ). There are a few major PRs left, and I’d like to have this feature included in Spark 4.0. On Wed, Jan 15, 2025 at 5:27 PM Bobby wrote: > I also have one: https://github.com/apache/spark/p

Re: [Spark SQL]: Are SQL User-Defined Functions on the Roadmap?

Hi Frank, I am actively working on SPARK-46057 . You should be able to try it out once the Spark 4.0 RC is out. Thanks, Allison On Wed, Feb 5, 2025 at 5:37 PM Reynold Xin wrote: > There's already one here https://issues.apache.org/jira/browse/

Re: [VOTE] Release Spark 3.5.6 (RC1)

+1 On Fri, May 23, 2025 at 10:15 AM Hyukjin Kwon wrote: > Oh it's actually a test and also to release. Let me know if you have any > concern! > > On Fri, May 23, 2025 at 11:25 PM Mridul Muralidharan > wrote: > >> Hi Hyukjin, >> >> This thread is to test the automated release, right ? >> Not t