Fwd: Inquiry: Extending Spark ML Support via Spark Connect to Scala/Java APIs (SPARK-50812 Analogue)

Excuse the thumb typos -- Forwarded message - From: Daniel Filev Date: Fri, 30 May 2025 at 5:07 AM Subject: Inquiry: Extending Spark ML Support via Spark Connect to Scala/Java APIs (SPARK-50812 Analogue) To: Dear Apache Spark Community/Development Team, I hope this message fi

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 (non-binding) —Sent from my iPhonePardon the dumb thumb typos :)On May 30, 2025, at 12:39 PM, Mark Hamstra wrote:A soft real-time system still defines an interval or frame within which results should be available, and often provides explicit warning or error-handling mechanisms when frame rate

Re: [VOTE] Release Spark 4.0.0 (RC7)

+ 1 (non-binding) —Sent from my iPhonePardon the dumb thumb typos :)On May 19, 2025, at 5:26 PM, Gengliang Wang wrote:+1On Mon, May 19, 2025 at 5:21 PM Jungtaek Lim wrote:+1 (non-binding)On Tue, May 20, 2025 at 8:47 AM Ruifeng Zheng wrote:+1On

Re: [VOTE] Release Apache Spark Connect Swift Client 0.2.0 (RC1)

+ 1 (non-binding) —Sent from my iPhonePardon the dumb thumb typos :)—Sent from my iPhonePardon the dumb thumb typos :)On May 17, 2025, at 4:32 PM, Zhou Jiang wrote:+1 (non-binding)On May 17, 2025, at 16:28, Hyukjin Kwon wrote:+1On Sun, 18 May 2025 at 07:47, L. C. Hsieh wrote

Re: [VOTE] Release Apache Spark Connect Swift Client 0.2.0 (RC1)

+ 1 (non-binding) —Sent from my iPhonePardon the dumb thumb typos :)On May 17, 2025, at 4:32 PM, Zhou Jiang wrote:+1 (non-binding)On May 17, 2025, at 16:28, Hyukjin Kwon wrote:+1On Sun, 18 May 2025 at 07:47, L. C. Hsieh wrote:+1 Thanks Dongjoon. On Sat, May 17, 2025 at 5:40

Re: [DISCUSS] New Spark Connect Client repository for Rust language

Wed, May 14, 2025 at 3:45 AM Jules Damji <jules.da...@gmail.com> wrote:+1 in this effort.—Sent from my iPhonePardon the dumb thumb typos :)On May 9, 2025, at 1:53 AM, Renjie Liu <liurenjie2...@gmail.com> wrote:Hi, All:I'd like to propose to add a new Apache Spark repository for `

Re: [DISCUSS] New Spark Connect Client repository for Rust language

May 14, 2025 at 3:45 AM Jules Damji wrote: > >> +1 in this effort. >> — >> Sent from my iPhone >> Pardon the dumb thumb typos :) >> >> On May 9, 2025, at 1:53 AM, Renjie Liu wrote: >> >>  >> Hi, All: >> >> I'd like to prop

Re: [DISCUSS] New Spark Connect Client repository for Rust language

>> >>> It seems there is no objection about this proposal, would some >>> committer/PMC member help to create the repo? >>> >>> On Wed, May 14, 2025 at 3:45 AM Jules Damji >>> wrote: >>> >>>> +1 in this effort. >>>>

Re: [VOTE] Release Spark 4.0.0 (RC6)

+1 (non-binding)—Sent from my iPhonePardon the dumb thumb typos :)On May 14, 2025, at 11:29 AM, Denny Lee wrote:+1 (non-binding)On Wed, May 14, 2025 at 10:34 AM Chao Sun wrote:+1On Wed, May 14, 2025 at 10:30 AM Holden Karau wrote:+1On Wed, May 14, 202

Re: [DISCUSS] New Spark Connect Client repository for Rust language

+1 in this effort.—Sent from my iPhonePardon the dumb thumb typos :)On May 9, 2025, at 1:53 AM, Renjie Liu wrote:Hi, All:I'd like to propose to add a new Apache Spark repository for `Spark Connect Client for Rust`.https://github.com/apache/spark-connect-rustThere are already some efforts for buil

Re: [VOTE] SPIP: Add geospatial types to Spark

+1 (Non-biding? Excuse the thumb typos On Mon, 05 May 2025 at 6:50 PM, Xiao Li wrote: > +1 > > On Mon, May 5, 2025 at 18:35 Yuming Wang wrote: > >> +1 >> >> On Tue, May 6, 2025 at 9:12 AM Denny Lee wrote: >> >>> +1 (non-binding) >>> >>> On Mon, May 5, 2025 at 18:03 Wenchen Fan wrote: >>> >>

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) Excuse the thumb typos On Wed, 09 Apr 2025 at 7:22 AM, Sandy Ryza wrote: > We started to get some votes on the discussion thread, so I'd like to move > to a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv

Re: [VOTE] SPIP: Constraints in DSv2

+1 (non-binding) — Sent from my iPhone Pardon the dumb thumb typos :) > On Mar 21, 2025, at 11:47 AM, Anton Okolnychyi wrote: > >  > Hi all, > > I would like to start a vote on adding support for constraints to DSv2. > > Discussion thread: > https://lists.apache.org/thread/njqjcryq0lot9rkbf

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+ 1 (non-binding) Generally speaking, it’s a good idea to separate repositories for all Spark Connect clients under Spark. - better organization - better visibility - easier for contribution - better for growth & extension of Spark Connect ecosystem Cheers Jules — Sent from my iPhone Pardon the

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+ 1 (non-binding) Generally speaking, it’s a good idea to separate repositories for all Spark Connect clients under Spark. - better organization - better visibility - easier for contribution - better for growth & extension of Spark Connect ecosystem Cheers Jules — Sent from my iPhone Pardon t

Re: [VOTE] Release Spark 4.0.0 (RC2)

Disregard the last message. I neglected to set SPARK_REMOTE to get pyspark to work correctly. Cheers Jules > On Mar 4, 2025, at 2:24 PM, Chris Nauroth wrote: > > -1 (non-binding) > > I think I found some missing license information in the binary distribution. > We may want to include this

Re: [VOTE] Release Spark 4.0.0 (RC2)

- 1 (non-binding) A ran into number of installation and launching problems. May be it’s my enviornment, even though I removed any old binaries and packages. 1. Pip installing pyspark4.0.0 and pyspark-connect-4.0 from .tz file workedl, launching pyspark results into 25/03/04 14:00:26 ERROR Spa

Re: [VOTE] SPIP: Add the TIME data type

+ 1 (non-binding) Excuse the thumb typos On Sun, 23 Feb 2025 at 7:50 AM, Max Gekk wrote: > Hi Spark devs, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > The SPIP aims to add a new data type TIME to Spark SQL types. New type > should conform to TIME(n) WITHOUT

Re: [VOTE] Release Apache Spark 3.5.5 deprecating `spark.databricks.*` configuration

+1 non-binding > On Feb 19, 2025, at 4:19 AM, Peter Toth wrote: > > +1 > > On Wed, Feb 19, 2025 at 10:20 AM Max Gekk > wrote: >> +1 >> >> On Wed, Feb 19, 2025 at 9:15 AM L. C. Hsieh > > wrote: >>> +1 >>> >>> On Tue, Feb 18, 2025 at 9:46 PM

Re: [VOTE][RESULT] Publish additional Spark distribution with Spark Connect enabled

= binding) > +1: > - Mridul Muralidharan * > - Hyukjin Kwon * > - Jungtaek Lim > - Xiao Li * > - DB Tsai * > - Sakthi > - Gengliang Wang * > - L. C. Hsieh * > - Yang Jie * > - Max Gekk * > - Yuming Wang * > - Mich Talebzadeh > - Huaxin Gao * > - Denny Lee >

Re: Extending Spark with a custom ExternalClusterManager

Yes, if this becomes a need that surfaces time and again, then it’s worthwhile to start a broader discussion in a manner of high-level proposal, which could trigger favorable discussion leading to next steps. CheersJules —Sent from my iPhonePardon the dumb thumb typos :)On Feb 7, 2025, at 8:00 AM,

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) Excuse the thumb typos On Tue, 04 Feb 2025 at 11:06 PM, Wenchen Fan wrote: > Hi all, > > Given the positive feedback in the previous DISCUSS email > , I'd > like to start the vote for the proposal "Publish addit

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) A way forward for Apache Spark, allowing developers to choose either option, offering community to share critical feedback for Spark Connect, and paving a path for Spark to be accessible from everywhere, from other non-jvm based languages. Cheers Jules — Sent from my iPhon

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

Thanks for update and looking into it. Excuse the thumb typos On Tue, 21 Jan 2025 at 4:09 PM, Hyukjin Kwon wrote: > Just a quick note on that: the major reason is 1. OOM we should figure out > and fix the CI environment. 2. structured streaming test failure that is > still in development. > I

RE: Re: Increasing Shading & Relocating for 4.0

On 2025/01/18 22:35:59 Mich Talebzadeh wrote: > I think your view highlights the need for a shift towards more stable and > version-independent APIs. Spark Connect IMO is a key enabler of this shift, > allowing users and developers to build applications and libraries that are > more resilient to ch

Re: [DISCUSS] Apache Spark 3.0.1 Release

+1 (non-binding) Sent from my iPhone Pardon the dumb thumb typos :) > On Jun 23, 2020, at 11:36 AM, Holden Karau wrote: > >  > +1 on a patch release soon > >> On Tue, Jun 23, 2020 at 10:47 AM Reynold Xin wrote: >> +1 on doing a new patch release soon. I saw some of these issues when >> prep

Re: [VOTE] Amend Spark's Semantic Versioning Policy

+1 (non-binding) Sent from my iPhone Pardon the dumb thumb typos :) > On Mar 6, 2020, at 7:09 PM, Sean Owen wrote: > > +1 > >> On Fri, Mar 6, 2020 at 8:59 PM Michael Armbrust >> wrote: >> >> I propose to add the following text to Spark's Semantic Versioning policy >> and adopt it as the

Re: [Proposal] Modification to Spark's Semantic Versioning Policy

+1 Well said! Sent from my iPhone Pardon the dumb thumb typos :) > On Feb 24, 2020, at 3:03 PM, Michael Armbrust wrote: > >  > Hello Everyone, > > As more users have started upgrading to Spark 3.0 preview (including myself), > there have been many discussions around APIs that have been br

Re: Request to document the direct relationship between other configurations

All are valid and valuable observations to put into practice: * structured and meaningful config names * explainable text or succinct description * easily accessible or searchable While these are aspirational but gradually doable if we make it part of the dev and review cycle. Often meaningfu

Re: More publicly documenting the options under spark.sql.*

It’s one thing to get the names/values of the configurations, via the Spark.sql(“set -v”), but another thing to understand what each achieves and when and why you’ll want to use it. A webpage with a table and description of each is huge benefit. Cheers Jules Sent from my iPhone Pardon the

Re: Should python-2 be supported in Spark 3.0?

Here’s the tweet from the horse’s mouth: https://twitter.com/gvanrossum/status/1133496146700058626?s=21 Cheers Jules — Sent from my iPhone Pardon the dumb thumb typos :) > On May 29, 2019, at 10:12 PM, Sean Owen wrote: > > Deprecated -- certainly and sooner than later. > I don't have a good

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

+ (non-binding) Sent from my iPhone Pardon the dumb thumb typos :) > On Apr 19, 2019, at 10:30 AM, Bryan Cutler wrote: > > +1 (non-binding) > >> On Thu, Apr 18, 2019 at 11:41 AM Jason Lowe wrote: >> +1 (non-binding). Looking forward to seeing better support for processing >> columnar data.

Re: Standardized Join Types for DataFrames

Also, Holden Karau conducts PR requests reviews and shows how you can contribute to this communal project. Attend one of her live PR sessions. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Feb 22, 2019, at 7:16 AM, Pooja Agrawal wrote: > > Hi, > > I am new to spark

Re: Welcome Jose Torres as a Spark committer

Congrats Jose! Sent from my iPhone Pardon the dumb thumb typos :) > On Jan 29, 2019, at 11:07 AM, shane knapp wrote: > > congrats, and welcome! > >> On Tue, Jan 29, 2019 at 11:07 AM Dean Wampler wrote: >> Congrats, Jose! >> >> Dean Wampler, Ph.D. >> VP, Fast Data Engineering at Lightbend >>

Re: [VOTE] [SPARK-25994] SPIP: DataFrame-based Property Graphs, Cypher Queries, and Algorithms

+1 (non-binding) (Heard their proposed tech-talk at Spark + A.I summit in London. Well attended & well received.) — Sent from my iPhone Pardon the dumb thumb typos :) > On Jan 29, 2019, at 7:30 AM, Denny Lee wrote: > > +1 > > yay - let's do it! > >> On Tue, Jan 29, 2019 at 6:28 AM Xiangrui

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

Indeed! Sent from my iPhone Pardon the dumb thumb typos :) > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > >> On Thu, Nov 8, 2018 at 11:24 AM Wenchen Fan wrote: >> + user list >>

Re: Python friendly API for Spark 3.0

+1 I think phasing out EOL of any feature or supported language is a better strategy if possible than a quick drop. With enough admonition, it can gradually be dropped in 3.x— of course, there are exceptions. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Sep 15, 2018,

Re: [discuss] replacing SPIP template with Heilmeier's Catechism?

+1 One could argue that the litany of the questions are really a double-click on the essence: why, what, how. The three interrogatives ought to be the essence and distillation of any proposal or technical exposition. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Aug 3

Re: [ANNOUNCE] Announcing Apache Spark 2.3.1

Matei & I own it. I normally tweet or handle Spark related PSAs Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Jun 14, 2018, at 11:45 AM, Marcelo Vanzin > wrote: > > Hi Jacek, > > I seriously have no idea... I don't even know who owns that account (I > hope they ha

Re: Generalised Spark-HBase integration

Brilliant! Will check it out. Cheers Jules -- The Best Ideas Are Simple Jules Damji Developer Relations & Community Outreach jda...@hortonworks.com http://hortonworks.com On 7/28/15, 8:59 AM, "Michal Haris" mailto:michal.ha...@visualdna.com>> wrote: Hi all, last couple