[DISCUSS] SPIP: Spark Connect - A client and server interface for Apache Spark.

Hi Everyone, We would like to start a discussion on the "Spark Connect" proposal. Please find the links below: *JIRA* - https://issues.apache.org/jira/browse/SPARK-39375 *SPIP Document* - https://docs.google.com/document/d/1Mnl6jmGszixLW4KcJU5j9IgpG9-UabS0dcM6PM2XGDc/edit#heading=h.wmsrrfealhrj

Re: [DISCUSS] SPIP: Spark Connect - A client and server interface for Apache Spark.

be supported in this? > > On Fri, Jun 3, 2022 at 1:52 PM Martin Grund > wrote: > >> Hi Everyone, >> >> We would like to start a discussion on the "Spark Connect" proposal. >> Please find the links below: >> >> *JIRA* - https://issues.apa

Re: [DISCUSS] SPIP: Spark Connect - A client and server interface for Apache Spark.

es the > API to various databases, for example Google BiqQuery is very efficient. I > am not sure what this proposal is to trying to address? > > HTH > > On Fri, 3 Jun 2022 at 18:46, Martin Grund ent server > wrote: > >> Hi Everyone, >> >> We would l

Re: [DISCUSS] SPIP: Spark Connect - A client and server interface for Apache Spark.

On Tue, Jun 7, 2022 at 3:54 PM Steve Loughran wrote: > > > On Fri, 3 Jun 2022 at 18:46, Martin Grund > wrote: > >> Hi Everyone, >> >> We would like to start a discussion on the "Spark Connect" proposal. >> Please find the links below: >>

Re: [VOTE][RESULT] SPIP: Spark Connect

Thanks everyone for your votes and thanks Herman for being the shepherd. On Fri 17. Jun 2022 at 02:23 Hyukjin Kwon wrote: > Awesome, I am excited to see this in Apache Spark. > > On Fri, 17 Jun 2022 at 08:37, Herman van Hovell > wrote: > >> The vote passes with 17 +1s (10 binding +1s). >> +1: >

Enforcing scalafmt on Spark Connect - connector/connect

Hi folks, I'm reaching out to ask to gather input / consensus on the following proposal: Since Spark Connect is effectively new code, I would like to enforce scalafmt explicitly *only* on this module by adding a check in `dev/lint-scala` that checks if there is a diff after running ./build/mvn -

Re: Enforcing scalafmt on Spark Connect - connector/connect

we now do this in PySpark, and it's >> pretty nice that you can just forget about formatting it manually by >> yourself. >> >> On Fri, 14 Oct 2022 at 16:37, Martin Grund >> wrote: >> >>> Hi folks, >>> >>> I'm reaching out t

[CONNECT] New Clients for Go and Rust

Hi folks, When Bo (thanks for the time and contribution) started the work on https://github.com/apache/spark/pull/41036 he started the Go client directly in the Spark repository. In the meantime, I was approached by other engineers who are willing to contribute to working on a Rust client for Spar

Re: [CONNECT] New Clients for Go and Rust

gt;> > >> > >> > >> 1. Different repository can maintain independent versions, different > release times, and faster bug fix releases. > >> > >> > >> > >> 2. Different languages have different build tools. Putting them in one > repository w

Re: [CONNECT] New Clients for Go and Rust

> Also, an elephant in the room is the future of the current API in Spark 4 > and onwards. As useful as connect is, it is not exactly a replacement for > many existing deployments. Furthermore, it doesn't make extending Spark > much easier and the current ecosystem is, subjectively sp

Re: [CONNECT] New Clients for Go and Rust

EADME file that this is an experimental client. Looking forward to all your contributions! On Tue, May 30, 2023 at 11:50 AM Martin Grund wrote: > I think it makes sense to split this discussion into two pieces. On the > contribution side, my personal perspective is that these new

Re: [CONNECT] New Clients for Go and Rust

ark/pull/41036)! > > Great to see we have a new repo for Spark Golang Connect client. Thanks > Hyukjin! > I am thinking to migrate my PR to this new repo. Would like to hear any > feedback or suggestion before I make the new PR :) > > Thanks, > Bo > > > > On

Re: [CONNECT] New Clients for Go and Rust

> > > On 5/30/23 11:50, Martin Grund wrote: > > I think it makes sense to split this discussion into two pieces. On > > the contribution side, my personal perspective is that these new > clients > are explicitly marked as experimental and unsupported until > we deem th

Re: [DISCUSS] SPIP: Python Data Source API

Hey, I would like to express my strong support for Python Data Sources even though they might not be immediately as powerful as Scala-based data sources. One element that is easily lost in this discussion is how much faster the iteration speed is with Python compared to Scala. Due to the dynamic n

Re: [VOTE][SPIP] Python Data Source API

+1 (non-binding) On Fri, Jul 7, 2023 at 12:05 AM Denny Lee wrote: > +1 (non-binding) > > On Fri, Jul 7, 2023 at 00:50 Maciej wrote: > >> +0 >> >> Best regards, >> Maciej Szymkiewicz >> >> Web: https://zero323.net >> PGP: A30CEF0C31A501EC >> >> On 7/6/23 17:41, Xiao Li wrote: >> >> +1 >> >> Xiao

Re: Spark Connect: API mismatch in SparkSesession#execute

Hi Stefan, There are some current limitations around how protobuf is embedded in Spark Connect. One of the challenges there is that for compatibility reasons we currently shade protobuf that then shades the `prototobuf.GeneramtedMessage` class. The way to work around this is to shade the protobuf

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

+1 (non binding) Tested Spark Connect fully isolated and with PySpark build. Tested as well some of the new PySpark ML Connect features On Tue 29. Aug 2023 at 18:25 Yuanjian Li wrote: > Please vote on releasing the following candidate(RC3) as Apache Spark > version 3.5.0. > > The vote is open u

Re: Write Spark Connection client application in Go

This is absolutely awesome! Thank you so much for dedicating your time to this project! On Wed, Sep 13, 2023 at 6:04 AM Holden Karau wrote: > That’s so cool! Great work y’all :) > > On Tue, Sep 12, 2023 at 8:14 PM bo yang wrote: > >> Hi Spark Friends, >> >> Anyone interested in using Golang to

Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)

+1 On Tue, Jul 2, 2024 at 7:19 AM yangjie01 wrote: > I have manually attempted to only modify the `assembly/pom.xml` and > examined the results of executing `dev/make-distribution.sh --tgz`. The > `spark-connect_2.13-4.0.0-SNAPSHOT.jar` is indeed included in the jars > directory. However, if rea

Re: [外部邮件] Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

+1 (non-binding) On Wed, Jul 3, 2024 at 07:25 Holden Karau wrote: > +1 > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > >

Re: [DISCUSS] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

Absolutely we should do that. I thought that the default rule was inclusive already so that once folks have their first contribution it would automatically allow kicking of the workflows. On Thu, Jul 4, 2024 at 04:20 Matthew Powers wrote: > Yea, this would be great. > > spark-connect-go is still

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 (non-binding) On Thu, Jul 4, 2024 at 7:15 PM Holden Karau wrote: > +1 > > Although given its a US holiday maybe keep the vote open for an extra day? > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

I took a quick look at the PR and would like to understand your concern better about: > SparkSession is heavier than SparkContext It looks like the PR is using the active SparkSession, not creating a new one etc. I would highly appreciate it if you could help me understand this situation better.

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

Mridul, I really just wanted to understand the concern from Dongjoon. What you're pointing at is a slightly different concern. So what I see is the following: > [...] they can initialize a SparkContext and work with RDD api: The current PR uses a potentially optional value without checking that i

Re: [DISCUSS] Differentiate Spark without Spark Connect from Spark Connect

+1 for classic. It's simple, easy to understand and it doesn't have the negative meanings like legacy for example. On Sun, Jul 21, 2024 at 23:48 Wenchen Fan wrote: > Classic SGTM. > > On Mon, Jul 22, 2024 at 1:12 PM Jungtaek Lim > wrote: > >> I'd propose not to change the name of "Spark Connect

Re: [外部邮件] [VOTE] Differentiate Spark without Spark Connect from Spark Connect

+1 On Tue, Jul 23, 2024 at 07:06 Dongjoon Hyun wrote: > +1 for the proposed definition. > > Thanks, > Dongjoon > > > On Tue, Jul 23, 2024 at 6:42 AM Xianjin YE wrote: > >> +1 (non-binding) >> >> On Jul 23, 2024, at 16:16, Jungtaek Lim >> wrote: >> >> +1 (non-binding) >> >> On Tue, Jul 23, 2024

[DISCUSS] Using Github Issues for Spark-Connect-Go _only_ issues.

Hi folks, I wanted to start a discussion for the following proposal: To make it easier for folks to contribute to the Spark Connect Go client, I was contemplating not requiring them to deal with two accounts (one for Jira) and one for Gihutb but allow using GitHub Issues for bugs and issues that a

Re: [DISCUSS] Using Github Issues for Spark-Connect-Go _only_ issues.

_London> >> London, United Kingdom >> >> *Disclaimer:* The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand &

Re: [外部邮件] Re: [DISCUSS] Using Github Issues for Spark-Connect-Go _only_ issues.

> > On Fri, Aug 9, 2024 at 11:56 PM bo yang wrote: > > +1 to start small as an experiment to see how people use GitHub issue... > > > > On Thu, Aug 8, 2024 at 11:54 PM Kent Yao wrote: > > +1 > > On 2024/08/08 23:24:32 Hyukjin Kwon wrote: > > SGTM > > >

[VOTE] Using Github Issues for Spark-Connect-Go _only_ issues.

Hi Folks, following the discussion in the previous thread, I would like to open the vote on the following proposal: Enabling the use of GitHub Issues for issue tracking for the Spark Connect Go repository. This will support the tracking of issues that are exclusively relevant to this project. For

Re: Welcoming a new PMC member

Congratulations! On Tue, Aug 13, 2024 at 9:37 AM Peter Toth wrote: > Congratulations! > > Mridul Muralidharan ezt írta (időpont: 2024. aug. 13., > K, 8:46): > >> >> Congratulations Kent ! >> >> Regards, >> Mridul >> >> On Mon, Aug 12, 2024 at 8:46 PM Dongjoon Hyun >> wrote: >> >>> Congratulati

Re: [DISCUSS] Deprecating SparkR

+1 On Tue, Aug 13, 2024 at 7:26 AM Ruifeng Zheng wrote: > +1 > > On Tue, Aug 13, 2024 at 1:08 PM Holden Karau > wrote: > >> +1 >> >> Are the sparklyr folks on this list? >> >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2M

Re: [VOTE] Using Github Issues for Spark-Connect-Go _only_ issues.

; >>>>>>view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>

Re: [VOTE] Deprecate SparkR

+1 On Wed, Aug 21, 2024 at 20:26 Xiangrui Meng wrote: > +1 > > On Wed, Aug 21, 2024, 10:24 AM Mridul Muralidharan > wrote: > >> +1 >> >> >> Regards, >> Mridul >> >> >> On Wed, Aug 21, 2024 at 11:46 AM Reynold Xin >> wrote: >> >>> +1 >>> >>> On Wed, Aug 21, 2024 at 6:42 PM Shivaram Venkataraman

Re: [VOTE] Document and Feature Preview via GitHub Pages

+1 On Wed, Sep 11, 2024 at 9:39 AM Kent Yao wrote: > Hi all, > > Following the discussion[1], I'd like to start the vote for 'Document and > Feature Preview via GitHub Pages' > > > Please vote for the next 72 hours:(excluding next weekend) > > [ ] +1: Accept the proposal > [ ] +0 > [ ]- 1: I

Re: Support for `sparkContext.broadcast` function in Spark Connect?

>From my rough high-level view, there is nothing stopping us from adding broadcast variables to Spark Connect; we essentially have to lift them to the Spark Session. This would not be different from what we're doing for artifact management or what we've done for job cancellation. If you're interes

Re: [DISCUSS] Support spark.ml on Spark Connect

Hi Bobby, Awesome to see the proposal! I'm very much looking forward to the contributions! Martin On Thu, Oct 10, 2024 at 4:16 AM Ángel wrote: > You have my vote (btw, great idea, ML is so sexy nowadays 😉) > > El jue, 10 oct 2024 a las 3:19, Bobby () escribió: > >> Hi, >> >> I'd like to start

Re: [外部邮件] Re: Spark Connect the default API in Spark 4.0

twitter.com/holdenkarau >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to

Re: Spark Connect the default API in Spark 4.0

+1 On Wed, Nov 27, 2024 at 5:21 AM bo yang wrote: > +1 > > On Tue, Nov 26, 2024 at 3:47 PM Hyukjin Kwon wrote: > >> +1 >> >> On Mon, 25 Nov 2024 at 23:33, Bjørn Jørgensen >> wrote: >> >>> +1 >>> >>> man. 25. nov. 2024 kl. 14:48 skrev Herman van Hovell >>> : >>> Hi All, I would l

Re: [外部邮件] Re: Spark Connect the default API in Spark 4.0

As part of the discussion on this topic, I would love to highlight the work that the community is currently doing to support SparkML, which is traditionally very RDD-heavy, natively in Spark Connect. Bobby's awesome work shows that, over time, we can extend the features of Spark Connect and support

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

I'm not a fan of this approach. Spark configuration keys are defined as string values in Spark and used as Strings everywhere. I don't necessarily see the benefit of changing conf["keyName"] vs conf.get("keyName") or even spark.conf.keyName Trying to wrap this into magic getattr calls is not ide

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

; that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Sat, 28 Dec 2024 at 13:18, Martin Grun

Re: [DISCUSS] Pythonic approach of setting Spark SQL configurations

On top proper deep integration with pytest would be awesome. On Sun, Dec 29, 2024 at 16:40 Martin Grund wrote: > Generally function and column composition can be quite verbose. Maybe this > is something to invest some brain power. I see to often folks fall back to > expr() or s

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 On Wed, Feb 5, 2025 at 17:15 bo yang wrote: > +1 (non-binding) > > On Wed, Feb 5, 2025 at 7:51 AM Jules Damji wrote: > >> +1 (non-binding) >> >> Excuse the thumb typos >> >> >> On Tue, 04 Feb 2025 at 11:06 PM, Wenchen Fan wrote: >> >>> Hi all, >>> >>> Given the positive feedback in the prev

Re: [外部邮件] Re: Spark Connect the default API in Spark 4.0

r.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>

Re: [DISCUSS] Use plain text logs by default

I'm generally supportive of this direction. However, I'm wondering if we can be more deliberate about when to use it. For example, for the common scenarios that you mention as "light" usage, we should switch to plain text logging. IMO, this would cover the cases where a user runs simply the pyspar

Re: FYI: A Hallucination about Spark Connect Stability in Spark 4

I'm very confused about how we use stability in CI as a measure to discuss the strategy of a particular feature, particularly because we call these "hallucinations." >From real-world experience, I can say that we have thousands of clients using Spark Connect across many different versions in our i

Re: [DISCUSS] Support spark.ml on Spark Connect

in.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* The information provided is correct to the best of my >>>> knowled

Re: Java Client for Spark Connect

You can use the JVM-based Scala client from Java. It's packaged as an extra artifact https://mvnrepository.com/artifact/org.apache.spark/spark-connect-client-jvm On Thu, Jan 23, 2025 at 9:36 AM Balaji Sudharsanam V wrote: > Hello Team, > > Do we already have a Java Client for Spark Connect simi

Re: FYI: SPARK-49700 Unified Scala Interface for Connect and Classic

Would it not have been mindful to wait for the original author to investigate the PR and do a forward fix instead of reverting such a big change? Since this was only blocking the Maven test we could have waited probably a few more days without any issues. On Mon, Jan 27, 2025 at 8:32 PM Dongjoon H

Re: [DISCUSS] New Spark Connect Client repository for Swift language

So I was just playing with the Swift client to build a Mac app and everything worked nicely! However, similar to the Go repository, my suggestion would be to handle issues directly in Github and not in Jira because the vast majority of initial issues will be simply compatibility issues. And creatin

Re: [DISCUSS] New Spark Connect Client repository for Swift language

tory. > > Could you tell me how the Apache Spark JIRA is blocking you, Martin? You > are the Apache Spark committer which is supposed to use Apache Spark JIRA > for Spark contribution. I'm interested what kind of difficulty blocks the > Apache Spark committer. > > Thanks, >

Re: [VOTE] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 8:43 PM Denny Lee wrote: > +1 (non-binding) > > On Wed, Apr 9, 2025 at 11:03 Chao Sun wrote: > >> +1 >> >> On Wed, Apr 9, 2025 at 10:55 AM L. C. Hsieh wrote: >> >>> +1 >>> >>> On Wed, Apr 9, 2025 at 7:22 AM Sandy Ryza wrote: >>> > >>> > We started to get some votes o

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 On Wed, Apr 9, 2025 at 9:37 AM Mich Talebzadeh wrote: > +1 > > Dr Mich Talebzadeh, > Architect | Data Science | Financial Crime | Forensic Analysis | GDPR > >view my Linkedin profile > > > > > > > On Wed, 9 Apr 2025 at 08:07, Pete

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

Not sure this counts as -1, but by cursory checking the code, I found that the way the TLS connection is set up is not always working: https://github.com/apache/spark-connect-swift/blob/v0.1.0-rc1/Sources/SparkConnect/DataFrame.swift#L276-L288 Shows that DataFrame operations explicitly set plaint

Re: [DISCUSS] New Spark Connect Client repository for Swift language

Great work and proposal! I'm supportive. On Sun, Mar 9, 2025 at 23:31 Dongjoon Hyun wrote: > Hi, All. > > I'd like to propose to add a new Apache Spark repository for `Spark > Connect Client for Swift` in Apache Spark 4.1.0 timeframe. > > https://github.com/apache/spark-connect-swift > > To do

Re: [VOTE] Release Apache Spark Connect Go Client 0.1.0

UST provide a LICENSE file and a NOTICE file ... > > [1] > https://github.com/apache/spark-connect-go/blob/v0.1.0-rc1/spark/version.go#L19 > [2] https://dist.apache.org/repos/dist/dev/spark > [3] https://www.apache.org/legal/release-policy.html#source-packages > [4] > https://www.ap

Re: [VOTE] Release Apache Spark Connect Go Client 0.1.0

using the dev KEYS file - Missing NOTICE file - Upload to the GitHub distribution Thanks Martin On Mon, Jun 9, 2025 at 8:54 AM Martin Grund wrote: > Thanks for the feedback, I'll address it shortly. > > On Mon, Jun 9, 2025 at 08:31 Cheng Pan wrote: > >> Hi Martin, >> &

[VOTE][RESULT] Release Apache Spark Connect Go Client 0.1.0

binding) >>>>> >>>>> I verified: >>>>> >>>>> 1. LICENSE/NOTICE are present >>>>> 2. Signatures is correct >>>>> 3. Build source code and run UT (I have to replace sparksrc folder >>>>> with the cont

[VOTE] Release Apache Spark Connect Go Client 0.1.0

Please vote on releasing the following candidate as Apache Spark Connect Go Client 0.1.0. The release candidate was tested and built against Spark 4.0.0. The repository contains a sample application for submitting jobs written in Go using a small JVM wrapper

Re: [VOTE] Release Apache Spark Connect Go Client 0.1.0

to build and test the release provided they have > access to the appropriate platform and tools. A source release SHOULD not > contain compiled code. > > [1] https://www.apache.org/legal/release-policy.html#publication > > Thanks, > Cheng Pan > > > > On Jun 9, 2025, at

Re: [VOTE] SPIP: Monthly preview release

+1 On Thu, Jul 3, 2025 at 9:32 AM Wenchen Fan wrote: > +1 > > On Thu, Jul 3, 2025 at 3:28 PM Xiao Li wrote: > >> +1 >> >> Dongjoon Hyun 于2025年7月3日周四 00:19写道： >> >>> +1 >>> >>> On 2025/07/03 06:44:32 "L. C. Hsieh" wrote: >>> > +1 >>> > >>> > On Wed, Jul 2, 2025 at 9:38 PM Hyukjin Kwon >>> wrot