Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 Jie Yang On 2025/02/05 07:38:08 Sakthi wrote: > +1 (non-binding) > > On Tue, Feb 4, 2025 at 11:25 PM DB Tsai wrote: > > > +1 > > > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > > > On Feb 4, 2025, at 11:13 PM, Xiao Li wrote: > > > > +1 > > > > Xiao > > > > Jungtaek Lim

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 On Tue, Feb 4, 2025 at 11:56 PM Gengliang Wang wrote: > > +1 > > On Tue, Feb 4, 2025 at 11:38 PM Sakthi wrote: >> >> +1 (non-binding) >> >> On Tue, Feb 4, 2025 at 11:25 PM DB Tsai wrote: >>> >>> +1 >>> >>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>> >>> On Feb 4, 2025, a

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 On Tue, Feb 4, 2025 at 11:38 PM Sakthi wrote: > +1 (non-binding) > > On Tue, Feb 4, 2025 at 11:25 PM DB Tsai wrote: > >> +1 >> >> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> >> On Feb 4, 2025, at 11:13 PM, Xiao Li wrote: >> >> +1 >> >> Xiao >> >> Jungtaek Lim 于2025年2月4日

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) On Tue, Feb 4, 2025 at 11:25 PM DB Tsai wrote: > +1 > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > On Feb 4, 2025, at 11:13 PM, Xiao Li wrote: > > +1 > > Xiao > > Jungtaek Lim 于2025年2月4日周二 23:11写道： > >> +1 (non-binding) >> >> Sounds like a great alternati

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 4, 2025, at 11:13 PM, Xiao Li wrote: > > +1 > > Xiao > > Jungtaek Lim > 于2025年2月4日周二 23:11写道： >> +1 (non-binding) >> >> Sounds like a great alternative! >> >> On Wed, Feb 5, 2025 a

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 Xiao Jungtaek Lim 于2025年2月4日周二 23:11写道： > +1 (non-binding) > > Sounds like a great alternative! > > On Wed, Feb 5, 2025 at 4:10 PM Hyukjin Kwon wrote: > >> +1 >> >> On Wed, 5 Feb 2025 at 16:06, Wenchen Fan wrote: >> >>> Hi all, >>> >>> Given the positive feedback in the previous DISCUSS em

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) Sounds like a great alternative! On Wed, Feb 5, 2025 at 4:10 PM Hyukjin Kwon wrote: > +1 > > On Wed, 5 Feb 2025 at 16:06, Wenchen Fan wrote: > >> Hi all, >> >> Given the positive feedback in the previous DISCUSS email >>

Re: Quick question: can you guys navigate 3.5.4 Java API documentation?

1.It seems to be a `CSP` issue, as follows: [image: d14447a60a387863b505f6b66b8e012a.png] 2.What is `CSP`? https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors https://developer.chrome.com/docs/privacy-security/csp 3.Temporary solution After installing

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 On Wed, 5 Feb 2025 at 16:06, Wenchen Fan wrote: > Hi all, > > Given the positive feedback in the previous DISCUSS email > , I'd > like to start the vote for the proposal "Publish additional Spark > distribution with Spark Conne

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 Regards, Mridul On Wed, Feb 5, 2025 at 1:06 AM Wenchen Fan wrote: > Hi all, > > Given the positive feedback in the previous DISCUSS email > , I'd > like to start the vote for the proposal "Publish additional Spark > distribut

[VOTE] Publish additional Spark distribution with Spark Connect enabled

Hi all, Given the positive feedback in the previous DISCUSS email , I'd like to start the vote for the proposal "Publish additional Spark distribution with Spark Connect enabled". Please vote for the next 72 hours: [ ] +1: Accept

Re: Quick question: can you guys navigate 3.5.4 Java API documentation?

It seems that there are issues with the Java documentation for version 3.5.x , not just the latest version, such as version 3.5.0: https://spark.apache.org/docs/3.5.0/api/java/index.html. The navigation on my side is also problematic. Using Google Developer Tools, we can see information simila

Quick question: can you guys navigate 3.5.4 Java API documentation?

Hi all, I was randomly navigating Spark documentation and realised that Java API documentation does not work. https://spark.apache.org/docs/latest/api/java/index.html I can open the page but when I navigate and visit pages, like 70% does not work. Is this specific to me? Or can it be reproduced

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 for the additional option. Agreed that we should keep on track with the schedule. If as mentioned earlier that there are no critical blockers, it should be fine. On Tue, Feb 4, 2025 at 8:05 PM Denny Lee wrote: > > +1 (non-binding) on this proposal. Just as long as there are no schedule > co

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) A way forward for Apache Spark, allowing developers to choose either option, offering community to share critical feedback for Spark Connect, and paving a path for Spark to be accessible from everywhere, from other non-jvm based languages. Cheers Jules — Sent from my iPhon

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 (non-binding) on this proposal. Just as long as there are no schedule concerns - similar to Mridul and Dongjoon’s call outs, then yes, I think this would be helpful for adoption.Thanks! On Tue, Feb 4, 2025 at 18:43 huaxin gao wrote: > I support publishing an additional Spark distributio

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

I support publishing an additional Spark distribution with Spark Connect enabled in Spark 4.0 to boost Spark adoption. I also share Dongjoon's concern regarding potential schedule delays. As long as we monitor the timeline closely and thoroughly document any PRs that do not make it into the RC, we

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 to new distribution mechanisms which will increase Spark adoption ! I do agree with Dongjoon’s concerns that this should not result in slipping the schedule; something to watch out for. Regards, Mridul On Tue, Feb 4, 2025 at 8:07 PM Hyukjin Kwon wrote: > I am fine with providing another o

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

I am fine with providing another option +1 with leaving others as are. Once the vote passes, we should probably make it ready ASAP - I don't think it will need a lot of changes in any event. On Wed, 5 Feb 2025 at 02:40, DB Tsai wrote: > Many of the remaining PRs relate to Spark ML Connect suppor

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

Many of the remaining PRs relate to Spark ML Connect support, but they are not critical blockers for offering an additional Spark distribution with Spark Connect enabled by default in Spark 4.0, allowing users to try it out and provide more feedback. I agree that we should not postpone the Spar

Re: [DISCUSS] Spark - How to improve our release processes

I still believe that the way to solve this is by splitting our Python build requirements into two: 1. Abstract dependencies: These capture the most open/flexible set of dependencies for the project. They are posted to PyPI. 2. Concrete build dependencies: These are derived automatically from the

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

Many new feature `Connect` patches are still landing `branch-4.0` during the QA period after February 1st. SPARK-49308 Support UserDefinedAggregateFunction in Spark Connect Scala Client SPARK-50104 Support SparkSession.executeCommand in Connect SPARK-50943 Support `Correlation` on Connect SPARK-50

Re: [DISCUSS] Spark - How to improve our release processes

+ @Hyukjin Kwon My understanding is that, in the PySpark CI we do not use fixed Python library versions as we want to test with the latest library versions as soon as possible. However, the release scripts use fixed Python library versions to make sure it's stable. This means that for almost ever

Re: [DISCUSS] Spark - How to improve our release processes

Hi all, I am trying to revive this thread - to work towards a better release process, and making sure we have no conflicts in the used artifacts like nicholas.cham...@gmail.com mentioned. @Wenchen Fan - can you please clarify - you state that the release scripts are using a different build and Do

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

Hi Dongjoon, This is a big decision but not a big project. We just need to update the release scripts to produce the additional Spark distribution. If people are positive about this, I can start to implement the script changes now and merge it after this proposal has been voted on and approved. T

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

Hi, Wenchen. I'm wondering if this implies any delay of the existing QA and RC1 schedule or not. If then, why don't we schedule this new alternative proposal on Spark 4.1 properly? Best regards, Dongjoon On Mon, Feb 3, 2025 at 23:31 Wenchen Fan wrote: > Hi all, > > There is partial agreement

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 This enables users to easily experiment with and provide feedback on Spark Connect, while also facilitating broader adoption and development in other languages like Rust, Go, or Scala 3. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 3, 2025, at 11:29 PM, Wenchen Fan