Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Niranjan Jayakar
+1 on the Rust client. We have a Swift, Go and soon a Rust client - I'm wondering if we should manage these in a single monorepo, rather than one per client, with just a separate top-level folder per language. This would make it easy to manage all clients, add features across clients, add more cl

Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Yuanjian Li
Happy to see the Rust client proposal and glad to help with the follow-up repo management work. Let’s use this thread for voting—posting my +1 here. Jules Damji : > I’ll ask them. > > Cheers > Jules > > Excuse the thumb typos > > > On Fri, 16 May 2025 at 2:38 AM, Renjie Liu > wrote: > >> Thank

Re: [VOTE] Release Spark 4.0.0 (RC6)

2025-05-16 Thread Anish Shrigondekar
Hi Adam, Thanks for reporting the issue. @Eric Marnadi has a PR for the fix here - https://github.com/apache/spark/pull/50924 @Wenchen Fan - would be good to include this in 4.0 RC Thanks, Anish On Fri, May 16, 2025 at 8:05 AM Adam Binford wrote: > I think I found a bug when trying to use t

Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Fokko Driesprong
Hi Niranjan, Thanks for jumping in here. I have some experience with monorepos in OSS, more precisely Avro, and I believe this complicates things. Combining everything complicates the build process, and I often find myself fixing things in the PHP client while we are trying to get a Java patch rel

Re: [VOTE] Release Spark 4.0.0 (RC6)

2025-05-16 Thread Jungtaek Lim
UPDATE: The fix is merged. @Adam Binford if you could help verifying this in latest branch-4.0, that would be great. Thanks for reporting the issue! On Sat, May 17, 2025 at 3:05 AM Anish Shrigondekar wrote: > Hi Adam, > > Thanks for reporting the issue. @Eric Marnadi > has a PR for the fix her

Re: [VOTE] Release Spark 4.0.0 (RC6)

2025-05-16 Thread Adam Binford
I think I found a bug when trying to use the new StateStore source reading from RocksDB. Not sure if it should be a blocker or not but I don't think it currently can read RocksDB state store checkpoints. This was from RC2 or RC3, but I don't see any relevant issues or fixes for this. Trying to test

Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Jules Damji
+1 yes, having in one place may be desirable but the release & build process could be onerous. I would keep them separate in their respective repos. Cheers Jules Excuse the thumb typos On Fri, 16 May 2025 at 6:53 AM, Fokko Driesprong wrote: > Hi Niranjan, > > Thanks for jumping in here. I hav

Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Jules Damji
I’ll ask them. Cheers Jules Excuse the thumb typos On Fri, 16 May 2025 at 2:38 AM, Renjie Liu wrote: > Thanks Denny and Jules for the response. > > It seems there is no objection about this proposal, would some > committer/PMC member help to create the repo? > > On Wed, May 14, 2025 at 3:45 A

Re: [DISCUSS] New Spark Connect Client repository for Rust language

2025-05-16 Thread Renjie Liu
Thanks Denny and Jules for the response. It seems there is no objection about this proposal, would some committer/PMC member help to create the repo? On Wed, May 14, 2025 at 3:45 AM Jules Damji wrote: > +1 in this effort. > — > Sent from my iPhone > Pardon the dumb thumb typos :) > > On May 9,