Got it. It seems there's potential benefits to have this work together with
Spark Connect (at least, there's two very similar wire protocols going on that
could possibly be consolidated). Have you shown this to the Spark community as
well?
You caught me in the middle of things, the PR to add th
1) David, thanks for mentioning that. tbh, this is the first time I'm
reading about it. If we are talking only about SparkFlightSql, It does seem
similar in the sense that both use Arrow for streaming data back to the
client. The difference is the Spark Connect client still has a dependency
on Spar
So this is now both a Flight SQL producer and consumer for Spark? That is very
cool.
A couple things I was wondering about:
- How do you think this compares to the Spark Connect proposal? [1]
- Have you considered ADBC [2] instead of Flight SQL for the DataSourceV2
implementation? While still u
This sounds pretty darn nifty!
I don't have much of value to offer, but the idea sounds like a great one
to me =)
On Sat, Jul 23, 2022 at 5:18 PM Tornike Gurgenidze
wrote:
> David, thank you for the reply.
>
> I recently managed to find the time to get back to the repo. I thought I
> would post
David, thank you for the reply.
I recently managed to find the time to get back to the repo. I thought I
would post the status update for anyone interested.
The project started out as just FlightSql implementation, but I ended up
splitting it into smaller components:
1. SparkFlightManager - a lo
Hi Tornike,
I'll have to take a closer look later when I can get back in front of a real
computer but I just want to say that this is super awesome, and thank you for
sharing!
I think we've kicked around the idea of "contrib" projects in the past. Maybe
this can be the impetus to take up that
Hi,
I'm not sure this is the right place to be posting this, so I apologize in
advance.
Recently I started a PoC for Arrow Flight SQL Server with Spark backend (
https://github.com/tokoko/SparkFlightSql). The main goal is to create a
SparkThriftServer alternative that will benefit from FlightSql
