Hey Matt, This is cool to see. To be clear, this is an implementation of Spark DataSourceV2 using Arrow Flight?
I think the questions I have are: - Does this belong under Arrow, or under Spark - I lean towards it being closer to Spark than Arrow; - How would we do the testing with Spark; - What else do you envision as being part of 'productionize'; - Whether this should perhaps target a higher level protocol like Flight SQL or ADBC instead. I'm also curious if you have a particular use case in mind for the connector. Finally, this may need to go through the IP clearance cycle since it's been developed for a while outside of the Apache organization. -David On Mon, Oct 17, 2022, at 15:06, Matt Phelps wrote: > Hi Arrow team, > > I’d like to make a contribution to the Arrow project by submitting an > existing Spark connector for Flight. Ryan Murray and Kyle Brooks wrote > a Flight implementation (“flight-spark”) of Spark’s DataSource V2 > interface found here: https://github.com/rymurr/flight-spark-source. We > would like to productionize this effort and are looking for advice on > where it should live within Arrow. We had envisioned that this > connector would align well with the existing functionality under > Arrow’s java/flight. > > Thanks, > Matt Phelps
