Re: [VOTE] Protocol for Dissociated Arrow IPC Transports

2024-03-02 Thread Paul Whalen
transports). I know of a > > third party that is interested in Arrow for HPC environments that could > be > > interested in the proposal and I can see if they're interested in > providing > > feedback. > > > > I glanced at the document before but I'l

Re: [VOTE] Protocol for Dissociated Arrow IPC Transports

2024-02-27 Thread Paul Whalen
As a potential "end user developer," (and aspiring contributor) this immediately excited me when I first saw it. I work at a trading firm, and my team has developed an IPC mechanism for efficiently transmitting pandas dataframes both remotely via TCP and locally via shared memory, where the interf

Re: [FLIGHT] Question about Flight Protocol Usage

2023-02-03 Thread Paul Whalen
My team uses Flight this way as well. We just use DoGet, DoPut, or DoExchange, depending on whether we're reading a stream of record batches, writing a stream, or need some sort of bidirectional communication. We also have full control over clients and servers, and simply use JSON in the Flight m

Re: Review request for Dataset Java API PRs

2021-08-04 Thread Paul Whalen
I would love to see a component or some standardization around Java using the C Data Interface. I’ve been prototyping JNI bindings for DataFusion in the last week or so with some success, and was getting ready to ask where/how such a thing might fit in. I’ll be sure to watch that JIRA. (I also

Re: [Flight Extension] Request for Comments

2021-06-01 Thread Paul Whalen
Hopefully this thread isn't too stale to pick back up with an open ended question. What interface would a Barrage client library expose? With Flight, application code cares about RecordBatches, but with Barrage it seems as though a client library ought to handle the updating of the table and expo

Re: [Go] Flight client app metadata access

2021-04-22 Thread Paul Whalen
Given that I'm doing a lot of updates to > the Go library already (and I'm the one who wrote the flight client/server > for Go) I can see about adding that functionality. > > --Matt > > -Original Message- > From: Paul Whalen > Sent: Wednesday, April 21, 20

[Go] Flight client app metadata access

2021-04-21 Thread Paul Whalen
Hi all, I'm using the Go Flight client, working off of this example: https://github.com/apache/arrow/blob/master/go/arrow/flight/flight_test.go I've found that there isn't a very convenient way to access the app metadata from a Flight stream, because the ipc.Reader you get from calling flight.New

Re: [Flight Extension] Request for Comments

2021-03-03 Thread Paul Whalen
I'm not an Arrow contributor (perhaps one day!) but as a close follower and user of the project for the last six months (Arrow Flight specifically), I kind of jumped out of my chair when I saw this today. It's *exactly* what my team is looking for and something I have been close to building mysel