While trying to implement and introduce the idea of adopting FlightSQL, the
largest challenge was the API itself
I know it's meant to be low-level. But I found that most of the development
time was in code to convert to/from
row-based data (IE Map<String, Object>) and Java types, and columnar data +
Arrow types.
I'm likely in the minority position here -- I know that Arrow and FlightSQL
users are largely looking at transferring large volumes of data and
servicing OLAP-type workloads
But the thing that excites me most about FlightSQL, isn't its performance
(always nice to have), but that it's a language-agnostic standard for data
access.
That has broad implications -- for all kinds of data-access workloads and
business usecases.
The challenge is that in trying to advocate for it, when presenting a
proof-of-concept,
rather than what a developer might expect to see, something like:
// FlightSQL handler code
List<Map<String, Object>> results = ....;
results.add(Map.of("id", 1, "name", "Person 1");
return results;
A significant portion of the code is in Arrow-specific implementation
details:
creating a VectorSchemaRoot, FieldVector, de-serializing the results on the
client, etc.
Just curious whether there is any interest/intention of possibly making a
higher level API around the basic FlightSQL one?
Maybe something closer to the traditional notion of a row-based "DataFrame"
or "Table", like:
DataFrame df = new DataFrame();
df.addColumn("id", ArrowTypes.Int);
df.addColumn("name", ArrowTypes.VarChar);
df.addRow(Map.of("id", 1, "name", "Person 1"));
VectorSchemaRoot root = df.toVectorSchemaRoot();
listener.setVectorSchemaRoot(root);
listener.sendVectorSchemaRootContents();
