Hello, I have a couple questions about using compression in Java Arrow Flight. I'll break it down into 2 parts: whether it's possible and whether it's useful.
*1) Is it possible to do in current Java APIs?* * I see that an ArrowMessage has a bodyCompression field that it derives from an ArrowRecordBatch or ArrowDictionaryBatch. * And then a FlightStream acts on these ArrowMessages, with some internal calls where converting ArrowMessages to ArrowRecordBatches is aware of compression via the MessageSerializer. * However, I haven't found a way where I can use these internal details to add compression in Flight. For example, my general workflow is to use a VectorSchemaRoot in tandem with a ServerStreamListener to handle DoGet RPCs - AFAIK I'm not meant to directly deal with ArrowRecordBatches, and thus haven't found the right spot to turn on compression. * Am I missing some APIs or other ways to enable compression in Java Flight? * I see that it's possible in C++ via David's answer here <https://stackoverflow.com/a/71802541>, but I don't see the same IpcOption in Java (whose IpcOption is more minimal <https://arrow.apache.org/docs/java/reference/index.html>, and not sure where that would go in Java API regardless). *2) Is compression useful in Flight?* >From the same SO linked above, David linked to this Issue <https://issues.apache.org/jira/browse/ARROW-10351> which has a really useful thread around examining whether Flight compression is helpful (I think in C++). It seems like based on the testing in that thread, the answer is broadly "unlikely". Perhaps in very network-constrained environments the story could change. If it's easy to enable in Java Flight with current APIs I'd like to do some more testing to see how it goes in my environment, but if not, it seems unclear if it's worth the effort to enable it. Thank you for any help!
