I think it may be possible with some upstream work but I don't think you can enable compression for Flight in Java out of the box. If you are interested in just seeing whether compression might help, I'd actually try enabling it at the gRPC layer first [1]. (I haven't tested this, though.) That might be a faster route to just testing whether it might help before considering what to do at the Arrow/Flight level (if anything).
Note that the gRPC-layer compression would operate on the entire payload, while the Arrow-level compression applies to individual buffers, so they aren't quite the same (I never tested whole-body compression). [1]: https://grpc.io/docs/guides/compression/ On Thu, Aug 24, 2023, at 17:41, Nathaniel Jones wrote: > Hello, > > I have a couple questions about using compression in Java Arrow Flight. > I'll break it down into 2 parts: whether it's possible and whether it's > useful. > > *1) Is it possible to do in current Java APIs?* > * I see that an ArrowMessage has a bodyCompression field that it derives > from an ArrowRecordBatch or ArrowDictionaryBatch. > * And then a FlightStream acts on these ArrowMessages, with some internal > calls where converting ArrowMessages to ArrowRecordBatches is aware of > compression via the MessageSerializer. > * However, I haven't found a way where I can use these internal details to > add compression in Flight. For example, my general workflow is to use a > VectorSchemaRoot in tandem with a ServerStreamListener to handle DoGet RPCs > - AFAIK I'm not meant to directly deal with ArrowRecordBatches, and thus > haven't found the right spot to turn on compression. > * Am I missing some APIs or other ways to enable compression in Java Flight? > * I see that it's possible in C++ via David's answer here > <https://stackoverflow.com/a/71802541>, but I don't see the same IpcOption > in Java (whose IpcOption is more minimal > <https://arrow.apache.org/docs/java/reference/index.html>, and not sure > where that would go in Java API regardless). > > *2) Is compression useful in Flight?* > From the same SO linked above, David linked to this Issue > <https://issues.apache.org/jira/browse/ARROW-10351> which has a really > useful thread around examining whether Flight compression is helpful (I > think in C++). It seems like based on the testing in that thread, the > answer is broadly "unlikely". Perhaps in very > network-constrained environments the story could change. > > If it's easy to enable in Java Flight with current APIs I'd like to do some > more testing to see how it goes in my environment, but if not, it seems > unclear if it's worth the effort to enable it. > > Thank you for any help!
