Hi Rong, IMO, the purpose of IPC spec is for wide interoperability to ensure data can easily be interchanged. Therefore, I don't think we should be adding a pluggable codecs that can be used to write Arrow IPC files to the main repo.
At the very least if we wanted to support custom codecs, we would need a way of encoding this information in the IPC metadata to not assume there is one and only one Custom Codec (which is what the referenced PR code seems to do, but I could be misreading it). Thanks, Micah On Thu, Jan 19, 2023 at 1:04 AM Ma, Rong <rong...@intel.com> wrote: > Hi all, > > We're using arrow IPC format for data transfer in our project. We have > done some performance optimization by using a custom compression method to > get better compression ratio and throughput. I've raised PR< > https://github.com/apache/arrow/pull/33690> to add a pluggable custom > codec, but it was rejected probably because it only made changes to the C++ > module. > Arrow IPC has very limited support for compression types, therefore it's > impossible to apply our optimizations without the patch. Is there any good > way to add a custom compression type in the IPC module? > > Thanks, > Rong >
