Re: Proposal for the plugin API to support user customized compression codec

2020-06-25 Thread Antoine Pitrou
framework. And we > can start with Parquet part first. > > Thanks > Cheng Xu > > From: Micah Kornfield > Sent: Tuesday, June 23, 2020 12:11 PM > To: dev > Cc: Xu, Cheng A ; Xie, Qi > Subject: Re: Proposal for the plugin API to support user customized > compressi

Re: Proposal for the plugin API to support user customized compression codec

2020-06-25 Thread Micah Kornfield
, June 23, 2020 12:11 PM > *To:* dev > *Cc:* Xu, Cheng A ; Xie, Qi > *Subject:* Re: Proposal for the plugin API to support user customized > compression codec > > > > It would be good to clarify the exact scope of this. If it is > particular to parquet then we should wa

RE: Proposal for the plugin API to support user customized compression codec

2020-06-25 Thread Xu, Cheng A
API to support user customized compression codec It would be good to clarify the exact scope of this. If it is particular to parquet then we should wait for the discussion on dev@parquet to conclude before moving forward. If it is more general to Arrow, then working through scenarios of how

Re: Proposal for the plugin API to support user customized compression codec

2020-06-22 Thread Micah Kornfield
It would be good to clarify the exact scope of this. If it is particular to parquet then we should wait for the discussion on dev@parquet to conclude before moving forward. If it is more general to Arrow, then working through scenarios of how this would be used for decompression when the Codec ca

Re: Proposal for the plugin API to support user customized compression codec

2020-06-22 Thread Wes McKinney
hi XieQi, Is the idea that your custom Gzip implementation would automatically override any places in the codebase where the built-in one would be used (like the Parquet codebase)? I see some things in the design doc about serializing the plugin information in the Parquet file metadata (assuming y