: Xu, Cheng A ; Dong, Xin ; Zhang,
Jie1
Subject: Re: [Discuss] Provide pluggable APIs to support user customized
compression codec
Hi,
Le 27/10/2020 à 09:55, Xie, Qi a écrit :
>
> The HW decompressor can't fall back automatically on SW decompression, but we
> can fallback t
> Sent: Thursday, October 22, 2020 7:20 PM
> To: dev@arrow.apache.org; Xie, Qi
> Cc: Xu, Cheng A ; Dong, Xin ;
> Zhang, Jie1
> Subject: Re: [Discuss] Provide pluggable APIs to support user customized
> compression codec
>
>
> Ok, thank you. Another question: why
u
> Sent: Thursday, October 22, 2020 4:30 PM
> To: Xie, Qi ; dev
> Cc: Xu, Cheng A ; Dong, Xin
> ; Zhang, Jie1
> Subject: Re: [Discuss] Provide pluggable APIs to support user
> customized compression codec
>
>
> Le 22/10/2020 à 05:38, Xie, Qi a écrit :
>> H
Thursday, October 22, 2020 4:30 PM
> To: Xie, Qi ; dev
> Cc: Xu, Cheng A ; Dong, Xin ;
> Zhang, Jie1
> Subject: Re: [Discuss] Provide pluggable APIs to support user customized
> compression codec
>
>
> Le 22/10/2020 à 05:38, Xie, Qi a écrit :
>> Hi,
>>
&g
4:30 PM
To: Xie, Qi ; dev
Cc: Xu, Cheng A ; Dong, Xin ; Zhang,
Jie1
Subject: Re: [Discuss] Provide pluggable APIs to support user customized
compression codec
Le 22/10/2020 à 05:38, Xie, Qi a écrit :
> Hi,
>
> I just tested with the Intel QuickAssist Technology, which pr
Le 22/10/2020 à 05:38, Xie, Qi a écrit :
> Hi,
>
> I just tested with the Intel QuickAssist Technology, which provide hardware
> accelerate to GZIP, you can see detail here
> https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html
>
>
: dev
Cc: anto...@python.org; Xu, Cheng A ; Dong, Xin
; Zhang, Jie1 ; Xie, Qi
Subject: Re: [Discuss] Provide pluggable APIs to support user customized
compression codec
Yes, I think he's asking about the motivation for the project. My understanding
is that Snappy is used more often than
t;
> Thanks,
> XieQi
>
> -Original Message-
> From: Antoine Pitrou
> Sent: Tuesday, October 20, 2020 10:38 PM
> To: dev@arrow.apache.org; Xie, Qi
> Cc: Xu, Cheng A ; Dong, Xin ;
> Zhang, Jie1
> Subject: Re: [Discuss] Provide pluggable APIs to support user customiz
pluggable APIs to support user customized
compression codec
Le 20/10/2020 à 12:09, Xie, Qi a écrit :
> Hi, Wes
>
> Yes currently the purpose of the key-value metadata is just a hint to
> indicate that the parquet file is compressed by plugin so that the parquet
> reader can
Le 20/10/2020 à 12:09, Xie, Qi a écrit :
> Hi, Wes
>
> Yes currently the purpose of the key-value metadata is just a hint to
> indicate that the parquet file is compressed by plugin so that the parquet
> reader can load the plugin library and use plugin to decompress the file.
> There are man
adata to distinguish it
from the standard gzip.
Thanks,
XieQi
-Original Message-
From: Wes McKinney
Sent: Tuesday, October 20, 2020 11:06 AM
To: dev
Cc: Xie, Qi ; Xu, Cheng A ; Dong, Xin
; Zhang, Jie1
Subject: Re: [Discuss] Provide pluggable APIs to support user customized
compre
What is the purpose of the key-value metadata aside from automatically
loading the plugin library if it's available (which seems like a
security risk if reading a data file can cause a shared library to be
loaded dynamically)? Is it necessary to have that metadata for it to
be safe to use the optim
Hi,
Again, I think the whole plugin concept falls outside of Arrow.
It should be much simpler to simply allow people to override the
compression codec factory. Then applications can define "plugins" if
they want to.
Regards
Antoine.
Le 19/10/2020 à 03:30, Xie, Qi a écrit :
> Hi, all
>
> A
Hi, all
Again as we discussed in the previous email, We are proposing an pluggable APIs
to support user customized compression codec in ARROW.
See proposal
https://docs.google.com/document/d/1W_TxVRN7WV1wBVOTdbxngzBek1nTolMlJWy6aqC6WG8/edit
We want to redefine the scope of the pluggable API and
14 matches
Mail list logo
