Re: Arrow on PyPy3 patch

2020-10-22 Thread Wes McKinney
Either way having a Dockerfile in the project to test with PyPy sounds like a good idea. On Thu, Oct 22, 2020 at 6:37 AM Antoine Pitrou wrote: > > We can, but we cannot be expected to act if something breaks. So this > would be wasting CPU resources for little use. > > Regards > > Antoine. > >

Re: Arrow on PyPy3 patch

2020-10-22 Thread Antoine Pitrou
We can, but we cannot be expected to act if something breaks. So this would be wasting CPU resources for little use. Regards Antoine. Le 22/10/2020 à 13:33, Krisztián Szűcs a écrit : > On Thu, Oct 22, 2020 at 1:21 PM Antoine Pitrou wrote: >> >> >> Hi Niklas, >> >> Le 22/10/2020 à 13:16, Nik

Re: Arrow on PyPy3 patch

2020-10-22 Thread Krisztián Szűcs
On Thu, Oct 22, 2020 at 1:21 PM Antoine Pitrou wrote: > > > Hi Niklas, > > Le 22/10/2020 à 13:16, Niklas B a écrit : > > > > Would the arrow project be open to a PR with the above patch, even though > > it doesn’t give you full PyPy support? > > I think that would be ok, but you'll have to provid

Re: Arrow on PyPy3 patch

2020-10-22 Thread Antoine Pitrou
Hi Niklas, Le 22/10/2020 à 13:16, Niklas B a écrit : > > Would the arrow project be open to a PR with the above patch, even though it > doesn’t give you full PyPy support? I think that would be ok, but you'll have to provide maintenance when needed, because I think we're unlikely to make PyPy

Re: [Discuss] Provide pluggable APIs to support user customized compression codec

2020-10-22 Thread Antoine Pitrou
Ok, thank you. Another question: why doesn't the HW decompressor fall back automatically on SW decompression when the window size is too large? That would avoid having to define metadata strings for this. Regards Antoine. Le 22/10/2020 à 10:38, Xie, Qi a écrit : > Yes, the HW-GZIP is able t

Arrow on PyPy3 patch

2020-10-22 Thread Niklas B
Hi, I’ve been (together with the PyPy team) working on getting arrow to build on PyPy3. I’m not looking for full feature capability, but specifically getting it to work with pandas read_parquet/to_parquet which it now does. There were a few roadblocks solved by the awesome Matti Picus on the Py

RE: [Discuss] Provide pluggable APIs to support user customized compression codec

2020-10-22 Thread Xie, Qi
Yes, the HW-GZIP is able to work on multiple threads too, but the test program lzbench https://github.com/inikep/lzbench seems work on single thread, so I can't run it with multiple threads. Thanks, XieQi -Original Message- From: Antoine Pitrou Sent: Thursday, October 22, 2020 4:30 P

Re: [Discuss] Provide pluggable APIs to support user customized compression codec

2020-10-22 Thread Antoine Pitrou
Le 22/10/2020 à 05:38, Xie, Qi a écrit : > Hi, > > I just tested with the Intel QuickAssist Technology, which provide hardware > accelerate to GZIP, you can see detail here > https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html > >