RE: [Discuss] Provide pluggable APIs to support user customized compression codec

Hi, I just tested with the Intel QuickAssist Technology, which provide hardware accelerate to GZIP, you can see detail here https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html Here is the benchmark result run on Intel(R) Xeon(R) Gol

Re: [Discuss] Provide pluggable APIs to support user customized compression codec

Yes, I think he's asking about the motivation for the project. My understanding is that Snappy is used more often than Gzip with Parquet On Wed, Oct 21, 2020 at 8:53 PM Xie, Qi wrote: > > Hi, Antoine > > Do you mean the performance data HW-GZIP compared with LZ4/ZSTD? > > Thanks, > XieQi > >

RE: [Discuss] Provide pluggable APIs to support user customized compression codec

Hi, Antoine Do you mean the performance data HW-GZIP compared with LZ4/ZSTD? Thanks, XieQi -Original Message- From: Antoine Pitrou Sent: Tuesday, October 20, 2020 10:38 PM To: dev@arrow.apache.org; Xie, Qi Cc: Xu, Cheng A ; Dong, Xin ; Zhang, Jie1 Subject: Re: [Discuss] Provide plu

Re: hadoop file system connect problem with pyarrow

Do either of these machines have a current Hadoop installation (and is that installation in the system path)? On Tue, Oct 20, 2020 at 9:53 AM 황세규 wrote: > > Dear Maintainer. My name is Joseph Hwang in South Korea. I need some advice > about PyArrow. > > I try to develop Hadoop File System client

Re: [VOTE] Release Apache Arrow 2.0.0 - RC2

We still have a couple of sections that need some content on the release announcement blog post: https://github.com/apache/arrow-site/pull/79 C++ and JavaScript are the main gaps needing attention. Thanks for your help. Neal On Wed, Oct 21, 2020 at 9:06 AM Uwe L. Korn wrote: > The r-arrow cond

Re: Experiment with DataFusion + Pyarrow

Very promising, congratulations :-) Le 20/10/2020 à 18:35, Jorge Cardoso Leitão a écrit : > Hi, > > Over the past few weeks I have been running an experiment whose main goal > is to run a query in (Rust's) DataFusion and use Python on it so that we > can embed the Python's ecosystem on the que

Re: [VOTE] Release Apache Arrow 2.0.0 - RC2

The r-arrow conda package also needs to be updated, I'll take care of that tomorrow. Mainly involves adding the new Windows patches to https://github.com/conda-forge/r-arrow-feedstock/pull/27 On Wed, Oct 21, 2020, at 4:06 PM, Krisztián Szűcs wrote: > There is one remaining task to update the hom

Re: [VOTE] Release Apache Arrow 2.0.0 - RC2

There is one remaining task to update the homebrew package: 1. [done] rebase master 2. [done] upload source 3. [done] upload binaries 4. [done] update website 5. [done] upload ruby gems 6. [done] upload js packages 8. [done] upload C# packages 9. [done] upload rust crates 10. [done] update

[NIGHTLY] Arrow Build Report for Job nightly-2020-10-21-0

Arrow Build Report for Job nightly-2020-10-21-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0-travis-gandiva-jar-osx - gandiva-jar-x

[NIGHTLY] Arrow Build Report for Job nightly-2020-10-21-0

Arrow Build Report for Job nightly-2020-10-21-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0-travis-gandiva-jar-osx - gandiva-jar-x

[NIGHTLY] Arrow Build Report for Job nightly-2020-10-21-0

Arrow Build Report for Job nightly-2020-10-21-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0-travis-gandiva-jar-osx - gandiva-jar-x

[NIGHTLY] Arrow Build Report for Job nightly-2020-10-21-0

Arrow Build Report for Job nightly-2020-10-21-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-10-21-0-travis-gandiva-jar-osx - gandiva-jar-x

[ANNOUNCE] Apache Arrow 2.0.0 released

The Apache Arrow community is pleased to announce the 2.0.0 release. The release includes 569 resolved issues ([1]) since the 1.0.0 release. The release is available now from our website, [2] and [3]: https://arrow.apache.org/install/ Release notes are available at: https://arrow.apache.o

Re: Rust CI failures

Hi, The "Post Cache Docker Volumes" stage is inserted by the github actions cache [1]. Seems like there is a newer version v2 available [2], we may want to bump the version across the github action configurations. Created a jira to track this [3]. Thanks for the heads up! - Krisztian [1]: https

Rust CI failures

Hi All; I am seeing spurious connection drops at "Post Cache Docker Volumes" stage. Not sure if this is a direct issue or something we can trivially fix. Want to notify you. Best, Mahmut