Re: Arrow 2.0.0 versioning

2020-11-02 Thread Krisztián Szűcs
On Tue, Nov 3, 2020 at 5:07 AM James Duong wrote: > > I can go ahead and make the change, but should this get force pushed > earlier in the history to be right after the Arrow 2 commit that was tagged? Something most have gone wrong during the maven release preparation. `mvn versions:set -DnewVers

Re: [Discuss] Arrow Release Schedule

2020-11-02 Thread Micah Kornfield
> > Are there any plans for a more frequent release cadence? Not to my knowledge. The release process is still relatively heavy weight. Do we have a guide for what goes into major releases vs. minor releases > vs. patch releases? In the current regime [1] we don't expect minor release. So maj

[Discuss] Arrow Release Schedule

2020-11-02 Thread James Duong
Hello, My understanding is that Arrow releases are 3 months apart. Are there any plans for a more frequent release cadence? Do we have a guide for what goes into major releases vs. minor releases vs. patch releases? Thanks, -- *James Duong* Lead Software Developer Bit Quill Technologies Inc. D

Re: Closing of Server Resources

2020-11-02 Thread James Duong
The concern I have about making this a custom action is that FlightClient (in Java) is a Closeable resource already. It's a bit unintuitive to require users to run an explicit cleanup operation, while also automating some of the clean-up as well. It seems error-prone. On Fri, Oct 30, 2020 at 3:56

Re: Arrow 2.0.0 versioning

2020-11-02 Thread James Duong
I can go ahead and make the change, but should this get force pushed earlier in the history to be right after the Arrow 2 commit that was tagged? The reason I ran into this was that I had a local build of master which had interface changes that weren't in the Arrow 2 maven artifacts. On Mon, Nov

Re: Using FlightClientOption `disable_server_verification` with TLS throws exception with pyarrow 2.0.0

2020-11-02 Thread Wes McKinney
OK, it seems then that the gRPC version in the manylinux1 build needs to be updated, and the error message is probably wrong also if version 1.29.1 is not high enough On Mon, Nov 2, 2020 at 5:44 PM Keerat Singh wrote: > > Thank you for the prompt reply Wes. Just wanted to give a quick update, I n

Re: Arrow 2.0.0 versioning

2020-11-02 Thread Wes McKinney
The parts of the release process involving Maven are brittle and error prone, so it probably just didn't flow through properly, or the commit updating the Java POM versions didn't get pushed to the release branch or something similar On Mon, Nov 2, 2020 at 6:12 PM James Duong wrote: > > Hi, > > I

Arrow 2.0.0 versioning

2020-11-02 Thread James Duong
Hi, I noticed that in the Java POM files, the version in master is still 2.0.0 even after the release. This patch updated versions to 3.0.0-SNAPSHOT in other languages: https://github.com/apache/arrow/commit/b1f36acca85d0845c1e64c0a3270651d4a1467b7 -- *James Duong* Lead Software Developer Bit

Re: Using FlightClientOption `disable_server_verification` with TLS throws exception with pyarrow 2.0.0

2020-11-02 Thread Keerat Singh
Thank you for the prompt reply Wes. Just wanted to give a quick update, I no longer see the exception when using pyarrow installed via `conda` and using the `disable_server_verification` option. Here are the details about the pyarrow build being used by conda. *# Name Version

Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
The ones in http://arrow.apache.org/docs/format/Versioning.html As far as the Arrow format itself is concerned, the change that you have described is not "breaking" because it was always permissible for the validity bitmap to be unallocated when the null count is 0. Previously it was handled inco

Re: null buffers in primitive arrays

2020-11-02 Thread Niranda Perera
I see. So, what are the backward compatibility guarantees Arrow has moving forward? On Mon, Nov 2, 2020 at 9:52 AM Wes McKinney wrote: > No, you'd have to follow the project's pull requests or JIRA issues, > that's the only place where these things are discussed (except > occasionally on the mai

Re: Using FlightClientOption `disable_server_verification` with TLS throws exception with pyarrow 2.0.0

2020-11-02 Thread Wes McKinney
Supposedly we are building with grpc 1.29.1 in the manylinux wheels (in conda-forge, we're at >= 1.33) https://github.com/apache/arrow/blob/master/python/manylinux201x/scripts/build_grpc.sh So someone will need to take a closer look at the wheel (or conda) builds to see what might be going wrong.

Using FlightClientOption `disable_server_verification` with TLS throws exception with pyarrow 2.0.0

2020-11-02 Thread Keerat Singh
Hi, I see that the commit to add the support for `*disable_server_verification`* was merged into the Arrow 2.0.0 release. - PR: https://github.com/apache/arrow/pull/8325 However, when trying to use the FlightClientOption `disable_server_verification` with TLS enabled using a Python applicatio

Best way to store ragged packet data in Parquet files

2020-11-02 Thread Jason Sachs
Hi all-- I've been getting started with Parquet as a storage alternative to HDF5 and it has a lot of attractive quantities including compression flexibility efficiency. But I'm stumped for storage efficiency in Parquet with one type of data that I have. This is a large series of "ragged" packets

Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
No, you'd have to follow the project's pull requests or JIRA issues, that's the only place where these things are discussed (except occasionally on the mailing list). On Mon, Nov 2, 2020 at 8:50 AM Niranda Perera wrote: > > Thanks Wes. Is there a document/ link for architectural decisions like >

Re: null buffers in primitive arrays

2020-11-02 Thread Niranda Perera
Thanks Wes. Is there a document/ link for architectural decisions like this? (apart from a release change log, that is) On Mon, Nov 2, 2020 at 9:26 AM Wes McKinney wrote: > Indeed, we made a change to cause buffers[0] to always be null when > the null count is 0, which has always been permitted

Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
Indeed, we made a change to cause buffers[0] to always be null when the null count is 0, which has always been permitted by the columnar format specification (and in 0.16.0 and prior it was inconsistently null depending on how the array was created). On Mon, Nov 2, 2020 at 8:22 AM Niranda Perera

null buffers in primitive arrays

2020-11-02 Thread Niranda Perera
Hi, We have been using arrow v0.16 and recently upgraded to v2.0. One main difference we saw was that in v2.0, primitive arrays' buffer vector contains an empty buffer at idx 0 (whereas in 0.16 this was a null bitmap IINM). Was this an architectural decision, may be to reduce the memory footprint

[NIGHTLY] Arrow Build Report for Job nightly-2020-11-02-0

2020-11-02 Thread Crossbow
Arrow Build Report for Job nightly-2020-11-02-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-02-0 Failed Tasks: - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-02-0-travis-gandiva-jar-xenial - test-co