Re: [Rust] Exposing Bitwise Helpers?

Aha. I was looking in the wrong place. Buffer does have `bitand`, `bitor` and `not` methods on it that seem to wrap the underlying `buffer_bin_and` and `buffer_bin_or`, etc. I'm still curious on whether it would make sense to offer some more variants of those (`A && !B`, for instance) to avoid mat

Re: [Rust] Exposing Bitwise Helpers?

Oh, another aspect of the issue that I forgot to mention is that `filter`(which I'm trying to use with these booleans) has this warning: "WARNING: the nulls of filter are ignored and the value on its slot is considered. Therefore, it is considered undefined behavior to pass filter with null values

[Rust] Exposing Bitwise Helpers?

I'm trying to implement something along the lines of "X if Y > Z", but treating the case of Y or Z as null as "false". Interestingly, this is difficult with the way the kernels are created: 1. `Y > Z` will treat `null > ???` as null. "Perform left > right operation on two arrays. Non-null values a

Re: Arrow Rust sync call February 10 at 12:00 US/Eastern, 17:00 UTC

Thanks for sharing the slides Rémi! That looks really cool. One question I have after this, do you plan to use S3 Select ( https://aws.amazon.com/blogs/aws/s3-glacier-select/)?Seems it would fit your architecture nicely and I think shouldn't be too hard to create the query from the filters/project

Re: [C++] Why are these two tables unequal?

Yup. That doesn’t change anything. I have just pushed this to https://github.com/apache/arrow/pull/8648 . Please take a look. Really thanks! TEST(TestAdapterWriteNested, writeList) { std::shared_ptr table_schema = schema({field("list", list(int32()))

Re: Arrow Rust sync call February 10 at 12:00 US/Eastern, 17:00 UTC

Thanks for the notes Andy. Here is the slide deck I presented, for further reference: https://docs.google.com/presentation/d/1uZ5PbazC1zCX24k0Hh-UItddIh9BRvD5GL7NUDgc9eQ/edit?usp=sharing If anyone wants to see how it works in practice and does not have an AWS account to try it out, feel free to re

Re: [C++] Why are these two tables unequal?

Just to be sure, have you tried to pass `check_metadata = true` as the optional argument for `DataType::Equals`? If that doesn't change anything, perhaps you can push your code somewhere so that I (or someone else) can take a look. Le 10/02/2021 à 18:39, Ying Zhou a écrit : > Not really. So wh

Re: [C++] Why are these two tables unequal?

Not really. So what’s really going on?! TEST(TestAdapterWriteNested, writeList) { std::shared_ptr table_schema = schema({field("list", list(int32()))}); int64_t num_rows = 1; arrow::random::RandomArrayGenerator rand(kRandomSeed); auto value_array = rand.ArrayOf(int32(), 5 * num_rows, 0

Re: Arrow Rust sync call February 10 at 12:00 US/Eastern, 17:00 UTC

Attendees - Andy Grove - Benjamin Blodgett - Marc Prud’Hommeaux - Mike Seddon - Jorge Leitao - Andrew Lamb - Fernando Herrera - Neville Dipale - Remi Dettai (Please let me know if I have misspelled anyone’s names) Topics Discussed

Re: [C++] Why are these two tables unequal?

Hmm, perhaps the types are unequal, then. Can you print them out (including field metadata)? Le 10/02/2021 à 18:03, Ying Zhou a écrit : > Thanks! Now we have an even weirder phenomenon. Even the null bitmaps and > offsets are equal. However the arrays aren’t! Does anyone know why? > > TEST(T

Re: [C++] Why are these two tables unequal?

Thanks! Now we have an even weirder phenomenon. Even the null bitmaps and offsets are equal. However the arrays aren’t! Does anyone know why? TEST(TestAdapterWriteNested, writeList) { std::shared_ptr table_schema = schema({field("list", list(int32()))}); int64_t num_rows = 1; arrow::ran

Arrow Rust sync call February 10 at 12:00 US/Eastern, 17:00 UTC

A quick reminder that the bi-weekly Arrow Rust sync call starts about an hour from now. Everyone is welcome. https://meet.google.com/ctp-yujs-aee Thanks, Andy.

Exposing low-level Parquet encryption to Python user (or, maybe not)

Hi, Since the PR for high-level C++ Parquet encryption API appears stalled (https://github.com/apache/arrow/pull/8023), I'm looking into exposing the low-level Parquet encryption API to Python. Arguments for doing this: the low-level API is all the users I'm talking to need, at the moment, so

[NIGHTLY] Arrow Build Report for Job nightly-2021-02-10-0

Arrow Build Report for Job nightly-2021-02-10-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-02-10-0 Failed Tasks: - gandiva-jar-ubuntu: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-02-10-0-github-gandiva-jar-ubuntu -

Re: [C++] Why are these two tables unequal?

Hi Ying, Hmm, yes, this may be related to the null bitmaps, or the offsets. Can you try to inspect or pretty-print the offsets arrays for the two list arrays? Regards Antoine. Le 10/02/2021 à 03:26, Ying Zhou a écrit : > Hi, > > This is an extremely weird phenomenon. There are two 2*1 table