Oh, another aspect of the issue that I forgot to mention is that `filter`(which I'm trying to use with these booleans) has this warning:
"WARNING: the nulls of filter are ignored and the value on its slot is considered. Therefore, it is considered undefined behavior to pass filter with null values." So, I guess a third option would be a variant of `filter` which treated `null` as `false`. On Wed, Feb 10, 2021 at 10:50 AM Ben Chambers <bchamb...@apache.org> wrote: > I'm trying to implement something along the lines of "X if Y > Z", but > treating the case of Y or Z as null as "false". Interestingly, this is > difficult with the way the kernels are created: > > 1. `Y > Z` will treat `null > ???` as null. > "Perform left > right operation on two arrays. Non-null values are greater > than null values." > > 2. Ok, so maybe we write that as `(Y > Z) && not_null(Y)`. > "If either left or right value is null then the result is also null." > > Oh. So if the LHS is null, there is *no way* to get a boolean array with a > non-null value. > > 3. Ok, I'll go write my own operator to do this (`null_to_false` or > something like that). > I can do this, but it requires iterating over the booleans and combining > them. It seems like it would be easy to do using `buffer_bin_and`, but that > is only visible within the Arrow crate. > > First, am I missing something with the above analysis? Is there some way > to provide non-null values for a boolean array that has nulls? > > Second, if not any thoughts on a solution? The two options I see (without > changing behavior of existing kernels) would be: > 1. Add kernel(s) that provide a value in place of `null` (a general case > of the `null_to_false`). These could be specialized in the boolean case to > use the `buffer_bin_and` as appropriate. > 2. Expose the `buffer_bin_and` and `buffer_bin_or` methods so that I (as a > user) can write the kernel myself. >
