I agree. Below are two notes from a similar discussion on the Rust implementation:
1. In SIMD, for performance reasons, operations are performed over the whole buffer irrespectively of the bitmap mask, and deal with the bitmap mask separately. If a slot contains an arbitrary value, the operation on that slot is usually undefined behavior. Even simple operations such as u32 + u32 where the user knows that the numbers are small can cause problems because a null slot may contain an (unitialized) number close to u32::MAX and the SIMD addition may overflow on the null slot. The tricky part is that users usually have no high-end APIs to change those numbers (as they are generally not intended to be accessible). 2. In Rust, it is undefined behavior to read from uninitialized memory. Therefore, we already always write 0s for null slots. Best, Jorge On Sat, Feb 20, 2021 at 9:18 PM Benjamin Kietzman <bengil...@gmail.com> wrote: > Original discussion at > https://github.com/apache/arrow/pull/9471#issuecomment-779944257 (PR for > https://issues.apache.org/jira/browse/ARROW-11595 ) > > Although the format does not specify what is contained in array slots > masked by null bits (for example the first byte in the data buffer of an > int8 array whose first slot is null), there are other considerations which > might motivate establishing conventions for some arrays created by the C++ > implementation: > - No spurious complaints from valgrind when running otherwise safe > element-wise compute kernels on values under null bits. In the case of > ARROW-11595, the values buffer of the result of casting from Type::NA to > Type::INT8 is left uninitialized but masked by an entirely-null validity > bitmap. When such an array is passed to a comparison kernel, a branch on > the uninitialized values triggered valgrind even though the results of that > branch were also masked by an empty validity bitmap. > - If the underlying values were allocated but not initialized they may leak > private information such as private keys, passwords, or tokens which were > placed in that memory then freed by an application without overwrite > - Improved compression of data buffers (for example in writing to the IPC > format), since a run of nulls would correspond to consistent, repeated > values in all buffers > - Deterministic output from operations which are unable to honor null > bitmaps, such as computing the checksum of an IPC file >
