Re: Crash with 0.15.1 when transposing dicts with nulls values

2020-02-29 Thread Antoine Pitrou
Hi Pierre, While the Arrow format doesn't mandate particular values under null slots, the Arrow C++ implementation should not create "undefined" values (for security reasons: failing to initialize data could lead to reveal confidential information that was previously at the same memory location)

Re: Crash with 0.15.1 when transposing dicts with nulls values

2020-02-29 Thread Pierre Belzile
Hi Wes, I guess the answer is that it is not fixed... At this line, https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/int_util.cc#L409, we have a utility for transposing a dict from its original indices to a new mapping that is used when unifying dictionaries. I use that when concate

Re: Crash with 0.15.1 when transposing dicts with nulls values

2020-02-29 Thread Wes McKinney
The Arrow format does not indicate any particular value "underneath" a null so I'm not sure what can be "fixed" here. What precisely are you doing with the data that is failing? On Fri, Feb 28, 2020 at 4:57 PM Pierre Belzile wrote: > > When I recover an array of type dictionary int32 -> string fr