Re: Tungsten gives unexpected results when selecting null elements in array

FYI, I filed a JIRA and submitted a PR for this issue: https://issues.apache.org/jira/browse/SPARK-12477 https://github.com/apache/spark/pull/10429 -- View this message in context: http://apach

Re: Tungsten gives unexpected results when selecting null elements in array

$eval1.numElements() || index < 0) { > > By: > > if (index >= $eval1.numElements() || index < 0 || ctx.getValue(eval1, > dataType, "index") == null ) { > > Does that make sense? > -------------- > View this message in context: Re: Tungs

Re: Tungsten gives unexpected results when selecting null elements in array

I believe the problem is that the generated code does not check if the selected item in the array is null.Naïvely, I think changing this line would solve this: https://github.com/apache/spark/blob/4af647c77ded6a0d3087ceafb2e30e01d97e7a06/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/exp

Re: Tungsten gives unexpected results when selecting null elements in array

For info, this is the generated code: GeneratedExpressionCode( cursor8 = 16; convertedStruct6.pointTo(buffer7, Platform.BYTE_ARRAY_OFFSET, 1, cursor8); /* input[0, ArrayType(StringType,true)][0] */ /* input[0, ArrayType(StringType,true)] */ boolean isNull2