Run an "explain" instead of show, i'm betting it's casting tier_id to a
small_int to do the comparison
On Wed, Feb 6, 2019 at 9:31 AM Artur Sukhenko <artur.sukhe...@gmail.com>
wrote:
> Hello guys,
> I am migrating from Spark 1.6 to 2.2 and have this issue:
> I am casting string to short and comparing them with equal .
> Original code is:
> ... when(col(fieldName).equalTo(castedValueCol), castedValueCol).
>
> otherwise(defaultErrorValueCol)
>
> Reproduce (version 2.3.0.cloudera4):
> scala> val df = Seq("3.15").toDF("tier_id")
> df: org.apache.spark.sql.DataFrame = [tier_id: string]
>
> scala> val colShort = col("tier_id").cast(ShortType)
> colShort: org.apache.spark.sql.Column = CAST(tier_id AS SMALLINT)
>
> scala> val colString = col("tier_id")
> colString: org.apache.spark.sql.Column = tier_id
>
> scala> res4.select(colString, colShort, colShort.equalTo(colString)).show
> +-------+-------+-------------------------------------+
> |tier_id|tier_id|(CAST(tier_id AS SMALLINT) = tier_id)|
> +-------+-------+-------------------------------------+
> | 3.15| 3| true|
> +-------+-------+-------------------------------------+
> scala>
>
> Why is this?
> --
> --
> Artur Sukhenko
>
