Probably it is wrong to compare StringType and ShortType.
I'll use something like this
df.select(colString, colShort,
colShort.equalTo(colString.cast(DecimalType(38,15)))).show
On Wed, Feb 6, 2019 at 6:32 PM Artur Sukhenko <artur.sukhe...@gmail.com>
wrote:
> scala> df.select(colString, colShort, colShort.equalTo(colString)).explain
> == Physical Plan ==
> LocalTableScan [tier_id#3, tier_id#56, (CAST(tier_id AS SMALLINT) =
> tier_id)#50]
>
>
> On Wed, Feb 6, 2019 at 6:19 PM Russell Spitzer <russell.spit...@gmail.com>
> wrote:
>
>> Run an "explain" instead of show, i'm betting it's casting tier_id to a
>> small_int to do the comparison
>>
>> On Wed, Feb 6, 2019 at 9:31 AM Artur Sukhenko <artur.sukhe...@gmail.com>
>> wrote:
>>
>>> Hello guys,
>>> I am migrating from Spark 1.6 to 2.2 and have this issue:
>>> I am casting string to short and comparing them with equal .
>>> Original code is:
>>> ... when(col(fieldName).equalTo(castedValueCol), castedValueCol).
>>>
>>> otherwise(defaultErrorValueCol)
>>>
>>> Reproduce (version 2.3.0.cloudera4):
>>> scala> val df = Seq("3.15").toDF("tier_id")
>>> df: org.apache.spark.sql.DataFrame = [tier_id: string]
>>>
>>> scala> val colShort = col("tier_id").cast(ShortType)
>>> colShort: org.apache.spark.sql.Column = CAST(tier_id AS SMALLINT)
>>>
>>> scala> val colString = col("tier_id")
>>> colString: org.apache.spark.sql.Column = tier_id
>>>
>>> scala> res4.select(colString, colShort, colShort.equalTo(colString)).show
>>> +-------+-------+-------------------------------------+
>>> |tier_id|tier_id|(CAST(tier_id AS SMALLINT) = tier_id)|
>>> +-------+-------+-------------------------------------+
>>> | 3.15| 3| true|
>>> +-------+-------+-------------------------------------+
>>> scala>
>>>
>>> Why is this?
>>> --
>>> --
>>> Artur Sukhenko
>>>
>> --
> --
> Artur Sukhenko
>
--
--
Artur Sukhenko
