Hello,
I was able to cast a timestamp into long using
df.withColumn("millis", $"eventTime".cast("long") * 1000)
in spark 1.3.0.
However, this statement returns a failure with spark 1.3.1. I got the
following exception:
Exception in thread "main" org.apache.spark.sql.types.DataTypeException:
Unsupported dataType: long. If you have a struct and a field name of it has
any special characters, please use backticks (`) to quote that field name,
e.g. `x+y`. Please note that backtick itself is not supported in a field
name.
at
org.apache.spark.sql.types.DataTypeParser$class.toDataType(DataTypeParser.scala:95)
at
org.apache.spark.sql.types.DataTypeParser$$anon$1.toDataType(DataTypeParser.scala:107)
at
org.apache.spark.sql.types.DataTypeParser$.apply(DataTypeParser.scala:111)
at org.apache.spark.sql.Column.cast(Column.scala:636)
Is there any change in the casting logic which may lead to this failure?
Thanks.
Justin
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/casting-timestamp-into-long-fail-in-Spark-1-3-1-tp22727.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
