This looks like a bug. Mind opening a JIRA?
On Thu, Apr 30, 2015 at 3:49 PM, Justin Yip <yipjus...@prediction.io> wrote:
> After some trial and error, using DataType solves the problem:
>
> df.withColumn("millis", $"eventTime".cast(
> org.apache.spark.sql.types.LongType) * 1000)
>
> Justin
>
> On Thu, Apr 30, 2015 at 3:41 PM, Justin Yip <yipjus...@prediction.io>
> wrote:
>
>> Hello,
>>
>> I was able to cast a timestamp into long using
>> df.withColumn("millis", $"eventTime".cast("long") * 1000)
>> in spark 1.3.0.
>>
>> However, this statement returns a failure with spark 1.3.1. I got the
>> following exception:
>>
>> Exception in thread "main" org.apache.spark.sql.types.DataTypeException:
>> Unsupported dataType: long. If you have a struct and a field name of it has
>> any special characters, please use backticks (`) to quote that field name,
>> e.g. `x+y`. Please note that backtick itself is not supported in a field
>> name.
>>
>> at
>> org.apache.spark.sql.types.DataTypeParser$class.toDataType(DataTypeParser.scala:95)
>>
>> at
>> org.apache.spark.sql.types.DataTypeParser$$anon$1.toDataType(DataTypeParser.scala:107)
>>
>> at
>> org.apache.spark.sql.types.DataTypeParser$.apply(DataTypeParser.scala:111)
>>
>> at org.apache.spark.sql.Column.cast(Column.scala:636)
>>
>> Is there any change in the casting logic which may lead to this failure?
>>
>> Thanks.
>>
>> Justin
>>
>> ------------------------------
>> View this message in context: casting timestamp into long fail in Spark
>> 1.3.1
>> <http://apache-spark-user-list.1001560.n3.nabble.com/casting-timestamp-into-long-fail-in-Spark-1-3-1-tp22727.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
