Hi All,
I am very new in Spark world. Just started some test coding from last week. I
am using spark-1.2.1-bin-hadoop2.4 and scala coding.
I am having issues while using Date and decimal data types. Following is my
code that I am simply running on scala prompt. I am trying to define a table
and point that to my flat file containing raw data (pipe delimited format).
Once that is done, I will run some SQL queries and put the output data in to
another flat file with pipe delimited format.
*******************************************************
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.createSchemaRDD
// Define row and table
case class ROW_A(
TSTAMP: Long,
USIDAN: String,
SECNT: Int,
SECT: String,
BLOCK_NUM: BigDecimal,
BLOCK_DEN: BigDecimal,
BLOCK_PCT: BigDecimal)
val TABLE_A =
sc.textFile("/Myhome/SPARK/files/table_a_file.txt").map(_.split("|")).map(p =>
ROW_A(p(0), p(1), p(2), p(3), p(4), p(5), p(6)))
TABLE_A.registerTempTable("TABLE_A")
***************************************************
The second last command is giving error, like following:
<console>:17: error: type mismatch;
found : String
required: Long
Looks like the content from my flat file are considered as String always and
not as Date or decimal. How can I make Spark to take them as Date or decimal
types?
Thanks & Regards
-----------------------
Ananda Basak
