Hello Experts,
After repeated attempt I am unable to run query on map json date string. I
tried two approaches:
*** Approach 1 *** created a Bean class with timespamp field. When I try to
run it I get scala.MatchError: class java.sql.Timestamp (of class
java.lang.Class). Here is the code:
import java.sql.Timestamp;
public class ComplexClaim {
private Timestamp timestamp;
public Timestamp getTimestamp() {
return timestamp;
}
public void setTimestamp(Timestamp timestamp) {
this.timestamp = timestamp;
}
}
JavaSparkContext ctx = getCtx(sc);
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) ,
ComplexClaim.class);
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
*** Approach 2 ***
Created a StructType to map the date field. I got scala.MatchError:
TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$).
here is the code:
public StructType createStructType() {
List<StructField> fields = new ArrayList<StructField>();
fields.add(DataType.createStructField("timestamp",
DataType.TimestampType, false));
return DataType.createStructType(fields);
}
public void testJsonStruct(SparkContext sc) {
JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc));
String path = "/hdd/spark/test.json";
JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType());
sqlCtx.registerRDDAsTable(test, "test");
execSql(sqlCtx, "select * from test", 1);
}
Input file has a single record:
{"timestamp":"2014-10-10T01:01:01"}
Thanks
Tridib
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
