Hello
I am trying to create a SchemaRDD from a RDD of case classes. Depending on
an argument to the program, the program reads data of specified type and
maps it to the correct case class. But this throws an exception. I am using
Spark version 1.1.0 and Scala version 2.10.4
The exception can be reproduced by:
val typee = "type 1"
import org.apache.spark.rdd.RDD
case class type_1(a: String, b: Int, c: List[Int])
case class type_2(a: String, b: Int, c: List[Int], d: String)
val data = sc.parallelize(Seq(("asd",1,List(1,2)))) // some data of type
type_1
var supportedTypes: RDD[Product] = null
import csc.createSchemaRDD // SQLContext.createSchemaRDD
typee match{
case "type 1" => supportedTypes = data.map(row=>type_1(row._1 + 1, row._2,
row._3))
case "type 2" => supportedTypes = data.map(row=>type_2(row._1, row._2,
row._3, "ghj"))
}
supportedTypes.schema
The stacktrace:
scala.ScalaReflectionException: <none> is not a method
at
scala.reflect.api.Symbols$SymbolApi$class.asMethod(Symbols.scala:279)
at
scala.reflect.internal.Symbols$SymbolContextApiImpl.asMethod(Symbols.scala:73)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:60)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:50)
at
org.apache.spark.sql.catalyst.ScalaReflection$.attributesFor(ScalaReflection.scala:44)
at
org.apache.spark.sql.execution.ExistingRdd$.fromProductRdd(basicOperators.scala:229)
at
org.apache.spark.sql.SQLContext.createSchemaRDD(SQLContext.scala:94)
Kind regards
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/scala-ScalaReflectionException-when-creating-SchemaRDD-tp23025.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
