Hi Evan, Patrick and Tobias,
So, It worked for what I needed it to do. I followed Yana's suggestion of
using parameterized type of [T <: Product:ClassTag:TypeTag]
more concretely, I was trying to make the query process a bit more fluent
-some pseudocode but with correct types
val table:SparkTable[POJO] = new
SparkTable[POJO](sqlContext,extractor:String=>POJO)
val data=
table.atLocation("hdfs://....")
.withName("tableName")
.makeRDD("SELECT * FROM tableName")
class SparkTable[T <: Product : ClassTag :TypeTag](val
sqlContext:SQLContext, val extractor: (String) => (T) ) {
private[this] var location:Option[String] =None
private[this] var name:Option[String]=None
private[this] val sc = sqlContext.sparkContext
def withName(name:String):SparkTable[T]={..}
def atLocation(path:String):SparkTable[T]={.. }
def makeRDD(sqlQuery:String):SchemaRDD={
...
import sqlContext._
val rdd:RDD[String] = sc.textFile(this.location.get)
val rddT:RDD[T] = rdd.map(extractor)
val schemaRDD= createSchemaRDD(rddT)
schemaRDD.registerAsTable(name.get)
val all = sqlContext.sql(sqlQuery)
all
}
}
Best,
Amit
On Tue, Aug 19, 2014 at 9:13 PM, Evan Chan <[email protected]> wrote:
> That might not be enough. Reflection is used to determine what the
> fields are, thus your class might actually need to have members
> corresponding to the fields in the table.
>
> I heard that a more generic method of inputting stuff is coming.
>
> On Tue, Aug 19, 2014 at 6:43 PM, Tobias Pfeiffer <[email protected]> wrote:
> > Hi,
> >
> > On Tue, Aug 19, 2014 at 7:01 PM, Patrick McGloin <
> [email protected]>
> > wrote:
> >>
> >> I think the type of the data contained in your RDD needs to be a known
> >> case class and not abstract for createSchemaRDD. This makes sense when
> you
> >> think it needs to know about the fields in the object to create the
> schema.
> >
> >
> > Exactly this. The actual message pointing to that is:
> >
> > "inferred type arguments [T] do not conform to method
> createSchemaRDD's
> > type parameter bounds [A <: Product]"
> >
> > All case classes are automatically subclasses of Product, but otherwise
> you
> > will have to extend Product and add the required methods yourself.
> >
> > Tobias
> >
>
