Hey Rakesh, To clarify, what I was referring to is when doing something like this:
sqlContext.applySchema(rdd, mySchema) mySchema must be a well-defined StructType, which presently does not allow for a recursive type. On Wed, May 20, 2015 at 5:39 PM Rakesh Chalasani <vnit.rak...@gmail.com> wrote: > Hi Jeremy: > > Row is a collect of 'Any'. So, you can be used as a recursive data type. > Is this what you were looking for? > > Example: > val x = sc.parallelize(Array.range(0,10)).map(x => Row(Row(x), > Row(x.toString))) > > Rakesh > > > > On Wed, May 20, 2015 at 7:23 PM Jeremy Lucas <jeremyalu...@gmail.com> > wrote: > >> Spark SQL has proven to be quite useful in applying a partial schema to >> large JSON logs and being able to write plain SQL to perform a wide variety >> of operations over this data. However, one small thing that keeps coming >> back to haunt me is the lack of support for recursive data types, whereby a >> member of a complex/struct value can be of the same type as the >> complex/struct value itself. >> >> I am hoping someone may be able to point me in the right direction of >> where to start to build out such capabilities, as I'd be happy to >> contribute, but am very new to this particular component of the Spark >> project. >> >
