subject:"SparkSQL Nested structure"

Re: SparkSQL Nested structure

2015-05-04 Thread Michael Armbrust

You are looking for LATERAL VIEW explode in HiveQL. On Mon, May 4, 2015 at 7:49 AM, Giovanni Paolo Gibilisco wrote: > Hi, I'm trying to parse log files generated by Spark using SparkSQL. > > In the JS

SparkSQL Nested structure

2015-05-04 Thread Giovanni Paolo Gibilisco

Hi, I'm trying to parse log files generated by Spark using SparkSQL. In the JSON elements related to the StageCompleted event we have a nested structre containing an array of elements with RDD Info. (see the log below as an example (omitting some parts). { "Event": "SparkListenerStageComplete