gt; 2. In my case, toJSON on RDD doesn't seem to help a lot. Attached a screen
> shot. Looks like I got the same data frame as my original one.
>
> Thanks much for these examples.
>
>
>
> On Wed, Nov 16, 2016 at 2:54 PM, Nathan Lande
> wrote:
>
>> I'm lookin
I'm looking forward to 2.1 but, in the meantime, you can pull out the
specific column into an RDD of JSON objects, pass this RDD into the
read.json() and then join the results back onto your initial DF.
Here is an example of what we do to unpack headers from Avro log data:
def jsonLoad(path):
We are currently pulling out the JSON columns, passing them through
read.json, and then joining them back onto the initial DF so something like
from_json would be a nice quality of life improvement for us.
On Wed, Sep 28, 2016 at 10:52 AM, Michael Armbrust
wrote:
> Spark SQL has great support fo
