llable = true)
|-- long: string (nullable = true)
|-- null: string (nullable = true)
|-- string: string (nullable = true)
Thanks,
Ewan
From: Yin Huai [mailto:yh...@databricks.com]
Sent: 01 October 2015 23:54
To: Ewan Leith
Cc: r...@databricks.com; dev@spark.apache.org
Subject: Re: Dataframe neste
ev@spark.apache.org>
Subject: Re: Dataframe nested schema inference from Json without type conflicts
Exactly, that's a much better way to put it.
Thanks,
Ewan
-- Original message--
From: Yin Huai
Date: Thu, 1 Oct 2015 23:54
To: Ewan Leith;
Cc:
r...@databricks.com;dev@spark.
on that data and includes it.
>
>
>
> Passing in a fixed schema loses that nice additional ability, though it's
> what we'll probably have to adopt if we can't come up with a way to keep
> the inference working.
>
>
>
> Thanks,
>
> Ewan
>
nested schema inference from Json without type conflicts
Exactly, that's a much better way to put it.
Thanks,
Ewan
-- Original message--
From: Yin Huai
Date: Thu, 1 Oct 2015 23:54
To: Ewan Leith;
Cc:
r...@databricks.com;dev@spark.apache.org<mailto:r...@databricks.com
Exactly, that's a much better way to put it.
Thanks,
Ewan
-- Original message--
From: Yin Huai
Date: Thu, 1 Oct 2015 23:54
To: Ewan Leith;
Cc: r...@databricks.com;dev@spark.apache.org;
Subject:Re: Dataframe nested schema inference from Json without type conflicts
Hi Ewan,
For y
015 22:12
>
> *To: *Ewan Leith;
>
> *Cc: *dev@spark.apache.org;
>
> *Subject:*Re: Dataframe nested schema inference from Json without type
> conflicts
>
>
> You can pass the schema into json directly, can't you?
>
> On Thu, Oct 1, 2015 at 10:33 AM, Ewan Le
We could, but if a client sends some unexpected records in the schema (which
happens more than I'd like, our schema seems to constantly evolve), its
fantastic how Spark picks up on that data and includes it.
Passing in a fixed schema loses that nice additional ability, though it's what
we'll p
You can pass the schema into json directly, can't you?
On Thu, Oct 1, 2015 at 10:33 AM, Ewan Leith
wrote:
> Hi all,
>
>
>
> We really like the ability to infer a schema from JSON contained in an
> RDD, but when we’re using Spark Streaming on small batches of data, we
> sometimes find that Spark
