: Thu, 1 Oct 2015 22:12
To: Ewan Leith;
Cc: dev@spark.apache.org<mailto:dev@spark.apache.org>;
Subject:Re: Dataframe nested schema inference from Json without type conflicts
You can pass the schema into json directly, can't you?
On Thu, Oct 1, 2015 at 10:33 AM, Ewan Leith
mailto:ewa
Thanks Yin, I'll put together a JIRA and a PR tomorrow.
Ewan
-- Original message--
From: Yin Huai
Date: Mon, 5 Oct 2015 17:39
To: Ewan Leith;
Cc: dev@spark.apache.org;
Subject:Re: Dataframe nested schema inference from Json without type conflicts
Hello Ewan,
Adding a
on that data and includes it.
>
>
>
> Passing in a fixed schema loses that nice additional ability, though it's
> what we'll probably have to adopt if we can't come up with a way to keep
> the inference working.
>
>
>
> Thanks,
>
> Ewan
>
;dev@spark.apache.org>;
Subject:Re: Dataframe nested schema inference from Json without type conflicts
Hi Ewan,
For your use case, you only need the schema inference to pick up the structure
of your data (basically you want spark sql to infer the type of complex values
like arrays and stru
Exactly, that's a much better way to put it.
Thanks,
Ewan
-- Original message--
From: Yin Huai
Date: Thu, 1 Oct 2015 23:54
To: Ewan Leith;
Cc: r...@databricks.com;dev@spark.apache.org;
Subject:Re: Dataframe nested schema inference from Json without type conflicts
Hi Ewan,
015 22:12
>
> *To: *Ewan Leith;
>
> *Cc: *dev@spark.apache.org;
>
> *Subject:*Re: Dataframe nested schema inference from Json without type
> conflicts
>
>
> You can pass the schema into json directly, can't you?
>
> On Thu, Oct 1, 2015 at 10:33 AM, Ewan Le
hat
we'll probably have to adopt if we can't come up with a way to keep the
inference working.
Thanks,
Ewan
-- Original message--
From: Reynold Xin
Date: Thu, 1 Oct 2015 22:12
To: Ewan Leith;
Cc: dev@spark.apache.org;
Subject:Re: Dataframe nested schema inference fr
You can pass the schema into json directly, can't you?
On Thu, Oct 1, 2015 at 10:33 AM, Ewan Leith
wrote:
> Hi all,
>
>
>
> We really like the ability to infer a schema from JSON contained in an
> RDD, but when we’re using Spark Streaming on small batches of data, we
> sometimes find that Spark
Hi all,
We really like the ability to infer a schema from JSON contained in an RDD, but
when we're using Spark Streaming on small batches of data, we sometimes find
that Spark infers a more specific type than it should use, for example if the
json in that small batch only contains integer value
