Hi Huai,
Exactly, I'll probably implement one using the new data source API when I
have time... I've found the utility functions in JsonRDD.
Jianshi
On Tue, Dec 9, 2014 at 3:41 AM, Yin Huai wrote:
> Hello Jianshi,
>
> You meant you want to convert a Map to a Struct, right? We can extract
> som
Hello Jianshi,
You meant you want to convert a Map to a Struct, right? We can extract some
useful functions from JsonRDD.scala, so others can access them.
Thanks,
Yin
On Mon, Dec 8, 2014 at 1:29 AM, Jianshi Huang
wrote:
> I checked the source code for inferSchema. Looks like this is exactly w
I checked the source code for inferSchema. Looks like this is exactly what
I want:
val allKeys = rdd.map(allKeysWithValueTypes).reduce(_ ++ _)
Then I can do createSchema(allKeys).
Jianshi
On Sun, Dec 7, 2014 at 2:50 PM, Jianshi Huang
wrote:
> Hmm..
>
> I've created a JIRA: https://issues.ap
Hmm..
I've created a JIRA: https://issues.apache.org/jira/browse/SPARK-4782
Jianshi
On Sun, Dec 7, 2014 at 2:32 PM, Jianshi Huang
wrote:
> Hi,
>
> What's the best way to convert RDD[Map[String, Any]] to a SchemaRDD?
>
> I'm currently converting each Map to a JSON String and do
> JsonRDD.inferS
Hi,
What's the best way to convert RDD[Map[String, Any]] to a SchemaRDD?
I'm currently converting each Map to a JSON String and do
JsonRDD.inferSchema.
How about adding inferSchema support to Map[String, Any] directly? It would
be very useful.
Thanks,
--
Jianshi Huang
LinkedIn: jianshi
Twitte
