It does generalize types, but only on the intersection of the columns it seems.
There might be a way to get the union of the columns too using HiveQL. Types
generalize up with string being the "most general".
Matei
> On Nov 1, 2014, at 6:22 PM, Daniel Mahler wrote:
>
> Thanks Matei. What does
Thanks Matei. What does unionAll do if the input RDD schemas are not 100%
compatible. Does it take the union of the columns and generalize the types?
thanks
Daniel
On Sat, Nov 1, 2014 at 6:08 PM, Matei Zaharia
wrote:
> Try unionAll, which is a special method on SchemaRDDs that keeps the
> schem
Try unionAll, which is a special method on SchemaRDDs that keeps the schema on
the results.
Matei
> On Nov 1, 2014, at 3:57 PM, Daniel Mahler wrote:
>
> I would like to combine 2 parquet tables I have create.
> I tried:
>
> sc.union(sqx.parquetFile("fileA"), sqx.parquetFile("fileB"))
>
