Re: Merging of parquet file schemas

2018-07-10 Thread Wes McKinney
hi Dan, Not yet -- the relevant JIRA is https://issues.apache.org/jira/browse/ARROW-843. We would appreciate some help with this Thanks On Tue, Jul 10, 2018 at 10:54 AM, Dan Amner wrote: > Hi, > > I am attempting to read a number of smaller parquet files and merge them into > a larger parquet

Merging of parquet file schemas

2018-07-10 Thread Dan Amner
Hi, I am attempting to read a number of smaller parquet files and merge them into a larger parquet file. The files are created by Spark jobs that run periodically throughout the day. The issue I have is that the small parquet files can have slightly different schemas and when I create the Data