Hi Benjamin,
As you might already know, I believe the Hadoop command automatically does not merge the column-based format such as ORC or Parquet but just simply concatenates them. I haven't tried this by myself but I remember I saw a JIRA in Parquet - https://issues.apache.org/jira/browse/PARQUET-460 It seems parquet-tools allows merge small Parquet files into one. Also, I believe there are command-line tools in Kite - https://github.com/kite-sdk/kite This might be useful. Thanks! 2016-12-23 7:01 GMT+09:00 Benjamin Kim <bbuil...@gmail.com>: > Has anyone tried to merge *.gz.parquet files before? I'm trying to merge > them into 1 file after they are output from Spark. Doing a coalesce(1) on > the Spark cluster will not work. It just does not have the resources to do > it. I'm trying to do it using the commandline and not use Spark. I will use > this command in shell script. I tried "hdfs dfs -getmerge", but the file > becomes unreadable by Spark with gzip footer error. > > Thanks, > Ben > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
