Re: Loading lots of parquet files into dataframe from s3

2015-06-18 Thread lovelylavs
e; } else { concatName+= "s3n://" + s3_bucket + "/" + fName + ","; } } -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-lots-of-parquet-files-into-dataframe-from-s

Re: Loading lots of parquet files into dataframe from s3

2015-06-17 Thread arnonrgo
) you can tell Spark to skip this check by specifying the option "mergeSchema" "false" as in read.format("parquet").option("mergeSchema","false").load("path") -- View this message in context: http://apache-spark-user-list.100156