I tried setting both the hdfs and parquet block size but write to parquet
did not seem to have had any effect on the total number of blocks or the
average block size. Here is what I did:
~~~~
sqlContext.setConf("dfs.blocksize", "134217728")
sqlContext.setConf("parquet.block.size", "134217728")
sqlContext.setConf("spark.broadcast.blockSize", "134217728")
df.write.parquet("/path/to/dest")
~~~~
I tried the same with different block sizes but none had any effect. Is
this the right way to set the properties using setConf()?
Thanks!
On Wed, Apr 6, 2016 at 11:28 PM, bdev <buntu...@gmail.com> wrote:
> I need to save the dataframe to parquet format and need some input on
> choosing the appropriate block size to help efficiently
> parallelize/localize
> the data to the executors. Should I be using parquet block size or hdfs
> block size and what is the optimal block size to use on a 100 node cluster?
>
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-to-parquet-using-hdfs-or-parquet-block-size-tp26693.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
