Hi Ayoub,
You could try using the sql format to set the compression type:
sc = SparkContext()
sqc = SQLContext(sc)
sqc.sql("SET spark.sql.parquet.compression.codec=gzip")
You get a notification on screen while running the spark job when you set
the compression codec like this. I havent compared it with different
compression methods, Please let the mailing list knows if this works for
you.
Best
Sahan
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Parquet-compression-codecs-not-applied-tp21058p21498.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
