Kaspar Sakmann created SPARK-20406:
--------------------------------------
Summary: Upgrade parquet to 1.9
Key: SPARK-20406
URL: https://issues.apache.org/jira/browse/SPARK-20406
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.1.0
Reporter: Kaspar Sakmann
Please upgrade parquet to 1.9.
with 500G memory 48 CPUs and a dataset about 1G in size
using Parquet 1.8 causes warnings:
"Total allocation exceeds 95.00% (2,147,483,647 bytes) of heap memory
Scaling row group sizes to 34.04% for 47 writers"
The issue appears to have been fixed with parquet 1.9, see
https://issues.apache.org/jira/browse/PARQUET-340
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
