big it would be, I assume it’s over 2 GB
From: Zhang, Jingyu [mailto:jingyu.zh...@news.com.au]
Sent: 16 November 2015 10:17
To: user
Subject: Size exceeds Integer.MAX_VALUE on EMR 4.0.0 Spark 1.4.1
I am using spark-csv to save files in s3, it shown Size exceeds. Please let me
know how to fix it
You can try increasing the number of partitions before writing it out.
Regards
Sab
On Mon, Nov 16, 2015 at 3:46 PM, Zhang, Jingyu
wrote:
> I am using spark-csv to save files in s3, it shown Size exceeds. Please let
> me know how to fix it. Thanks.
>
> df.write()
> .format("com.databricks.s
I am using spark-csv to save files in s3, it shown Size exceeds.
Please let me know how to fix it. Thanks.
df.write()
.format("com.databricks.spark.csv")
.option("header", "true")
.save("s3://newcars.csv");
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at
