ot;para_predictions")--
The error happens when saveAsTextFile. I changed the repartition number to 200
or 300. But still got this error.
Thanks for help.
Date: Thu, 8 Oct 2015 13:55:32 -0700
Subject: Re: ValueError: can not serialize object larger than 2G
From: yuzhih...@gmail.com
To: zxd_ci
ine 126, in
>> dump_stream
>> self._write_with_length(obj, stream)
>> File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
>> _write_with_length
>> raise ValueError("can not serialize object larger than 2G")
>> ValueE
on/pyspark/serializers.py", line 126, in
> dump_stream
> self._write_with_length(obj, stream)
> File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
> _write_with_length
> raise ValueError("can not serialize object larger than 2G")
>
alizers.py", line 126, in
dump_stream
self._write_with_length(obj, stream)
File "/home/hadoop/spark/python/pyspark/serializers.py", line 140, in
_write_with_length
raise ValueError("can not serialize object larger than 2G")
ValueError: can not serialize object lar
