For uniform partitioning, you can try custom Partitioner.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-Requested-array-size-exceeds-VM-limit-tp16809p26477.html
Sent from the Apache Spark User List mailing list archive at Nabble
"only option is to split you problem further by increasing parallelism" My
understanding is by increasing the number of partitions, is that right?
That didn't seem to help because it is seem the partitions are not uniformly
sized. My observation is when I increase the number of partitions, it
c
That's true Guillaume.
I'm currently aggregating documents considering a week as time range.
I will have to make it daily and aggregate the results later.
thanks for your hints anyway
Arian Pasquali
http://about.me/arianpasquali
2014-10-20 13:53 GMT+01:00 Guillaume Pitel :
> Hi,
>
> The arr
Hi,
The array size you (or the serializer) tries to allocate is just too big
for the JVM. No configuration can help :
https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit
The only option is to split you problem further by increasing parallelism.
Guillaume
Hi,
I’m using S
Try setting SPARK_EXECUTOR_MEMORY=5g (not sure how many workers you are
having), You can also set the executor memory while creating the
sparkContext (like *sparkContext.set("spark.executor.memory","5g")* )
Thanks
Best Regards
On Mon, Oct 20, 2014 at 5:01 PM, Arian Pasquali
wrote:
> Hi Akhil,
>
Hi Akhil,
thanks for your help
but I was originally running without xmx option. With that I was just
trying to push the limit of my heap size, but obviously doing it wrong.
Arian Pasquali
http://about.me/arianpasquali
2014-10-20 12:24 GMT+01:00 Akhil Das :
> Hi Arian,
>
> You will get this e
Hi Arian,
You will get this exception because you are trying to create an array that
is larger than the maximum contiguous block of memory in your Java VMs heap.
Here since you are setting Worker memory as *5Gb* and you are exporting the
*_OPTS as *8Gb*, your application actually thinks it has 8G
