Re: OutofMemoryError: Java heap space

Thanks, Kelvin :) The error seems to disappear after I decreased both spark.storage.memoryFraction and spark.shuffle.memoryFraction to 0.2 And, some increase on driver memory. Best, Yifan LI > On 10 Feb 2015, at 18:58, Kelvin Chu <2dot7kel...@gmail.com> wrote: > > Since the stacktrace

Re: OutofMemoryError: Java heap space

Since the stacktrace shows kryo is being used, maybe, you could also try increasing spark.kryoserializer.buffer.max.mb. Hope this help. Kelvin On Tue, Feb 10, 2015 at 1:26 AM, Akhil Das wrote: > You could try increasing the driver memory. Also, can you be more specific > about the data volume?

Re: OutofMemoryError: Java heap space

Yes, I have read it, and am trying to find some way to do that… Thanks :) Best, Yifan LI > On 10 Feb 2015, at 12:06, Akhil Das wrote: > > Did you have a chance to look at this doc > http://spark.apache.org/docs/1.2.0/tuning.html > > > Than

Re: OutofMemoryError: Java heap space

Did you have a chance to look at this doc http://spark.apache.org/docs/1.2.0/tuning.html Thanks Best Regards On Tue, Feb 10, 2015 at 4:13 PM, Yifan LI wrote: > Hi Akhil, > > Excuse me, I am trying a random-walk algorithm over a not that large > graph(~1GB raw dataset, including ~5million vertic

Re: OutofMemoryError: Java heap space

Hi Akhil, Excuse me, I am trying a random-walk algorithm over a not that large graph(~1GB raw dataset, including ~5million vertices and ~60million edges) on a cluster which has 20 machines. And, the property of each vertex in graph is a hash map, of which size will increase dramatically during

Re: OutofMemoryError: Java heap space

You could try increasing the driver memory. Also, can you be more specific about the data volume? Thanks Best Regards On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI wrote: > Hi, > > I just found the following errors during computation(graphx), anyone has > ideas on this? thanks so much! > > (I think