Thanks Ted for the input. I was able to get it working with pyspark shell
but the same job submitted via 'spark-submit' using client or cluster
deploy mode ends up with these errors:
~
java.lang.OutOfMemoryError: Java heap space
at java.lang.Object.clone(Native Method)
at akka.util.CompactByte
Looks like the exception occurred on driver.
Consider increasing the values for the following config:
conf.set("spark.driver.memory", "10240m")
conf.set("spark.driver.maxResultSize", "2g")
Cheers
On Sat, Apr 9, 2016 at 9:02 PM, Buntu Dev wrote:
> I'm running it via pyspark against yarn in cli
I'm running it via pyspark against yarn in client deploy mode. I do notice
in the spark web ui under Environment tab all the options I've set, so I'm
guessing these are accepted.
On Sat, Apr 9, 2016 at 5:52 PM, Jacek Laskowski wrote:
> Hi,
>
> (I haven't played with GraphFrames)
>
> What's your
Hi,
(I haven't played with GraphFrames)
What's your `sc.master`? How do you run your application --
spark-submit or java -jar or sbt run or...? The reason I'm asking is
that few options might not be in use whatsoever, e.g.
spark.driver.memory and spark.executor.memory in local mode.
Pozdrawiam,
I'm running this motif pattern against 1.5M vertices (5.5mb) and 10M (60mb)
edges:
tgraph.find("(a)-[]->(b); (c)-[]->(b); (c)-[]->(d)")
I keep running into Java heap space errors:
~
ERROR actor.ActorSystemImpl: Uncaught fatal error from thread
[sparkDriver-akka.actor.default-dispatcher-33]
