Shuai, What did " ps aux | grep spark-submit" reveal?
When you compare using _JAVA_OPTIONS and without using it, where do you see the difference? Thanks Christopher -----Original Message----- From: Shuai Zheng [mailto:szheng.c...@gmail.com] Sent: Wednesday, April 01, 2015 11:12 AM To: 'Sean Owen' Cc: 'Akhil Das'; user@spark.apache.org Subject: RE: --driver-memory parameter doesn't work for spark-submmit on yarn? Nice. But when my case shows that even I use Yarn-Client, I have same issue. I do verify it several times. And I am running 1.3.0 on EMR (use the version dispatch by installSpark script from AWS). I agree _JAVA_OPTIONS is not a right solution, but I will use it until 1.4.0 out :) Regards, Shuai -----Original Message----- From: Sean Owen [mailto:so...@cloudera.com] Sent: Wednesday, April 01, 2015 10:51 AM To: Shuai Zheng Cc: Akhil Das; user@spark.apache.org Subject: Re: --driver-memory parameter doesn't work for spark-submmit on yarn? I feel like I recognize that problem, and it's almost the inverse of https://issues.apache.org/jira/browse/SPARK-3884 which I was looking at today. The spark-class script didn't seem to handle all the ways that driver memory can be set. I think this is also something fixed by the new launcher library in 1.4.0. _JAVA_OPTIONS is not a good solution since it's global. On Wed, Apr 1, 2015 at 3:21 PM, Shuai Zheng <szheng.c...@gmail.com> wrote: > Hi Akhil, > > > > Thanks a lot! > > > > After set export _JAVA_OPTIONS="-Xmx5g", the OutOfMemory exception > disappeared. But this make me confused, so the driver-memory options > doesn’t work for spark-submit to YARN (I haven’t check other clusters), is it > a bug? > > > > Regards, > > > > Shuai > > > > > > From: Akhil Das [mailto:ak...@sigmoidanalytics.com] > Sent: Wednesday, April 01, 2015 2:40 AM > To: Shuai Zheng > Cc: user@spark.apache.org > Subject: Re: --driver-memory parameter doesn't work for spark-submmit > on yarn? > > > > Once you submit the job do a ps aux | grep spark-submit and see how > much is the heap space allocated to the process (the -Xmx params), if > you are seeing a lower value you could try increasing it yourself with: > > > > export _JAVA_OPTIONS="-Xmx5g" > > > Thanks > > Best Regards > > > > On Wed, Apr 1, 2015 at 1:57 AM, Shuai Zheng <szheng.c...@gmail.com> wrote: > > Hi All, > > > > Below is the my shell script: > > > > /home/hadoop/spark/bin/spark-submit --driver-memory=5G > --executor-memory=40G --master yarn-client --class > com.***.FinancialEngineExecutor /home/hadoop/lib/my.jar > s3://bucket/vriscBatchConf.properties > > > > My driver will load some resources and then broadcast to all executors. > > > > That resources is only 600MB in ser format, but I always has out of > memory exception, it looks like the driver doesn’t allocate right > memory to my driver. > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > at java.lang.reflect.Array.newArray(Native Method) > > at java.lang.reflect.Array.newInstance(Array.java:70) > > at > java.io.ObjectInputStream.readArray(ObjectInputStream.java:1670) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) > > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199 > 0) > > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17 > 98) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > > at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > > at > com.***.executor.support.S3FileUtils.loadCache(S3FileUtils.java:68) > > > > Do I do anything wrong here? > > > > And no matter how much I set for --driver-memory value (from 512M to > 20G), it always give me error on the same line (that line try to load > a 600MB java serialization file). So looks like the script doesn’t > allocate right memory to driver in my case? > > > > Regards, > > > > Shuai > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
