Thanks. using this command and parameters,it works. *spark-submit --master yarn-client --executor-memory 8g --executor-cores 4 /home/hadoop/spark/main.py*
2015-04-08 5:45 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: > 李铖: > w.r.t. #5, you can use --executor-cores when invoking spark-submit > > Cheers > > On Tue, Apr 7, 2015 at 2:35 PM, java8964 <java8...@hotmail.com> wrote: > >> It is hard to guess why OOM happens without knowing your application's >> logic and the data size. >> >> Without knowing that, I can only guess based on some common experiences: >> >> 1) increase "spark.default.parallelism" >> 2) Increase your executor-memory, maybe 6g is not just enough >> 3) Your environment is kind of unbalance between cup cores and available >> memory (8 cores vs 12G). Each core should have 3G for Spark. >> 4) If you cache RDD, using "MEMORY_ONLY_SER" instead of "MEMORY_ONLY" >> 5) Since your cores is much more compared with your available memory, >> lower the cores for executor by set "-Dspark.deploy.defaultCores=". When >> you have not enough memory, reduce the concurrency of your executor, it >> will lower the memory requirement, with running in a slower speed. >> >> Yong >> >> ------------------------------ >> Date: Wed, 8 Apr 2015 04:57:22 +0800 >> Subject: Re: 'Java heap space' error occured when query 4G data file from >> HDFS >> From: lidali...@gmail.com >> To: user@spark.apache.org >> >> Any help?please. >> >> Help me do a right configure. >> >> >> 李铖 <lidali...@gmail.com>于2015年4月7日星期二写道： >> >> In my dev-test env .I have 3 virtual machines ,every machine have 12G >> memory,8 cpu core. >> >> Here is spark-defaults.conf,and spark-env.sh.Maybe some config is not >> right. >> >> I run this command :*spark-submit --master yarn-client --driver-memory >> 7g --executor-memory 6g /home/hadoop/spark/main.py* >> exception rised. >> >> *spark-defaults.conf* >> >> spark.master spark://cloud1:7077 >> spark.default.parallelism 100 >> spark.eventLog.enabled true >> spark.serializer >> org.apache.spark.serializer.KryoSerializer >> spark.driver.memory 5g >> spark.driver.maxResultSize 6g >> spark.kryoserializer.buffer.mb 256 >> spark.kryoserializer.buffer.max.mb 512 >> spark.executor.memory 4g >> spark.rdd.compress true >> spark.storage.memoryFraction 0 >> spark.akka.frameSize 50 >> spark.shuffle.compress true >> spark.shuffle.spill.compress false >> spark.local.dir /home/hadoop/tmp >> >> * spark-evn.sh* >> >> export SCALA=/home/hadoop/softsetup/scala >> export JAVA_HOME=/home/hadoop/softsetup/jdk1.7.0_71 >> export SPARK_WORKER_CORES=1 >> export SPARK_WORKER_MEMORY=4g >> export HADOOP_CONF_DIR=/opt/cloud/hadoop/etc/hadoop >> export SPARK_EXECUTOR_MEMORY=4g >> export SPARK_DRIVER_MEMORY=4g >> >> *Exception:* >> >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on >> cloud3:38109 (size: 162.7 MB) >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on >> cloud3:38109 (size: 162.7 MB) >> 15/04/07 18:11:03 INFO TaskSetManager: Starting task 31.0 in stage 1.0 >> (TID 31, cloud3, NODE_LOCAL, 1296 bytes) >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on >> cloud2:49451 (size: 163.7 MB) >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on >> cloud2:49451 (size: 163.7 MB) >> 15/04/07 18:11:03 INFO TaskSetManager: Starting task 30.0 in stage 1.0 >> (TID 32, cloud2, NODE_LOCAL, 1296 bytes) >> 15/04/07 18:11:03 ERROR Utils: Uncaught exception in thread >> task-result-getter-0 >> java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:61) >> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) >> at >> org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:58) >> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> at >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >> at >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:81) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:73) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) >> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:48) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Exception in thread "task-result-getter-0" java.lang.OutOfMemoryError: >> Java heap space >> at >> org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:61) >> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985) >> at >> org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:58) >> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) >> at >> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) >> at >> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:81) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:73) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:49) >> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460) >> at >> org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:48) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_28 on disk on >> cloud3:38109 (size: 162.7 MB) >> 15/04/07 18:11:03 INFO BlockManagerInfo: Added taskresult_29 on disk on >> cloud2:49451 (size: 163.7 MB) >> 15/04/07 18:11:05 ERROR Utils: Uncaught exception in thread >> task-result-getter-4 >> java.lang.OutOfMemoryError: Java heap space >> Exception in thread "task-result-getter-4" java.lang.OutOfMemoryError: >> Java heap space >> 15/04/07 18:11:07 INFO BlockManagerInfo: Added taskresult_31 on disk on >> cloud3:38109 (size: 87.9 MB) >> 15/04/07 18:11:07 INFO BlockManagerInfo: Added taskresult_31 on disk on >> cloud3:38109 (size: 87.9 MB) >> 15/04/07 18:11:08 WARN TransportChannelHandler: Exception in connection >> from cloud3/192.168.0.95:38109 >> java.lang.OutOfMemoryError: Java heap space >> 15/04/07 18:11:08 ERROR TransportResponseHandler: Still have 1 requests >> outstanding when connection from cloud3/192.168.0.95:38109 is closed >> 15/04/07 18:11:08 ERROR OneForOneBlockFetcher: Failed while starting >> block fetches >> java.lang.OutOfMemoryError: Java heap space >> 15/04/07 18:11:08 ERROR RetryingBlockFetcher: Failed to fetch block >> taskresult_31, and will not retry (0 retries) >> java.lang.OutOfMemoryError: Java heap space >> 15/04/07 18:11:08 ERROR TransportClient: Failed to send RPC >> 7722440433247749491 to cloud3/192.168.0.95:38109: >> java.nio.channels.ClosedChannelException >> java.nio.channels.ClosedChannelException >> 15/04/07 18:11:08 ERROR OneForOneBlockFetcher: Failed while starting >> block fetches >> java.io.IOException: Failed to send RPC 7722440433247749491 to cloud3/ >> 192.168.0.95:38109: java.nio.channels.ClosedChannelException >> at >> org.apache.spark.network.client.TransportClient$2.operationComplete(TransportClient.java:158) >> at >> org.apache.spark.network.client.TransportClient$2.operationComplete(TransportClient.java:145) >> at >> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) >> at >> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) >> at >> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) >> at >> io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:745) >> at >> io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:646) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1054) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658) >> at >> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:716) >> at >> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:651) >> at >> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:112) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658) >> at >> io.netty.channel.AbstractChannelHandlerContext.access$2000(AbstractChannelHandlerContext.java:32) >> at >> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:939) >> at >> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:991) >> at >> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:924) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.nio.channels.ClosedChannelException >> 15/04/07 18:11:08 INFO BlockManagerInfo: Added taskresult_30 on disk on >> cloud1:44029 (size: 163.5 MB) >> 15/04/07 18:11:08 INFO BlockManagerInfo: Added taskresult_30 on disk on >> cloud1:44029 (size: 163.5 MB) >> 15/04/07 18:11:08 ERROR Utils: Uncaught exception in thread >> task-result-getter-6 >> java.lang.OutOfMemoryError: Java heap space >> Exception in thread "task-result-getter-6" java.lang.OutOfMemoryError: >> Java heap space >> 15/04/07 18:11:08 ERROR TaskResultGetter: Exception while getting task >> result >> java.util.concurrent.ExecutionException: Boxed Error >> at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) >> at >> scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) >> at >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) >> at scala.concurrent.Promise$class.complete(Promise.scala:55) >> at >> scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) >> at scala.concurrent.Promise$class.failure(Promise.scala:107) >> at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:153) >> at >> org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchFailure(BlockTransferService.scala:92) >> at >> org.apache.spark.network.shuffle.RetryingBlockFetcher$RetryingBlockFetchListener.onBlockFetchFailure(RetryingBlockFetcher.java:230) >> at >> org.apache.spark.network.shuffle.OneForOneBlockFetcher.failRemainingBlocks(OneForOneBlockFetcher.java:123) >> at >> org.apache.spark.network.shuffle.OneForOneBlockFetcher.access$300(OneForOneBlockFetcher.java:43) >> at >> org.apache.spark.network.shuffle.OneForOneBlockFetcher$1.onFailure(OneForOneBlockFetcher.java:114) >> at >> org.apache.spark.network.client.TransportResponseHandler.failOutstandingRequests(TransportResponseHandler.java:84) >> at >> org.apache.spark.network.client.TransportResponseHandler.exceptionCaught(TransportResponseHandler.java:108) >> at >> org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:69) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:271) >> at >> io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:768) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.OutOfMemoryError: Java heap space >> >> >
