Hi,
When running a Spark job, the following warning message displays and the job
seems no longer progressing.
(Detailed log message are at the bottom of this message.)
---------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
---------------
The specifics of the job is as follows:
- It reads 168016 files on the HDFS, by calling
sc.textFile("hdfs://cluster01/user/data/*/*/*.csv")
- The total size of the files is 164,111,123,686 bytes. (164GB)
- The max size of the files is 8,546,230 bytes. (8.5MB)
- The min size of the files is 3,920 bytes. (3KB)
- The command line options are: --master yarn-client --executor-memory 14G
--executor-cores 7 --num-executors 3
The WARN occurred when processing reduceByKey().
On Spark Stage web page, the number of tasks for reduceByKey() was 168016.
(same as the number of the files)
The WARN occurred when reduceByKey() was progressing about 10%. (not exact)
Actual method call sequence is:
sc.textFile(...).filter().mapToPair().reduceByKey().map().saveAsTextFile().
How can I fix this?
Detailed log messages are as follows: (IP and hostname was replaced.)
----------------------------------------
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:16 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(<driver>, DriverHostName, 63548, 0) with no recent heart
beats: 63641ms exceeds 45000ms
14/07/02 17:00:14 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
14/07/02 17:00:14 ERROR Utils: Uncaught exception in thread Result resolver
thread-2
14/07/02 17:00:37 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at
org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at
org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChanne
lBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at
org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferF
actory.java:68)
at
org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChanne
lBufferFactory.java:48)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWork
er.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelect
or.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.j
ava:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Result resolver thread-2" java.lang.OutOfMemoryError:
Java heap space
14/07/02 17:00:19 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(3, cluster03, 45202, 0) with no recent heart beats: 65296ms
exceeds 45000ms
14/07/02 17:00:41 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:00:43 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(1, cluster04, 55341, 0) with no recent heart beats: 65328ms
exceeds 45000ms
14/07/02 17:00:47 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(2, cluster02, 54328, 0) with no recent heart beats: 69631ms
exceeds 45000ms
Exception in thread "SparkListenerBus" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "Connection manager future execution context-1" 14/07/02
17:00:55 WARN AbstractNioSelector: Unexpected exception in the selector
loop.
14/07/02 17:01:00 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:01:48 WARN DefaultChannelPipeline: An exception was thrown by a
user handler while handling an exception event ([id: 0xee94ecdb,
/111.111.111.12:32842 => /111.111.111.199:63545] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
14/07/02 17:02:28 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:02:29 WARN ServletHandler: Error for /stages/
14/07/02 17:02:28 WARN AbstractNioSelector: Failed to accept a connection.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp1359288682-100" java.lang.OutOfMemoryError: Java
heap space
Exception in thread "qtp1359288682-63" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1359288682-57" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1492586839-40" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "qtp1359288682-106" java.lang.OutOfMemoryError: Java
heap space
14/07/02 17:02:55 WARN AbstractHttpConnection: /stages/
14/07/02 17:03:16 WARN nio: handle failed
14/07/02 17:03:33 WARN nio: handle failed
14/07/02 17:07:58 WARN nio:
14/07/02 17:10:43 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:18:34 WARN AbstractNioSelector: Unexpected exception in the
selector loop.
14/07/02 17:19:32 WARN ServletHandler: Error for /stages/
java.lang.OutOfMemoryError: Java heap space
14/07/02 17:20:05 WARN AbstractHttpConnection: /stages/
java.lang.OutOfMemoryError: Java heap space
Thanks in advance.
