I removed the use of numactl but left in starting two TaskManagers and am
still seeing TaskManagers crash.
>From the JobManager log:
17:36:06,412 WARN
akka.remote.ReliableDeliverySupervisor - Association
with remote system [akka.tcp://flink@10.0.88.140:45742] has failed, address
is now gated for [5000] ms. Reason is: [Disassociated].
17:36:06,567 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (370/2322)
(cac9927a8568c2ad79439262a91478af) switched from RUNNING to FAILED
17:36:06,572 INFO
org.apache.flink.runtime.jobmanager.JobManager - Status of
job 14d946015fd7b35eb801ea6fee5af9e4 (Flink Java Job at Thu Oct 29 17:34:48
UTC 2015) changed to FAILING.
java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
(Compute scores) -> FlatMap (checksum())' , caused an error: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated
due to an exception: Connection unexpectedly closed by remote task manager
'ip-10-0-88-140/10.0.88.140:58558'. This might indicate that the remote
task manager was lost.
at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:465)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
Thread 'SortMerger Reading Thread' terminated due to an exception:
Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
10.0.88.140:58558'. This might indicate that the remote task manager was
lost.
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1089)
at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:459)
... 3 more
Caused by: java.io.IOException: Thread 'SortMerger Reading Thread'
terminated due to an exception: Connection unexpectedly closed by remote
task manager 'ip-10-0-88-140/10.0.88.140:58558'. This might indicate that
the remote task manager was lost.
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by:
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
10.0.88.140:58558'. This might indicate that the remote task manager was
lost.
at
org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
at
io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
at
io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
17:36:06,587 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (367/2322)
(d63c681a18b8164bc24936df1ecb159b) switched from RUNNING to FAILED
On Thu, Oct 29, 2015 at 1:00 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi Greg!
>
> Interesting... When you say the TaskManagers are dropping, are the
> TaskManager processes crashing, or are they loosing connection to the
> JobManager?
>
> Greetings,
> Stephan
>
>
> On Thu, Oct 29, 2015 at 9:56 AM, Greg Hogan <c...@greghogan.com> wrote:
>
> > I recently discovered that AWS uses NUMA for its largest nodes. An
> example
> > c4.8xlarge:
> >
> > $ numactl --hardware
> > available: 2 nodes (0-1)
> > node 0 cpus: 0 1 2 3 4 5 6 7 8 18 19 20 21 22 23 24 25 26
> > node 0 size: 29813 MB
> > node 0 free: 24537 MB
> > node 1 cpus: 9 10 11 12 13 14 15 16 17 27 28 29 30 31 32 33 34 35
> > node 1 size: 30574 MB
> > node 1 free: 22757 MB
> > node distances:
> > node 0 1
> > 0: 10 20
> > 1: 20 10
> >
> > I discovered yesterday that Flink performed ~20-30% faster on large
> > datasets by running two NUMA-constrained TaskManagers per node. The
> > JobManager node ran a single TaskManager. Resources were divided in half
> > relative to running a single TaskManager.
> >
> > The changes from the tail of /bin/taskmanager.sh:
> >
> > -"${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP taskmanager "${args[@]}"
> > +numactl --membind=0 --cpunodebind=0 "${FLINK_BIN_DIR}"/flink-daemon.sh
> > $STARTSTOP taskmanager "${args[@]}"
> > +numactl --membind=1 --cpunodebind=1 "${FLINK_BIN_DIR}"/flink-daemon.sh
> > $STARTSTOP taskmanager "${args[@]}"
> >
> > After reverting this change the system is again stable. I had not
> > experienced issues using numactl when running 16 nodes.
> >
> > Greg
> >
>
