I have memory logging enabled. Tail of TaskManager log on 10.0.88.140:
17:35:26,415 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 341, GC COUNT: 3], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:27,415 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 576/1917/1917 MB, NON HEAP: 56/58/-1 MB
(used/committed/max)]
17:35:27,415 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 18/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:27,415 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 341, GC COUNT: 3], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:28,012 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (938/2322)
17:35:28,015 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(938/2322)
17:35:28,016 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(938/2322) [DEPLOYING]
17:35:28,065 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (938/2322) switched to
RUNNING
17:35:28,100 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2304/2322)
17:35:28,116 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2304/2322)
17:35:28,116 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2304/2322) [DEPLOYING]
17:35:28,132 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2304/2322) switched
to RUNNING
17:35:28,255 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (939/2322)
17:35:28,263 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(939/2322)
17:35:28,263 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(939/2322) [DEPLOYING]
17:35:28,304 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2062/2322)
17:35:28,311 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2062/2322)
17:35:28,311 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2062/2322) [DEPLOYING]
17:35:28,323 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (939/2322) switched to
RUNNING
17:35:28,386 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2062/2322) switched
to RUNNING
17:35:28,396 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1775/2322)
17:35:28,401 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1775/2322)
17:35:28,402 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1775/2322) [DEPLOYING]
17:35:28,416 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 747/1917/1917 MB, NON HEAP: 56/58/-1 MB
(used/committed/max)]
17:35:28,416 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 18/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:28,416 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 341, GC COUNT: 3], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:28,419 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1775/2322) switched
to RUNNING
17:35:28,475 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2158/2322)
17:35:28,475 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2158/2322)
17:35:28,476 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2158/2322) [DEPLOYING]
17:35:28,509 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1463/2322)
17:35:28,860 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1463/2322)
17:35:28,861 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1463/2322) [DEPLOYING]
17:35:28,862 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2158/2322) switched
to RUNNING
17:35:28,878 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1463/2322) switched
to RUNNING
17:35:28,892 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1154/2322)
17:35:28,893 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1154/2322)
17:35:28,893 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1154/2322) [DEPLOYING]
17:35:28,914 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1154/2322) switched
to RUNNING
17:35:28,916 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1429/2322)
17:35:28,917 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1429/2322)
17:35:28,917 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1429/2322) [DEPLOYING]
17:35:28,942 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1078/2322)
17:35:28,942 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1078/2322)
17:35:28,942 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1078/2322) [DEPLOYING]
17:35:28,943 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1429/2322) switched
to RUNNING
17:35:28,955 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1078/2322) switched
to RUNNING
17:35:28,959 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (524/2322)
17:35:28,995 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(524/2322)
17:35:28,995 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(524/2322) [DEPLOYING]
17:35:29,000 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2021/2322)
17:35:29,000 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2021/2322)
17:35:29,000 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2021/2322) [DEPLOYING]
17:35:29,012 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (524/2322) switched to
RUNNING
17:35:29,039 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2022/2322)
17:35:29,039 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2021/2322) switched
to RUNNING
17:35:29,043 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2022/2322)
17:35:29,043 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2022/2322) [DEPLOYING]
17:35:29,076 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1464/2322)
17:35:29,081 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1464/2322)
17:35:29,081 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1464/2322) [DEPLOYING]
17:35:29,095 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2022/2322) switched
to RUNNING
17:35:29,108 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (1095/2322)
17:35:29,110 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1464/2322) switched
to RUNNING
17:35:29,112 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1095/2322)
17:35:29,112 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(1095/2322) [DEPLOYING]
17:35:29,140 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2306/2322)
17:35:29,142 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2306/2322)
17:35:29,142 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2306/2322) [DEPLOYING]
17:35:29,147 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (1095/2322) switched
to RUNNING
17:35:29,152 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (974/2322)
17:35:29,153 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2306/2322) switched
to RUNNING
17:35:29,155 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(974/2322)
17:35:29,155 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(974/2322) [DEPLOYING]
17:35:29,166 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Received
task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum()) (2305/2322)
17:35:29,167 INFO
org.apache.flink.runtime.taskmanager.Task - Loading JAR
files for task CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2305/2322)
17:35:29,167 INFO
org.apache.flink.runtime.taskmanager.Task - Registering
task at network: CHAIN GroupReduce (Compute scores) -> FlatMap (checksum())
(2305/2322) [DEPLOYING]
17:35:29,176 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (974/2322) switched to
RUNNING
17:35:29,205 INFO
org.apache.flink.runtime.taskmanager.Task - CHAIN
GroupReduce (Compute scores) -> FlatMap (checksum()) (2305/2322) switched
to RUNNING
17:35:29,417 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 590/1917/1917 MB, NON HEAP: 57/59/-1 MB
(used/committed/max)]
17:35:29,417 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 18/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:29,417 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:30,418 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 614/1917/1917 MB, NON HEAP: 57/59/-1 MB
(used/committed/max)]
17:35:30,418 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 18/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:30,418 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:31,418 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 634/1917/1917 MB, NON HEAP: 57/59/-1 MB
(used/committed/max)]
17:35:31,418 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:31,419 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:32,419 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 638/1917/1917 MB, NON HEAP: 57/59/-1 MB
(used/committed/max)]
17:35:32,419 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:32,419 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:33,487 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 648/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:33,494 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:33,522 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:34,523 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 662/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:34,523 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:34,523 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:35,523 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 670/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:35,524 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:35,524 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:36,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 717/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:36,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:36,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:37,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 737/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:37,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/19/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:37,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:38,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 747/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:38,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:38,525 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:39,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 817/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:39,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:39,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:40,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 832/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:40,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:40,526 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:41,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 840/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:41,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:41,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:42,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 847/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:42,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:42,527 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 689, GC COUNT: 4], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:43,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 450/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:43,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:43,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:44,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 508/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:44,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:44,599 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:45,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 517/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:45,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:45,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:46,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 528/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:46,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:46,600 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:47,663 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 541/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:47,664 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:47,664 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:48,791 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 554/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:48,791 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:48,791 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:49,794 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 562/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:49,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:49,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:50,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 569/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:50,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:50,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:51,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 582/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:51,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:51,795 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:52,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 593/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:52,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:52,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:53,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 600/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:53,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:53,796 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:54,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 604/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:54,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:54,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:55,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 610/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:55,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:55,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:56,797 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 615/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:56,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:56,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:57,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 624/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:57,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:57,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:58,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 636/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:58,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:58,798 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:35:59,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 641/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:35:59,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:35:59,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:36:00,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 648/1917/1917 MB, NON HEAP: 58/59/-1 MB
(used/committed/max)]
17:36:00,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/34/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:36:00,799 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:36:01,821 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 655/1917/1917 MB, NON HEAP: 58/60/-1 MB
(used/committed/max)]
17:36:01,936 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/35/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:36:01,936 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:36:02,937 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 665/1917/1917 MB, NON HEAP: 58/60/-1 MB
(used/committed/max)]
17:36:02,937 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/35/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:36:02,937 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
17:36:03,944 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Memory
usage stats: [HEAP: 666/1917/1917 MB, NON HEAP: 58/60/-1 MB
(used/committed/max)]
17:36:03,950 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Off-heap
pool stats: [Code Cache: 19/20/240 MB (used/committed/max)], [Metaspace:
34/35/-1 MB (used/committed/max)], [Compressed Class Space: 4/4/1024 MB
(used/committed/max)]
17:36:03,951 INFO
org.apache.flink.runtime.taskmanager.TaskManager - Garbage
collector stats: [PS Scavenge, GC TIME (ms): 797, GC COUNT: 5], [PS
MarkSweep, GC TIME (ms): 974, GC COUNT: 1]
On Thu, Oct 29, 2015 at 1:55 PM, Till Rohrmann <trohrm...@apache.org> wrote:
> What does the log of the failed TaskManager 10.0.88.140 say?
>
> On Thu, Oct 29, 2015 at 6:44 PM, Greg Hogan <c...@greghogan.com> wrote:
>
> > I removed the use of numactl but left in starting two TaskManagers and am
> > still seeing TaskManagers crash.
> > From the JobManager log:
> >
> > 17:36:06,412 WARN
> > akka.remote.ReliableDeliverySupervisor -
> Association
> > with remote system [akka.tcp://flink@10.0.88.140:45742] has failed,
> > address
> > is now gated for [5000] ms. Reason is: [Disassociated].
> > 17:36:06,567 INFO
> > org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN
> > GroupReduce (Compute scores) -> FlatMap (checksum()) (370/2322)
> > (cac9927a8568c2ad79439262a91478af) switched from RUNNING to FAILED
> > 17:36:06,572 INFO
> > org.apache.flink.runtime.jobmanager.JobManager - Status of
> > job 14d946015fd7b35eb801ea6fee5af9e4 (Flink Java Job at Thu Oct 29
> 17:34:48
> > UTC 2015) changed to FAILING.
> > java.lang.Exception: The data preparation for task 'CHAIN GroupReduce
> > (Compute scores) -> FlatMap (checksum())' , caused an error: Error
> > obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated
> > due to an exception: Connection unexpectedly closed by remote task
> manager
> > 'ip-10-0-88-140/10.0.88.140:58558'. This might indicate that the remote
> > task manager was lost.
> > at
> > org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:465)
> > at
> > org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
> > Thread 'SortMerger Reading Thread' terminated due to an exception:
> > Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
> > 10.0.88.140:58558'. This might indicate that the remote task manager was
> > lost.
> > at
> >
> >
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
> > at
> >
> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1089)
> > at
> >
> >
> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
> > at
> > org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:459)
> > ... 3 more
> > Caused by: java.io.IOException: Thread 'SortMerger Reading Thread'
> > terminated due to an exception: Connection unexpectedly closed by remote
> > task manager 'ip-10-0-88-140/10.0.88.140:58558'. This might indicate
> that
> > the remote task manager was lost.
> > at
> >
> >
> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
> > Caused by:
> >
> >
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> > Connection unexpectedly closed by remote task manager 'ip-10-0-88-140/
> > 10.0.88.140:58558'. This might indicate that the remote task manager was
> > lost.
> > at
> >
> >
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:119)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
> > at
> >
> >
> io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
> > at
> >
> >
> io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:306)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
> > at
> >
> >
> io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
> > at
> >
> >
> io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
> > at
> >
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
> > at
> >
> >
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
> > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> > at
> >
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> > at java.lang.Thread.run(Thread.java:745)
> > 17:36:06,587 INFO
> > org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN
> > GroupReduce (Compute scores) -> FlatMap (checksum()) (367/2322)
> > (d63c681a18b8164bc24936df1ecb159b) switched from RUNNING to FAILED
> >
> >
> > On Thu, Oct 29, 2015 at 1:00 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi Greg!
> > >
> > > Interesting... When you say the TaskManagers are dropping, are the
> > > TaskManager processes crashing, or are they loosing connection to the
> > > JobManager?
> > >
> > > Greetings,
> > > Stephan
> > >
> > >
> > > On Thu, Oct 29, 2015 at 9:56 AM, Greg Hogan <c...@greghogan.com>
> wrote:
> > >
> > > > I recently discovered that AWS uses NUMA for its largest nodes. An
> > > example
> > > > c4.8xlarge:
> > > >
> > > > $ numactl --hardware
> > > > available: 2 nodes (0-1)
> > > > node 0 cpus: 0 1 2 3 4 5 6 7 8 18 19 20 21 22 23 24 25 26
> > > > node 0 size: 29813 MB
> > > > node 0 free: 24537 MB
> > > > node 1 cpus: 9 10 11 12 13 14 15 16 17 27 28 29 30 31 32 33 34 35
> > > > node 1 size: 30574 MB
> > > > node 1 free: 22757 MB
> > > > node distances:
> > > > node 0 1
> > > > 0: 10 20
> > > > 1: 20 10
> > > >
> > > > I discovered yesterday that Flink performed ~20-30% faster on large
> > > > datasets by running two NUMA-constrained TaskManagers per node. The
> > > > JobManager node ran a single TaskManager. Resources were divided in
> > half
> > > > relative to running a single TaskManager.
> > > >
> > > > The changes from the tail of /bin/taskmanager.sh:
> > > >
> > > > -"${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP taskmanager
> "${args[@]}"
> > > > +numactl --membind=0 --cpunodebind=0
> "${FLINK_BIN_DIR}"/flink-daemon.sh
> > > > $STARTSTOP taskmanager "${args[@]}"
> > > > +numactl --membind=1 --cpunodebind=1
> "${FLINK_BIN_DIR}"/flink-daemon.sh
> > > > $STARTSTOP taskmanager "${args[@]}"
> > > >
> > > > After reverting this change the system is again stable. I had not
> > > > experienced issues using numactl when running 16 nodes.
> > > >
> > > > Greg
> > > >
> > >
> >
>
