2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) - execution #0 is
assigned resource container_1554366508934_0084_01_000004_3 with
d5363785f9f7f1965851ba54de935e7d
2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (3/4) - execution #0 is assigned resource
container_1554366508934_0084_01_000004_1 with 1f1217c617a27211157883e437a6bd6e
2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (2/4) - execution #0 is assigned resource
container_1554366508934_0084_01_000004_4 with cd351063b4a86182ad17b7b025541bb4
2019-04-09 09:58:03.013 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) - execution #0 is
assigned resource container_1554366508934_0084_01_000004_5 with
e6742f700171d110fe27258bf77a789f
2019-04-09 09:58:03.013 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (4/4) - execution #0 is assigned resource
container_1554366508934_0084_01_000004_2 with 408ccaad63574e30b5fe622b87822fc4
2019-04-09 09:58:03.013 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (1/4) (e972cd212cc3845b114deb5b86a229b6) switched from
SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.015 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source:
KafkaJsonTableSource (1/4) (attempt #0) to slot
container_1554366508934_0084_01_000004_0 on bd129120
2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (2/4) (96a8bf35b6a85502b2b1afa400c2f0bd) switched from
SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source:
KafkaJsonTableSource (2/4) (attempt #0) to slot
container_1554366508934_0084_01_000004_4 on bd129120
2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (3/4) (1f69f483771696ca5352edb9e172de6f) switched from
SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source:
KafkaJsonTableSource (3/4) (attempt #0) to slot
container_1554366508934_0084_01_000004_1 on bd129120
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (4/4) (8c5bfc27c50c37587d5b4439ab3a4a26) switched from
SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source:
KafkaJsonTableSource (4/4) (attempt #0) to slot
container_1554366508934_0084_01_000004_2 on bd129120
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2)
(482b5c9e8f24afcac81e95372b1c02d3) switched from SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) (attempt #0) to
slot container_1554366508934_0084_01_000004_5 on bd129120
2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2)
(4c0abbf6e5dc19b1bd51e650364a2c26) switched from SCHEDULED to DEPLOYING.
2019-04-09 09:58:03.018 [jobmanager-future-thread-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) (attempt #0) to
slot container_1554366508934_0084_01_000004_3 on bd129120
2019-04-09 09:58:04.081 [flink-akka.actor.default-dispatcher-27] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2)
(482b5c9e8f24afcac81e95372b1c02d3) switched from DEPLOYING to RUNNING.
2019-04-09 09:58:04.086 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
SourceConversion(table:[builtin, default, sourceStreamTable, source:
[KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI,
EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name],
schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id,
prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on:
(TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id,
DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink:
JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2)
(4c0abbf6e5dc19b1bd51e650364a2c26) switched from DEPLOYING to RUNNING.
2019-04-09 09:58:04.087 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (2/4) (96a8bf35b6a85502b2b1afa400c2f0bd) switched from
DEPLOYING to RUNNING.
2019-04-09 09:58:04.089 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (3/4) (1f69f483771696ca5352edb9e172de6f) switched from
DEPLOYING to RUNNING.
2019-04-09 09:58:04.091 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (1/4) (e972cd212cc3845b114deb5b86a229b6) switched from
DEPLOYING to RUNNING.
2019-04-09 09:58:04.092 [flink-akka.actor.default-dispatcher-3] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
KafkaJsonTableSource (4/4) (8c5bfc27c50c37587d5b4439ab3a4a26) switched from
DEPLOYING to RUNNING.
2019-04-09 09:58:07.508 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000003 finished with exit code 31
2019-04-09 09:58:07.508 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 2.
2019-04-09 09:58:07.509 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000005 finished with exit code 31
2019-04-09 09:58:07.509 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 3.
2019-04-09 09:58:08.012 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000006 - Remaining pending container requests: 2
2019-04-09 09:58:08.013 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000006 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129139:8042
2019-04-09 09:58:08.015 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:08.047 [pool-1-thread-4] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:08.157 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:08.158 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:08.518 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000007 - Remaining pending container requests: 1
2019-04-09 09:58:08.519 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000007 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129144:8042
2019-04-09 09:58:08.520 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000009 - Remaining pending container requests: 0
2019-04-09 09:58:08.520 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:08.521 [pool-1-thread-1] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000009 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129136:8042
2019-04-09 09:58:08.523 [pool-1-thread-1] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:08.555 [pool-1-thread-1] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:08.555 [pool-1-thread-3] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:08.661 [pool-1-thread-1] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:08.661 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:08.662 [pool-1-thread-1] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:08.662 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:14.023 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000006 finished with exit code 31
2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 1.
2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000007 finished with exit code 31
2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 2.
2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000009 finished with exit code 31
2019-04-09 09:58:14.025 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 3.
2019-04-09 09:58:19.533 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000014 - Remaining pending container requests: 2
2019-04-09 09:58:19.534 [pool-1-thread-2] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000014 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129139:8042
2019-04-09 09:58:19.534 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000015 - Remaining pending container requests: 1
2019-04-09 09:58:19.535 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000016 - Remaining pending container requests: 0
2019-04-09 09:58:19.535 [pool-1-thread-2] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:19.535 [pool-1-thread-5] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000015 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129144:8042
2019-04-09 09:58:19.535 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000016 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129136:8042
2019-04-09 09:58:19.536 [pool-1-thread-5] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:19.536 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:19.565 [pool-1-thread-4] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:19.565 [pool-1-thread-5] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:19.565 [pool-1-thread-2] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:19.665 [pool-1-thread-2] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:19.665 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:19.665 [pool-1-thread-5] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:19.665 [pool-1-thread-2] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:19.665 [pool-1-thread-4] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:19.665 [pool-1-thread-5] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:25.037 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000014 finished with exit code 31
2019-04-09 09:58:25.038 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 1.
2019-04-09 09:58:25.038 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000015 finished with exit code 31
2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 2.
2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Container
container_1554366508934_0084_01_000016 finished with exit code 31
2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container
with resources <memory:2048, vCores:6>. Number pending requests 3.
2019-04-09 09:58:30.547 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000017 - Remaining pending container requests: 2
2019-04-09 09:58:30.548 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000018 - Remaining pending container requests: 1
2019-04-09 09:58:30.548 [AMRM Callback Handler Thread] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Received new container:
container_1554366508934_0084_01_000019 - Remaining pending container requests: 0
2019-04-09 09:58:30.548 [pool-1-thread-6] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000017 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129139:8042
2019-04-09 09:58:30.549 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000019 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129136:8042
2019-04-09 09:58:30.548 [pool-1-thread-7] INFO
org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor
container_1554366508934_0084_01_000018 will be started with container size 2048
MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit
128 MB on cdh129144:8042
2019-04-09 09:58:30.549 [pool-1-thread-6] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:30.550 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:30.550 [pool-1-thread-7] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab
to the AM container local resource bucket
2019-04-09 09:58:30.575 [pool-1-thread-3] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:30.576 [pool-1-thread-6] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:30.576 [pool-1-thread-7] INFO org.apache.flink.yarn.Utils -
Use the beforehand copied resource
hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml
(the corresponding local path:
file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml).
Visibility: APPLICATION.
2019-04-09 09:58:30.661 [pool-1-thread-6] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:30.661 [pool-1-thread-7] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:30.661 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch
context for TaskManagers
2019-04-09 09:58:30.662 [pool-1-thread-6] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:30.662 [pool-1-thread-7] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
2019-04-09 09:58:30.662 [pool-1-thread-3] INFO
org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
________________________________
[email protected]
发件人： 苏 欣<mailto:[email protected]>
发送时间： 2019-04-09 10:36
收件人： user-zh<mailto:[email protected]>
主题： 回复: 答复: blink提交yarn卡在一直重复分配container
________________________________
[email protected]
发件人： 苏 欣<mailto:[email protected]>
发送时间： 2019-04-09 10:30
收件人： [email protected]<mailto:[email protected]>
主题： 答复: blink提交yarn卡在一直重复分配container
不好意思，已补充yarn的日志文件。
出现问题的原因我已经找到了，在配置flink-conf.yaml中的下面三项后，会出现分配不了资源的问题
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /home/hive.keytab
security.kerberos.login.principal: hive/cdh129135@MYCDH
如果在客户机使用kinit命令后再提交，yarn资源可以正常分配。
现在我有几个问题请教大佬们：
1、 提交作业到带有kerberos认证的yarn，除了kinit方式之外还有其他方式吗，为什么读配置文件中的票据会出现code 31？
2、 taskmanager.cpu.core与slot数量在yarn上面他们是相等的吗？有没有一个core分配多个slot的情况？
发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用
发件人: Zili Chen<mailto:[email protected]>
发送时间: 2019年4月8日 19:29
收件人: [email protected]<mailto:[email protected]>
主题: Re: blink提交yarn卡在一直重复分配container
你好，apache 的邮件列表不支持内嵌图片，请以附件或链接方式引用。
Best,
tison.
苏 欣 <[email protected]> 于2019年4月8日周一 上午10:17写道：
> 我以per-job方式提交了一个作业到yarn上面，发现会出现不断重复分配container的现象。
>
> 现象为从yarn的web ui上看一瞬间tm的container分配成功了，但是立刻变为只剩一个jm的container，接着会继续分配tm的
> container。不断的重复这个过程直到作业调度不到资源而失败。
>
> 我查了一下exit code没找到31代表是什么意思，有没有大佬帮忙分析下，非常感谢！
>
>
>
> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>
>
>
