[
https://issues.apache.org/jira/browse/FLINK-34576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823864#comment-17823864
]
chenyuzhi commented on FLINK-34576:
-----------------------------------
I have check our operaotr's log， got some error log：
{code:java}
2024-03-05 04:35:55,147 ERROR
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher
[gdc-gdc-sa/logstream-grand-grand-dep456-g96-production] - Error during event
processing ExecutionScope{ resource id:
ResourceID{name='logstream-grand-grand-dep456-g96-production',
namespace='gdc-gdc-sa'}, version: 3628981709} failed.
org.apache.flink.kubernetes.operator.exception.ReconciliationException:
org.apache.flink.kubernetes.operator.exception.StatusConflictException: Status
have been modified externally in version 3628984691 Previous:
{"jobStatus":{"jobName":"logstream-dep456-grand-grand_dep456_g96","jobId":"706cae800e89eb15911760c173a40c05","state":"RECONCILING","startTime":"1709203055328","updateTime":"1709203103773","savepointInfo":{"lastSavepoint":{"timeStamp":1709203025670,"location":"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340","triggerType":"UNKNOWN","formatType":"UNKNOWN","triggerNonce":null},"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0}},"error":"{\"type\":\"org.apache.flink.kubernetes.operator.exception.ReconciliationException\",\"message\":\"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException: Failed to load
configuration\",\"additionalMetadata\":{},\"throwableList\":[{\"type\":\"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException\",\"message\":\"java.lang.RuntimeException:
Failed to load
configuration\",\"additionalMetadata\":{}},{\"type\":\"java.lang.RuntimeException\",\"message\":\"Failed
to load
configuration\",\"additionalMetadata\":{}}]}","lifecycleState":"STABLE","clusterInfo":{"flink-revision":"b6d20ed
@
2023-12-20T10:01:39+01:00","flink-version":"1.14.0-GDC1.6.0","total-cpu":"1.6","total-memory":"4294967296"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1709203026998,"lastReconciledSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":{"labelSelector":"component=taskmanager,app=logstream-grand-grand-dep456-g96-production","replicas":1}}
Latest:
{"jobStatus":{"jobName":"logstream-dep456-grand-grand_dep456_g96","jobId":"706cae800e89eb15911760c173a40c05","state":"RECONCILING","startTime":"1709203055328","updateTime":"1709203103773","savepointInfo":{"lastSavepoint":{"timeStamp":1709203025670,"location":"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340","triggerType":"UNKNOWN","formatType":"UNKNOWN","triggerNonce":null},"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"flink-revision":"b6d20ed
@
2023-12-20T10:01:39+01:00","flink-version":"1.14.0-GDC1.6.0","total-cpu":"1.6","total-memory":"4294967296"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1709203026998,"lastReconciledSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":{"labelSelector":"component=taskmanager,app=logstream-grand-grand-dep456-g96-production","replicas":1}}
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)
[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by:
org.apache.flink.kubernetes.operator.exception.StatusConflictException: Status
have been modified externally in version 3628984691 Previous:
{"jobStatus":{"jobName":"logstream-dep456-grand-grand_dep456_g96","jobId":"706cae800e89eb15911760c173a40c05","state":"RECONCILING","startTime":"1709203055328","updateTime":"1709203103773","savepointInfo":{"lastSavepoint":{"timeStamp":1709203025670,"location":"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340","triggerType":"UNKNOWN","formatType":"UNKNOWN","triggerNonce":null},"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0}},"error":"{\"type\":\"org.apache.flink.kubernetes.operator.exception.ReconciliationException\",\"message\":\"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException: Failed to load
configuration\",\"additionalMetadata\":{},\"throwableList\":[{\"type\":\"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException\",\"message\":\"java.lang.RuntimeException:
Failed to load
configuration\",\"additionalMetadata\":{}},{\"type\":\"java.lang.RuntimeException\",\"message\":\"Failed
to load
configuration\",\"additionalMetadata\":{}}]}","lifecycleState":"STABLE","clusterInfo":{"flink-revision":"b6d20ed
@
2023-12-20T10:01:39+01:00","flink-version":"1.14.0-GDC1.6.0","total-cpu":"1.6","total-memory":"4294967296"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1709203026998,"lastReconciledSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":{"labelSelector":"component=taskmanager,app=logstream-grand-grand-dep456-g96-production","replicas":1}}
Latest:
{"jobStatus":{"jobName":"logstream-dep456-grand-grand_dep456_g96","jobId":"706cae800e89eb15911760c173a40c05","state":"RECONCILING","startTime":"1709203055328","updateTime":"1709203103773","savepointInfo":{"lastSavepoint":{"timeStamp":1709203025670,"location":"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340","triggerType":"UNKNOWN","formatType":"UNKNOWN","triggerNonce":null},"triggerId":null,"triggerTimestamp":null,"triggerType":null,"formatType":null,"savepointHistory":[],"lastPeriodicSavepointTimestamp":0}},"error":null,"lifecycleState":"STABLE","clusterInfo":{"flink-revision":"b6d20ed
@
2023-12-20T10:01:39+01:00","flink-version":"1.14.0-GDC1.6.0","total-cpu":"1.6","total-memory":"4294967296"},"jobManagerDeploymentStatus":"READY","reconciliationStatus":{"reconciliationTimestamp":1709203026998,"lastReconciledSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","lastStableSpec":"{\"spec\":{\"job\":{\"jarURI\":\"local:///opt/flink/usrlib/usercode.jar\",\"parallelism\":1,\"entryClass\":\"com.netease.gdc.streaming.serverdump.Main\",\"args\":[\"-id\",\"grand-grand_dep456_g96\",\"-server\",\"http://logstreamapi-in.nie.netease.com:9099/\",\"-auth_key\",\"3e0e9294802f4b868dc89e8f32ae43ab\",\"-auth_user\",\"_loghub\",\"-auth_project\",\"dep456\",\"-type\",\"logstream\",\"-meta_api\",\"https://meta.gdc.nie.netease.com/\",\"-datahub_api\",\"http://datahub-\"],\"state\":\"running\",\"savepointTriggerNonce\":1,\"initialSavepointPath\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947/savepoint-0c18ba-008f65afd340\",\"upgradeMode\":\"savepoint\",\"allowNonRestoredState\":true},\"restartNonce\":null,\"flinkConfiguration\":{\"containerized.master.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.master.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.master.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.master.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.master.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.JAVA_LIBRARY_PATH\":\"$JAVA_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_APP_NAME\":\"logstream-grand-grand_dep456_g96-production\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_DEPARTMENT\":\"sa\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_LAMBDA_ID\":\"659fa2ec5ce541a5e2d68947\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG_PROJECT\":\"gdc\",\"containerized.taskmanager.env.LAMBDA_METRICS_TAG__share_project_\":\"gdc\",\"containerized.taskmanager.env.LD_LIBRARY_PATH\":\"$LD_LIBRARY_PATH:/home/hadoop/hadoop/lib/native\",\"env.hadoop.conf.dir\":\"/home/hadoop/gdcconf/hadoop/mogra\",\"env.java.opts\":\"-Duser.timezone=GMT+08
-XX:+UseG1GC -Xloggc:log/gc.log -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2
-XX:GCLogFileSize=10M\",\"execution.checkpointing.externalized-checkpoint-retention\":\"RETAIN_ON_CANCELLATION\",\"execution.checkpointing.interval\":\"30s\",\"execution.checkpointing.tolerable-failed-checkpoints\":\"3\",\"execution.shutdown-on-application-finish\":\"false\",\"high-availability\":\"zookeeper\",\"high-availability.kubernetes.leader-election.lease-duration\":\"60
s\",\"high-availability.kubernetes.leader-election.renew-deadline\":\"60
s\",\"high-availability.kubernetes.leader-election.retry-period\":\"30
s\",\"high-availability.storageDir\":\"hdfs://mogra/flink/gdc_sa/ha/\",\"high-availability.zookeeper.path.root\":\"/flink\",\"high-availability.zookeeper.quorum\":\"mogra2.realtime.zk.gdc.x.netease.com:2181\",\"io.tmp.dirs\":\"/tmp/flink-io\",\"jobmanager.archive.fs.dir\":\"hdfs://mogra/flink/gdc_sa/completed-jobs\",\"jobmanager.retrieve-taskmanager-hostname\":\"false\",\"metrics.latency.interval\":\"10000\",\"metrics.reporter.jmx.factory.class\":\"org.apache.flink.metrics.jmx.JMXReporterFactory\",\"metrics.reporter.jmx.port\":\"8800-8880\",\"metrics.reporter.prom.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporter\",\"metrics.reporter.prom.factory.class\":\"org.apache.flink.metrics.prometheus.PrometheusReporterFactory\",\"metrics.reporter.prom.factory.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.factory.port\":\"9900-10099\",\"metrics.reporter.prom.filterLabelValueCharacters\":\"false\",\"metrics.reporter.prom.port\":\"9900-10099\",\"rest.flamegraph.enabled\":\"true\",\"restart-strategy\":\"failure-rate\",\"restart-strategy.failure-rate.delay\":\"1
min\",\"restart-strategy.failure-rate.failure-rate-interval\":\"10
min\",\"restart-strategy.failure-rate.max-failures-per-interval\":\"3\",\"security.kerberos.login.keytab\":\"/opt/scheduler/keytab/gdc_sa.keytab\",\"security.kerberos.login.principal\":\"gdc_sa/scheduler\",\"state.checkpoints.dir\":\"hdfs://mogra/flink/gdc_sa/checkpoints/659fa2ec5ce541a5e2d68947\",\"state.savepoints.dir\":\"hdfs://mogra/flink/gdc_sa/savepoints/659fa2ec5ce541a5e2d68947\",\"taskmanager.numberOfTaskSlots\":\"1\",\"taskmanager.slot.timeout\":\"60
s\",\"web.cancel.enable\":\"true\"},\"image\":\"ncr.nie.netease.com/v1-gdcstreaming/streamfly-job-659fa2ec5ce541a5e2d68947:latest\",\"imagePullPolicy\":null,\"serviceAccount\":\"streamfly\",\"flinkVersion\":\"v1_14\",\"ingress\":{\"template\":\"/{{namespace}}/{{name}}(/|$)(.*)\",\"className\":\"nginx\",\"annotations\":{\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\"}},\"podTemplate\":{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"clusterSelector\":\"{\\\"affinity\\\":{\\\"nodeAffinity\\\":{\\\"preferredDuringSchedulingIgnoredDuringExecution\\\":
[{\\\"weight\\\": 2, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"hdd\\\",\\\"operator\\\":\\\"Exists\\\"}]}},{\\\"weight\\\":
1, \\\"preference\\\":
{\\\"matchExpressions\\\":[{\\\"key\\\":\\\"ssd\\\",\\\"operator\\\":\\\"Exists\\\"}]}}]}}}\",\"gdc-logs\":\"{\\\"server-log-gdc-flink\\\":{\\\"type\\\":\\\"gdc\\\",\\\"size\\\":\\\"10\\\",\\\"logType\\\":\\\"server\\\"}}\",\"initializer.kubernetes.io/gdc-logservice\":\"true\",\"streamfly.nie.netease.com/lambda_biztype\":\"logstream\",\"streamfly.nie.netease.com/lambda_department\":\"sa\",\"streamfly.nie.netease.com/lambda_id\":\"659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_jm_cores\":\"\\\\\\\"0.8\\\\\\\"\",\"streamfly.nie.netease.com/lambda_job_url\":\"https://streamfly.nie.netease.com/_gdc_sa/lambda/jobs/659fa2ec5ce541a5e2d68947\",\"streamfly.nie.netease.com/lambda_name\":\"logstream-grand-grand_dep456_g96-production\",\"streamfly.nie.netease.com/lambda_owner\":\"_loghub\",\"streamfly.nie.netease.com/lambda_parallelism\":\"\\\\\\\"1\\\\\\\"\",\"streamfly.nie.netease.com/lambda_project\":\"gdc\",\"streamfly.nie.netease.com/lambda_tags\":\"_cost_dep456_bu\",\"streamfly.nie.netease.com/lambda_tm_cores\":\"\\\\\\\"0.8\\\\\\\"\"},\"labels\":{\"app.kubernetes.io/instance\":\"659fa2ec5ce541a5e2d68947\",\"app.kubernetes.io/managed-by\":\"streamfly\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"node-role.kubernetes.io/vk_streamfly\",\"operator\":\"Exists\"}]}]}}},\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"FLINK_POD_ID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}},{\"name\":\"FLINK_POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"FLINK_LOG_DIR\",\"value\":\"/flink-data/gdc-sa/logstream/659fa2ec5ce541a5e2d68947/$(FLINK_POD_IP)/$(FLINK_POD_ID)\"},{\"name\":\"AUTH_PROJECT\",\"value\":\"gdc\"},{\"name\":\"AUTH_DEPT\",\"value\":\"sa\"},{\"name\":\"LAMBDA_JOB_NAME\",\"value\":\"logstream-grand-grand_dep456_g96-production\"},{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"k596.elk.x.netease.com\"}],\"imagePullPolicy\":\"Always\",\"name\":\"flink-main-container\",\"ports\":[{\"containerPort\":9900,\"name\":\"flink-metric\",\"protocol\":\"TCP\"}],\"resources\":{\"limits\":{\"ephemeral-storage\":\"20Gi\"},\"requests\":{\"ephemeral-storage\":\"10Gi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"SYS_PTRACE\"]}},\"volumeMounts\":[{\"mountPath\":\"/flink-data\",\"name\":\"server-log-gdc-flink\"}]}],\"imagePullSecrets\":[{\"name\":\"ncr-pull-secret\"}],\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"virtual-kubelet.io/provider\",\"value\":\"whale-virtual-kubelet\"}],\"volumes\":[{\"emptyDir\":{},\"name\":\"server-log-gdc-flink\"}]}},\"jobManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":1,\"podTemplate\":null},\"taskManager\":{\"resource\":{\"cpu\":0.8,\"memory\":\"2048m\",\"ephemeralStorage\":null},\"replicas\":null,\"podTemplate\":null},\"logConfiguration\":null,\"mode\":null},\"resource_metadata\":{\"apiVersion\":\"flink.apache.org/v1beta1\",\"metadata\":{\"generation\":2},\"firstDeployment\":true}}","state":"DEPLOYED"},"taskManager":{"labelSelector":"component=taskmanager,app=logstream-grand-grand-dep456-g96-production","replicas":1}}
at
org.apache.flink.kubernetes.operator.utils.StatusRecorder.replaceStatus(StatusRecorder.java:164)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
org.apache.flink.kubernetes.operator.utils.StatusRecorder.patchAndCacheStatus(StatusRecorder.java:101)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:135)
~[flink-kubernetes-operator-1.6.1-shaded.jar:1.6.1]
... 13 more {code}
It was logged in the newLeader after leader switching.
> Flink deployment keep staying at RECONCILING/STABLE status
> ----------------------------------------------------------
>
> Key: FLINK-34576
> URL: https://issues.apache.org/jira/browse/FLINK-34576
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.6.1
> Reporter: chenyuzhi
> Priority: Major
> Attachments: image-2024-03-05-15-13-11-032.png
>
>
> The HA mode of flink-kubernetes-operator is being used. When one of the pods
> of flink-kubernetes-operator restarts, flink-kubernetes-operator switches the
> leader. However, some flinkdeployments have been in the
> *JOB_STATUS=RECONCILING&LIFECYCLE_STATE=STABLE* state for a long time.
> Through the cmd "kubectl describe flinkdeployment xxx", can see the following
> error, but there are no exceptions in the flink-kubernetes-operator log.
>
> {code:java}
> Status:
> Cluster Info:
> Flink - Revision: b6d20ed @ 2023-12-20T10:01:39+01:00
> Flink - Version: 1.14.0-GDC1.6.0
> Total - Cpu: 7.0
> Total - Memory: 30064771072
> Error:
> {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Failed to load
> configuration","additionalMetadata":{},"throwableList":[{"type":"org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException","message":"java.lang.RuntimeException:
> Failed to load
> configuration","additionalMetadata":{}},{"type":"java.lang.RuntimeException","message":"Failed
> to load configuration","additionalMetadata":{}}]}
> Job Manager Deployment Status: READY
> Job Status:
> Job Id: cf44b5e73a1f263dd7d9f2c82be5216d
> Job Name: noah_stream_studio_1754211682_2218100380
> Savepoint Info:
> Last Periodic Savepoint Timestamp: 0
> Savepoint History:
> Start Time: 1705635107137
> State: RECONCILING
> Update Time: 1709272530741
> Lifecycle State: STABLE {code}
>
> !image-2024-03-05-15-13-11-032.png!
>
> version：
> flink-kubernetes-operator: 1.6.1
> flink: 1.14.0/1.15.2 (flinkdeployment 1200+)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
