liang yu created FLINK-36540:
--------------------------------
Summary: Add Support for Hadoop Caller Context when using Flink on
yarn
Key: FLINK-36540
URL: https://issues.apache.org/jira/browse/FLINK-36540
Project: Flink
Issue Type: Improvement
Components: Runtime / Task
Environment: Hadoop 3.3.4
Flink 1.14.4
Reporter: liang yu
*Description:* As HDFS mentioned in HDFS-9184, for a given HDFS operation, it's
very helpful to track which upper level job issues it. But now Flink does not
support to set callerContext when trying to operate on Hadoop fileSystem.
*implementation:* Add a new HadoopOption which can be set in flink.yaml. By
setting this option to true, flink can search for the classpath if we can
invoke the method
{code:java}
callerContext.Builder().build().setCurrent(context){code}
and then set the jobId and TaskName with subtask id so that we can find
exactly who did the operation to Hadoop file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
