Hi, We run flink jobs through yarn on hadoop clusters. One challenge that we are facing is to simplify flink job log access.
The flink job logs can be accessible using "yarn logs $application_id". That approach has a few limitations: 1. It is not straightforward to find yarn application id based on flink job id. 2. It is difficult to find the corresponding container id for the flink sub tasks. 3. For jobs that have many tasks, it is inefficient to use "yarn logs ..." as it mixes logs from all task managers. Any suggestions on the best practice to get logs for completed flink job that run on yarn? Regards, -Yu
