Hi Gyula, Sorry for the late reply. I think it is definitely a challenge in terms of log visibility. However, for your requirement I think you can customize your Flink job by utilizing a customized log formatter/encoder (e.g. log4j.properties or logback.xml) and a suitable logger implementation.
One example you can follow is to provide customFields in your log encoding [1,2] and utilizing a supported Appender to append your log to a file. You can also utilize a more customized appender to log the data into some external database (for example, ElasticSearch and access via Kibana). One challenge you might face is how to configure these contextual information dynamically. In our setup, these contextual information are configured as system env params when job launches. so loggers can dynamically resolve them during start time. Please let me know if any of the suggestions above helps. Cheers, Rong [1] https://github.com/logstash/logstash-logback-encoder/blob/master/src/test/resources/logback-test.xml#L13 [2] https://github.com/logstash/logstash-logback-encoder On Thu, Oct 3, 2019 at 1:56 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > Hi all! > > We have been thinking that it would be a great improvement to add > contextual information to the Flink logs: > > - Container / yarn / host info to JM/TM logs > - Job info (job id/ jobname) to task logs > > I this should be similar to how the metric scopes are set up and should be > able to provide the same information for logs. Ideally it would be user > configurable. > > We are wondering what would be the best way to do this, and would like to > ask for opinions or past experiences. > > Our natural first thought was setting NDC / MDC in the different threads > but it seems to be a somewhat fragile mechanism as it can be easily > "cleared" or deleted by the user. > > What do you think? > > Gyula >
