If you have set the environment FLINK_CONF_DIR, then it will have a higher priority. I think that could be why you changed the log4j.properties in the conf directory but it does not take effect.
Yes, if you have changed the log4j.properties, you need to relaunch the Flink application. Although we have a ticket to support updating the log level via rest API[1], we do not have much process on this. And I think you could keep an eye on this and share your thoughts. [1]. https://issues.apache.org/jira/browse/FLINK-16478 Best, Yang Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月12日周四 上午5:38写道： > HI Yang, > > I'm able to see taskmanage and jobmanager logs after I changed the > log4j.properties file (/usr/lib/flink/conf). Thank you! > I updated the file as shown below. I had to kill the app ( yarn > application -kill <appid> ) and start flink job again to get the logs. This > doesn't seem like an efficient way. I was wondering if there's a more > simpler way to do it in production. let me know, please! > > *Actual* > log4j.rootLogger=INFO,file > > # Log all infos in the given file > log4j.appender.file=org.apache.log4j.FileAppender > log4j.appender.file.file=${log.file} > log4j.appender.file.append=false > log4j.appender.file.layout=org.apache.log4j.PatternLayout > log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} > %-5p %-60c %x - %m%n > > # suppress the irrelevant (wrong) warnings from the netty channel handler > log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file > > > *modified : *commented the above and added new logging from actual flink > application log4.properties file > > #log4j.rootLogger=INFO,file > > # Log all infos in the given file > #log4j.appender.file=org.apache.log4j.FileAppender > #log4j.appender.file.file=${log.file} > #log4j.appender.file.append=false > #log4j.appender.file.layout=org.apache.log4j.PatternLayout > #log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} > %-5p %-60c %x - %m%n > > # suppress the irrelevant (wrong) warnings from the netty channel handler > #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file > > # This affects logging for both user code and Flink > rootLogger.level = INFO > rootLogger.appenderRef.file.ref = MainAppender > > # Uncomment this if you want to _only_ change Flink's logging > #logger.flink.name = org.apache.flink > #logger.flink.level = INFO > > # The following lines keep the log level of common libraries/connectors on > # log level INFO. The root logger does not override this. You have to > manually > # change the log levels here. > logger.akka.name = akka > logger.akka.level = INFO > logger.kafka.name= org.apache.kafka > logger.kafka.level = INFO > logger.hadoop.name = org.apache.hadoop > logger.hadoop.level = INFO > logger.zookeeper.name = org.apache.zookeeper > logger.zookeeper.level = INFO > > # Log all infos in the given file > appender.main.name = MainAppender > appender.main.type = File > appender.main.append = false > appender.main.fileName = ${sys:log.file} > appender.main.layout.type = PatternLayout > appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - > %m%n > > # Suppress the irrelevant (wrong) warnings from the Netty channel handler > logger.netty.name = > org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline > logger.netty.level = OFF > > > > On Tue, Nov 3, 2020 at 4:56 AM Yang Wang <danrtsey...@gmail.com> wrote: > >> You could issue "ps -ef | grep container_id_for_some_tm". And then you >> will find the >> following java options about log4j. >> >> >> -Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log >> -Dlog4j.configuration=file:./log4j.properties >> -Dlog4j.configurationFile=file:./log4j.properties >> >> Best, >> Yang >> >> Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月2日周一 下午11:37写道： >> >>> Sure. I will check that and get back to you. could you please share how >>> to check java dynamic options? >>> >>> Best, >>> Diwakar >>> >>> On Mon, Nov 2, 2020 at 1:33 AM Yang Wang <danrtsey...@gmail.com> wrote: >>> >>>> If you have already updated the log4j.properties, and it still could >>>> not work, then I >>>> suggest to log in the Yarn NodeManager machine and check the >>>> log4j.properties >>>> in the container workdir is correct. Also you could have a look at the >>>> java dynamic >>>> options are correctly set. >>>> >>>> I think it should work if the log4j.properties and java dynamic options >>>> are set correctly. >>>> >>>> BTW, could you share the new yarn logs? >>>> >>>> Best, >>>> Yang >>>> >>>> Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月2日周一 下午4:32写道： >>>> >>>>> >>>>> >>>>> Hi Yang, >>>>> >>>>> Thank you so much for taking a look at the log files. I changed my >>>>> log4j.properties. Below is the actual file that I got from EMR 6.1.0 >>>>> distribution of flink 1.11. I observed that it is different from Flink >>>>> 1.11 >>>>> that i downloaded so i changed it. Still I didn't see any logs. >>>>> >>>>> *Actual* >>>>> log4j.rootLogger=INFO,file >>>>> >>>>> # Log all infos in the given file >>>>> log4j.appender.file=org.apache.log4j.FileAppender >>>>> log4j.appender.file.file=${log.file} >>>>> log4j.appender.file.append=false >>>>> log4j.appender.file.layout=org.apache.log4j.PatternLayout >>>>> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd >>>>> HH:mm:ss,SSS} %-5p %-60c %x - %m%n >>>>> >>>>> # suppress the irrelevant (wrong) warnings from the netty channel >>>>> handler >>>>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file >>>>> >>>>> >>>>> *modified : *commented the above and added new logging from >>>>> actual flink application log4.properties file >>>>> >>>>> #log4j.rootLogger=INFO,file >>>>> >>>>> # Log all infos in the given file >>>>> #log4j.appender.file=org.apache.log4j.FileAppender >>>>> #log4j.appender.file.file=${log.file} >>>>> #log4j.appender.file.append=false >>>>> #log4j.appender.file.layout=org.apache.log4j.PatternLayout >>>>> #log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd >>>>> HH:mm:ss,SSS} %-5p %-60c %x - %m%n >>>>> >>>>> # suppress the irrelevant (wrong) warnings from the netty channel >>>>> handler >>>>> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file >>>>> >>>>> # This affects logging for both user code and Flink >>>>> rootLogger.level = INFO >>>>> rootLogger.appenderRef.file.ref = MainAppender >>>>> >>>>> # Uncomment this if you want to _only_ change Flink's logging >>>>> #logger.flink.name = org.apache.flink >>>>> #logger.flink.level = INFO >>>>> >>>>> # The following lines keep the log level of common >>>>> libraries/connectors on >>>>> # log level INFO. The root logger does not override this. You have to >>>>> manually >>>>> # change the log levels here. >>>>> logger.akka.name = akka >>>>> logger.akka.level = INFO >>>>> logger.kafka.name= org.apache.kafka >>>>> logger.kafka.level = INFO >>>>> logger.hadoop.name = org.apache.hadoop >>>>> logger.hadoop.level = INFO >>>>> logger.zookeeper.name = org.apache.zookeeper >>>>> logger.zookeeper.level = INFO >>>>> >>>>> # Log all infos in the given file >>>>> appender.main.name = MainAppender >>>>> appender.main.type = File >>>>> appender.main.append = false >>>>> appender.main.fileName = ${sys:log.file} >>>>> appender.main.layout.type = PatternLayout >>>>> appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c >>>>> %x - %m%n >>>>> >>>>> # Suppress the irrelevant (wrong) warnings from the Netty channel >>>>> handler >>>>> logger.netty.name = >>>>> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline >>>>> logger.netty.level = OFF >>>>> >>>>> ********************************** >>>>> I also think its related to the log4j setting but I'm not able to >>>>> figure it out. >>>>> Please let me know if you want any other log files or configuration. >>>>> >>>>> Thanks. >>>>> >>>>> On Sun, Nov 1, 2020 at 10:06 PM Yang Wang <danrtsey...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Diwakar Jha, >>>>>> >>>>>> From the logs you have provided, everything seems working as >>>>>> expected. The JobManager and TaskManager >>>>>> java processes have been started with correct dynamic options, >>>>>> especially for the logging. >>>>>> >>>>>> Could you share the content of $FLINK_HOME/conf/log4j.properties? I >>>>>> think there's something wrong with the >>>>>> log4j config file. For example, it is a log4j1 format. But we are >>>>>> using log4j2 in Flink 1.11. >>>>>> >>>>>> >>>>>> Best, >>>>>> Yang >>>>>> >>>>>> Diwakar Jha <diwakar.n...@gmail.com> 于2020年11月2日周一 上午1:57写道： >>>>>> >>>>>>> Hi >>>>>>> I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running >>>>>>> fine but i'm not seeing any taskmanager/jobmanager logs. >>>>>>> I see the below error in stdout. >>>>>>> 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR >>>>>>> org.apache.flink.runtime.rest.handler.taskmanager. >>>>>>> TaskManagerLogFileHandler - Failed to transfer file from >>>>>>> TaskExecutor container_1604033334508_0001_01_000004. >>>>>>> java.util.concurrent.CompletionException: org.apache.flink.util. >>>>>>> FlinkException: The file LOG does not exist on the TaskExecutor. >>>>>>> >>>>>>> I'm stuck at this step for a couple of days now and not able to >>>>>>> migrate to Flink 1.11. I would appreciate it if anyone can help me. >>>>>>> i have the following setup : >>>>>>> a) i'm deploying flink using yarn. I have attached yarn application >>>>>>> id logs. >>>>>>> c) stsd setup >>>>>>> >>>>>>> metrics.reporters: stsd >>>>>>> metrics.reporter.stsd.factory.class: >>>>>>> org.apache.flink.metrics.statsd.StatsDReporterFactory >>>>>>> metrics.reporter.stsd.host: localhost >>>>>>> metrics.reporter.stsd.port: 8125 >>>>>>> >>>>>>>
