Hi Gyula, How about putting the appender class in lib folder, but choosing the appender only for user code through log4j.properties? That way only the user logger would be sent to Kafka. It seems what you want...
Thanks, Biao /'bɪ.aʊ/ On Thu, Aug 1, 2019 at 2:25 PM Gyula Fóra <gyula.f...@gmail.com> wrote: > Hi Biao, > > Thanks for the input! > This is basically where we got ourselves as well. > > We are trying to avoid putting things in the lib folder so separating the > loggers would be great but we don't have a solution for it yet. > > Thanks > Gyula > > On Thu, 1 Aug 2019, 07:09 Biao Liu <mmyy1...@gmail.com wrote: > >> Hi Gyula, >> >> I guess it should work if you put log appender jar under $flink/lib >> folder. >> >> There are two different kinds of classloader in your case. One is for >> Flink framework, the other is for user code. The framework classloader is >> the parent of user classloader. The parent classloader could not find class >> from sub-classloader due to the delegation model [1]. In your case, the ` >> ClusterEntrypoint` is in parent classloader, the log appender class in >> fatjar is in sub-classloader. >> >> So I think there are two ways to solve this. >> 1. Put your appender class under $flink/lib folder. >> 2. Try to avoid using user-defined appender for Flink framework class. >> (I'm not sure it could be done or not) >> >> You could find more informations in document [2] about Flink classloading >> strategy. >> >> 1. https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html >> 2. >> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html >> >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <gyula.f...@gmail.com> wrote: >> >>> Hi All! >>> >>> We are trying to configure a custom Kafka log appender for our YARN >>> application and we hit the following problem. >>> >>> We included the log appender dependency in the fatjar of the application >>> because in YARN that should be part of the system class path. >>> >>> However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint, >>> YarnTaskExecutorRunner) these only seem to have the contents of lib on >>> their classpath. Does someone have any pointers to how the classpath is >>> configured for running the entrypoints? >>> >>> The exception is this: >>> >>> java.lang.ClassNotFoundException: >>> org.apache.kafka.log4jappender.KafkaLog4jAppender >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:264) >>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) >>> at >>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) >>> at >>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) >>> at >>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) >>> at >>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) >>> at >>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) >>> at >>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) >>> at >>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) >>> at >>> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) >>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) >>> at >>> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81) >>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329) >>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349) >>> at >>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104) >>> >>> >>> Thanks, >>> Gyula >>> >>> >>>
