It is finding the file though, the problem is that the lib/ might not be on
the classpath when the file is being parsed, thus the YAML file is not
recognized as being parsable.
Is there a way to differ the inference from BashJavaUtils to the actual
bootstrap of the app? or perhaps add the lib to the utils classpath?
On Thu, Jul 29, 2021, 17:55 Chesnay Schepler <ches...@apache.org> wrote:
> That could be...could you try configuring "env.java.opts:
> -Dlog4j.configurationFile=..."?
>
> On 29/07/2021 15:18, Yuval Itzchakov wrote:
>
> Perhaps because I'm passing it as an argument to the job with
> -Dlog4j.configurationFile=... ?
>
> On Thu, Jul 29, 2021 at 3:27 PM Chesnay Schepler <ches...@apache.org>
> wrote:
>
>> Looking once more at the stacktrace this happens in the BashJavaUtils,
>> which only puts flink-dist and not the entire lib/ directory on the
>> classpath.
>> What I don't understand is why it would pick up your log4j file. It
>> should only use the file that is embedded within BashJavaUtils.jar.
>>
>> On 29/07/2021 13:11, Yuval Itzchakov wrote:
>>
>> Hi Chesnay,
>>
>> So I looked into it, and jackson-databind is definitely in my UBER jar as
>> a transitive dependency. Additionally, this uber JAR is added to the `lib/`
>> directly for Flink before I start the job, yet it still fails.
>> Are there any other paths I should explore?
>>
>> One thing to add is that I'm using the "classloader.resolve-order" =
>> "parent-first" flag due to my lib having some deps that otherwise cause
>> runtime collision errors in Kafka.
>>
>>
>> On Wed, Jul 7, 2021 at 12:25 PM Yuval Itzchakov <yuva...@gmail.com>
>> wrote:
>>
>>> Interesting, I don't have bind explicitly on the classpath, will give it
>>> a try.
>>>
>>> Although locally it is working properly.
>>>
>>> On Wed, Jul 7, 2021, 12:19 Chesnay Schepler <ches...@apache.org> wrote:
>>>
>>>> According to the log4j documentation you need both jackson-databind and
>>>> jackson-dataformat-yaml to be on the classpath.
>>>> Did you make sure that that this is the case in 1.13.1? It could very
>>>> well be that in 1.9 databind was on the classpath by chance.
>>>>
>>>> On 07/07/2021 10:22, Arvid Heise wrote:
>>>>
>>>> Hi Yuval,
>>>>
>>>> For some reason the YamlConfigurationFactory is not correctly loaded
>>>> and the fallback XmlConfigurationFactory is used unsuccessfully.
>>>> You could try to force DEBUG on that YamlConfigurationFactory and check
>>>> for some output like
>>>> "Missing dependencies for Yaml support, ConfigurationFactory {} is
>>>> inactive"
>>>>
>>>> In particular, it checks for these classes
>>>> "com.fasterxml.jackson.databind.ObjectMapper",
>>>> "com.fasterxml.jackson.databind.JsonNode",
>>>> "com.fasterxml.jackson.core.JsonParser",
>>>> "com.fasterxml.jackson.dataformat.yaml.YAMLFactory"
>>>>
>>>> I'm assuming that with improved classloader separation, you actually
>>>> need to put your dependency into `lib/` instead of putting it into your
>>>> user jar. But I'm pulling in @Chesnay Schepler <ches...@apache.org>
>>>> who has much more insights.
>>>>
>>>> On Sun, Jul 4, 2021 at 9:45 PM Yuval Itzchakov <yuva...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am attempting to upgrade Flink from 1.9 to 1.13.1
>>>>> I am using a YAML based log4j file. In 1.9, it worked perfectly fine
>>>>> by adding the following dependency to the classpath (I deploy with an uber
>>>>> JAR):
>>>>>
>>>>> "com.fasterxml.jackson.dataformat" % "jackson-dataformat-yaml"
>>>>> % "2.12.3"
>>>>>
>>>>> However, with Flink 1.13.1 I get the following error at runtime (only
>>>>> when running in a Cluster, running locally works fine):
>>>>>
>>>>>
>>>>>
>>>>> *flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager [Fatal Error] log4j2.yaml:1:1: Content is not allowed in
>>>>> prolog. flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager ERROR StatusLogger Error parsing /opt/foo/conf/log4j2.yaml
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager
>>>>> org.xml.sax.SAXParseException; systemId:
>>>>> file:///opt/foo/conf/log4j2.yaml;
>>>>> lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.*
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> java.xml/com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:261)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> java.xml/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:458)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:385)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:293)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:647)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.LogManager.getContext(LogManager.java:155)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager at
>>>>> org.apache.flink.runtime.util.bash.BashJavaUtils.<clinit>(BashJavaUtils.java:45)
>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>> taskmanager ERROR StatusLogger No logging configuration
>>>>>
>>>>> This indicates that for some reason, the jackson dataformat YAML
>>>>> library is not getting properly loaded from my uber JAR at runtime.
>>>>>
>>>>> Has anyone run into this? Any possible workarounds?
>>>>> --
>>>>> Best Regards,
>>>>> Yuval Itzchakov.
>>>>>
>>>>
>>>>
>>
>> --
>> Best Regards,
>> Yuval Itzchakov.
>>
>>
>>
>
> --
> Best Regards,
> Yuval Itzchakov.
>
>
>
