I am new to Flink and the days I used java was long ago. I am trying to write
a Flink app that utilizes Table API with a source table used the MySQL CDC
connector 3.3.0 to talk to a Postgres table using JDBC driver. I am using
Gradle to build a shadowJar which I submit to my Flink containers by running
./bin/flink run <jar>.
I am getting this error:
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.flink.cdc.connectors.mysql.table.MySqlDeserializationConverterFactory.createStringConverter(MySqlDeserializationConverterFactory.java:72)
at
org.apache.flink.cdc.connectors.mysql.table.MySqlDeserializationConverterFactory.access$100(MySqlDeserializationConverterFactory.java:45)
at
org.apache.flink.cdc.connectors.mysql.table.MySqlDeserializationConverterFactory$1.createUserDefinedConverter(MySqlDeserializationConverterFactory.java:60)
…
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.databind.ObjectMapper
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
Environment:
Java: OpenJDK 11.0.26
Flink 1.20 / Java 11 running on docker container locally
Gradle 9
Flink MySQL Connector 3.3.0
MySQL Connector Java 8.0.33
After researching this issue and getting no results, I tried working with
Copilot (for good or bad, not pertinent to this discussion) for several hours
trying lots of things and not succeeding.
To summarize things I have tried:
* Initially I only had these lines in the gradle file:
flinkShadowJar “org.apache.flink:flink-connector-mysql-cdc:3.3.0”
flinkShadowJar “mysql:mysql-connector-java:8.0.33”
fllinkShadowJar “com.fasterxml.jackson.core:Jackson-databind:2.12.7.1”
flinkShadowJar “com.fasterxml.jackson.core:Jackson-core:2.12.7”
flinkShadowJar “com.fasterxml.jackson.core:Jackson-annotations::2.12.7.1”
* I have tried also adding those same lines but with the word
implementation instead of shadowJar (so both kinds of references were in the
gradle file)
* I have tried various ways to manipulate the shadowjar trying to do things
like:
* Relocate com.fasterxml.jackson to
org.apache.flink.cdc.connectors.mysql.shaded.com.fasterxml.jackson
* Forcing resolution strategy
* The shadowjar portion of my build.gradle looks like this now after
trying lots of things:
shadowJar {
configurations = [project.configurations.flinkShadowJar]
mergeServiceFiles()
// Add this transformer to handle service files
transform(com.github.jengelman.gradle.plugins.shadow.transformers.AppendingTransformer)
{
resource = 'META-INF/services/org.apache.flink.table.factories.Factory'
}
// // Add Java bytecode version check
//
transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer)
// Relocate Jackson to avoid conflicts
// relocate 'com.fasterxml.jackson',
'org.apache.flink.cdc.connectors.mysql.shaded.com.fasterxml.jackson'
manifest {
attributes 'Implementation-Version': project.version,
'Build-Jdk-Spec': '11',
'Multi-Release': 'true'
}
// Exclude any META-INF files from dependencies that might cause conflicts
exclude 'META-INF/*.SF'
exclude 'META-INF/*.DSA'
exclude 'META-INF/*.RSA'
}
// Add version constraints
// configurations.all {
// resolutionStrategy {
// force 'com.fasterxml.jackson.core:jackson-databind:2.12.7.1'
// force 'com.fasterxml.jackson.core:jackson-core:2.12.7'
// force 'com.fasterxml.jackson.core:jackson-annotations:2.12.7'
// }
// }
// Remove the version constraints section as we're handling it in dependencies
configurations.all {
resolutionStrategy.eachDependency { DependencyResolveDetails details ->
if (details.requested.group == 'com.fasterxml.jackson.core') {
if (details.requested.name == 'jackson-databind') {
details.useVersion '2.12.7.1'
} else {
details.useVersion '2.12.7'
}
}
}
}
Last thing I want to mention: after all the changes I made, I made sure gradle
build successfully. I examined the jar file by running this command:
jar tvf build/libs/myapp-0.2-all.jar | grep -i Jackson
amongst all the com/fasterxml/Jackson entries I did find:
com/fasterxml/jackson/databind/ObjectReader.class
Has anyone else ran into this issue and can offer advice on how to resolve it?
[Graphical user interface, text Description automatically generated]
+1 (316) 348.5461<tel:+13163485461> : office
weavix.com<http://www.weavix.com/>
Disclaimer
The information contained in this communication from the sender is
confidential. It is intended solely for use by the recipient and others
authorized to receive it. If you are not the recipient, you are hereby notified
that any disclosure, copying, distribution or taking action in relation of the
contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been
automatically archived by Mimecast, a leader in email security and cyber
resilience. Mimecast integrates email defenses with brand protection, security
awareness training, web security, compliance and other essential capabilities.
Mimecast helps protect large and small organizations from malicious activity,
human error and technology failure; and to lead the movement toward building a
more resilient world. To find out more, visit our website.
