On Thu, Oct 5, 2023 at 2:05 PM L. C. <chen....@gmail.com> wrote: > I'm getting class not found error while running the word count example on > Dataproc 2.1 with Beam 2.50.0. The class exists under the jar. Does > anyone know how to resolve this? > > This is a list of dependency versions: > <beam.version>2.50.0</beam.version> > > <bigquery.version>v2-rev20230520-2.0.0</bigquery.version> > <google-api-client.version>2.0.0</google-api-client.version> > <guava.version>32.1.2-jre</guava.version> > <hamcrest.version>2.1</hamcrest.version> > <jackson.version>2.14.1</jackson.version> > <joda.version>2.10.10</joda.version> > <junit.version>4.13.1</junit.version> > <kafka.version>2.4.1</kafka.version> > <libraries-bom.version>26.22.0</libraries-bom.version> > <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version> > <maven-exec-plugin.version>1.6.0</maven-exec-plugin.version> > <maven-jar-plugin.version>3.0.2</maven-jar-plugin.version> > <maven-shade-plugin.version>3.1.0</maven-shade-plugin.version> > <mockito.version>3.7.7</mockito.version> > <pubsub.version>v1-rev20220904-2.0.0</pubsub.version> > <slf4j.version>1.7.30</slf4j.version> > <spark.version>3.2.2</spark.version> > <hadoop.version>2.10.2</hadoop.version> > <maven-surefire-plugin.version>3.0.0-M5</maven-surefire-plugin.version> > <nemo.version>0.1</nemo.version> > <flink.artifact.name>beam-runners-flink-1.16</flink.artifact.name> > > > I used this to build a shaded jar: > $ mvn compile -Pspark-runner package > > Here's the stack trace: >
Given that this raised NoClassDefFoundError (and not ClassNotFoundException) it's possible that the class initialization failed. Is there another exception before this one (may be at the first occurrence of NoClassDefFoundError) ? > Waiting for job output... > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/beam/sdk/coders/CoderProviderRegistrar > at java.base/java.lang.ClassLoader.defineClass1(Native Method) > at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) > at > java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) > at > java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) > at > java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) > at > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:576) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:398) > at > java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1210) > at > java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) > at > java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) > at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) > at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) > at > org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterators.addAll(Iterators.java:366) > at > org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Lists.newArrayList(Lists.java:146) > at > org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Lists.newArrayList(Lists.java:132) > at > org.apache.beam.sdk.coders.CoderRegistry.<clinit>(CoderRegistry.java:168) > at org.apache.beam.sdk.Pipeline.getCoderRegistry(Pipeline.java:334) > at > org.apache.beam.sdk.values.PCollection.finishSpecifyingOutput(PCollection.java:94) > at > org.apache.beam.sdk.runners.TransformHierarchy.setOutput(TransformHierarchy.java:173) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:546) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:479) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:175) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:150) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:134) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:545) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:496) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) > at org.apache.beam.sdk.io.TextIO$Read.expand(TextIO.java:413) > at org.apache.beam.sdk.io.TextIO$Read.expand(TextIO.java:275) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:545) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:496) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:190) > at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:201) > at org.apache.beam.examples.WordCount.main(WordCount.java:213) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at org.apache.spark.deploy.SparkSubmit.org > $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.sdk.coders.CoderProviderRegistrar > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > at > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > ... 53 more >
