Hello, I sent an email sometime ago about java external transforms in a java job. Seems like things are for the most part working, but we have run into this error that I have attached. At the base cause it says [debug] Caused by: java.lang.ClassNotFoundException: com.spotify.ingestion.recentactivity.RecentActivityFeature [debug] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) [debug] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) [debug] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) [debug] at java.base/java.lang.Class.forName0(Native Method)
for a class used in the external transform. We find that it seems to be related to this code isJavaSDKCompatible method <https://github.com/apache/beam/blob/66ffc0b9fe99ba7d305d00c2f93e29979b1b2123/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java#L419> and maybe this PR <https://github.com/apache/beam/pull/11018/files>. We monkey patched the file and made the method return false, which seemed to get further. That was just us messing around to see what happens and maybe runs into more issues down the line. Any guidance here on what is going on and how we might be able to fix this? Thanks, Sahith
xception in thread "main" org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) [debug] at org.apache.beam.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:168) [debug] at org.apache.beam.runners.core.construction.External$ExpandableTransform.lambda$expand$3(External.java:326) [debug] at java.base/java.util.Map.forEach(Map.java:661) [debug] at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1505) [debug] at org.apache.beam.runners.core.construction.External$ExpandableTransform.expand(External.java:321) [debug] at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548) [debug] at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482) [debug] at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:363) [debug] at com.spotify.oasis.OasisMaterializerJob$.run(OasisMaterializerJob.scala:60) [debug] at com.spotify.oasis.OasisMaterializerJob$.main(OasisMaterializerJob.scala:26) [debug] at com.spotify.oasis.OasisMaterializerJob.main(OasisMaterializerJob.scala) [debug] Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes [debug] at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:78) [debug] at org.apache.beam.runners.core.construction.CoderTranslation.fromCustomCoder(CoderTranslation.java:176) [debug] at org.apache.beam.runners.core.construction.CoderTranslation.fromProto(CoderTranslation.java:143) [debug] at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:87) [debug] at org.apache.beam.runners.core.construction.RehydratedComponents$2.load(RehydratedComponents.java:82) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) [debug] at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) [debug] ... 14 more [debug] Caused by: java.lang.ClassNotFoundException: com.spotify.ingestion.recentactivity.RecentActivityFeature [debug] at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) [debug] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) [debug] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) [debug] at java.base/java.lang.Class.forName0(Native Method) [debug] at java.base/java.lang.Class.forName(Class.java:398) [debug] at java.base/java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:756) [debug] at org.apache.beam.sdk.util.SerializableUtils$ContextualObjectInputStream.resolveClass(SerializableUtils.java:193) [debug] at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1997) [debug] at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1864) [debug] at java.base/java.io.ObjectInputStream.readClass(ObjectInputStream.java:1827) [debug] at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1652) [debug] at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490) [debug] at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384) [debug] at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222) [debug] at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681) [debug] at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490) [debug] at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384) [debug] at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222) [debug] at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681) [debug] at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:493) [debug] at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:451) [debug] at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:75)
