I am on Flink 1.7.1 and K8S. I said "suddenly" because my program worked fine until I added a new MapFunction. I do not know the details, but I think I know what is causing it
=== Start of Program === val stream: DataStream[MaxwellEvent] = <get the stream> steam.map(new ProblemFunction()) will cause the issue class ProblemFunction(stringParam: String)(implicit datadog: DatadogClient) extends MapFunction[MaxwellEvent, MaxwellEvent] === End of Program === I think the class taking curry params caused the issue, after I give up on the curry format, the error disappeared. I am using https://github.com/sbt/sbt-assembly to assemble the fat jar. There might be some issue, or config issue with that as well. I am reading this article, it is a good start for me as well https://heapanalytics.com/blog/engineering/missing-scala-class-noclassdeffounderror Hao Sun Team Lead 1019 Market St. 7F San Francisco, CA 94103 On Thu, Jan 3, 2019 at 1:08 AM Timo Walther <twal...@apache.org> wrote: > Hi Hao, > > which Flink version are you using? What do you mean with "suddenly", did > it work before? > > Regards, > Timo > > > Am 03.01.19 um 07:13 schrieb Hao Sun: > > Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow > could not find it suddenly > > javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar > 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45' > Compiled from "ConnectedStreams.scala" > public final class > com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 > extends > org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> > { > public com.zendesk.fraudprevention.datatypes.MaxwellEvent > createInstance(java.lang.Object[]); > public > org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> > createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, > org.apache.flink.api.common.typeutils.TypeSerializer<?>[]); > public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase > createSerializerInstance(java.lang.Class, > org.apache.flink.api.common.typeutils.TypeSerializer[]); > public java.lang.Object createInstance(java.lang.Object[]); > public > com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, > org.apache.flink.api.common.typeutils.TypeSerializer[]); > } > > Hao Sun > Team Lead > 1019 Market St. 7F > San Francisco, CA 94103 > > > On Wed, Jan 2, 2019 at 6:04 PM qi luo <luoqi...@gmail.com> wrote: > >> Hi Hao, >> >> Since Flink is using Child-First class loader, you may try search for the >> class >> "*com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” >> in your fat JAR. Is that an inner class?* >> >> *Best,* >> *Qi* >> >> On Jan 3, 2019, at 7:01 AM, Hao Sun <ha...@zendesk.com> wrote: >> >> Hi, >> >> I am wondering if there are any protips to figure out what class is not >> found? >> >> ===== Logs ==== >> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not >> instantiate chained outputs. >> at >> org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324) >> at >> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292) >> at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>( >> http://OperatorChain.java:133 >> <http://OperatorChain.java:133>) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) >> at java.lang.Thread.run(Thread.java:748) >> *Caused by: java.lang.ClassNotFoundException: >> com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45* >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at >> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at >> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77) >> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866) >> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) >> at java.util.ArrayList.readObject(ArrayList.java:797) >> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498) >> at >> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459) >> at >> org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321) >> ... 5 more >> >> >> Hao Sun >> Team Lead >> 1019 Market St. 7F >> San Francisco, CA 94103 >> >> >> >
