Re: ClassNotFound exception on a reduce group (Cluster mode)

[Thomas Peel]

Hi,

Thank you for your help. It was actually an issue between the Flink 
version I used to build my quickstart jar and the Flink version on the 
cluster.

Cheers,
Thomas

Le 16/06/15 10:10, Till Rohrmann a écrit :
I also tested your code with an additional sink and it is executed on a
cluster without problems.

Cheers,
Till

On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park <chiwanp...@icloud.com> wrote:

Hi. What flink version is running now in cluster?
I copied your code and packaging with flink quickstart archetype. There is
another error because you don’t add any data sink in the program.

Regards,
Chiwan Park

On Jun 16, 2015, at 4:17 PM, Thomas Peel <thomas.p...@euranova.eu>
wrote:


Hi guys,

I am having troubles with a portion of code that throws an exception
when executed on a cluster (through the web interface) but not when
executed localy.

The code :

object DorotheaLassoRegression {

def main(args: Array[String]): Unit = {

val env = ExecutionEnvironment.getExecutionEnvironment

val y = env.readTextFile(
"hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels"
).setParallelism(1).map(x => x.toDouble)

val Y = y.reduceGroup(iterator => iterator.toArray)
env.execute()
}
}

The exception:
An error occurred while invoking the program:

The main method caused an error.

java.lang.ClassNotFoundException:
eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:270) at

org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317)
at
com.esotericsoftware.reflectasm.shaded.org
.objectweb.asm.ClassReader.accept(Unknown
Source) at
com.esotericsoftware.reflectasm.shaded.org
.objectweb.asm.ClassReader.accept(Unknown
Source) at

org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93)
at

org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113)
at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at
org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at
org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at

eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42)
at

eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at

org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at

org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at
org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229)
at

org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
at

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
at

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
at

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:352) at

org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at

org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
at
org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
at

org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
at

org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:745)

Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a
version problem ?

Thank you for your help, kind regards.
Thomas

-------------------------
Thomas Peel

Data Scientist
(M) +32 478 98 93 81

EURA NOVA

Rue Emile Francqui, 4

1435 Mont-Saint-Guibert

(T) +32 10 75 02 00

euranova.eu