I also tested your code with an additional sink and it is executed on a
cluster without problems.
Cheers,
Till
On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park <chiwanp...@icloud.com> wrote:
> Hi. What flink version is running now in cluster?
> I copied your code and packaging with flink quickstart archetype. There is
> another error because you don’t add any data sink in the program.
>
> Regards,
> Chiwan Park
>
> > On Jun 16, 2015, at 4:17 PM, Thomas Peel <thomas.p...@euranova.eu>
> wrote:
> >
> >
> >
> > Hi guys,
> >
> > I am having troubles with a portion of code that throws an exception
> > when executed on a cluster (through the web interface) but not when
> > executed localy.
> >
> > The code :
> >
> > object DorotheaLassoRegression {
> >
> > def main(args: Array[String]): Unit = {
> >
> > val env = ExecutionEnvironment.getExecutionEnvironment
> >
> > val y = env.readTextFile(
> > "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels"
> > ).setParallelism(1).map(x => x.toDouble)
> >
> > val Y = y.reduceGroup(iterator => iterator.toArray)
> > env.execute()
> > }
> > }
> >
> > The exception:
> > An error occurred while invoking the program:
> >
> > The main method caused an error.
> >
> > java.lang.ClassNotFoundException:
> > eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at
> > java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
> > java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
> > java.lang.Class.forName0(Native Method) at
> > java.lang.Class.forName(Class.java:270) at
> >
> org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317)
> > at
> > com.esotericsoftware.reflectasm.shaded.org
> .objectweb.asm.ClassReader.accept(Unknown
> > Source) at
> > com.esotericsoftware.reflectasm.shaded.org
> .objectweb.asm.ClassReader.accept(Unknown
> > Source) at
> >
> org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93)
> > at
> >
> org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113)
> > at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at
> > org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at
> > org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at
> >
> eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42)
> > at
> >
> eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606) at
> >
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> > at
> >
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> > at
> > org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229)
> > at
> >
> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at
> > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> > at
> > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
> > at org.eclipse.jetty.server.Server.handle(Server.java:352) at
> >
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
> > at
> >
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
> > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at
> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
> > at
> > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
> > at
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a
> > version problem ?
> >
> > Thank you for your help, kind regards.
> > Thomas
> >
> > -------------------------
> > Thomas Peel
> >
> > Data Scientist
> > (M) +32 478 98 93 81
> >
> > EURA NOVA
> >
> > Rue Emile Francqui, 4
> >
> > 1435 Mont-Saint-Guibert
> >
> > (T) +32 10 75 02 00
> >
> > euranova.eu
> >
> >
>
>
>
>
