Re: starting flink job from bash script with maven

2015-07-24 Thread Stephan Ewen
Thanks for letting us know! The problem with Java Serialization is that they often swallow exceptions and you only see a "corrupted byte stream" in the end. So far, I have found no workaround for that. Stephan On Fri, Jul 24, 2015 at 11:31 AM, Stefano Bortoli wrote: > It seems there is a prob

Re: starting flink job from bash script with maven

2015-07-24 Thread Stefano Bortoli
It seems there is a problem with the maven class loading. I have created the uberjar and then executed with traditional java -cp uberjar.jar args and it worked with no problems. It could be interesting to investigate the reason, as maven exec is very convenient. However, with the uberjar the proble

Re: starting flink job from bash script with maven

2015-07-24 Thread Stefano Bortoli
HI Stephan, I think I may have found a possible root of the problem. I do not build the fat jar, I simply execute the main with maven exec:java with default install and compile. No uberjar created shading. I will try that and report. The fact that it runs in eclipse so easily makes it confusing so

Re: starting flink job from bash script with maven

2015-07-24 Thread Stephan Ewen
Hi! There is probably something going wrong in MongoOutputFormat or MongoHadoop2OutputFormat. Something fails, but Java swallows the problem during Serialization. It may be a classloading issue that gets not reported. Are the MongoOutputFormat and the MongoHadoop2OutputFormat both in the fat jar?

Re: starting flink job from bash script with maven

2015-07-24 Thread Stefano Bortoli
I have implemented this test without any exception: package org.tagcloud.persistence.batch.test; import java.io.IOException; import org.apache.commons.lang.SerializationUtils; import org.apache.hadoop.mapreduce.Job; import org.tagcloud.persistence.batch.MongoHadoop2OutputFormat; import com.mong

Re: starting flink job from bash script with maven

2015-07-24 Thread Stephan Ewen
Hi! The user code object (the output format here) has a corrupt serialization routine. We use default Java Serialization for these objects. Either the MongoHadoopOutputFormat cannot be serialized and swallows an exception, or it overrides the readObject() / writeObject() methods (from Java Serial

starting flink job from bash script with maven

2015-07-24 Thread Stefano Bortoli
Hi guys! I could program a data maintenance job using Flink on MongoDB. The job runs smoothly if I start it from eclipse. However, when I try to run it using a bash script invoking a maven exec:java I have a serialization exception: org.apache.flink.runtime.client.JobExecutionException: Cannot ini