Re: Flink gelly dependency in transient EMR cluster

2020-03-16 Thread Till Rohrmann
Alternatively, you could also bundle the Gelly dependency with your user code jar by creating an uber jar. The downside of this approach would be an increased jar size which needs to be uploaded to the cluster. Cheers, Till On Thu, Mar 12, 2020 at 4:13 PM Antonio Martínez Carratalá < amarti...@al

Re: Flink gelly dependency in transient EMR cluster

2020-03-12 Thread Antonio Martínez Carratalá
I reply to myself with the solution in case someone else is having the same question It is only needed to add a copy command to copy the jar from flink/opt to flink/lib, in my case: StepConfig addGellyStep = new StepConfig() .withName("add-gelly-step") .withHadoopJarStep(new Hadoo

Flink gelly dependency in transient EMR cluster

2020-03-12 Thread Antonio Martínez Carratalá
Hello, I'm trying to run a flink job that works with graphs in a transient cluster in EMR, here is my code: -- HadoopJarStepConfig copyJarStepConf = new HadoopJarStepConfig() .withJar("command-runner.jar") .withArgs("bash", "-c", "aws s3 cp s3://" + BUCKET_NAME