Re: Moving from single-node, Maven examples to cluster execution

All right, I figured I’d have to do shading, but hadn’t gotten around to experimenting. I’ll try it out. Prez Cannady p: 617 500 3378 e: revp...@opencorrelate.org GH: https://github.com/opencorrelate LI: https://www.li

Re: Moving from single-node, Maven examples to cluster execution

Hi Prez, You need to build a jar with all your dependencies bundled inside. With maven you can use maven-assembly-plugin for this, or with SBT there's sbt-assembly. Once you've done this, you can login to the JobManager node of your Flink cluster, copy the jar across and use the Flink command lin

Moving from single-node, Maven examples to cluster execution

Having a hard time trying to get my head around how to deploy my Flink programs to a pre-configured, remote Flink cluster setup. My Mavenized setup uses Spring Boot (to simplify class path handling and generate pretty logs) to execute provision a StreamExecutionEnvironment with Kafka sources an

Re: cluster execution

xD… a simple "hdfs dfs -chmod -R 777 /users" fixed it! > Am 01.02.2016 um 12:17 schrieb Till Rohrmann : > > Hi Lydia, > > I looks like that. I guess you should check your hdfs access rights. > > Cheers, > Till > > On Mon, Feb 1, 2016 at 11:28 AM, Lydia Ickler

Re: cluster execution

Hi Lydia, I looks like that. I guess you should check your hdfs access rights. Cheers, Till On Mon, Feb 1, 2016 at 11:28 AM, Lydia Ickler wrote: > Hi Till, > > thanks for your reply! > I tested it with the Wordcount example. > Everything works fine if I run the command: > ./flink run -p 3 /hom

Re: cluster execution

Hi Till, thanks for your reply! I tested it with the Wordcount example. Everything works fine if I run the command: ./flink run -p 3 /home/flink/examples/WordCount.jar Then the program gets executed by my 3 workers. If I want to save the output to a file: ./flink run -p 3 /home/flink/examples/Wor

Re: cluster execution

Hi Lydia, what do you mean with master? Usually when you submit a program to the cluster and don’t specify the parallelism in your program, then it will be executed with the parallelism.default value as parallelism. You can specify the value in your cluster configuration flink-config.yaml file. Al

cluster execution

Hi all, I am doing some operations on a DataSet> … (see code below) When I run my program on a cluster with 3 machines I can see within the web client that only my master is executing the program. Do I have to specify somewhere that all machines have to participate? Usually the cluster execute

Re: Cluster Execution - log

Hi Stephan, yes, now it is solved. I was running an older version of the client by mistake. Thanks Juan On Thu, 2015-07-23 at 15:31 +0200, Stephan Ewen wrote: > Hi! > > > Seems that you have different versions of the code running locally and > on the cluster. Is it possible that you did a co

Re: Cluster Execution - log

Hi! Seems that you have different versions of the code running locally and on the cluster. Is it possible that you did a code update locally (client), and forgot to update the cluster (or the other way around)? Greetings, Stephan On Thu, Jul 23, 2015 at 3:10 PM, Juan Fumero < juan.jose.fumero.a

Cluster Execution - log

Hi, When I execute from Java the app in a cluster, the user app is blocked in this point: log4j:WARN No appenders could be found for logger (org.apache.flink.api.java.ExecutionEnvironment). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.

Re: Cluster execution -jar files-

As the JavaDoc explains: >* @param jarFiles The JAR files with code that needs to be shipped to > the cluster. If the program uses >* user-defined functions, user-defined input formats, > or any libraries, those must be >* provided in the J

Re: Cluster execution -jar files-

Missing reference: [1] https://ci.apache.org/projects/flink/flink-docs-release-0.9/apis/cluster_execution.html On Don, 2015-07-16 at 16:04 +0200, Juan Fumero wrote: > Hi, > I would like to use the createRemoteEnvironment to run the application > in a cluster and I have some questions. Followin

Cluster execution -jar files-

Hi, I would like to use the createRemoteEnvironment to run the application in a cluster and I have some questions. Following the documentation in [1] It is not clear to me how to use it. What should be the content of the jar file? All the external libraries that I use? or need to include the pr