How large a Flink cluster can have?

2016-07-13 Thread Yan Chou Chen
FAQ[1], mailing list[2], and the powered by page[3] doesn't find related information. Just out of curiosity, what is the current largest Flink cluster size running in production? For instance, long time ago yahoo [4] ran 4k hadoop nodes in production. Thanks [1]. https://flink.apache.org/faq.html

Stoppable Job And Web UI Questions

2016-06-22 Thread Yan Chou Chen
Several new questions: - Stoppable job I read threads mentioning that a streaming job can be stopped [1][2]. However looks like it can only be called through command line. Is it possible to programmatically stop the streaming job from within the job itself? For instance, a Kafka consumer streaming

Re: How MapFunction gets executed?

2016-06-22 Thread Yan Chou Chen
Thanks for clarifying that helps me identify the root cause. The problem comes from my code which is not related to Flink. Now the problem is solved. Thank you again for the advice! On 16 June 2016 at 23:49, Till Rohrmann wrote: > Hi Yan Chou Chen, > > Flink does not instantiate for ea

How MapFunction gets executed?

2016-06-16 Thread Yan Chou Chen
A quick question. When running a stream job that executes DataStream.map(MapFunction) , after data is read from Kafka, does each MapFunction is created per item or based on parallelism? For instance, for the following code snippet val env = StreamExecutionEnvironment.getExeutionEnvironment val st