Passing records between two jobs

2018-06-18 Thread Avihai Berkovitz
Hello, We are planning a system that will be comprised of 3 different jobs: 1. Getting a stream of events, adding some metadata to the events, and outputting them to a temporary message queue. 2. Performing some calculations on the events we got from job 1, as required for product A. 3.

Standalone cluster layout

2016-12-13 Thread Avihai Berkovitz
Hi folks, I am setting up a Flink cluster for testing, and I have a few questions regarding memory allocations: 1. Is there a recommended limit to the size of a TaskManager heap? I saw that Flink uses G1GC, so we can use dozens of GB. 2. Following the above question, should I use only one

RE: Standalone cluster layout

2016-12-15 Thread Avihai Berkovitz
3, 2016 at 5:41 PM, Avihai Berkovitz mailto:avihai.berkov...@microsoft.com>> wrote: Hi folks, I am setting up a Flink cluster for testing, and I have a few questions regarding memory allocations: 1. Is there a recommended limit to the size of a TaskManager heap? I saw that Flink uses

Possible JVM native memory leak

2017-01-17 Thread Avihai Berkovitz
Hello, I am running a streaming job on a small cluster, and after a few hours I noticed that my TaskManager processes are being killed by the OOM killer. The processes were using too much memory. After a bit of monitoring, I have the following status: * The maximum heap size (Xmx) is 4