Only a single message processed

2018-02-17 Thread Niclas Hedhman
Hi, I am pretty new to Flink, and I like what I see and have started to build my first application using it. I must be missing something very fundamental. I have a FlinkKafkaConsumer011, followed by a handful of filter, map and flatMap functions and terminated with the standard CassandraSink. I hav

Re: Correlation between number of operators and Job manager memory requirements

2018-02-17 Thread Shailesh Jain
Oops, hit send by mistake. In the configuration section, it is mentioned that for "many operators" heap size should be increased. "JVM heap size (in megabytes) for the JobManager. You may have to increase the heap size for the JobManager if you are running very large applications (with many opera

Correlation between number of operators and Job manager memory requirements

2018-02-17 Thread Shailesh Jain
Hi, I have flink job with almost 300 operators, and every time I'm trying to submit the job, the cluster crashes with OutOfMemory exception. I have 1 job manager and 1 task manager with 2 GB heap space allocated to both. In the configuration section of the documentation

Re: Variable time gap in Session Window

2018-02-17 Thread Fabian Hueske
Hi Teena, I think we just added exactly the feature you are asking for [1]. Dynamic session windows will be release with Flink 1.5.0. It would be great if you could try the feature and check if it fits your requirements and works as you'd expect it to work. Thanks, Fabian [1] https://issues.apa