Re: Memory management in Flink

2015-03-14 Thread Robert Metzger
Hi Emmanuel, Flink is not starting new JVMs on the workers when submitting a new topology. When starting Flink using the "start-cluster.sh" script, it will create the Flink cluster and its JVMs. Your topologies are then started as Threads inside these JVMs. So the overhead per topology is actuall

Memory management in Flink

2015-03-13 Thread Emmanuel
Hello, In Storm, when running a new topology, a new JVM is started and this takes some memory. In my use case, I may need to run many different topologies. How does it work in Flink? Is the Flink run command spinning a new JVM each time?If I run multiple topologies it seems like the additional pr