Hey,
for our CI/CD cycle I'd like to try out our Flink Jobs in an development
environment without running them against a huge EMR cluster (which is
what we do for production), so something like a standalone mode.
Until now, for this standalone running, I just started the job jar. As
the "env.execute()" is in the main-method, this works. I think this is
callled "Local Mode" by the Flink Devs. I packaged the whole thing in a
docker container so I have a deployable artefact.
The problem with that is, that the memory constraint seem to be
difficult to control: Setting Xmx and Xms for the job doesn't seem to
limit the memory. This is most likely due to flinks off-heap memory
allocation.
Now, I got as feedback that perhaps the MiniCluster is the way to go
instead of the "Local Mode".
My questions:
1. Is the MiniCluster better than the local mode? What are the use-cases
in which you would choose one over the other?
2. Is there an example how to use the MiniCluster? I see that I need a
JobGraph, how do I get one?
3. What are the tuning parameters to limit the memory consumption of the
MiniCluster (and maybe the local mode)?
Thanks for your help,
Dominik
