Hi, We have a computation heavy streaming flink job which will be processing around 100 million message at peak time and around 1 million messages in non peak time. We need the capability to dynamically scale so that the computation operator can scale up and down during high or low work loads respectively without restarting the job in order to lower the machine costs.
Is there an ETA on when the rescaling a single operator without restart feature will be released? Is it possible to auto scale one of the operators with docker swarm or Amazon ECS auto scaling based on kafka consumer lag or cpu consumption? If so can I get some documentation or steps to achieve this behaviour. Also is there any document on what are the tasks of a job manager apart from scheduling and reporting status? Since there is just one job manager we just wanted to check if there would be any potential scaling limitations as the processing capacity increases. Thanks Govind
