Re: [DISCUSS]Rethink the rescale operation, can we do it async

2018-11-08 Thread Stefan Richter
Hi, in general, I think it is clearly a good idea to make rescaling as cheap and as dynamic as possible and this is a goal that is surely on the long term roadmap of Flink. However, from a technical point of view I think that things are not as simple if you go into details, and details is what

[DISCUSS]Rethink the rescale operation, can we do it async

2018-11-07 Thread shimin yang
Currently, the rescale operation is to stop the whole job and restart it with different parrellism. But the rescale operation cost a lot and took lots of time to recover if the state size is quite big. And a long-time rescale might cause other problems like latency increase and back pressure. For