Re: runtime memory management

2020-08-31 Thread Xintong Song
Well, that's a long story. In general, there are 2 steps. 1. *Which operators are deployed in the same slot?* Operators are first *chained*[1] together, then a *slot sharing strategy*[2] is applied by default. 2. *Which task managers are slots allocated from?* 1. For active deplo

Re: runtime memory management

2020-08-31 Thread lec ssmi
Thanks. When the program starts, how is each operator allocated in taskmanager? For example, if I have 2 taskmanagers and 10 operators, 9 operators are allocated to tm-A and the remaining one is placed in tm-B, the utilization of resources will be very low. Xintong Song 于2020年8月31日周一 下午2:45写道:

Re: runtime memory management

2020-08-30 Thread Xintong Song
Hi, For a complex streaming job, is there any way to tilt the memory towards > stateful operators? If streaming jobs are interested, the quick answer is no. Memory is fetched on demand for all operators. Currently, only managed memory for batch jobs are pre-planned for each operator. Thank you~