Re: runtime memory management

2020-08-31 Thread Xintong Song
Well, that's a long story. In general, there are 2 steps. 1. *Which operators are deployed in the same slot?* Operators are first *chained*[1] together, then a *slot sharing strategy*[2] is applied by default. 2. *Which task managers are slots allocated from?* 1. For active deplo

Re: runtime memory management

2020-08-31 Thread lec ssmi
Thanks. When the program starts, how is each operator allocated in taskmanager? For example, if I have 2 taskmanagers and 10 operators, 9 operators are allocated to tm-A and the remaining one is placed in tm-B, the utilization of resources will be very low. Xintong Song 于2020年8月31日周一 下午2:45写道:

Re: runtime memory management

2020-08-30 Thread Xintong Song
Hi, For a complex streaming job, is there any way to tilt the memory towards > stateful operators? If streaming jobs are interested, the quick answer is no. Memory is fetched on demand for all operators. Currently, only managed memory for batch jobs are pre-planned for each operator. Thank you~

runtime memory management

2020-08-30 Thread lec ssmi
HI: Generally speaking, when we submitting the flink program, the number of taskmanager and the memory of each tn will be specified. And the smallest real execution unit of flink should be operator. Since the calculation logic corresponding to each operator is different, some need to save the