Thanks. When the program starts, how is each operator allocated in taskmanager? For example, if I have 2 taskmanagers and 10 operators, 9 operators are allocated to tm-A and the remaining one is placed in tm-B, the utilization of resources will be very low.
Xintong Song <tonysong...@gmail.com> 于2020年8月31日周一 下午2:45写道： > Hi, > > For a complex streaming job, is there any way to tilt the memory towards >> stateful operators? > > If streaming jobs are interested, the quick answer is no. Memory is > fetched on demand for all operators. > > Currently, only managed memory for batch jobs are pre-planned for each > operator. > > Thank you~ > > Xintong Song > > > > On Mon, Aug 31, 2020 at 1:33 PM lec ssmi <shicheng31...@gmail.com> wrote: > >> HI: >> Generally speaking, when we submitting the flink program, the number of >> taskmanager and the memory of each tn will be specified. And the smallest >> real execution unit of flink should be operator. >> Since the calculation logic corresponding to each operator is >> different, some need to save the state, and some don't. Therefore, the >> memory size required by each operator should be different. How does the >> flink program allocate taskmanager memory to the operator by default? >> In our production practice, with the increase of traffic, some >> operators (mainly stateful such as join and groupby) often have >> insufficient memory, resulting in slower calculations. The usual approach >> is to increase the entire taskmanager memory. But will this part of the >> increased memory be allocated to the map-like operator, or that the memory >> itself is fetched on demand in the same taskmanager whoever needs the >> memory will fetch it until the memory is used up, in other words, there is >> no preset memory allocation ratio. For a complex streaming job, is there >> any way to tilt the memory towards stateful operators? >> >> Thanks. >> >> >> >>
