Re: Balancing operation across machines

2017-10-21 Thread Tony Wei
Hi, I think this patch[1] is what you need and it might be merged in 1.4.0 release. AFAIK, there is no way to change the behavior of scheduler from public API. One work around that I could come up with is to give each TM one available slot, make all operators in the same SlotSharingGroup and set t

Balancing operation across machines

2017-10-21 Thread Le Xu
Hello! Just wondering if there is a way to load-balancing all executors on one operator across the machine. I have a reduce operator that has less parallelism level than the total number of slots in the cluster. Currently flink tries to compact executors in an operator and schedule them to some of