Hi Matthias, Yes, all the task managers have the same hardware/memory configuration.
Aeden On Fri, Mar 12, 2021 at 3:25 AM Matthias Pohl <matth...@ververica.com> wrote: > > Hi Aeden, > just to be sure: All task managers have the same hardware/memory > configuration, haven't they? I'm not 100% sure whether this affects the slot > selection in the end, but it looks like this parameter has also an influence > on the slot matching strategy preferring slots with less utilization of > resources [1]. > > I'm gonna add Chesnay to the thread. He might have more insights here. > @Chesnay are there any other things that might affect the slot selection when > actually trying to evenly spread out the slots? > > Matthias > > [1] > https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerConfiguration.java#L141 > > On Fri, Mar 12, 2021 at 12:58 AM Aeden Jameson <aeden.jame...@gmail.com> > wrote: >> >> Hi Arvid, >> >> Thanks for responding. I did check the configuration tab of the job >> manager and the setting cluster.evenly-spread-out-slots: true is >> there. However I'm still observing unevenness in the distribution of >> source tasks. Perhaps this additional information could shed light. >> >> Version: 1.12.1 >> Deployment Mode: Application >> Deployment Type: Standalone, Docker on Kubernetes using the Lyft >> Flink operator https://github.com/lyft/flinkk8soperator >> >> I did place the setting under the flinkConfig section, >> >> apiVersion: flink.k8s.io/v1beta1 >> .... >> spec: >> flinkConfig: >> cluster.evenly-spread-out-slots: true >> high-availability: zookeeper >> ... >> state.backend: filesystem >> ... >> jobManagerConfig: >> envConfig: >> .... >> >> Would you explain how the setting ends up evenly distributing active >> kafka consumers? Is it a result of just assigning tasks toTM1, TM2, >> TM3 ... TM18 in order and starting again. In my case I have 36 >> partitions and 18 nodes so after the second pass in assignment I would >> end up with 2 subtasks in the consumer group on each TM. And then >> subsequent passes result in inactive consumers. >> >> >> Thank you, >> Aeden >> >> On Thu, Mar 11, 2021 at 5:26 AM Arvid Heise <ar...@apache.org> wrote: >> > >> > Hi Aeden, >> > >> > the option that you mentioned should have actually caused your desired >> > behavior. Can you double-check that it's set for the job (you can look at >> > the config in the Flink UI to be 100% sure). >> > >> > Another option is to simply give all task managers 2 slots. In that way, >> > the scheduler can only evenly distribute. >> > >> > On Wed, Mar 10, 2021 at 7:21 PM Aeden Jameson <aeden.jame...@gmail.com> >> > wrote: >> >> >> >> I have a cluster with 18 task managers 4 task slots each running a >> >> job whose source/sink(s) are declared with FlinkSQL using the Kafka >> >> connector. The topic being read has 36 partitions. The problem I'm >> >> observing is that the subtasks for the sources are not evenly >> >> distributed. For example, 1 task manager will have 4 active source >> >> subtasks and other TM's none. Is there a way to force each task >> >> manager to have 2 active source subtasks. I tried using the setting >> >> cluster.evenly-spread-out-slots: true , but that didn't have the >> >> desired effect. >> >> >> >> -- >> >> Thank you, >> >> Aeden
