Re: Flink & Kafka multi-node config

2018-04-18 Thread Tzu-Li (Gordon) Tai
Hi, The partition-to-subtask assignment of is not locality aware. There were discussions to expose functionality for custom user-defined assignment methods, which it might be possible to leverage that for a locality aware assignment. Unfortunately, this feature is not implemented, yet. The rrela

Flink & Kafka multi-node config

2018-04-16 Thread TechnoMage
If I use defaults for the most part but configure flink to have parallelism 5 and kafka to have 5 brokers (one of each on 5 nodes) will the connector and kafka be smart enough to use the kafka partition on the same node as the flink task manager for the 5 partitions? Do I need to explicitly ass