Hi All, We have a set of logstash consumer groups running under the same set of instances, we have decided to run separate consumer groups subscribing multiple topics instead of running single consumer group for all topics(the reasoning behind this decision is because of how our elasticsearch cluster is designed).
Since we are running multiple consumer groups, sometimes we have detected that a few ec2 nodes are receiving multiple high throughput topics in different consumer groups. which was expected based on the implementation of round robin assignor. So I've decided to make a partition assignor which will consider the assignment based on other consumer group assignment. Could you please give me some pointers on how to proceed. This is my initial ideas on the problem. Solution #0: write an assignor, and use a specific consumer id pattern across all consumer groups, and in the assignor do a describe on all consumer groups and based on the topic throughput and the other consumer group assignment decide the assignment of this topic Thanks
