I saw the example at https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example and it answered most of my questions. I am still trying to figure out the pattern to be used when I want to use a single simple consumer to fetch messages from different partitions (possibly from different offsets) and possibly managed by different leaders.
My use case right now is that I have a consumer which dynamically decides which partitions it is responsible for and decides to fetch messages from them at potentially different offsets. Right now it seems like I would need a new SimpleConsumer for each broker since the SimpleConsumer takes the leadBroker in it's constructor. Then I'd have to build a FetchRequest for each broker and ensure that the addFetch(...) calls are made with partitions that correspond to the leader broker that a SimpleConsumer is managing. Finally I'll need to make up to numBrokers number of consumer.fetch(req) calls since each request is for a separate broker. Is there a better way to do this where I can build one big requests with broker -> partition mappings and call a consumer.fetch() with one giant request? Otherwise if I am doing this in a single thread I have head of line blocking with one request blocking another. Thanks!
