Hello Experts & Kafka Team, Its existing to learn and work on using Kafka. I have been going through lot of pages and Q&A.
We are building an infra & topology using Kafka for events processing in our application. We need some advice about designing the Producer and Consumer. *Please find attached file/below picture* of our current setup that we are thinking of. [image: Inline image 1] *1) Producer:* I understand that from 0.8.1, the message balancing is done in a fashion that the broker will choose a partition after every meta refresh(the default of which is 10 mins) Questions are: a.* Is there any other mechanism other than changing meta refresh ?* (I understand that the logic implementation using custom class is no longer supported in 0.8.1) b. We ultimately want the message to be evenly distributed across partitions so that consumer's load is also evenly distributed thus paving scalability & reduce lag and will help us scale easily as we can just add partition with a corresponding consumer node attached to it. Is this advised ? And to achieve this, *what is the optimized meta refresh time without affecting performance ?* *2) Consumer* a. I was in a though that SimpleConsumer has more flexibillity and features. But after reading Neha's below JavaDoc, I am liking KafkaConsumer features and the less need to handle at granular level. *What is the adviced Consumer, SimpleCosumer or KafkaConsumer ?* Neha's KafkaCosumer JavaDoc: http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/kafka/clients/consumer/KafkaConsumer.html b. For keeping track of the Offset at each Consumer node, I am thinking to manually control the Offset commit(to ensure that processing a message is neither missed nor duplicated). On failure or exception, I would also log the current Offset in a file or something before exiting so that when I start my consumer again I can start from the Offset where I left. *Is this a good design ?* Thanks for the time and really appreciate the effort for making Kafka amazing :) Thanks, KR
<<inline: image.png>>
