Hi, It is a little long, since I wanted to explain the use case and then ask questions, so thanks for your attention
Use case: We have a use case where everything in the queue has to be consumed at least once. So the consumer has to have "consumed" (saved in some destination database) the message before confirming consumption to kafka (or ZK). Now it is possible and from what I have read so far we will have consumer groups and partitions. Here are some facts/numbers for our case * We will potentially have messages with peaks of 5k /second. * We can play with the message size if that makes any difference (keep it < 100 bytes for a link or put the entire message avg size of 2-5K bytes). * We do not need replication, but might have a kafka cluster to handle the load. * Also work consumption will take anywhere from 300-500ms, generally we would like the consumer to be not behind by more than 1-2 minutes. So if the message shows up in a queue, it should show up in the database within 2 minutes. The questions I have are * If this has been covered before, please point me to it. Thanks * Is that possible/recommended "controlled commit per consumed message" for this load (have read about some concerns on ZK issues)? * Are there any recommendations on configurations in terms of partitions to number of messages OR consumers? Maybe more queues/topics * Anything else that we might need to watch out for? * As for the client, I should be able to do this (control when the offset commit happens) with high level consumer I suppose? Thanks Anand
