Re: Consuming a large number of partitions in a deterministic way

2014-10-29 Thread Rajiv Kurian
Thanks Jun, looks like I am on the right track. On Wed, Oct 29, 2014 at 6:51 PM, Jun Rao wrote: > 1), 2), 3) Yes. If you use SimpleConsumer, you have to figure out the > leader of each partition and connect to the right broker. Each fetch > request can send multiple partitions (if with same lead

Re: Consuming a large number of partitions in a deterministic way

2014-10-29 Thread Jun Rao
1), 2), 3) Yes. If you use SimpleConsumer, you have to figure out the leader of each partition and connect to the right broker. Each fetch request can send multiple partitions (if with same leader) and you need to examine the error code per partition. Not all those error codes are applicable to th

Consuming a large number of partitions in a deterministic way

2014-10-29 Thread Rajiv Kurian
I am planning to use the current Java API and have the following use case: i) A single topic with about 1024 partitions. ii) A number of processes that want to consume these partitions in a deterministic way. The machine -> partitions assignment is done outside of kafka. During the lifetime of a p