Hi, The new consumer is single threaded. You can layer multi-threaded processing on top of it, but you'll definitely need to be careful about how offset commits are handled to ensure a) processing of a message is actually *complete* not just passed off to another thread before committing an offset and b) that all threads have finished processing up to the offset to commit (e.g. message 10 may take a long time to process, and completing processing message 11 in another thread should *not* allow you to commit offset 11). And, of course, as soon as you process messages in multiple threads, you're going to lose ordering guarantees.
Given that this can get relatively complicated and has some significant tradeoffs, this definitely isn't recommended for normal usage but could be reasonable in certain situations. Depending on where time is being spent, you might be able to use a slightly simpler approach where you let the consumer run in one thread and only have one other thread that processes messages in order. This requires some coordination, but is less complicated and still preserves ordering. It is useful if a significant chunk of time is spent in the consumer itself (and, e.g., the deserializer). -Ewen On Wed, Jan 20, 2016 at 1:45 PM, sagarcasual . <sagarcas...@gmail.com> wrote: > Hello, > > We are using Kafka 0.9 version and are having large number of messages > pushed to specific partition within a kafka topic. And there are multiple > such partitions within this topic. We have one consumer assigned per > partition within this topic, and we are maintaining offset manually within > the topic-partition in an outside datastore. I wanted to know if we start > getting really large number of messages in a topic-partition, is it > possible to have consumer dealing with a topic-partition to be > multi-threaded. Because it might not be possible that a consumer instance > assigned to a partition is able to finish processing all the records in the > time-span we want it finish. Is it possible to have such kind of > multi-threaded consumer with a partition. Is it recommended? Also if the > answer is YES, then how multiple threads can do offset management, because > all these threads are dealing with messages within the same partition. Any > sample snippet available? > > Please note: I am asking about "*consumer dealing with a single partition > within a topic*", I am *not *looking at the consumer group for a topic > across partitions in it. > -- Thanks, Ewen
