Sounds like the parallel consumer is what you want: https://github.com/confluentinc/parallel-consumer
Or at least decide why that won't work before writing something yourself. - alex On Tue, Oct 3, 2023 at 4:20 PM Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com> wrote: > Hey Neeraj > I had a kafka assignment where I had to consume from a single partition. > When I was looking to scale the application, I was planning to consume the > partition with more than one consumer when I came across this issue. > > Regards, > Sree Sanjeev. > > On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya > <neeraj.vai...@yahoo.co.in.invalid> wrote: > > > Hi Sree Sanjeev, > > Other than the info from Colt, why can't you just increase partitions and > > scale the number of consumers accordingly ? > > Or build your application as a KStreams application and take advantage of > > horizontal and vertical scalability ? > > > > Regards, > > Neeraj > > > > > > On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy < > > c...@littlehorse.io> wrote: > > > > Hello, > > > > Currently, you cannot have more than one consumer consuming from a > > partition in Kafka. However, there is active discussion about a proposal > to > > enable queueing semantics for Kafka, which is a superset of the feature > you > > are talking about. Feel free to check KIP-932: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > > > > Hope this helps, > > Colt McNealy > > > > *Founder, LittleHorse.dev* > > > > > > On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < > > sreesanjee...@gmail.com> wrote: > > > > > I'm interested in knowing if we can achieve parallelism by allowing > > > multiple consumers to subscribe to a single partition of a topic. > > > > > > To explore potential solutions, I'm considering the following approach. > > > > > > 1. Implementing a locking mechanism to control access to the offsets > in a > > > partition to avoid concurrent processing issues. > > > > > > My primary concern is maintaining message ordering within the > partition, > > as > > > Kafka guarantees ordering within a partition. and message ordering is > not > > > the primary concern for many applications, so can we have a flag to > > enable > > > multiple consumers to a single partition ? > > > > > > I appreciate any insights, advice, or experiences that the Kafka > > community > > > can share on this topic. Thank you! > > > > > > Regards, > > > Sree Sanjeev. > > > > > >
