Hey Neeraj I had a kafka assignment where I had to consume from a single partition. When I was looking to scale the application, I was planning to consume the partition with more than one consumer when I came across this issue.
Regards, Sree Sanjeev. On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya <neeraj.vai...@yahoo.co.in.invalid> wrote: > Hi Sree Sanjeev, > Other than the info from Colt, why can't you just increase partitions and > scale the number of consumers accordingly ? > Or build your application as a KStreams application and take advantage of > horizontal and vertical scalability ? > > Regards, > Neeraj > > > On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy < > c...@littlehorse.io> wrote: > > Hello, > > Currently, you cannot have more than one consumer consuming from a > partition in Kafka. However, there is active discussion about a proposal to > enable queueing semantics for Kafka, which is a superset of the feature you > are talking about. Feel free to check KIP-932: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > > Hope this helps, > Colt McNealy > > *Founder, LittleHorse.dev* > > > On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < > sreesanjee...@gmail.com> wrote: > > > I'm interested in knowing if we can achieve parallelism by allowing > > multiple consumers to subscribe to a single partition of a topic. > > > > To explore potential solutions, I'm considering the following approach. > > > > 1. Implementing a locking mechanism to control access to the offsets in a > > partition to avoid concurrent processing issues. > > > > My primary concern is maintaining message ordering within the partition, > as > > Kafka guarantees ordering within a partition. and message ordering is not > > the primary concern for many applications, so can we have a flag to > enable > > multiple consumers to a single partition ? > > > > I appreciate any insights, advice, or experiences that the Kafka > community > > can share on this topic. Thank you! > > > > Regards, > > Sree Sanjeev. > > >
