Sounds like the parallel consumer is what you want:
https://github.com/confluentinc/parallel-consumer
Or at least decide why that won't work before writing something yourself.
- alex
On Tue, Oct 3, 2023 at 4:20 PM Sree Sanjeev Kandasamy Gokulrajan <
sreesanjee...@gmail.com> wrote:
> Hey Neeraj
Hey Neeraj
I had a kafka assignment where I had to consume from a single partition.
When I was looking to scale the application, I was planning to consume the
partition with more than one consumer when I came across this issue.
Regards,
Sree Sanjeev.
On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya
Hi Sree Sanjeev,
Other than the info from Colt, why can't you just increase partitions and scale
the number of consumers accordingly ?
Or build your application as a KStreams application and take advantage of
horizontal and vertical scalability ?
Regards,
Neeraj
On Wednesday, 4 October,
Hi Sree.
Yeah, parallel-processing-per-partition requirement often arises
particularly when the throughput is limited due to external I/O latency and
there are some solutions:
- https://github.com/line/decaton
* provides per-key ordering guarantee (or can be unordered if no
ordering is necess
Hello,
Currently, you cannot have more than one consumer consuming from a
partition in Kafka. However, there is active discussion about a proposal to
enable queueing semantics for Kafka, which is a superset of the feature you
are talking about. Feel free to check KIP-932:
https://cwiki.apache.org/
I'm interested in knowing if we can achieve parallelism by allowing
multiple consumers to subscribe to a single partition of a topic.
To explore potential solutions, I'm considering the following approach.
1. Implementing a locking mechanism to control access to the offsets in a
partition to avoi
