same partitions being
backed up, just being populated more slowly barring some weird partitioning bug.
Again, thanks.
From: Joe Stein [joe.st...@stealth.ly]
Sent: Friday, November 21, 2014 12:15 AM
To: users@kafka.apache.org
Subject: Re: Elastsic Scaling
Meant to say "burst 1,000,000 messages per second on those X partitions for
10 minutes"
On Fri, Nov 21, 2014 at 12:12 AM, Joe Stein wrote:
> You need to be thoughtful about adding more partitions. This is paramount
> if you are doing semantic partitioning in which case adding more partitions
> c
You need to be thoughtful about adding more partitions. This is paramount
if you are doing semantic partitioning in which case adding more partitions
could break things downstream.
If you average lets say 100,000 messages per second and at full tilt
consumer 1:1 for each partition you can process
While it’s good to plan ahead for growth, Kafka will still let you add more
partitions to a topic
https://kafka.apache.org/081/ops.html#basic_ops_modify_topic. This will
rebalance the hashing if you are partitioning by your key, and consumers will
probably end up with different partitions, but
If you plan ahead of time with enough partitions then you won't fall into
an issue of backed up consumers when you scale them up.
If you have 100 partitions 20 consumers can read from them (each could read
from 5 partitions). You can scale up to 100 consumers (one for each
partition) as the upper
