We don't have such throttling right now. Could you file a jira? Thanks,
Jun On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez <mjua...@gmail.com>wrote: > Hi, > > We have several Kafka clusters in production, and we've had to reassign > replication a few times now in production. Some of our topic/partitions > are pretty large, up to 32 partitions per topic, and 16GB per partition, so > adding a new broker and/or repairing a broker that had been down for some > time turns out to be a major undertaking. > > Today, when we attempt to replicate a single partition, it pegs the disk > IO, and uses a significant chunk of the 10Gbps interface for a good ~5 > minutes. This is causing problems for our downstream consumers, which rely > on having a consistent stream of realtime data being sent to them. > > Is there a way to throttle Kafka replication between nodes, so that instead > of it going full blast, it will replicate at a fixed rate in megabytes or > activities/batches per second? Or maybe is this planned for a future > release, maybe 0.9? > > Thanks, > > Marcos Juarez >
