The following may help: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Howtoimprovethethroughputofaremoteconsumer%3F
Thanks, Jun On Thu, Aug 29, 2013 at 4:47 PM, Walter King <wal...@adroll.com> wrote: > I'm having a lot of trouble getting throughput of mirrormaker to be > consistent. On ec2, I have ~10 topics with pretty different throughputs, > mirroring from 4 regions into 1. Total of about 20MBps. Sometimes it seems > to be going fine, and then it seems like one of the consumers stops > altogether, and restarting the process will let it catch back up. > Sometimes I'll see errors in the logs about mirrormaker threads being > stopped, and it doesn't seem like they ever get replaced. > > Whats the recommended setup for mirrormaker? I've tried changing number of > streams, producers, adding servers, splitting processes by topic or region, > all to varying degrees of effect but still seem to just lose a partition > periodically. > > Thanks >
