Your partitions are balanced, but is your data being evenly written across all the partitions? How are you producing data? Are you producing them with keys? Is it possible that the majority of the messages being written to just a few partitions, and so the brokers for those partitions are seeing more load than the others?
-James > On Mar 4, 2016, at 9:15 AM, Cees de Groot <c...@pagerduty.com> wrote: > > We're seeing something funny in one of our production clusters that I > cannot explain away. Everything works fine, but as we're ramping up on > Kafka, I really want to get at the root cause before we push a ton of > traffic through it :) > > We have 6 nodes over three DCs in the cluster. Currently it's running a > light load of two topics, one with small (KB) messages, one with variable > sized (KB-MB) messages, both with 64 partitions and 3 replicas. All topics, > including __consumer_offsets, have been rebalanced with a script we wrote > to make sure that the replicas are spread out over the three datacenters > and that leadership is evenly balanced, so we can continue to operate if we > lose one DC. Producers use Consul to find an initial broker (round-robin > through the local DC), Consumers use the 0.9.0.1 client. > > The funny thing is that in each DC, one broker graphs "normal" JVM heap > behavior - a sawtooth of the expected garbage creation/collection cycle. > The other one essentially stays flat. The flat-lining brokers also show > less incoming traffic when graphing the OS' received bytes. Everything else > - incoming, outgoing messages, etcetera, shows up as essentially the same > on the graphs. > > I've been digging around for a bit, but can't find anything obvious that > would cause the differences in memory pressure. Assuming that Kafka brokers > pre-allocate buffers, I'd expect not too much garbage being generated. Is > the flatline the expected behavior and the sawtooth the unexpected one? > What could cause the difference? > > Thanks for any pointers :-) > > > -- > > *Cees de Groot* > PRINCIPAL SOFTWARE ENGINEER > [image: PagerDuty logo] <http://pagerduty.com/> > pagerduty.com > c...@pagerduty.com <m...@pagerduty.com> > +1(416)435-4085 > > [image: Twitter] <http://twitter.com/pagerduty>[image: FaceBook] > <https://www.facebook.com/PagerDuty>[image: Google+] > <https://plus.google.com/114339089137644062989>[image: LinkedIn] > <https://www.linkedin.com/company/pagerduty>[image: Blog] > <https://blog.pagerduty.com/> ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
