Can you enable DEBUG logging in log4j and see what requests are coming in? -Jay
On Tue, Feb 4, 2014 at 9:51 PM, Carl Lerche <m...@carllerche.com> wrote: > Hi Jay, > > I do not believe that I have changed the replica.fetch.wait.max.ms > setting. Here I have included the kafka config as well as a snapshot > of jnettop from one of the servers. > > https://gist.github.com/carllerche/4f2cf0f0f6d1e891f482 > > The bottom row (89.9K/s) is the producer (it lives on a Kafka server). > The top two rows are Kafkas on other servers, you can see the combined > throughput is ~80MB/s > > On Tue, Feb 4, 2014 at 9:36 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > No this is not normal. > > > > Checking twice a second (using 500ms default) for new data shouldn't > cause > > high network traffic (that should be like < 1KB of overhead). I don't > think > > that explains things. Is it possible that setting has been overridden? > > > > -Jay > > > > > > On Tue, Feb 4, 2014 at 9:25 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > >> Hi Carl, > >> > >> For each partition the follower will also fetch data from the leader > >> replica, even if there is no new data in the leader replicas. > >> > >> One thing you can try to increase replica.fetch.wait.max.ms (default > value > >> 500ms) so that the followers's fetching request frequency to the leader > can > >> be reduced, and see if that has some effect on the traffic. > >> > >> Guozhang > >> > >> > >> On Tue, Feb 4, 2014 at 8:46 PM, Carl Lerche <m...@carllerche.com> wrote: > >> > >> > Hello, > >> > > >> > I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is > >> > for is not in full production yet, so the data written to cluster is > >> > minimal (seems to average between 100kb/s -> 300kb/s per server). I > >> > have configured Kafka to have a 3 replicas. I am noticing that each > >> > Kafka server is talking to all the others at a data rate of 40MB/s for > >> > each server (so, a total of 80MB/s for each server). This > >> > communication is constant. > >> > > >> > Is this normal? This seems like very strange behavior and I'm not > >> > exactly sure how to debug. > >> > > >> > Thanks, > >> > Carl > >> > > >> > >> > >> > >> -- > >> -- Guozhang > >> >
