The only other thing being written to these disks is log4j (kafka.out), so technically it is not dedicated to the data logs. The disks are 250GB SATA.
On Fri, Apr 26, 2013 at 6:35 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > > - Decreased num.partitions and log.flush.interval on the brokers from > > 64/10k to 32/100 in order to lower the average flush time (we were > > previously always hitting the default flush interval since no > partitions > > Hmm, that is a pretty low value for flush interval leading to higher disk > usage. Do you use dedicated disks for kafka data logs ? Also what sort of > disks do you use ? > > Thanks, > Neha > > > > > > > On Tue, Apr 23, 2013 at 7:53 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > You can run kafka.tools.ConsumerOffsetChecker to check the consumer > lag. If > > > the consumer is lagging, this indicates a problem on the consumer side. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Mon, Apr 22, 2013 at 9:13 PM, Andrew Neilson <arsneil...@gmail.com > > > >wrote: > > > > > > > Hmm it is highly unlikely that that is the culprit... There is lots > of > > > > bandwidth available for me to use. I will definitely keep that in > mind > > > > though. I was working on this today and have some tidbits of > additional > > > > information and thoughts that you might be able to shed some light > on: > > > > > > > > - I mentioned I have 2 consumers, but each consumer is running > with 8 > > > > threads for this topic (and each consumer has 8 cores available). > > > > - When I initially asked for help the brokers were configured with > > > > num.partitions=1, I've since tried higher numbers (3, 64) and > haven't > > > > seen > > > > much of an improvement aside from forcing both consumer apps to > handle > > > > messages (with the overall performance not changing much). > > > > - I ran into this article > > > > > > > > > > > > > http://riccomini.name/posts/kafka/2012-10-05-kafka-consumer-memory-tuning/and > > > > tried a variety of options of queuedchunks.max and fetch.size with no > > > > significant results (simply meaning it did not achieve the goal of > me > > > > constantly processing hundreds or thousands of messages per > second, > > > > which > > > > is similar to the rate of input). I would not be surprised if I'm > > > wrong > > > > but > > > > this made me start to think that the problem may lie outside of > the > > > > consumers > > > > - Would the combination of a high number of partitions (64) and a > high > > > > log.flush.interval (10k) prevent logs from flushing as often as > they > > > > need > > > > to for my desired rate of consumption (even with > > > > log.default.flush.interval.ms=1000?) > > > > > > > > Despite the changes I mentioned the behaviour is still the consumers > > > > receiving larger spikes of messages mixed with periods of complete > > > > inactivity and overall a long delay between messages being written > and > > > > messages being read (about 2 minutes). Anyway... as always I greatly > > > > appreciate any help. > > > > > > > > On Sun, Apr 21, 2013 at 8:50 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > Is your network shared? Is so, another possibility is that some > other > > > > apps > > > > > are consuming the bandwidth. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Sun, Apr 21, 2013 at 12:23 PM, Andrew Neilson < > arsneil...@gmail.com > > > > > >wrote: > > > > > > > > > > > Thanks very much for the reply Neha! So I swapped out the > consumer > > > that > > > > > > processes the messages with one that just prints them. It does > indeed > > > > > > achieve a much better rate at peaks but can still nearly zero out > (if > > > > not > > > > > > completely zero out). I plotted the messages printed in graphite > to > > > > show > > > > > > the behaviour I'm seeing (this is messages printed per second): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.dropbox.com/s/7u7uyrefw6inetu/Screen%20Shot%202013-04-21%20at%2011.44.38%20AM.png > > > > > > > > > > > > The peaks are over ten thousand per second and the troughs can go > > > below > > > > > 10 > > > > > > per second just prior to another peak. I know that there are > plenty > > > of > > > > > > messages available because the ones currently being processed are > > > still > > > > > > from Friday afternoon, so this may or may not have something to > do > > > with > > > > > > this pattern. > > > > > > > > > > > > Is there anything I can do to avoid the periods of lower > performance? > > > > > > Ideally I would be processing messages as soon as they are > written. > > > > > > > > > > > > > > > > > > On Sun, Apr 21, 2013 at 8:49 AM, Neha Narkhede < > > > > neha.narkh...@gmail.com > > > > > > >wrote: > > > > > > > > > > > > > Some of the reasons a consumer is slow are - > > > > > > > 1. Small fetch size > > > > > > > 2. Expensive message processing > > > > > > > > > > > > > > Are you processing the received messages in the consumer ? Have > you > > > > > > > tried running console consumer for this topic and see how it > > > performs > > > > > > > ? > > > > > > > > > > > > > > Thanks, > > > > > > > Neha > > > > > > > > > > > > > > On Sun, Apr 21, 2013 at 1:59 AM, Andrew Neilson < > > > > arsneil...@gmail.com> > > > > > > > wrote: > > > > > > > > I am currently running a deployment with 3 brokers, 3 ZK, 3 > > > > > producers, > > > > > > 2 > > > > > > > > consumers, and 15 topics. I should first point out that this > is > > > my > > > > > > first > > > > > > > > project using Kafka ;). The issue I'm seeing is that the > > > consumers > > > > > are > > > > > > > only > > > > > > > > processing about 15 messages per second from what should be > the > > > > > largest > > > > > > > > topic it is consuming (we're sending 200-400 ~300 byte > messages > > > per > > > > > > > second > > > > > > > > to this topic). I should note that I'm using a high level ZK > > > > consumer > > > > > > and > > > > > > > > ZK 3.4.3. > > > > > > > > > > > > > > > > I have a strong feeling I have not configured things properly > so > > > I > > > > > > could > > > > > > > > definitely use some guidance. Here is my broker > configuration: > > > > > > > > > > > > > > > > brokerid=1 > > > > > > > > port=9092 > > > > > > > > socket.send.buffer=1048576 > > > > > > > > socket.receive.buffer=1048576 > > > > > > > > max.socket.request.bytes=104857600 > > > > > > > > log.dir=/home/kafka/data > > > > > > > > num.partitions=1 > > > > > > > > log.flush.interval=10000 > > > > > > > > log.default.flush.interval.ms=1000 > > > > > > > > log.default.flush.scheduler.interval.ms=1000 > > > > > > > > log.retention.hours=168 > > > > > > > > log.file.size=536870912 > > > > > > > > enable.zookeeper=true > > > > > > > > zk.connect=XXX > > > > > > > > zk.connectiontimeout.ms=1000000 > > > > > > > > > > > > > > > > Here is my producer config: > > > > > > > > > > > > > > > > zk.connect=XXX > > > > > > > > producer.type=async > > > > > > > > compression.codec=0 > > > > > > > > > > > > > > > > Here is my consumer config: > > > > > > > > > > > > > > > > zk.connect=XXX > > > > > > > > zk.connectiontimeout.ms=100000 > > > > > > > > groupid=XXX > > > > > > > > autooffset.reset=smallest > > > > > > > > socket.buffersize=1048576 > > > > > > > > fetch.size=10485760 > > > > > > > > queuedchunks.max=10000 > > > > > > > > > > > > > > > > Thanks for any assistance you can provide, > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > >
