yeh mostly they are up always, sometimes when we do new deployment they might be down for about 15 mins or so. seems to have become slightly better now, i changed the replica socket buffer size and also the number of replica fetcher threads.
On Mon, May 23, 2016 at 6:24 PM, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > Are you sure consumers are always up, when they are behind they could > generate a lot of traffic in a small amount of time? > > On Mon, May 23, 2016 at 9:11 AM Anishek Agarwal <anis...@gmail.com> wrote: > > > additionally all the read / writes are happening via storm topologies. > > > > On Mon, May 23, 2016 at 12:17 PM, Anishek Agarwal <anis...@gmail.com> > > wrote: > > > > > Hello, > > > > > > we are using 4 kafka machines in production with 4 topics and each > topic > > > either 16/32 partitions and replication factor of 2. each machine has 3 > > > disks for kafka logs. > > > > > > we see a strange behaviour where we see high disk usage spikes on one > of > > > the disks on all machines. it varies over time, with different disks > > > showing spikes over time. is this normal .. i thought the partition per > > > topic is distributed per disk and hence unless traffic is such that its > > > hitting one partition this should not happen. > > > > > > any suggestions on how to verify what is happening will be helpful. > > > > > > Thanks > > > anishek > > > > > > > > >
