> Any reason why cassandra might be reading a lot from the data disks(not > the commit log disk) more than usual? On the new node or all nodes ?
Maybe cold Key Cache or cold memmapped files due to a change in the data distribution ? Did it settle down ? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 14/05/2013, at 5:06 AM, "Hiller, Dean" <dean.hil...@nrel.gov> wrote: > Ah, okay iostat -x NEEDS a number like "iostat -x 5" works better(first > one always shows 4% util while second one shows 100%). Iotop seems a bit > better here. > > So we know that since we added our new node, we are slammed with read and > no one is running compations according to "clush -g datanodes nodetool > compactionstats" > > Any reason why cassandra might be reading a lot from the data disks(not > the commit log disk) more than usual? > > Thanks, > Dean > > On 5/13/13 10:46 AM, "Hiller, Dean" <dean.hil...@nrel.gov> wrote: > >> We running a pretty consistent load on our cluster and added a new node >> to a 6 node cluster Friday(QA worked great, but production not so much). >> One mistake that was made was starting up the new node, then disabling >> the firewall :( which allowed nodes to discover it BEFORE the node >> bootstrapped itself. We shutdown the node and booted him up and he >> bootstrapped himself streaming all the data in. >> >> After that though, all the ndoes have really really high load numbers >> now. We are trying to figure out what is going on still. >> >> Is there any way to get the number of reads/second and writes/second >> through JMX or something? The only way I can see of on doing this is >> manually calculating it by timing the read count and dividing by my >> manual stop watches start/stop times(timerange). >> >> Also, while my load is load average: 20.31, 19.10, 19.72 , what does a >> normal iostat look like? My iostat await time is 13.66 ms which I think >> is kind of bad, but not that bad to cause a load of 20.31? >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s >> avgrq-sz avgqu-sz await svctm %util >> sda 0.02 0.07 11.70 1.96 1353.67 702.88 >> 150.58 0.19 13.66 3.61 4.93 >> sdb 0.00 0.02 0.11 0.46 20.72 97.54 >> 206.70 0.00 1.33 0.67 0.04 >> >> Thanks, >> Dean >
