Hi,
On 20/10/2015 19:48, Carlos Alonso wrote:
> I think also having the output of cfhistograms could help. I'd like to
> know how many sstables are being hit during reads.
Ooops, my bad I thought you mentioned proxyhistograms.
Here's the worst node cfhistograms:
nodetool cfhistograms akka messages
akka/messages histograms
Percentile SSTables Write Latency Read Latency Partition Size
Cell Count
(micros) (micros) (bytes)
50% 1.00 60.00 310.00 535
1
75% 2.00 103.00 9887.00 11864
17
95% 3.00 14237.00 126934.00 73457
215
98% 3.00 51012.00 219342.00 263210
446
99% 3.00 61214.00 1131752.00 454826
924
Min 0.00 6.00 18.00 73
0
Max 6.00 263210.00 5839588.00 10090808
1109
> Also, which CL are you reading with?
For the test request in cqlsh, I'm using ONE, but the application in which I
first observed the issue was using QUORUM.
> cfstats is a local command, so maybe that node you've printed is working
> fine but there's another that is causing the latency. Can you check that
> command in all nodes?
I checked and on all nodes, the read latency and read local latency are within
15 to 40ms.
I also noticed that C* was taking a fair bit of CPU on some of the nodes
(ranging from 60% to 200%), looking at ttop output it was mostly taken by
SharedPool-Worker threads, which I assume are the thread that are doing the
real query work.
Well, I'm puzzled, and I'll keep searching, thanks for your help!
--
Brice Figureau
