It may be IO, it's a good place to look http://spyced.blogspot.com/2010/01/linux-performance-basics.html
Look at the logs, when messages are dropped you will see logs from StatusLogger that include TP stats and some other things. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 7/01/2012, at 8:35 AM, Daning Wang wrote: > Thanks for your reply. > > Nodes are equally balanced. and it is RandomPartitioner. I think that machine > is slower, Are you saying it is IO issue? > > Daning > > On Fri, Jan 6, 2012 at 10:25 AM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > Are all your nodes equally balanced in terms of read requests? Are you > using RandomPartitioner? Are you reading using indexes? > > First thing you can do is compare iostat -x output between the 2 nodes > to rule out any io issues assuming your read requests are equally > balanced. > > On Fri, Jan 6, 2012 at 10:11 AM, Daning Wang <dan...@netseer.com> wrote: > > Hi all, > > > > We have 5 nodes cluster(0.8.6), but the performance from one node is way > > behind others, I checked tpstats, It always show non-zero pending ReadStage, > > I don't see this problem on other nodes. > > > > What caused the problem? I/O? Memory? Cpu usage is still low. How to fix > > this problem? > > > > ~/bin/nodetool -h localhost tpstats > > Pool Name Active Pending Completed Blocked All > > time blocked > > ReadStage 11 15 56960 > > 0 0 > > RequestResponseStage 0 0 606695 > > 0 0 > > MutationStage 0 0 538634 > > 0 0 > > ReadRepairStage 0 0 199997 > > 0 0 > > ReplicateOnWriteStage 0 0 0 > > 0 0 > > GossipStage 0 0 5734 > > 0 0 > > AntiEntropyStage 0 0 0 > > 0 0 > > MigrationStage 0 0 0 > > 0 0 > > MemtablePostFlusher 0 0 7 > > 0 0 > > StreamStage 0 0 0 > > 0 0 > > FlushWriter 0 0 8 > > 0 0 > > MiscStage 0 0 0 > > 0 0 > > FlushSorter 0 0 0 > > 0 0 > > InternalResponseStage 0 0 0 > > 0 0 > > HintedHandoff 1 4 0 > > 0 0 > > > > Message type Dropped > > RANGE_SLICE 0 > > READ_REPAIR 0 > > BINARY 0 > > READ 9082 > > MUTATION 0 > > REQUEST_RESPONSE 0 > > > > Thanks you in advance. > > > > Daning > > >
