I *think* this may be ghost rows which have not being compacted. How many SSTables are on disk for the HintedHandoff CF ?
Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 19/10/2012, at 7:16 AM, David Daeschler <david.daesch...@gmail.com> wrote: > Hi Steve, > > Also confirming this. After having a node go down on Cassandra 1.0.8 > there seems to be hinted handoff between two of our 4 nodes every 10 > minutes. Our setup also shows 0 rows. It does not appear to have any > effect on the operation of the ring, just fills up the log files. > > - David > > > > On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <spie...@verifyle.com> wrote: >> I installed Cassandra on three nodes. I then ran a test suite against them >> to generate load. The test suite is designed to generate the same type of >> load that we plan to have in production. As one of many tests, I reset one >> of the nodes to check the failure/recovery modes. Cassandra worked just >> fine. >> >> >> >> I stopped the load generation, and got distracted with some other >> project/problem. A few days later, I noticed something strange on one of the >> nodes. On this node hinted handoff starts every ten minutes, and while it >> seems to finish without any errors, it will be started again in ten minutes. >> None of the nodes has any traffic, and hasn’t for several days. I checked >> the logs, and this goes back to the initial failure/recovery testing: >> >> >> >> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java >> (line 294) Started hinted handoff for token: >> 113427455640312821154458202477256070484 with IP: /192.168.128.136 >> >> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java >> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136 >> >> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java >> (line 294) Started hinted handoff for token: >> 113427455640312821154458202477256070484 with IP: /192.168.128.136 >> >> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java >> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136 >> >> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java >> (line 294) Started hinted handoff for token: >> 113427455640312821154458202477256070484 with IP: /192.168.128.136 >> >> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java >> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136 >> >> >> >> The other nodes are happy and don’t show this behavior. All the test data is >> readable, and everything is fine, but I’m curious why hinted handoff is >> running on one node all the time. >> >> >> >> I searched the bug database, and I found a bug that seems to have the same >> symptoms: >> >> https://issues.apache.org/jira/browse/CASSANDRA-3733 >> >> Although it’s been marked fixed in 0.6, this describes my problem exactly. >> >> >> >> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0: >> >> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm >> >> >> >> Is anyone else seeing this behavior? What can I do to provide more >> information? >> >> >> >> Steve >> >> > > > > -- > David Daeschler
