If I trigger hint delivery using JMX, it works. I see in the log: 2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for endpoint /172.16.108.19 However, it does not work automatically (i.e. after node 3 is up again).
I tested it with Cassandra 0.8.6 and there it works as expected. Unfortunately, we have 0.7.7. in production. Thanks! Rene From: aaron morton [mailto:aa...@thelastpickle.com] Sent: donderdag 6 oktober 2011 22:51 To: user@cassandra.apache.org Subject: Re: Hinted handoff question At debug level you should see some log messages such as... "Checking remote schema before delivering hints..." "Sleeping {}ms to stagger hint delivery..." "Endpoint {} died before hint delivery, aborting..." "Started hinted handoff for endpoint..." If you want to move on, you can deliver the hints using JMX or the node should start to deliver them if you restart it (i think). Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6/10/2011, at 11:40 PM, Rene Kochen wrote: Node 3 is up (using ring on node 1). There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0). This is the log from node 1 when nodes 3 starts: 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted, now UP again 2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal, token 170141183460469000000000000000000000000 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to normal 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints 2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes -> empty pending ranges for Traxis 2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to /172.16.108.19 2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ... 2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ... 2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ... The countPendingHints on node 1 is {172.16.108.19=5176} Thanks Rene From: aaron morton [mailto:aa...@thelastpickle.com] Sent: donderdag 6 oktober 2011 12:13 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Hinted handoff question Check that node 1 sees node 3 as UP (via ring). Check the tpstats on node 1, is there an active HH task ? Take another crawl through the logs. Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 6/10/2011, at 10:35 PM, Rene Kochen wrote: I'am using Cassandra 0.7.7 and have a question about hinted handoff. I have a cluster of three nodes. I stop node 3. I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709). However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery. If I look at the countPendingHints on node 1, it still says 28709. hinted_handoff_enabled is enabled. Thanks!
