Hi,
We are running a small test cluster and recently installed Cassandra 1.1 and
started with a new clean database. We keep seeing these messages in the log
on just one of our nodes:
INFO [HintedHandoff:1] 2012-05-21 09:49:56,757 HintedHandOffManager.java (line
294) Started hinted handoff for token: 85070591730234615865843651857942052864
with IP: /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 09:49:56,775 HintedHandOffManager.java (line
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 09:59:56,756 HintedHandOffManager.java (line
294) Started hinted handoff for token: 42535295865117307932921825928971026432
with IP: /10.0.0.69
INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java (line
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
INFO [HintedHandoff:1] 2012-05-21 09:59:56,757 HintedHandOffManager.java (line
294) Started hinted handoff for token: 85070591730234615865843651857942052864
with IP: /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 09:59:56,775 HintedHandOffManager.java (line
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 10:09:56,757 HintedHandOffManager.java (line
294) Started hinted handoff for token: 42535295865117307932921825928971026432
with IP: /10.0.0.69
INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java (line
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.69
INFO [HintedHandoff:1] 2012-05-21 10:09:56,758 HintedHandOffManager.java (line
294) Started hinted handoff for token: 85070591730234615865843651857942052864
with IP: /10.0.0.73
INFO [HintedHandoff:1] 2012-05-21 10:09:56,879 HintedHandOffManager.java (line
382) Finished hinted handoff of 0 rows to endpoint /10.0.0.73
All four nodes are up:
-bash-4.1$ nodetool ring -h localhost
Note: Ownership information does not include topology, please specify a
keyspace.
Address DC Rack Status State Load Owns
Token
127605887595351923798765477786913079296
10.0.0.65 datacenter1 rack1 Up Normal 244.41 MB 25.00%
0
10.0.0.69 datacenter1 rack1 Up Normal 155.39 MB 25.00%
42535295865117307932921825928971026432
10.0.0.73 datacenter1 rack1 Up Normal 220.42 MB 25.00%
85070591730234615865843651857942052864
10.0.0.77 datacenter1 rack1 Up Normal 296.14 MB 25.00%
127605887595351923798765477786913079296
This has been going on for days. Note that it's just two key's in the log
that keep repeating. No recent messages about HintedHandOff in the logs
on the other nodes.
Let me know if you need more info.
Arend-Jan
--
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl
