I suspect these partial/invalid hints are left over from a failed
hints delivery from before you upgraded to 1.0.3 and not something
created by 1.0.3.
Try to clear HintsColumnFamily (by removing the sstables for example)
first and then see if you still can reproduce this issue afterwards.
it still does not work, but there is an improvement over 1.0.2.
currently it looks like this:
INFO [HintedHandoff:1] 2011-11-15 06:42:59,794
HintedHandOffManager.java (line 334) Finished hinted handoff of 0 rows
to endpoint /
but there is lot of data not delivered. forced compaction shows it:
112) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-63-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-61-Data.db')]
INFO [CompactionExecutor:41] 2011-11-15 07:08:24,034
CompactionController.java (line 129) Compacting large row
system/HintsColumnFamily:0758480e325aa09951c555e1249a69ba (237411479
bytes) incrementally
INFO [CompactionExecutor:41] 2011-11-15 07:08:33,418
CompactionTask.java (line 213) Compacted to
[/var/lib/cassandra/data/system/HintsColumnFamily-hb-64-Data.db,].
237,411,643 to 237,411,505 (~99% of original) bytes for 1 keys at
24.091642MB/s. Time: 9,398ms.
after node restart it still shows 0 rows delivered. hints should be
newly written by 1.0.3. i deleted old rows from older versions in hints
- only 2 tombstones left there before another batch of hints testing.
hints inside are not from supercolumn CF. I remember to have 2 rows in
HintsColumnFamily, one was successfully delivered and this one seems to
be stuck. These hints are older than 1 hour.
Did you tested outgoing hints delivery with more than row there? its
probably 1 key = 1 hints delivery session and if node is back up for
while then second hints session is created. Try to do your testing with
about 500k hints with multiple sessions.
