Hello,
I thought hints are being stored on /node_A/ every time /node_B/ is
unavailable for whatever the reason. I also thought that these hints are
being delivered from /node_A/ to /node_B/ when /node_B/ is back and this
is true for a /period = max_hint_window_in_ms/. After that hints are
dropped and therefore never delivered to /node_B/.
Obviously I am wrong, because occasionally we get alerted from our
monitoring system that hints are being stored and delivered, which as
far as I know indicates a problem. Now, when that happens I cannot
correlate it with any network issues (all nodes are on the same LAN
anyway) or other problems. The output from /system.log/ looks like this:
INFO [CompactionExecutor:109085] 2015-01-17 15:35:13,536
CompactionTask.java (line 262) Compacted 2 sstables to
[/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quoteby
quotereference-ic-89765,]. 222,905,570 bytes to 222,881,859 (~99% of
original) in 91,850ms = 2.314172MB/s. 161,259 total rows, 161,253
unique. Row merge counts were {1:161247, 2:6, }
INFO [CompactionExecutor:109090] 2015-01-17 15:35:13,537
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-
quotebyquotereference-ic-89750-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quotebyquotereference-ic-89765-Data.db')]
INFO [HintedHandoff:2] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
2ae2c679-8769-44da-a713-3bc21c670620 with IP: /10.3.5.3
INFO [HintedHandoff:1] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
0bb63124-6333-43fa-b1c8-3a8f6627b85a with IP: /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,967
HintedHandOffManager.java (line 326) Finished hinted handoff of 17 rows
to endpoint /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,968 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1779218028(614406/2848765
serialized/live bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:38,969 Memtable.java (line
398) Writing Memtable-hints@1779218028(614406/2848765 serialized/live
bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:39,192 Memtable.java (line
436) Completed flushing
/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db (176861
bytes) for commitlog position ReplayPosition(segmentId=1418136927153,
position=20201767)
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,194
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db')]
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,485
CompactionTask.java (line 262) Compacted 1 sstables to
[/var/lib/cassandra/data/system/hints/system-hints-ic-90,]. 176,861
bytes to 177,355 (~100% of original) in 290ms = 0.583238MB/s. 4 total
rows, 3 unique. Row merge counts were {1:4, }
INFO [HintedHandoff:1] 2015-01-17 15:35:39,485
HintedHandOffManager.java (line 294) Started hinted handoff for host:
6b99058f-ba48-42b9-baa1-a878a74338cc with IP: /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,084
HintedHandOffManager.java (line 326) Finished hinted handoff of 22 rows
to endpoint /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,085 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1204004752(2356/10923
serialized/live bytes, 62 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:40,085 Memtable.java (line
398) Writing Memtable-hints@1204004752(2356/10923 serialized/live bytes,
62 ops)
Is there any other occasion that hints are stored and then being sent in
a cluster, other than network or other temporary or permanent failure?
Could it be that the client responsible for establishing a connection is
causing this? We use the Datastax C# driver for connecting to the
cluster and we run C* 1.2.18 on Ubuntu 12.04.
Many thanks,
Vasilis
- Should one expect to see hints being stored/delivered oc... Vasileios Vlachos
-
