Hi Rahul,
Thanks for your reply.
I have never seen message like "Timed out replaying hints to...", which is
a good thing then, I suppose ;)
Normally, I do see the "Finished hinted handoff..." log message. However,
every now and then this message is not logged, not even after several
hours. This is the problem I'm trying to solve.
The log messages you describe are quite course-grained; they only tell you
that a task has started or finished, but not how this task is progressing.
And that's exactly what I would like to know if I see that a task has
started, but has not finished after a reasonable amount of time.
So I guess the only way to see learn the progress is to look inside the
'hints' column family then.I'll give that a try.
Thanks,
Tom
On Tue, Dec 3, 2013 at 1:43 PM, Rahul Menon <ra...@apigee.com> wrote:
> Tom,
>
> You should check the size of the hints column family to determine how much
> are present. The hints are a super column family and its keys are
> destination tokens. You could look at it if you would like.
>
> Hints send and timedouts are logged, you should be seeing something like
>
> Timed out replaying hints to {}; aborting ({} delivered
>
>
>
> OR
>
> Finished hinted handoff of {} rows to endpoint {}
>
>
>
> Thanks
> Rahul
>
>
> On Tue, Dec 3, 2013 at 2:36 PM, Tom van den Berge <t...@drillster.com>wrote:
>
>> Hi,
>>
>> Is there a way to monitor the progress of a hinted handoff task?
>>
>> I found the following two mbeans providing some info:
>>
>> org.apache.cassandra.internal:type=HintedHandoff, which tells me that
>> there is 1 active task, and
>> org.apache.cassandra.db:type=HintedHandoffManager#countPendingHints(),
>> which quite often gives a timeout when executed.
>>
>> Ideally, I would like to see how many hints have been sent (e.g. over the
>> last minute or so), and how many hints are still to be sent (although I
>> assume that's what countPendingHints normally does?)
>>
>> I'm experiencing hinted handoff tasks that are started, but never finish,
>> so I would like to know what the task is doing.
>>
>> My log shows this:
>>
>> INFO [HintedHandoff:1] 2013-12-02 13:49:05,325 HintedHandOffManager.java
>> (line 297) Started hinted handoff for host:
>> 6f80b942-5b6d-4233-9827-3727591abf55 with IP: /10.55.156.66
>> (nothing more for [HintedHandoff:1])
>>
>> The node is up and running, the network connection is ok, no gossip
>> messages appear in the logs.
>>
>> Any idea is welcome.
>> (Casandra 1.2.3)
>>
>>
>>
>>
>> --
>>
>> Drillster BV
>> Middenburcht 136
>> 3452MT Vleuten
>> Netherlands
>>
>> +31 30 755 5330
>>
>> Open your free account at www.drillster.com
>>
>
>
--
Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands
+31 30 755 5330
Open your free account at www.drillster.com
