Thanks for your reply Rob, I am back to this after a while...
I am not sure if this is different in 1.2.18, but I remember from older
versions that GC pauses would only be logged in the /system.log/ if
their duration /was >= 200ms/. Also, when hints are detected, we cannot
correlate it with GC pauses. We are thinking of tweaking the GC logging
settings in the /cassandra-env/ file, but we are unsure as to which ones
are going to be heavy for the server and which ones are safer to modify.
Would you be able to advice on this?
The hints issue we seem to have, is not catastrophic in the sense that
it is not causing serious/obvious problems to the clients, but makes us
feel rather uncomfortable with the overall cluster health because, as
you said, is a warning sign that something is wrong. It doesn't happen
very often either, but I don't think this makes the situation any
better. Apart from increasing the GC logging, I don't see any other way
of debugging this further.
Thanks for your input,
Vasilis
On 20/01/15 22:53, Robert Coli wrote:
On Sat, Jan 17, 2015 at 3:32 PM, Vasileios Vlachos
<vasileiosvlac...@gmail.com <mailto:vasileiosvlac...@gmail.com>> wrote:
Is there any other occasion that hints are stored and then being
sent in a cluster, other than network or other temporary or
permanent failure? Could it be that the client responsible for
establishing a connection is causing this? We use the Datastax C#
driver for connecting to the cluster and we run C* 1.2.18 on
Ubuntu 12.04.
Other than restarting nodes manually (which I consider a "temporary
failure" for the purposes of this question), no. Seeing hints being
stored and delivered outside of this context is a warning sign that
something may be wrong with your cluster.
Probably what is happening is that you have stop the world GCs long
enough to trigger queueing of hints via timeouts during these GCs.
=Rob
