Sure I can fill in the ticket. Here is what I have noticed so far, the count of HH is not going up, which is good. I think what must have happened is that after I restarted the cluster, no new hints were added just the old one's are still around and not cleaned up, is that possible? Cannot say for sure, since I only looked at this JMX bean abt 36 hours after restart.
Can I just clean this up using the JMX call? I do not want to turn off HH, since that can handle the intermittent network hiccups well, right? On Thu, Aug 25, 2011 at 2:47 PM, aaron morton <aa...@thelastpickle.com>wrote: > Could you put together some information on this in a ticket and references > this one https://issues.apache.org/jira/browse/CASSANDRA-3071 > > The short term fix is to disable HH. You will still get consistent reads. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 25/08/2011, at 3:22 AM, Anand Somani wrote: > > So I have looked at the cluster from > > - Cassandra-client - describe cluster => shows correctly - 3 nodes > - used the StorageService - JMX bean =>UnreachableNodes - shows 0 > > > If all these show the correct ring state, why are hints being maintained, > looks like that is the only way to find out about "phantom" nodes. > > On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <meatfor...@gmail.com>wrote: > >> So, I restarted the cluster (not rolling), but it is still maintaining >> hints for the IP's that are no longer part of the ring. nodetool ring shows >> things correctly (as only 3 nodes). >> When I check thru the jmx hintedhandoff manager, it shows it is >> maintaining the hints for those non existent IP's. So the question is >> - How can I remove these IP permanently, so hints do not get saved? >> - Not all nodes see the same list of IP's >> >> >> >> >> On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aa...@thelastpickle.com>wrote: >> >>> Yup, you can check the what HH is doing via JMX. >>> >>> there is a bug in 0.7 that can result in log files not been deleted >>> https://issues.apache.org/jira/browse/CASSANDRA-2829 >>> >>> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 22/08/2011, at 4:56 AM, Anand Somani wrote: >>> >>> We have a lot of space on /data, and looks like it was flushing data fine >>> from file timestamps. >>> >>> We did have a bit of goofup with IP's when bringing up a down node (and >>> the commit files have been around since then). Wonder if that is what >>> triggered it and we have a bunch of hinted handoff's being backed up. >>> >>> For hinted hand off - how do I check if the nodes are collecting hints ( >>> I do have it turned on). I noticed console bean HintedHandManager, is that >>> the only way to find out. >>> >>> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller < >>> peter.schul...@infidyne.com> wrote: >>> >>>> > When does the actual commit-data file get deleted. >>>> > >>>> > The flush interval on all my memtables is 60 minutes >>>> >>>> They *should* be getting deleted when they no longer contain any data >>>> that has not been flushed to disk. Are flushes definitely still >>>> happening? Is it possible flushing has started failing (e.g. out of >>>> disk)? >>>> >>>> The only way I can think of over nodes directly affecting the commit >>>> log size on your node would be e.g. hinted handoff resulting in burst >>>> of writes. >>>> >>>> -- >>>> / Peter Schuller (@scode on twitter) >>>> >>> >>> >>> >> > >
