Could you put together some information on this in a ticket and references this one https://issues.apache.org/jira/browse/CASSANDRA-3071
The short term fix is to disable HH. You will still get consistent reads. Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 25/08/2011, at 3:22 AM, Anand Somani wrote: > So I have looked at the cluster from > Cassandra-client - describe cluster => shows correctly - 3 nodes > used the StorageService - JMX bean =>UnreachableNodes - shows 0 > > If all these show the correct ring state, why are hints being maintained, > looks like that is the only way to find out about "phantom" nodes. > > On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani <meatfor...@gmail.com> wrote: > So, I restarted the cluster (not rolling), but it is still maintaining hints > for the IP's that are no longer part of the ring. nodetool ring shows things > correctly (as only 3 nodes). > When I check thru the jmx hintedhandoff manager, it shows it is maintaining > the hints for those non existent IP's. So the question is > - How can I remove these IP permanently, so hints do not get saved? > - Not all nodes see the same list of IP's > > > > > On Sun, Aug 21, 2011 at 3:10 PM, aaron morton <aa...@thelastpickle.com> wrote: > Yup, you can check the what HH is doing via JMX. > > there is a bug in 0.7 that can result in log files not been deleted > https://issues.apache.org/jira/browse/CASSANDRA-2829 > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 22/08/2011, at 4:56 AM, Anand Somani wrote: > >> We have a lot of space on /data, and looks like it was flushing data fine >> from file timestamps. >> >> We did have a bit of goofup with IP's when bringing up a down node (and the >> commit files have been around since then). Wonder if that is what triggered >> it and we have a bunch of hinted handoff's being backed up. >> >> For hinted hand off - how do I check if the nodes are collecting hints ( I >> do have it turned on). I noticed console bean HintedHandManager, is that the >> only way to find out. >> >> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller >> <peter.schul...@infidyne.com> wrote: >> > When does the actual commit-data file get deleted. >> > >> > The flush interval on all my memtables is 60 minutes >> >> They *should* be getting deleted when they no longer contain any data >> that has not been flushed to disk. Are flushes definitely still >> happening? Is it possible flushing has started failing (e.g. out of >> disk)? >> >> The only way I can think of over nodes directly affecting the commit >> log size on your node would be e.g. hinted handoff resulting in burst >> of writes. >> >> -- >> / Peter Schuller (@scode on twitter) >> > > >
