JMX ended up just with lots more IOErrors. Did a rolling restart of the cluster and removed the HH family in the mean time. That seemed to do the trick. Thanks!
/Janne On Mar 14, 2013, at 06:58 , aaron morton <aa...@thelastpickle.com> wrote: >> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? >> Truncate from CLI? > There is a JMX command to do it for a particular node. > But if you just want to remove all of them, stop and delete the files. > >> the only one with zero size are the -tmp- files. It seems odd… > Temp files are created during compaction and flushing sstables. > > Cheers > > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 11/03/2013, at 11:19 PM, Janne Jalkanen <janne.jalka...@ecyrd.com> wrote: > >> >> Oops, forgot to mention that, did I… Cass 1.1.10. >> >> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? >> Truncate from CLI? >> >> This is ls -l of my /system/HintsColumnFamily/ btw - the only one with zero >> size are the -tmp- files. It seems odd… >> >> -rw-rw-r-- 1 ubuntu ubuntu 86373144 Jan 26 21:39 >> system-HintsColumnFamily-hf-11-Data.db >> -rw-rw-r-- 1 ubuntu ubuntu 80 Jan 26 21:39 >> system-HintsColumnFamily-hf-11-Digest.sha1 >> -rw-rw-r-- 1 ubuntu ubuntu 976 Jan 26 21:39 >> system-HintsColumnFamily-hf-11-Filter.db >> -rw-rw-r-- 1 ubuntu ubuntu 11 Jan 26 21:39 >> system-HintsColumnFamily-hf-11-Index.db >> -rw-rw-r-- 1 ubuntu ubuntu 4348 Jan 26 21:39 >> system-HintsColumnFamily-hf-11-Statistics.db >> -rw-rw-r-- 1 ubuntu ubuntu 569 Feb 27 08:33 >> system-HintsColumnFamily-hf-23-Data.db >> -rw-rw-r-- 1 ubuntu ubuntu 80 Feb 27 08:33 >> system-HintsColumnFamily-hf-23-Digest.sha1 >> -rw-rw-r-- 1 ubuntu ubuntu 1936 Feb 27 08:33 >> system-HintsColumnFamily-hf-23-Filter.db >> -rw-rw-r-- 1 ubuntu ubuntu 11 Feb 27 08:33 >> system-HintsColumnFamily-hf-23-Index.db >> -rw-rw-r-- 1 ubuntu ubuntu 4356 Feb 27 08:33 >> system-HintsColumnFamily-hf-23-Statistics.db >> -rw-rw-r-- 1 ubuntu ubuntu 5500155 Feb 27 08:57 >> system-HintsColumnFamily-hf-24-Data.db >> -rw-rw-r-- 1 ubuntu ubuntu 80 Feb 27 08:57 >> system-HintsColumnFamily-hf-24-Digest.sha1 >> -rw-rw-r-- 1 ubuntu ubuntu 16 Feb 27 08:57 >> system-HintsColumnFamily-hf-24-Filter.db >> -rw-rw-r-- 1 ubuntu ubuntu 26 Feb 27 08:57 >> system-HintsColumnFamily-hf-24-Index.db >> -rw-rw-r-- 1 ubuntu ubuntu 4340 Feb 27 08:57 >> system-HintsColumnFamily-hf-24-Statistics.db >> -rw-rw-r-- 1 ubuntu ubuntu 0 Feb 27 08:57 >> system-HintsColumnFamily-tmp-hf-25-Data.db >> -rw-rw-r-- 1 ubuntu ubuntu 0 Feb 27 08:57 >> system-HintsColumnFamily-tmp-hf-25-Index.db >> >> >> /Janne >> >> On Mar 12, 2013, at 08:07 , aaron morton <aa...@thelastpickle.com> wrote: >> >>> What version of cassandra are you using? >>> I would stop each node and delete the hints. If it happens again I could >>> either indicate a failing disk or a bug. >>> >>> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Consultant >>> New Zealand >>> >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 11/03/2013, at 2:13 PM, Robert Coli <robert.d.a.c...@gmail.com> wrote: >>> >>>> On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen >>>> <janne.jalka...@ecyrd.com> wrote: >>>>> I keep seeing these in my log. Three-node cluster, one node is working >>>>> fine, but two other nodes have increased latencies and these in the error >>>>> logs (might of course be unrelated). No obvious GC pressure, no disk >>>>> errors that I can see. Ubuntu 12.04 on EC2, Java 7. Repair is run >>>>> regularly. >>>>> >>>>> My two questions: 1) should I worry, and 2) what might be going on, and >>>>> 3) is there any way to get rid of these? Can I just blow my HintedHandoff >>>>> table to smithereens? >>>> >>>> http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassandra/io/sstable/IndexHelper.java >>>> " >>>> public static Filter defreezeBloomFilter(FileDataInput file, long >>>> maxSize, boolean useOldBuffer) throws IOException >>>> { >>>> int size = file.readInt(); >>>> if (size > maxSize || size <= 0) >>>> throw new EOFException("bloom filter claims to be " + size >>>> + " bytes, longer than entire row size " + maxSize); >>>> ByteBuffer bytes = file.readBytes(size); >>>> " >>>> >>>> Based on the above, I would suspect either a zero byte -Filter.db file >>>> or a corrupt one. Probably worry a little bit, but only a little bit >>>> unless your cluster is RF=1. >>>> >>>> =Rob >>> >> >
