JMX ended up just with lots more IOErrors. Did a rolling restart of the cluster 
and removed the HH family in the mean time. That seemed to do the trick. Thanks!

/Janne

On Mar 14, 2013, at 06:58 , aaron morton <aa...@thelastpickle.com> wrote:

>> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? 
>> Truncate from CLI?
> There is a JMX command to do it for a particular node. 
> But if you just want to remove all of them, stop and delete the files. 
> 
>>  the only one with zero size are the -tmp- files.  It seems odd…
> Temp files are created during compaction and flushing sstables. 
> 
> Cheers
> 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 11/03/2013, at 11:19 PM, Janne Jalkanen <janne.jalka...@ecyrd.com> wrote:
> 
>> 
>> Oops, forgot to mention that, did I… Cass 1.1.10. 
>> 
>> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? 
>> Truncate from CLI?
>> 
>> This is ls -l of my /system/HintsColumnFamily/ btw - the only one with zero 
>> size are the -tmp- files.  It seems odd…
>> 
>> -rw-rw-r--  1 ubuntu ubuntu 86373144 Jan 26 21:39 
>> system-HintsColumnFamily-hf-11-Data.db
>> -rw-rw-r--  1 ubuntu ubuntu       80 Jan 26 21:39 
>> system-HintsColumnFamily-hf-11-Digest.sha1
>> -rw-rw-r--  1 ubuntu ubuntu      976 Jan 26 21:39 
>> system-HintsColumnFamily-hf-11-Filter.db
>> -rw-rw-r--  1 ubuntu ubuntu       11 Jan 26 21:39 
>> system-HintsColumnFamily-hf-11-Index.db
>> -rw-rw-r--  1 ubuntu ubuntu     4348 Jan 26 21:39 
>> system-HintsColumnFamily-hf-11-Statistics.db
>> -rw-rw-r--  1 ubuntu ubuntu      569 Feb 27 08:33 
>> system-HintsColumnFamily-hf-23-Data.db
>> -rw-rw-r--  1 ubuntu ubuntu       80 Feb 27 08:33 
>> system-HintsColumnFamily-hf-23-Digest.sha1
>> -rw-rw-r--  1 ubuntu ubuntu     1936 Feb 27 08:33 
>> system-HintsColumnFamily-hf-23-Filter.db
>> -rw-rw-r--  1 ubuntu ubuntu       11 Feb 27 08:33 
>> system-HintsColumnFamily-hf-23-Index.db
>> -rw-rw-r--  1 ubuntu ubuntu     4356 Feb 27 08:33 
>> system-HintsColumnFamily-hf-23-Statistics.db
>> -rw-rw-r--  1 ubuntu ubuntu  5500155 Feb 27 08:57 
>> system-HintsColumnFamily-hf-24-Data.db
>> -rw-rw-r--  1 ubuntu ubuntu       80 Feb 27 08:57 
>> system-HintsColumnFamily-hf-24-Digest.sha1
>> -rw-rw-r--  1 ubuntu ubuntu       16 Feb 27 08:57 
>> system-HintsColumnFamily-hf-24-Filter.db
>> -rw-rw-r--  1 ubuntu ubuntu       26 Feb 27 08:57 
>> system-HintsColumnFamily-hf-24-Index.db
>> -rw-rw-r--  1 ubuntu ubuntu     4340 Feb 27 08:57 
>> system-HintsColumnFamily-hf-24-Statistics.db
>> -rw-rw-r--  1 ubuntu ubuntu        0 Feb 27 08:57 
>> system-HintsColumnFamily-tmp-hf-25-Data.db
>> -rw-rw-r--  1 ubuntu ubuntu        0 Feb 27 08:57 
>> system-HintsColumnFamily-tmp-hf-25-Index.db
>> 
>> 
>> /Janne
>> 
>> On Mar 12, 2013, at 08:07 , aaron morton <aa...@thelastpickle.com> wrote:
>> 
>>> What version of cassandra are you using?
>>> I would stop each node and delete the hints. If it happens again I could 
>>> either indicate a failing disk or a bug. 
>>> 
>>> Cheers
>>> 
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>> 
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>> 
>>> On 11/03/2013, at 2:13 PM, Robert Coli <robert.d.a.c...@gmail.com> wrote:
>>> 
>>>> On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen
>>>> <janne.jalka...@ecyrd.com> wrote:
>>>>> I keep seeing these in my log.  Three-node cluster, one node is working 
>>>>> fine, but two other nodes have increased latencies and these in the error 
>>>>> logs (might of course be unrelated). No obvious GC pressure, no disk 
>>>>> errors that I can see.  Ubuntu 12.04 on EC2, Java 7. Repair is run 
>>>>> regularly.
>>>>> 
>>>>> My two questions: 1) should I worry, and 2) what might be going on, and 
>>>>> 3) is there any way to get rid of these? Can I just blow my HintedHandoff 
>>>>> table to smithereens?
>>>> 
>>>> http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassandra/io/sstable/IndexHelper.java
>>>> "
>>>> public static Filter defreezeBloomFilter(FileDataInput file, long
>>>> maxSize, boolean useOldBuffer) throws IOException
>>>>    {
>>>>        int size = file.readInt();
>>>>        if (size > maxSize || size <= 0)
>>>>            throw new EOFException("bloom filter claims to be " + size
>>>> + " bytes, longer than entire row size " + maxSize);
>>>>        ByteBuffer bytes = file.readBytes(size);
>>>> "
>>>> 
>>>> Based on the above, I would suspect either a zero byte -Filter.db file
>>>> or a corrupt one. Probably worry a little bit, but only a little bit
>>>> unless your cluster is RF=1.
>>>> 
>>>> =Rob
>>> 
>> 
> 

Reply via email to