Re: Hintedhandoff mutation

2016-08-17 Thread Chris Lohfink
Probably question better suited for the dev@ list. But I afaik the answer is there is no way to tell the difference, but probably safe to look at the created time, HHs tend to be older. Chris On Wed, Aug 17, 2016 at 5:02 AM, Stone Fang wrote: > Hi All, > > I want to differ hintedhandoff mutatio

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread Allan C
 Certainly makes sense to not allow it. Any idea why the node would be holding hints for tokens that don’t exist? -Allan On January 20, 2014 at 1:09:51 PM, sankalp kohli (kohlisank...@gmail.com) wrote: Yes as per code you cannot delete hints for endpoints which are not part of the ring.   if

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread sankalp kohli
Yes as per code you cannot delete hints for endpoints which are not part of the ring. if (!StorageService.instance.getTokenMetadata().isMember(endpoint)) return; On Mon, Jan 20, 2014 at 12:34 PM, Allan C wrote: > There are 3 other nodes that have a mild case. This is one node is w

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread Allan C
There are 3 other nodes that have a mild case. This is one node is worse by an order of magnitude. deleteHintsForEndpoint fails with the same error  on any of the affected nodes. -Allan On January 20, 2014 at 12:24:33 PM, sankalp kohli (kohlisank...@gmail.com) wrote: Is this happening in one

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread sankalp kohli
Is this happening in one node or all. Did you try to delete the hints via JMX in other nodes? On Mon, Jan 20, 2014 at 12:18 PM, Allan C wrote: > Hi , > > I’m hitting a very odd issue with HintedHandoff on 1 node in my 12 node > cluster running 1.2.13. Somehow it’s holding a large amount of hint

Re: HintedHandoff process does not finish

2013-09-30 Thread Aaron Morton
> What can be the reason for the handoff process not to finish? Check for other errors about timing out during hint reply. > What would be the best way to recover from this situation? If they are really causing trouble drop the hints via HintedHandoffManager JMX MBean or stopping the node and de

Re: HintedHandoff

2013-05-09 Thread aaron morton
> ·If node ‘X ‘ in DC1 which is a ‘replica’ node is down and a write > comes with CL =1 to DC1, the co-ordinator node will write the hint and also > the data will be written to the other ‘replica’ node in DC2 ? Is this correct > ? Writes always go to all UP replicas. So yes. > ·

RE: HintedHandoff

2013-05-08 Thread Kanwar Sangha
Is this correct guys ? From: Kanwar Sangha [mailto:kan...@mavenir.com] Sent: 07 May 2013 14:07 To: user@cassandra.apache.org Subject: HintedHandoff Hi -I had a question on hinted-handoff. We have 2 DCs configured with overall RF = 2 (DC1:1, DC2:1) and 4 nodes in each DC (total - 8 nodes across

Re: HintedHandoff IOError?

2013-03-15 Thread Janne Jalkanen
JMX ended up just with lots more IOErrors. Did a rolling restart of the cluster and removed the HH family in the mean time. That seemed to do the trick. Thanks! /Janne On Mar 14, 2013, at 06:58 , aaron morton wrote: >> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? >

Re: HintedHandoff IOError?

2013-03-13 Thread aaron morton
> What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? > Truncate from CLI? There is a JMX command to do it for a particular node. But if you just want to remove all of them, stop and delete the files. > the only one with zero size are the -tmp- files. It seems odd… Temp fi

Re: HintedHandoff IOError?

2013-03-11 Thread Janne Jalkanen
Oops, forgot to mention that, did I… Cass 1.1.10. What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? Truncate from CLI? This is ls -l of my /system/HintsColumnFamily/ btw - the only one with zero size are the -tmp- files. It seems odd… -rw-rw-r-- 1 ubuntu ubuntu 863731

Re: HintedHandoff IOError?

2013-03-11 Thread aaron morton
What version of cassandra are you using? I would stop each node and delete the hints. If it happens again I could either indicate a failing disk or a bug. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 11/03/2013,

Re: HintedHandoff IOError?

2013-03-11 Thread Robert Coli
On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen wrote: > I keep seeing these in my log. Three-node cluster, one node is working fine, > but two other nodes have increased latencies and these in the error logs > (might of course be unrelated). No obvious GC pressure, no disk errors that I > can

Re: HintedHandoff increases in read?

2011-03-20 Thread aaron morton
Can we get some more information... - What is countPendingHints showing ? Are they all for the same row ? - What about listEndpointsPendingHints are their different end points listed there ? - Can you turn up the logging to DEBUG on one of the machines that has the increasing number of hints ?

Re: HintedHandoff and ReplicationFactor with a downed node

2010-10-22 Thread Dan Washusen
The last time this came up on the list Jonathan Ellis said (something along the lines of) if your application can't tolerate stale data then you should read with a consistency level of QUORUM. It would be nice if there was some sort of middle ground for an application that can tolerate slightly st

Re: HintedHandoff and ReplicationFactor with a downed node

2010-10-22 Thread Rob Coli
On 10/22/10 2:55 PM, Craig Ching wrote: Even better, I'd love a way to not allow B to be available until replication is complete, can I detect that somehow? Proposed and rejected a while back : https://issues.apache.org/jira/browse/CASSANDRA-768 =Rob

Re: Hintedhandoff will never complete when a BIG rowmutation

2010-07-15 Thread Schubert Zhang
Yes, I think current HintedHandOff implementation in 0.6.x cannot support large hints, it is a risk in a production system. On Tue, Jun 29, 2010 at 12:31 AM, albert_e wrote: > In 0.6.2, HH sending MUTATION message using the same OutboundTcpConnection > with READ message. When HH transfering big

Re: Hintedhandoff will never complete when a BIG rowmutation

2010-06-28 Thread Robert Coli
On 6/28/10 3:29 AM, Lu Ming wrote: > Every one hour HintedHandOffManager will check hintedhandoff > ColumnFamily then send out the big rowmutations to alive nodes, > It fails again because of the TimeoutException, so the task will never > finish and the big rowmutation is sending again and again. >

Re: Hintedhandoff will never complete when a BIG rowmutation

2010-06-28 Thread Jonathan Ellis
Yes, you should increase your timeout if you are hinting big mutations (or big rows that were built from smaller mutations). 2010/6/28 Lu Ming : > Hi: > These days I found my Cassandra is strange, much slower than before. > And I Spent much time to figure it out and today I got the answer. >