After some more investigation it does not appear to be the CL issue. Every time I am starting up the node in other datacenter with 1sec delay my throughput starts degrading, even with CL=ONE and CL=LOCAL_QUORUM.
I will put the logs on debug and investigate more and report back the findings. On Mon, Jun 3, 2013 at 3:37 PM, Hiller, Dean <dean.hil...@nrel.gov> wrote: > Our badness threshold is 0.1 currently(just checked). Our website used to > get slow during a slow node time until we rolled our own patch out. > > Dean > > From: srmore <comom...@gmail.com<mailto:comom...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Monday, June 3, 2013 2:31 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Re: Consistency level for multi-datacenter setup > > We observed that as well, please let us know what you find out it would be > extremely helpful. There is also this property that you can play with to > take care of slow nodes > dynamic_snitch_badness_threshold. > > > http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold > > Thanks ! > > > On Mon, Jun 3, 2013 at 3:24 PM, Hiller, Dean <dean.hil...@nrel.gov<mailto: > dean.hil...@nrel.gov>> wrote: > Also, we had to put a fix into cassandra so it removed "slow nodes" from > the list of nodes to read from. With that fix our QUOROM(not local quorom) > started working again and would easily take the other DC nodes out of the > list of reading from for you as well. I need to circle back to with my > teammate to check if he got his fix posted to the dev list or not. > > Later, > Dean > > From: srmore <comom...@gmail.com<mailto:comom...@gmail.com><mailto: > comom...@gmail.com<mailto:comom...@gmail.com>>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org > ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > Date: Monday, June 3, 2013 2:09 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto: > user@cassandra.apache.org<mailto:user@cassandra.apache.org>>> > Subject: Consistency level for multi-datacenter setup > > I am a bit confused when using the consistency level for multi datacenter > setup. Following is my setup: > > I have 4 nodes the way these are set up are > Node 1 DC 1 - N1DC1 > Node 2 DC 1 - N2DC1 > > Node 1 DC 2 - N1DC2 > Node 2 DC 2 - N2DC2 > > I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one > way) > > I am observing that when I use consistency level 2 for some reason the > coordinate node is picking up the nodes from other datacenter. My > understanding was that Cassandra picks up nodes which are close by (from > local datacenter), determined by Gossip but looks like that's not the case. > > I found the following comment on Datastax website : > > "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in > the same data center as the coordinator node must respond to the client > request in order for the request to succeed." > > Does this mean that for multi datacenter we can only use ONE or > LOCAL_QUORUM if we want to use the local datacenter to avoid cross > datacenter latency. > > I am using the GossipingPropertyFileSnitch. > > Thanks ! > > >
