Thanks Rob, we are using Cassandra 2.0.2, CQL3, native protocol. tpstats is nearly all zeros from what I can tell. Even running a load of 100rps I can only ever see 1 or 2 in the active or pending counters, never anything in the blocked. Even in the "blocked all time" column it is zero in all cases except for 4 flush writers that have been there for weeks. It is hard for me to "catch it in the act" since it only lasts maybe 500ms.
Excuse my ignorance, but where would I look for the GC info? What logs contain this? I will start looking for log files and more clues in them. Thanks for the quick response! -Thunder On Thu, Jan 2, 2014 at 2:15 PM, Robert Coli <rc...@eventbrite.com> wrote: > (D'oh, missed your details in the PS.. :D) > > I don't know whether the .NET client uses thrift or native protocol.. > > Re 2.0.2 in production : > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > > > =Rob > > > On Thu, Jan 2, 2014 at 2:13 PM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Thu, Jan 2, 2014 at 2:05 PM, Thunder Stumpges < >> thunder.stump...@gmail.com> wrote: >> >>> I am seeing a read operation delay in our small (3 node) cluster where I >>> am testing. The "normal" latency for these operations is < 2ms as recorded >>> by our load client. This holds easily beyond several hundred qps. However >>> there are times when all incoming queries (on a node-by-node basis) are >>> stalled anywhere from ~100-500ms, and then all "clear" and return at the >>> same time. This behavior is independent of the amount of load applied; Just >>> more queries get stalled at higher loads :). It seems like a "stall" >>> condition happens maybe every 30 seconds or so. >>> >> >> What version of cassandra, what configuration for thrift server if >> relevant, what protocol being used? >> >> The JVM can be expected to pause for 100-500ms while doing GC, cassandra >> logs the various GC types, what do you see in those logs? What does >> "nodetool tpstats" say when it happens? >> >> =Rob >> >> > >
