Sounds like GC from your description of fast->slow->fast. Collect GC times from both the client and server side and plot against your application timing.
If you uncomment the verbose GC entries in the cassandra-env.sh file you should get timing for the server side, pass in the same arguments for your client. Align time across the 3 files and plot to see if GC is the cause. Sridhar On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani <roshandawr...@gmail.com>wrote: > Hi, > > I am in the middle of some load testing on a 1-node Cassandra setup. We are > not on very high loads yet. We have recorded the timings taken up by > mutator.execute() calls and we see this kind of variation during the test > run: > > So, 25% of the times, execute() calls come back in 25 milli-seconds, but > the longer calls go upto 4 seconds. > > Can someone please provide some pointers on what and where to focus on in > my Hector / Cassandra setup? We are mostly on the default Cassandra > configuration at this time - only change is the max connection pool size > (CassandraHostConfigurator.maxActive) is changed to 300 from a default of > 50. > > I would also like to add that the time increase is not linear - it starts > fast, goes, slow, very slow, and becomes faster again. > > ------------------------ > 25% 29 > 50% 105 > 66% 185 > 70% 208 > 75% 240 > 80% 297 > 90% 510 > 95% 854 > 98% 1075 > 99% 1215 > 100% 4442 > ------------------------ > > -- > Roshan > Blog: http://roshandawrani.wordpress.com/ > Twitter: @roshandawrani <http://twitter.com/roshandawrani> > Skype: roshandawrani > >
