Garbage is one more issue we are having with counters. We are operating under very heavy load. Counters are spread over 7 nodes with SSD drives and we often seeing CPU usage between 90-100%. We are doing mostly reads. Latency is very important for us so GC pauses taking longer than 10ms (often around 50-100ms) are very annoying.
I don't have actual numbers right now, but we've also got the impressions that cassandra generates "too much" garbage. Is there a possible that counters are somehow guilty? @Rohit: Did you tried something more stressful? Like sending more traffic to a node that it can actually handle, turning nodes up and down, changing the topology (moving/adding nodes)? I believe our problems comes from very high load and some operations like this (adding new nodes, replacing dead ones etc...). I was expecting that cassandra will fail some request, loose consistency temporarily or something like that in such cases, but generation highly incorrect values was very disappointing. Thanks, Bartek On Tue, Sep 18, 2012 at 9:30 AM, Robin Verlangen <ro...@us2.nl> wrote: > @Rohit: We also use counters quite a lot (lets say 2000 increments / sec), > but don't see the 50-100KB of garbage per increment. Are you sure that > memory is coming from your counters? > > Best regards, > > Robin Verlangen > Software engineer > > W http://www.robinverlangen.nl > E ro...@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > > > 2012/9/18 rohit bhatia <rohit2...@gmail.com> >> >> We use counters in a 8 node cluster with RF 2 in cassandra 1.0.5. >> We use phpcassa and execute cql queries through thrift to work with >> composite types. >> >> We do not have any problem of overcounts as we tally with RDBMS daily. >> >> It works fine but we are having some GC pressure for young generation. >> Per my calculation around 50-100 KB of garbage is generated every >> counter increment. >> Is this memory usage expected of counters? >> >> On Tue, Sep 18, 2012 at 7:16 AM, Bartłomiej Romański <b...@sentia.pl> wrote: >> > Hi, >> > >> > Does anyone have any experience with using Cassandra counters in >> > production? >> > >> > We rely heavily on them and recently we've got a few very serious >> > problems. Our counters values suddenly became a few times higher than >> > expected. From the business point of view this is a disaster :/ Also >> > there a few open major bugs related to them. Some of them for quite >> > long (months). >> > >> > We are seriously considering going back to other solutions (e.g. SQL >> > databases). We simply cannot afford incorrect counter values. We can >> > tolerate loosing a few increments from time to time, but we cannot >> > tolerate having counters suddenly 3 times higher or lower than the >> > expected values. >> > >> > What is the current status of counters? Should I consider them a >> > production-ready feature and we just have some bad luck? Or should I >> > rather consider them as a experimental-feature and look for some other >> > solutions? >> > >> > Do you have any experiences with them? Any comments would be very >> > helpful for us! >> > >> > Thanks, >> > Bartek > >
