On Mon, Jun 15, 2015 at 2:52 PM, Dan Kinder <dkin...@turnitin.com> wrote:
> Potentially relevant facts: > - Recently upgraded to 2.1.6 from 2.0.14 > - This table has ~million rows, low contention, and fairly high increment > rate > Can you repro on a counter that was created after the upgrade? > Mainly wondering: > > - Is this known or expected? I know Cassandra counters have had issues but > thought by now it should be able to keep a consistent counter or at least > repair it... > All counters which haven't been written to after 2.1 "new counters" are still on disk as "old counters" and will remain that way until UPDATEd and then compacted together with all old shards. "Old counters" can exhibit this behavior. > - Any way to "reset" this counter? > Per Aleksey (in IRC) you can turn a replica for an old counter into a new counter by UPDATEing it once. In order to do that without modifying the count, you can [1] : UPDATE tablename SET countercolumn = countercolumn +0 where id = 1; The important caveat that this must be done at least once per shard, with one shard per RF. The only way one can be sure that all shards have been UPDATEd is by contacting each replica node and doing the UPDATE + 0 there, because local writes are preferred. To summarize, the optimal process to upgrade your pre-existing counters to 2.1-era "new counters" : 1) get a list of all counter keys 2) get a list of replicas per counter key 3) connect to each replica for each counter key and issue an UPDATE + 0 for that counter key 4) run a major compaction As an aside, Aleksey suggests that the above process is so heavyweight that it may not be worth it. If you just leave them be, all counters you're actually used will become progressively more accurate over time. =Rob [1] Special thanks to Jeff Jirsa for verifying that this syntax works.
