We switched to to parallel repairs, and now our repairs in 2.0 are behaving like the repairs in 1.2.
The change from parallel to sequential is very dramatic. For a small cluster with 3 nodes, using cassandra 2.0.10, a parallel repair takes 2 hours, and io throughput peaks at 6 mb/s. Sequential repair takes 40 hours, with average io around 27 mb/s. Should I file a jira? Sean On Wed, Oct 15, 2014 at 9:23 PM, Sean Bridges <sean.brid...@gmail.com> wrote: > Thanks Robert. Does the switch to sequential from parallel explain why IO > increases, we see significantly higher IO with 2.10. > > The nodetool docs [1] hint at the reason for defaulting to sequential, > > "This allows the dynamic snitch to maintain performance for your > application via the other replicas, because at least one replica in the > snapshot is not undergoing repair." > > Sean > > [1] > http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsRepair.html > > > On Wed, Oct 15, 2014 at 5:36 PM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Wed, Oct 15, 2014 at 4:54 PM, Sean Bridges <sean.brid...@gmail.com> >> wrote: >> >>> We upgraded a cassandra cluster from 1.2.18 to 2.0.10, and it looks like >>> repair is significantly more expensive now. Is this expected? >>> >> >> It depends on what you mean by "expected." Operators usually don't expect >> defaults with such dramatic impacts to change without them understanding >> why, but there is a reason for it. >> >> In 2.0 the default for repair was changed to be non-parallel. To get the >> old behavior, you need to supply -par as an argument. >> >> The context-free note you missed the significance of in NEWS.txt for >> version 2.0.2 says : >> >> - Nodetool defaults to Sequential mode for repair operations >> >> What this doesn't say is how almost certainly unreasonable this is as a >> default, because this means that repair is predictably slower in direct >> relationship to your replication factor, and the default for >> gc_grace_seconds (the time box in which one must complete a repair) did not >> change at the same time. The ticket where the change happens [1] does not >> specify a rationale, so your guess is as good as mine as to the reasoning >> which not only felt the change was necessary but reasonable. >> >> Leaving aside the problem you've encountered ("upgraders notice that >> their repairs (which already took forever) are suddenly WAY SLOWER") this >> default is also quite pathological for anyone operating with a RF over 3, >> which are valid, if very uncommon, configurations. >> >> In summary, if, as an operator, you disagree that making repair slower by >> default as a factor of replication factor is reasonable, I suggest filing a >> JIRA and letting the project know. At least in that case there is a chance >> they might explain the rationale for so blithely making a change that has >> inevitable impact on operators... ? >> >> =Rob >> [1] https://issues.apache.org/jira/browse/CASSANDRA-5950 >> http://twitter.com/rcolidba >> > >
