On Thu, May 22, 2014 at 2:16 PM, Phil Luckhurst < phil.luckhu...@powerassure.com> wrote:
> Thanks Rob, I didn't realize that you could use the initial_token when > using > vnodes. > I am planning to write a blog post on the always-specify-initial_token best practice. It protects users in a variety of cases, at the cost of comparatively little management overhead. For reference, the format of this when using vnodes is a comma delimited list. > I see what you mean now that with RF=N having multiple vnodes is not > actually achieving anything unless we add further nodes, we hadn't really > considered that when we initially installed with the default yaml file. > Cassandra "design" and its implicit perspective on defaults often seem to not be able to make up its mind : are they for the developer running a single node on his laptop, or the operator deploying a reasonably sized production cluster? The answer seems to be "it depends on who is asking for what and who is merging the patch." It is my view that vnodes as a default is unreasonable because most clusters are unreasonably small and will stay that way, and vnodes only lose in that case. Vnodes only help you if RF is some multiple of N, but 100% of vnode users pay fixed overhead costs. As it is clear that vnodes lose in many common cases, I strongly doubt there was any formal-esque argument made for it as a reasonable default for all users. The paper trail suggests that virtually no public justification was made for making it the default, perhaps there was some discussion not immediately visible in google : https://issues.apache.org/jira/browse/CASSANDRA-5869 - "Make vnodes default in 2.0" " Description _Click To Add Description_ " http://grokbase.com/t/cassandra/dev/1383gva1fm/time-for-a-release-candidate " We hadn't actually set vnodes to be on by default, which we should do after having 1.2 default to old single token per node. Fixed in CASSANDRA-5869 so let's roll an rc2. " The stated rationale here appears to be that a major version has passed, and therefore it is appropriate to change the default. This decision does not seem related in any way to the volume of still quite serious problems vnodes had as of the 2.0 rc1 era. In real reality, vnodes were almost certainly set to default in 2.0 so that they could be hardened by both early adopters (cassandra experts) and noobs (cassandra anti-experts) encountering problems with this default. As Eric Evans mentioned at the time in a related post, this is to some extent the reality of open source software development. Where he and I appear to disagree is on whether it is reasonable to set new features as defaults and thereby use noobs as part of your QA process for same. If a Cassandra Operator notices something new and shiny has been made default or is being pushed heavily in a brand new version (vnodes, level compaction at 5mb, serial repair, etc., etc.) they should do their own evaluation and determine whether they want to participate in the QA process for this new feature, in production. My view is that it would be a lot more reasonable to continue to promote such new things heavily but with VERY LARGE REPEATED CAVEATS so that only the former class of willful early adopter experts will catch these bullets. tl;dr - Prematurely and optimistically making immature features like vnodes the default seems to mostly ensure that noobs such as yourself get a very bad initial experience with Cassandra. For a small cluster, e.g. 9 nodes with RF=3 would you actually recommend > using vnodes at all and if so how many? The key consideration when deciding vnodes or not is not really how many nodes you initially deploy, it's how many nodes you are likely to ever have. If you have a reasonable expectation of ever having over RF*3 nodes where RF>=3, you probably want to use vnodes, so that you have a faster and less degrading rebuild experience when 1/9 nodes fails. If you don't, it's probably not worth the trouble. The caveat here is that it is non-trivial to get from non-vnodes to vnodes if RF<N, which is why I phrase the choice whether to use vnodes or not as relating to the expectation of actually winning by using them. There is one final caveat, which is that significant work is being done to reduce the fixed costs and inefficiencies associated with vnodes. Each of these improvements lowers the threshold at which vnodes will become a reasonable default. =Rob
