On Tue, Jan 19, 2016 at 10:44 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote:
> > Consider a scenario where I have a 20 node clsuter, RF=5, Read/Write > Quorum, gc grace period=20. My cluster is fault tolerant and it can afford > 2 node failure. Suddenly, one node goes down due to some hardware issue. > Its 10 days since my node is down, none of the 19 nodes are being repaired > and now its decision time. I am not sure how soon issue would be fixed may > be 8 days before gc grace, so I shouldnt remove node early and add node > back as it would cause unnecessary streaming. At the same time, if I dont > remove the failed node, my entire system health would be in question and it > would be a panic situation as no data got repaired in last 10 days and gc > grace is approaching. I need sufficient time to repair 19 nodes. > > What looked like a fault tolerant system which can afford 2 node failure, > required urgent attention and manual decision making when a single node > went down. Why cant we just go ahead and repair remaining replicas if some > replicas are down? If failed node comes up before gc grace period, we would > run repair to fix inconsistencies and otheriwse we would discard data and > bootstrap. I think that would be a really robust fault tolerant system. > That makes sense. It seems like having the option to ignore down replicas during repair could be at least somewhat helpful, although it may be tricky to decide how this should interact with incremental repairs. If there isn't a jira ticket for this already, can you open one with the scenario above? -- Tyler Hobbs DataStax <http://datastax.com/>
