Yes Carlos same issue in our case. We had some cloud issue and node wasn’t marked down. We did have some schema changes a few days back but didn’t notice because one node was still serving and when that stopped serving, we saw issues.
So, it WAS due to schema changes that happened few days back and not propagated to all three replicas. Thank you everybody for sharing their experience. > On May 6, 2017, at 7:15 AM, Carlos Rolo <r...@pythian.com> wrote: > > Sometimes "resetlocalschema" (happened to me, didn't check why) will not > work, and you need to stop the offending nodes and bring them back one by > one.. That solved the issue. > > In the cases I've seen this happened the clusters in question where either: > > a) A couple of nodes down (cloud provider caused a partial outage) with an > application that would create/change tables every now and then > b) Massive GC pauses around the cluster with a schema change being tried over > and over. > > CQLSH stopped working, your nodes died? GC Pauses? Operator mistake? > > Regards, > > Carlos Juzarte Rolo > Cassandra Consultant / Datastax Certified Architect / Cassandra MVP > > Pythian - Love your data > > rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: > linkedin.com/in/carlosjuzarterolo > <http://linkedin.com/in/carlosjuzarterolo> > Mobile: +351 918 918 100 > www.pythian.com <http://www.pythian.com/> > On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth <ni...@bamlabs.com > <mailto:ni...@bamlabs.com>> wrote: > Thank you Jeff. > > James, > > We started getting insufficient replica errors. Cqlsh stopped working for two > nodes . > > Sent from my iPhone > > > On May 5, 2017, at 7:13 PM, James Rothering <jrother...@codojo.me > > <mailto:jrother...@codojo.me>> wrote: > > > > I've heard about this ... how did the problem present itself? > > > > Sent from my iPhone > > > >> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jji...@apache.org > >> <mailto:jji...@apache.org>> wrote: > >> > >> > >> > >>> On 2017-05-05 11:00 (-0700), Nitan Kainth <ni...@bamlabs.com > >>> <mailto:ni...@bamlabs.com>> wrote: > >>> Hi Experts, > >>> > >>> We found schema version mismatch in our cluster. We fixed it by bouncing > >>> C* on nodes where version was mismatched. Can someone suggest, what are > >>> the possible reasons for this? We are trying to figure out the root cause. > >>> > >> > >> Do all of your versions match? You didn't accidentally upgrade half the > >> cluster did you? > >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > >> <mailto:user-unsubscr...@cassandra.apache.org> > >> For additional commands, e-mail: user-h...@cassandra.apache.org > >> <mailto:user-h...@cassandra.apache.org> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > > <mailto:user-unsubscr...@cassandra.apache.org> > > For additional commands, e-mail: user-h...@cassandra.apache.org > > <mailto:user-h...@cassandra.apache.org> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > <mailto:user-unsubscr...@cassandra.apache.org> > For additional commands, e-mail: user-h...@cassandra.apache.org > <mailto:user-h...@cassandra.apache.org> > > > > -- > > > > >
