Re: Dropping down replication factor

2017-08-15 Thread Erick Ramirez
I would discourage dropping to RF=2 because if you're using CL=*QUORUM, it won't be able to tolerate a node outage. You mentioned a couple of days ago that there's an index file that is corrupted on 10.40.17.114. Could you try moving out the sstable set associated with that corrupt file and try ag

Re: Dropping down replication factor

2017-08-13 Thread Brian Spindler
Thanks Kurt. We had one sstable from a cf of ours. I am actually running a repair on that cf now and then plan to try and join the additional nodes as you suggest. I deleted the opscenter corrupt sstables as well but will not bother repairing that before adding capacity. Been keeping an eye acr

Re: Dropping down replication factor

2017-08-13 Thread kurt greaves
On 14 Aug. 2017 00:59, "Brian Spindler" wrote: Do you think with the setup I've described I'd be ok doing that now to recover this node? The node died trying to run the scrub; I've restarted it but I'm not sure it's going to get past a scrub/repair, this is why I deleted the other files as a bru

Re: Dropping down replication factor

2017-08-13 Thread Brian Spindler
for >>> sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack >>> probably not necessary, streaming is being marked as failed and it’s >>> turning itself off. Not sure why streaming is marked as failing, though, >>> anything on the sending sides? >

Re: Dropping down replication factor

2017-08-13 Thread Jeff Jirsa
econdary index build. Hard to say for >>> sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack >>> probably not necessary, streaming is being marked as failed and it’s >>> turning itself off. Not sure why streaming is marked as failing, though, >>> anythin

Re: Dropping down replication factor

2017-08-13 Thread Brian Spindler
e jstack >> probably not necessary, streaming is being marked as failed and it’s >> turning itself off. Not sure why streaming is marked as failing, though, >> anything on the sending sides? >> >> >> >> >> >> From: Brian Spindler >> Reply-T

Re: Dropping down replication factor

2017-08-12 Thread Brian Spindler
> > > From: Brian Spindler > Reply-To: > Date: Saturday, August 12, 2017 at 6:34 PM > To: > Subject: Re: Dropping down replication factor > > Thanks for replying Jeff. > > Responses below. > > On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa wrote: > >> Answers

Re: Dropping down replication factor

2017-08-12 Thread Jeffrey Jirsa
itself off. Not sure why streaming is marked as failing, though, anything on the sending sides? From: Brian Spindler Reply-To: Date: Saturday, August 12, 2017 at 6:34 PM To: Subject: Re: Dropping down replication factor Thanks for replying Jeff. Responses below. On Sat, Aug 12, 2017 at

Re: Dropping down replication factor

2017-08-12 Thread Brian Spindler
Thanks for replying Jeff. Responses below. On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa wrote: > Answers inline > > -- > Jeff Jirsa > > > > On Aug 12, 2017, at 2:58 PM, brian.spind...@gmail.com wrote: > > > > Hi folks, hopefully a quick one: > > > > We are running a 12 node cluster (2.1.15) in AW

Re: Dropping down replication factor

2017-08-12 Thread Jeff Jirsa
Answers inline -- Jeff Jirsa > On Aug 12, 2017, at 2:58 PM, brian.spind...@gmail.com wrote: > > Hi folks, hopefully a quick one: > > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all in > one region but spread across 3 availability zones. It was nicely balanced > w

Dropping down replication factor

2017-08-12 Thread brian . spindler
Hi folks, hopefully a quick one: We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all in one region but spread across 3 availability zones. It was nicely balanced with 4 nodes in each. But with a couple of failures and subsequent provisions to the wrong az we now have a