Re: Repair Issues

2019-10-26 Thread Ben Mills
Thanks Ghiyasi. On Sat, Oct 26, 2019 at 9:17 AM Hossein Ghiyasi Mehr wrote: > If the problem exist still, and all nodes are up, reboot them one by one. > Then try to repair one node. After that repair other nodes one by one. > > On Fri, Oct 25, 2019 at 12:56 AM Ben Mills wrote: > >> >> Thanks J

Re: Repair Issues

2019-10-26 Thread Hossein Ghiyasi Mehr
If the problem exist still, and all nodes are up, reboot them one by one. Then try to repair one node. After that repair other nodes one by one. On Fri, Oct 25, 2019 at 12:56 AM Ben Mills wrote: > > Thanks Jon! > > This is very helpful - allow me to follow-up and ask a question. > > (1) Yes, inc

Re: Repair Issues

2019-10-24 Thread Ben Mills
Thanks Jon! This is very helpful - allow me to follow-up and ask a question. (1) Yes, incremental repairs will never be used (unless it becomes viable in Cassandra 4.x someday). (2) I hear you on the JVM - will look into that. (3) Been looking at Cassandra version 3.11.x though was unaware that 3

Re: Repair Issues

2019-10-24 Thread Ben Mills
Hi Reid, Many thanks - I have seen that article though will definitely give it another read. Note that nodetool scrub has been tried (no effect) and sstablescrub cannot currently be run with the Cassandra image in use (though certainly a new image that allows the server to be stopped but keeps th

Re: Repair Issues

2019-10-24 Thread Jon Haddad
There's some major warning signs for me with your environment. 4GB heap is too low, and Cassandra 3.7 isn't something I would put into production. Your surface area for problems is massive right now. Things I'd do: 1. Never use incremental repair. Seems like you've already stopped doing them,

Re: Repair Issues

2019-10-24 Thread Ben Mills
Hi Sergio, No, not at this time. It was in use with this cluster previously, and while there were no reaper-specific issues, it was removed to help simplify investigation of the underlying repair issues I've described. Thanks. On Thu, Oct 24, 2019 at 4:21 PM Sergio wrote: > Are you using Cass

Re: Repair Issues

2019-10-24 Thread Reid Pinchback
Ben, you may find this helpful: https://blog.pythian.com/so-you-have-a-broken-cassandra-sstable-file/ From: Ben Mills Reply-To: "user@cassandra.apache.org" Date: Thursday, October 24, 2019 at 3:31 PM To: "user@cassandra.apache.org" Subject: Repair Issues Message from External Sender Greeting

Re: Repair Issues

2019-10-24 Thread Sergio
Are you using Cassandra reaper? On Thu, Oct 24, 2019, 12:31 PM Ben Mills wrote: > Greetings, > > Inherited a small Cassandra cluster with some repair issues and need some > advice on recommended next steps. Apologies in advance for a long email. > > Issue: > > Intermittent repair failures on two