ok thank you On Thu, May 11, 2017 at 1:11 PM, Jeff Jirsa <jji...@gmail.com> wrote:
> No, it's not expected, but it's pretty obvious from reading the code > what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA- > 13526 > > > > > > On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> Yes I have many keyspaces which are not spread across all the data >> centers(expected by design). >> In this case, is this the expected behavior cleanup will not work for all >> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest >> versions? >> >> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup >> keyspacename" >> >> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa <jji...@gmail.com> wrote: >> >>> If you didn't explicitly remove a keyspace from one of your datacenters, >>> the next most likely cause is that you have one keyspace that's NOT >>> replicated to one of the datacenters. You can work around this by running >>> 'nodetool cleanup <ks>' on all of your other keyspaces individually, >>> skipping the one that isn't replicated to that datacenter. >>> >>> >>> >>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada < >>> jaibheem...@gmail.com> wrote: >>> >>>> Thanks Jeff, >>>> >>>> I have a C* cluster spread across multiple datacenter. >>>> reason for cleanup : I added multiple nodes to cluster and need to run >>>> cleanup on old nodes so that the redundant data is cleaned-up. >>>> >>>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa <jji...@apache.org> wrote: >>>> >>>>> >>>>> >>>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada < >>>>> jaibheem...@gmail.com> wrote: >>>>> > Hello, >>>>> > >>>>> > I am running into an issue where *nodetool cleanup *fails to cleanup >>>>> data. >>>>> > We are running 2.1.16 version of Cassandra. >>>>> > >>>>> > >>>>> > [user@host ~]$ nodetool cleanup >>>>> > Aborted cleaning up atleast one column family in keyspace user, check >>>>> > server logs for more information. >>>>> > Aborted cleaning up atleast one column family in keyspace org, check >>>>> server >>>>> > logs for more information. >>>>> > error: nodetool failed, check server logs >>>>> > -- StackTrace -- >>>>> > java.lang.RuntimeException: nodetool failed, check server logs >>>>> > at >>>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool >>>>> .java:294) >>>>> > at org.apache.cassandra.tools.Nod >>>>> eTool.main(NodeTool.java:206) >>>>> > >>>>> > *Logs:* >>>>> > >>>>> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987 >>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has >>>>> joined >>>>> > the ring >>>>> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010 >>>>> > CompactionManager.java:415 - Cleanup cannot run before a node has >>>>> joined >>>>> > the ring >>>>> > >>>>> > All the nodes in the cluster are up and running. We tried doing a >>>>> rolling >>>>> > restart of all nodes and no luck. >>>>> > >>>>> > After looking at the Cassandra JIRA : >>>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like >>>>> the issue >>>>> > is fixed with 2.2.6 and 3.0 version. >>>>> > While we have plans to upgrade to the latest versions(which might >>>>> take >>>>> > longer time), does any know if there is any work around to mitigate >>>>> the >>>>> > issue? >>>>> > >>>>> >>>>> Are you running multiple datacenters, and you just removed a specific >>>>> datacenter from a keyspace (and that's why you want to run cleanup)? If >>>>> that's the case, I fear the fix for 10991 isn't really going to fix it in >>>>> the way you hope (we may need a follow-up jira). What you'll almost >>>>> certainly need to do is remove the data on disk manually, which is quite >>>>> unfortunate as it'll require you to >>>>> stop+delete-data-for-that-keyspace+start >>>>> each node in the datacenter for which you removed replication. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>>> >>>>> >>>> >>> >> >
