The nodetool cleanup ran successfully after setting the CLASSPATH variable to 
the kubernetes-cassandra.jar.

Thanks.

> On 09-Feb-2017, at 2:23 PM, Srinath Reddy <ksre...@gmail.com> wrote:
> 
> Alex,
> 
> Thanks for reply.  I will try the workaround and post an update.
> 
> Regards,
> 
> Srinath Reddy
> 
>> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <oleksandr.shul...@zalando.de 
>> <mailto:oleksandr.shul...@zalando.de>> wrote:
>> 
>> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ksre...@gmail.com 
>> <mailto:ksre...@gmail.com>> wrote:
>> Hi,
>> 
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm 
>> getting this error when running nodetool cleanup. The Cassandra cluster is 
>> running in a Kubernetes cluster.
>> 
>> Cassandra version is 2.2.8
>> 
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException: 
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>>      at 
>> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>>      at 
>> org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
>>      at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>>      at 
>> org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>>      at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>>      at 
>> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>>      at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>> 
>> Hi,
>> 
>> From the above stacktrace it looks like you're hitting the following TODO 
>> item:
>> 
>> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>>  
>> <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282>
>> 
>> That is, nodetool needs to know concurrent_compactors setting's value before 
>> starting cleanup, but doesn't use JMX and tries to parse the configuration 
>> file instead.  That fails because your custom SeedProvider class is not on 
>> classpath for nodetool.
>> 
>> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found 
>> by java when running nodetool script, see 
>> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>>  
>> <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108>
>> 
>> Proper fix: get rid of the TODO and really query the value using JMX, 
>> especially since the latest tick-tock release of Cassandra (3.10) added a 
>> way to modify it with JMX.
>> 
>> --
>> Alex
> 

Attachment: signature.asc
Description: Message signed with OpenPGP

Reply via email to