Jai Bheemsen Rao Dhanwada created CASSANDRA-15449:
-----------------------------------------------------
Summary: Credentials out of sync after replacing the nodes
Key: CASSANDRA-15449
URL: https://issues.apache.org/jira/browse/CASSANDRA-15449
Project: Cassandra
Issue Type: Bug
Reporter: Jai Bheemsen Rao Dhanwada
Attachments: Screen Shot 2019-12-12 at 11.13.52 AM.png
Hello,
We are seeing a strange issue where, after replacing multiple C* nodes from the
clusters intermittently we see an issue where few nodes doesn't have any
credentials and the client queries fail.
Here are the sequence of steps
1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in
one DC.
2. The approach we took to replace the nodes is kill one node and launch a new
node with {{-Dcassandra.replace_address=}} and proceed with next node once the
node is bootstrapped and CQL is enabled.
3. This process works fine and all of a sudden, we started seeing our
application started failing with the below errors in the logs
{quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has
no SELECT permission on <table ks.cf> or any of its parents at
com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59)
at
com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25)
at
{quote}
4. At this stage we see that 3 nodes in the cluster takes zero traffic, while
rest of the nodes are serving ~100 requests. (attached the metrics)
!Screen Shot 2019-12-12 at 11.13.52 AM.png!
5. We suspect some credentials sync issue and manually synced the credentials
and restarted the nodes with 0 requests, which fixed the problem.
Also, one few C* nodes we see below exception immediately after the bootstrap
is completed and the process dies. is this contributing to the credentials
issue?
NOTE: The C* nodes with zero traffic and the nodes with the below exception
are not the same.
{quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 -
Exception encountered during startup
java.lang.AssertionError:
org.apache.cassandra.exceptions.InvalidRequestException: Undefined name
salted_hash in selection clause
at
org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202)
~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.Auth.setup(Auth.java:144)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.16.jar:2.1.16]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined
name salted_hash in selection clause
at
org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198)
~[apache-cassandra-2.1.16.jar:2.1.16]
... 7 common frames omitted
{quote}
Not sure why this is happening, is this a potential bug or any other pointers
to fix the problem.
C* Version: 2.1.16
Client: Datastax Java Driver.
system_auth RF: 3, dc-1:3 and dc-2:3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
