RF of one means all auth requests go to the same node, so they’re more likely to time out if that host is overloaded or restarts
Increasing it distributed the queries among more hosts -- Jeff Jirsa > On May 12, 2018, at 6:11 AM, Abdul Patel <abd786...@gmail.com> wrote: > > Yeah found that all had 3 replication factor and system_auth had 1 , chnaged > to 3 now ..so was this issue due to system_auth replication facyor mismatch? > >> On Saturday, May 12, 2018, Hannu Kröger <hkro...@gmail.com> wrote: >> Hi, >> >> Did you check replication strategy and amounts of replicas of system_auth >> keyspace? >> >> Hannu >> >>> Abdul Patel <abd786...@gmail.com> kirjoitti 12.5.2018 kello 5.21: >>> >>> No applicatiom isnt impacted ..no complains .. >>> Also its an 4 node cluster in lower non production and all are on same >>> version. >>> >>>> On Friday, May 11, 2018, Jeff Jirsa <jji...@gmail.com> wrote: >>>> The read is timing out - is the cluster healthy? Is it fully upgraded or >>>> mixed versions? Repeated isn’t great, but is the application impacted? >>>> >>>> -- >>>> Jeff Jirsa >>>> >>>> >>>>> On May 12, 2018, at 6:17 AM, Abdul Patel <abd786...@gmail.com> wrote: >>>>> >>>>> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if >>>>> this is repeatedly coming , whats solution for this? >>>>> >>>>> WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 >>>>> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize >>>>> #<User cas_sar_dit> for <table et_session.session> >>>>> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 >>>>> ErrorMessage.java:384 - Unexpected exception during request >>>>> com.google.common.util.concurrent.UncheckedExecutionException: >>>>> java.lang.RuntimeException: >>>>> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >>>>> - received only 0 responses. >>>>> at >>>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>>>> ~[guava-18.0.jar:na] >>>>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>>>> ~[guava-18.0.jar:na] >>>>> at >>>>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>>>> ~[guava-18.0.jar:na] >>>>> at >>>>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>>>> ~[guava-18.0.jar:na] >>>>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> at >>>>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) >>>>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>>>> >>>>>> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa <jji...@gmail.com> wrote: >>>>>> That looks like Cassandra 3.10 not 3.11.2 >>>>>> >>>>>> It’s also just the auth cache failing to refresh - if it’s transient >>>>>> it’s probably not a big deal. If it continues then there may be an issue >>>>>> with the cache refresher. >>>>>> >>>>>> -- >>>>>> Jeff Jirsa >>>>>> >>>>>> >>>>>>> On May 12, 2018, at 5:55 AM, Abdul Patel <abd786...@gmail.com> wrote: >>>>>>> >>>>>>> HI All, >>>>>>> >>>>>>> Seen below stack trace messages , in errorlog one day after upgrade. >>>>>>> one of the blogs said this might be due to old drivers, but not sure on >>>>>>> it. >>>>>>> >>>>>>> FYI : >>>>>>> >>>>>>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>>>>>> OutboundTcpConnection.java:510 - Handshaking version with >>>>>>> /10.152.205.150 >>>>>>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>>>>>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to >>>>>>> /10.152.205.150 >>>>>>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >>>>>>> ErrorMessage.java:384 - Unexpected exception during request >>>>>>> com.google.common.util.concurrent.UncheckedExecutionException: >>>>>>> com.google.common.util.concurrent.UncheckedExecutionException: >>>>>>> java.lang.RuntimeException: >>>>>>> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve >>>>>>> consistency level LOCAL_ONE >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) >>>>>>> [apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) >>>>>>> [apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >>>>>>> [netty-all-4.0.39.Final.jar:4.0.39.Final] >>>>>>> at >>>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) >>>>>>> [netty-all-4.0.39.Final.jar:4.0.39.Final] >>>>>>> at >>>>>>> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) >>>>>>> [netty-all-4.0.39.Final.jar:4.0.39.Final] >>>>>>> at >>>>>>> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) >>>>>>> [netty-all-4.0.39.Final.jar:4.0.39.Final] >>>>>>> at >>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>> [na:1.8.0_51] >>>>>>> at >>>>>>> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) >>>>>>> [apache-cassandra-3.10.jar:3.10] >>>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] >>>>>>> Caused by: >>>>>>> com.google.common.util.concurrent.UncheckedExecutionException: >>>>>>> java.lang.RuntimeException: >>>>>>> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve >>>>>>> consistency level LOCAL_ONE >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.PermissionsCache$$Lambda$269/275740217.apply(Unknown >>>>>>> Source) ~[na:na] >>>>>>> at >>>>>>> org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> at >>>>>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) >>>>>>> ~[guava-18.0.jar:na] >>>>>>> ... 26 common frames omitted >>>>>>> Caused by: java.lang.RuntimeException: >>>>>>> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve >>>>>>> consistency level LOCAL_ONE >>>>>>> at >>>>>>> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:513) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:481) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:285) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) >>>>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>>>> at >>>>>>> org.apache.cassandra.auth.RolesCache$$Lambda$280/1091297529.apply(Unknown >>>>>>> Source) ~[na:na] >>>>>>> >>>>>
