No, I am not On Wed, Feb 7, 2018 at 11:35 AM, Jeff Jirsa <jji...@gmail.com> wrote:
> Are you using internode ssl? > > > -- > Jeff Jirsa > > > On Feb 7, 2018, at 8:24 AM, Mike Torra <mto...@salesforce.com> wrote: > > Thanks for the feedback guys. That example data model was indeed > abbreviated - the real queries have the partition key in them. I am using > RF 3 on the keyspace, so I don't think a node being down would mean the key > I'm looking for would be unavailable. The load balancing policy of the > driver seems correct (https://docs.datastax.com/en/ > developer/nodejs-driver/3.4/features/tuning-policies/# > load-balancing-policy, and I am using the default `TokenAware` policy > with `DCAwareRoundRobinPolicy` as a child), but I will look more closely at > the implementation. > > It was an oversight of mine to not include `nodetool disablebinary`, but I > still experience the same issue with that. > > One other thing I've noticed is that after restarting a node and seeing > application latency, I also see that the node I just restarted sees many > other nodes in the same DC as being down (ie status 'DN'). However, > checking `nodetool status` on those other nodes shows all nodes as > up/normal. To me this could kind of explain the problem - node comes back > online, thinks it is healthy but many others are not, so it gets traffic > from the client application. But then it gets requests for ranges that > belong to a node it thinks is down, so it responds with an error. The > latency issue seems to start roughly when the node goes down, but persists > long (ie 15-20 mins) after it is back online and accepting connections. It > seems to go away once the bounced node shows the other nodes in the same DC > as up again. > > As for speculative retry, my CF is using the default of '99th percentile'. > I could try something different there, but nodes being seen as down seems > like an issue. > > On Tue, Feb 6, 2018 at 6:26 PM, Jeff Jirsa <jji...@gmail.com> wrote: > >> Unless you abbreviated, your data model is questionable (SELECT without >> any equality in the WHERE clause on the partition key will always cause a >> range scan, which is super inefficient). Since you're doing LOCAL_ONE and a >> range scan, timeouts sorta make sense - the owner of at least one range >> would be down for a bit. >> >> If you actually have a partition key in your where clause, then the next >> most likely guess is your clients aren't smart enough to route around the >> node as it restarts, or your key cache is getting cold during the bounce. >> Double check your driver's load balancing policy. >> >> It's also likely the case that speculative retry may help other nodes >> route around the bouncing instance better - if you're not using it, you >> probably should be (though with CL: LOCAL_ONE, it seems like it'd be less >> of an issue). >> >> We need to make bouncing nodes easier (or rather, we need to make drain >> do the right thing), but in this case, your data model looks like the >> biggest culprit (unless it's an incomplete recreation). >> >> - Jeff >> >> >> On Tue, Feb 6, 2018 at 10:58 AM, Mike Torra <mto...@salesforce.com> >> wrote: >> >>> Hi - >>> >>> I am running a 29 node cluster spread over 4 DC's in EC2, using C* >>> 3.11.1 on Ubuntu. Occasionally I have the need to restart nodes in the >>> cluster, but every time I do, I see errors and application (nodejs) >>> timeouts. >>> >>> I restart a node like this: >>> >>> nodetool disablethrift && nodetool disablegossip && nodetool drain >>> sudo service cassandra restart >>> >>> When I do that, I very often get timeouts and errors like this in my >>> nodejs app: >>> >>> Error: Cannot achieve consistency level LOCAL_ONE >>> >>> My queries are all pretty much the same, things like: "select * from >>> history where ts > {current_time}" >>> >>> The errors and timeouts seem to go away on their own after a while, but >>> it is frustrating because I can't track down what I am doing wrong! >>> >>> I've tried waiting between steps of shutting down cassandra, and I've >>> tried stopping, waiting, then starting the node. One thing I've noticed is >>> that even after `nodetool drain`ing the node, there are open connections to >>> other nodes in the cluster (ie looking at the output of netstat) until I >>> stop cassandra. I don't see any errors or warnings in the logs. >>> >>> What can I do to prevent this? Is there something else I should be doing >>> to gracefully restart the cluster? It could be something to do with the >>> nodejs driver, but I can't find anything there to try. >>> >>> I appreciate any suggestions or advice. >>> >>> - Mike >>> >> >> >
