We serialize the other caches to disk to avoid cold-start problems, I don't see why we couldn't also serialize the chunk cache? Seems worth a JIRA to me.
Until then, you can probably use the dynamic snitch (badness + severity) to route around newly started hosts. I'm actually pretty surprised the chunk cache is that effective, sort of nice to know. On Tue, Mar 21, 2023 at 10:17 AM Carlos Diaz <crdiaz...@gmail.com> wrote: > Hi Team, > > We are heavy users of Cassandra at a pretty big bank. Security measures > require us to constantly refresh our C* nodes every x number of days. We > normally do this in a rolling fashion, taking one node down at a time and > then refreshing it with a new instance. This process has been working for > us great for the past few years. > > However, we recently started having issues when a newly refreshed instance > comes back online, our automation waits a few minutes for the node to > become "ready (UN)" and then moves on to the next node. The problem that > we are facing is that when the node is ready, the chunk cache is still > empty so when the node starts accepting new connections, queries that go to > take much longer to respond and this causes errors for our apps. > > I was thinking that it would be great if we had a nodetool command that > would allow us to prefetch a certain table or a set of tables to preload > the chunk cache. Then we could simply add another check (nodetool info?), > to ensure that the chunk cache has been preloaded enough to handle queries > to this particular node. > > Would love to hear others' feedback on the feasibility of this idea. > > Thanks! > > > >
