We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but nothing helped . Our hints related parameters are as below, if you don't find any parameter below then it is not set in our environment and should be of the default value.
max_hint_window_in_ms: 10800000 # 3 hours hinted_handoff_enabled: true hinted_handoff_throttle_in_kb: 100 max_hints_delivery_threads: 8 hints_directory: /var/lib/cassandra/hints hints_flush_period_in_ms: 10000 max_hints_file_size_in_mb: 128 On Mon, 27 Jan 2020 at 18:34, Jeff Jirsa <jji...@gmail.com> wrote: > > The high cpu is probably the hints getting replayed slamming the write path > > Slowing it down with the hint throttle may help > > It’s not instant. > > On Jan 27, 2020, at 6:05 PM, Erick Ramirez <flightc...@gmail.com> wrote: > > > >> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than >> 3 hours, perhaps 6 hours. If the issue still persists networking may need >> to be tested for bandwidth issues. >> > > Just a note of warning about bumping up the hint window without > understanding the pros and cons. Be aware that doubling it means: > > - you'll end up doubling the size of stored hints in > the hints_directory > - there'll be twice as much hints to replay when node(s) come back > online > > There's always 2 sides to fiddling with the knobs in C*. Cheers! > >
