Jai Bheemsen Rao Dhanwada created CASSANDRA-20571:
-----------------------------------------------------
Summary: CPU Spikes during the Streaming of data
Key: CASSANDRA-20571
URL: https://issues.apache.org/jira/browse/CASSANDRA-20571
Project: Apache Cassandra
Issue Type: Bug
Reporter: Jai Bheemsen Rao Dhanwada
Attachments: async_profiler_cpu.html
Hello Team,
We are seeing an issue where there is a huge spike in CPU on the node which is
streaming data (adding a new node or replacing a node or running a nodetool
rebuild). Essentially anytime when there is a Streaming is involved the CPU
spike is very huge. This does not happen in all the clusters but we
occasionally see this issue on specific cluster.
C* version: 4.1.6 (> 4.1.0)
Schema: All the tables use counter data types.
CPU Cores: 16
The same worksloads + clusters types do not show this behavior with the 4.0.x
version of cassandra, hence we suspect something changed in 4.1.6. Looking at
the top threads it's mostly the StreamDeserialize + compaction.
{code:java}
top - 17:01:29 up 18:42, 2 users, load average: 51.75, 13.61, 4.79
Threads: 741 total, 54 running, 687 sleeping, 0 stopped, 0 zombie
%Cpu(s): 91.5 us, 4.9 sy, 0.0 ni, 1.4 id, 0.7 wa, 1.1 hi, 0.4 si, 0.0 st
MiB Mem : 31176.5 total, 8762.5 free, 11028.0 used, 11386.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 19334.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
305763 xxxxxx 20 0 18.6g 9.8g 446524 R 30.8 32.2 0:04.69
Stream-Deserial
305815 xxxxxx 20 0 18.6g 9.8g 446524 R 28.6 32.2 0:04.81
Stream-Deserial
300600 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:04.73
CompactionExecu
305678 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:03.98
Stream-Deserial
305602 xxxxxx 20 0 18.6g 9.8g 446524 R 27.6 32.2 0:04.65
Stream-Deserial
305563 xxxxxx 20 0 18.6g 9.8g 446524 R 27.3 32.2 0:04.02
CompactionExecu
305687 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.28
Stream-Deserial
305707 xxxxxx 20 0 18.6g 9.8g 446524 S 26.9 32.2 0:04.29
Stream-Deserial
305714 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.91
Stream-Deserial
305569 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:05.69
Stream-Deserial
305771 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:03.99
Stream-Deserial
305817 xxxxxx 20 0 18.6g 9.8g 446524 R 26.3 32.2 0:03.79
Stream-Deserial
305566 xxxxxx 20 0 18.6g 9.8g 446524 R 26.0 32.2 0:04.64
CompactionExecu {code}
Initial Hypothesis was if streaming_stats are playing a role here based on:
https://issues.apache.org/jira/browse/CASSANDRA-18110. However we turned the
streaming_stats: false and still see a spike in CPU. Post the streaming is
complete the cluster is back to normal state where we don't see a spike in CPU
but we would like to understand what's causing the huge CPU spikes. I have
profiler attached during the time of CPU.
Please let me know if you need any other details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
