Hi Joshua, # uname -a Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)
On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith < jgalbra...@newrelic.com.invalid> wrote: > Renoy, > > Out of curiosity, which kernel version are your nodes running? > > You may find this old message on the mailing list helpful: > http://mail-archives.apache.org/mod_mbox/cassandra-user/ > 201602.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2w > m_u5kaqoro...@mail.gmail.com%3E > > On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elli...@backblaze.com> > wrote: > >> You might have more luck trying to analyze at the Java level, either via >> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra >> tools like "nodetool tpstats" >> >> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephus...@gmail.com> >> wrote: >> >>> Hi, >>> >>> i'm having a 5 node cluster with cassandra 3.0.13. >>> >>> i could see the cassandra process has too many threads. >>> >>> *# pstree -p `pgrep java` | wc -l* >>> *453* >>> >>> And almost all of those threads are in *sleeping* state and wait at >>> *# cat /proc/166022/task/1698913/wchan* >>> *futex_wait_queue_me* >>> >>> Some more info: >>> *# strace -e trace=all -p 166022* >>> *strace: Process 166022 attached* >>> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* >>> >>> # cat /proc/166022/stack >>> [<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130 >>> [<ffffffff810f5deb>] futex_wait+0x17b/0x280 >>> [<ffffffff810f7b26>] do_futex+0x106/0x5a0 >>> [<ffffffff810f8040>] SyS_futex+0x80/0x180 >>> [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b >>> [<ffffffffffffffff>] 0xffffffffffffffff >>> >>> >>> What is the reason cassandra is having these many threads? is it the >>> normal behavior of cassandra? Is there a way to reduce this thread count? >>> will there be any performance impact because of this (our platform experts >>> suspects so)? >>> >>> Regards, >>> Renoy Paulose >>> >>> >> > > > -- > *Joshua Galbraith *| Lead Software Engineer | New Relic >
