Hello!
We have a cluster of 25 c3.4xlarge nodes (16 cores, 32 GiB) with attached 1.5 
TB 4000 PIOPS EBS drive.
Sometimes one or two nodes user cpu spikes to 100%, load average to 20-30 - 
read requests drops of.
Only restart of this cassandra services helps.
Please advice.

One big table with wide rows. 600 Gb per node.
LZ4Compressor
LeveledCompaction

concurrent compactors: 4
compactor throughput: tried from 16 to 128
Concurrent_readers: from 16 to 32
Concurrent_writers: 128


https://gist.github.com/rskvazh/de916327779b98a437a6


 JvmTop 0.8.0 alpha - 06:51:10,  amd64, 16 cpus, Linux 3.14.44-3, load avg 19.35
 http://code.google.com/p/jvmtop

 Profiling PID 9256: org.apache.cassandra.service.CassandraDa

  95.73% (     4.31s) ....google.common.collect.AbstractIterator.tryToComputeN()
   1.39% (     0.06s) com.google.common.base.Objects.hashCode()
   1.26% (     0.06s) io.netty.channel.epoll.Native.epollWait()
   0.85% (     0.04s) net.jpountz.lz4.LZ4JNI.LZ4_compress_limitedOutput()
   0.46% (     0.02s) net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast()
   0.26% (     0.01s) com.google.common.collect.Iterators$7.computeNext()
   0.06% (     0.00s) io.netty.channel.epoll.Native.eventFdWrite()


ttop:

2016-02-12T08:20:25.605+0000 Process summary
  process cpu=1565.15%
  application cpu=1314.48% (user=1354.48% sys=-40.00%)
  other: cpu=250.67%
  heap allocation rate 146mb/s
[000405] user=76.25% sys=-0.54% alloc=     0b/s - SharedPool-Worker-9
[000457] user=75.54% sys=-1.26% alloc=     0b/s - SharedPool-Worker-14
[000451] user=73.52% sys= 0.29% alloc=     0b/s - SharedPool-Worker-16
[000311] user=76.45% sys=-2.99% alloc=     0b/s - SharedPool-Worker-4
[000389] user=70.69% sys= 2.62% alloc=     0b/s - SharedPool-Worker-6
[000388] user=86.95% sys=-14.28% alloc=     0b/s - SharedPool-Worker-5
[000404] user=70.69% sys= 0.10% alloc=     0b/s - SharedPool-Worker-8
[000390] user=72.61% sys=-1.82% alloc=     0b/s - SharedPool-Worker-7
[000255] user=87.86% sys=-17.87% alloc=     0b/s - SharedPool-Worker-1
[000444] user=72.21% sys=-2.30% alloc=     0b/s - SharedPool-Worker-12
[000310] user=71.50% sys=-2.31% alloc=     0b/s - SharedPool-Worker-3
[000445] user=69.68% sys=-0.83% alloc=     0b/s - SharedPool-Worker-13
[000406] user=72.61% sys=-4.40% alloc=     0b/s - SharedPool-Worker-10
[000446] user=69.78% sys=-1.65% alloc=     0b/s - SharedPool-Worker-11
[000452] user=66.86% sys= 0.22% alloc=     0b/s - SharedPool-Worker-15
[000256] user=69.08% sys=-2.42% alloc=     0b/s - SharedPool-Worker-2
[004496] user=29.99% sys= 0.59% alloc=   30mb/s - CompactionExecutor:15
[004906] user=29.49% sys= 0.74% alloc=   39mb/s - CompactionExecutor:16
[010143] user=28.58% sys= 0.25% alloc=   26mb/s - CompactionExecutor:17
[000785] user=27.87% sys= 0.70% alloc=   38mb/s - CompactionExecutor:12
[012723] user= 9.09% sys= 2.46% alloc= 2977kb/s - RMI TCP 
Connection(2673)-127.0.0.1
[000555] user= 5.35% sys=-0.08% alloc=  474kb/s - SharedPool-Worker-24
[000560] user= 3.94% sys= 0.07% alloc=  434kb/s - SharedPool-Worker-22
[000557] user= 3.94% sys=-0.17% alloc=  339kb/s - SharedPool-Worker-25
[000447] user= 2.73% sys= 0.60% alloc=  436kb/s - SharedPool-Worker-19
[000563] user= 3.33% sys=-0.04% alloc=  460kb/s - SharedPool-Worker-20
[000448] user= 2.73% sys= 0.27% alloc=  414kb/s - SharedPool-Worker-21
[000554] user= 1.72% sys= 0.70% alloc=  232kb/s - SharedPool-Worker-26
[000558] user= 1.41% sys= 0.39% alloc=  213kb/s - SharedPool-Worker-23
[000450] user= 1.41% sys=-0.03% alloc=  158kb/s - SharedPool-Worker-17

Reply via email to