Hi Romain, I am using Cassandra version 3.0.9 and here is the generated report <http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMS8tLWpzdGFja19kdW1wLm91dC0tMi0yNC00OA==> (Graphical view) of my thread dump as well!. Just send this over in case if it helps.
Thanks, kant On Tue, Feb 28, 2017 at 7:51 PM, Kant Kodali <k...@peernova.com> wrote: > Hi Romain, > > Thanks again. My response are inline. > > kant > > On Tue, Feb 28, 2017 at 10:04 AM, Romain Hardouin <romainh...@yahoo.fr> > wrote: > >> > we are currently using 3.0.9. should we use 3.8 or 3.10 >> >> No, don't use 3.X in production unless you really need a major feature. >> I would advise to stick to 3.0.X (i.e. 3.0.11 now). >> You can backport CASSANDRA-11966 easily but of course you have to deploy >> from source as a prerequisite. >> > > * By backporting you mean I should cherry pick CASSANDRA-11966 commit > and compile from source?* > >> >> > I haven't done any tuning yet. >> >> So it's a good news because maybe there is room for improvement >> >> > Can I change this on a running instance? If so, how? or does it require >> a downtime? >> >> You can throttle compaction at runtime with "nodetool >> setcompactionthroughput". Be sure to read all nodetool commmands, some of >> them are really useful for a day to day tuning/management. >> >> If GC is fine, then check other things -> "[...] different pool sizes for >> NTR, concurrent reads and writes, compaction executors, etc. Also check if >> you can improve network latency (e.g. VF or ENA on AWS)." >> >> Regarding thread pools, some of them can be resized at runtime via JMX. >> >> > 5000 is the target. >> >> Right now you reached 1500. Is it per node or for the cluster? >> We don't know your setup so it's hard to say it's doable. Can you provide >> more details? VM, physical nodes, #nodes, etc. >> Generally speaking LWT should be seldom used. AFAIK you won't achieve >> 10,000 writes/s per node. >> >> Maybe someone on the list already made some tuning for heavy LWT workload? >> > > * 1500 total cluster. * > > * I have a 8 node cassandra cluster. Each node is AWS m4.xlarge > instance (so 4 vCPU, 16GB, 1Gbit network=125MB/s)* > > > > * I have 1 node (m4.xlarge) for my application which just inserts a > bunch of data and each insert is an LWT I tested the network throughput > of the node. I can get up 98 MB/s.* > > * Now, when I start my application. I see that Cassandra nodes Receive > rate/ throughput is about 4MB/s (yes it is in Mega Bytes. I checked this by > running sudo iftop -B). The Disk I/O is also same and the Cassandra process > CPU usage is about 360% (the max is 400% since it is a 4 core machine). The > application node transmission throughput is about 6MB/s. so even with 4MB/s > receive throughput at Cassandra node the CPU is almost maxed out. I am not > sure what this says about Cassandra? But, what I can tell is that Network > is way underutilized and that 8 nodes are unnecessary so we plan to bring > it down to 4 nodes except each node this time will have 8 cores. All said, > I am still not sure how to scale up from 1500 writes/sec? * > > >> >> Best, >> >> Romain >> >> >
