What's your CPU looking like? If it's low, check your IO with iostat or dstat. I know some people have used Ebs and say it's fine but ive been burned too many times. On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang <y...@kryptoncloud.com> wrote:
> Hi Riccardo, > > Very low IO-wait. About 0.3%. > No stolen CPU. It is a casssandra only instance. I did not see any dropped > messages. > > > ubuntu@cassandra1:/mnt/data$ nodetool tpstats > Pool Name Active Pending Completed Blocked > All time blocked > MutationStage 1 1 929509244 0 > 0 > ViewMutationStage 0 0 0 0 > 0 > ReadStage 4 0 4021570 0 > 0 > RequestResponseStage 0 0 731477999 0 > 0 > ReadRepairStage 0 0 165603 0 > 0 > CounterMutationStage 0 0 0 0 > 0 > MiscStage 0 0 0 0 > 0 > CompactionExecutor 2 55 92022 0 > 0 > MemtableReclaimMemory 0 0 1736 0 > 0 > PendingRangeCalculator 0 0 6 0 > 0 > GossipStage 0 0 345474 0 > 0 > SecondaryIndexManagement 0 0 0 0 > 0 > HintsDispatcher 0 0 4 0 > 0 > MigrationStage 0 0 35 0 > 0 > MemtablePostFlush 0 0 1973 0 > 0 > ValidationExecutor 0 0 0 0 > 0 > Sampler 0 0 0 0 > 0 > MemtableFlushWriter 0 0 1736 0 > 0 > InternalResponseStage 0 0 5311 0 > 0 > AntiEntropyStage 0 0 0 0 > 0 > CacheCleanupExecutor 0 0 0 0 > 0 > Native-Transport-Requests 128 128 347508531 2 > 15891862 > > Message type Dropped > READ 0 > RANGE_SLICE 0 > _TRACE 0 > HINT 0 > MUTATION 0 > COUNTER_MUTATION 0 > BATCH_STORE 0 > BATCH_REMOVE 0 > REQUEST_RESPONSE 0 > PAGED_RANGE 0 > READ_REPAIR 0 > > > > > > On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari <ferra...@gmail.com> > wrote: > >> Hi Yuan, >> >> You machine instance is 4 vcpus that is 4 threads (not cores!!!), aside >> from any Cassandra specific discussion a system load of 10 on a 4 threads >> machine is way too much in my opinion. If that is the running average >> system load I would look deeper into system details. Is that IO wait? Is >> that CPU Stolen? Is that a Cassandra only instance or are there other >> processes pushing the load? >> What does your "nodetool tpstats" say? Hoe many dropped messages do you >> have? >> >> Best, >> >> On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang <y...@kryptoncloud.com> wrote: >> >>> Thanks Ben! For the post, it seems they got a little better but similar >>> result than i did. Good to know it. >>> I am not sure if a little fine tuning of heap memory will help or not. >>> >>> >>> On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater <ben.sla...@instaclustr.com> >>> wrote: >>> >>>> Hi Yuan, >>>> >>>> You might find this blog post a useful comparison: >>>> >>>> https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/ >>>> >>>> Although the focus is on Spark and Cassandra and multi-DC there are >>>> also some single DC benchmarks of m4.xl clusters plus some discussion of >>>> how we went about benchmarking. >>>> >>>> Cheers >>>> Ben >>>> >>>> >>>> On Fri, 8 Jul 2016 at 07:52 Yuan Fang <y...@kryptoncloud.com> wrote: >>>> >>>>> Yes, here is my stress test result: >>>>> Results: >>>>> op rate : 12200 [WRITE:12200] >>>>> partition rate : 12200 [WRITE:12200] >>>>> row rate : 12200 [WRITE:12200] >>>>> latency mean : 16.4 [WRITE:16.4] >>>>> latency median : 7.1 [WRITE:7.1] >>>>> latency 95th percentile : 38.1 [WRITE:38.1] >>>>> latency 99th percentile : 204.3 [WRITE:204.3] >>>>> latency 99.9th percentile : 465.9 [WRITE:465.9] >>>>> latency max : 1408.4 [WRITE:1408.4] >>>>> Total partitions : 1000000 [WRITE:1000000] >>>>> Total errors : 0 [WRITE:0] >>>>> total gc count : 0 >>>>> total gc mb : 0 >>>>> total gc time (s) : 0 >>>>> avg gc time(ms) : NaN >>>>> stdev gc time(ms) : 0 >>>>> Total operation time : 00:01:21 >>>>> END >>>>> >>>>> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla <r...@foundev.pro> wrote: >>>>> >>>>>> Lots of variables you're leaving out. >>>>>> >>>>>> Depends on write size, if you're using logged batch or not, what >>>>>> consistency level, what RF, if the writes come in bursts, etc, etc. >>>>>> However, that's all sort of moot for determining "normal" really you >>>>>> need a >>>>>> baseline as all those variables end up mattering a huge amount. >>>>>> >>>>>> I would suggest using Cassandra stress as a baseline and go from >>>>>> there depending on what those numbers say (just pick the defaults). >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Jul 7, 2016, at 4:39 PM, Yuan Fang <y...@kryptoncloud.com> wrote: >>>>>> >>>>>> yes, it is about 8k writes per node. >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle <daeme...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Are you saying 7k writes per node? or 30k writes per node? >>>>>>> >>>>>>> >>>>>>> *.......* >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 >>>>>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872 >>>>>>> <%28%2B44%29%20%280%29%2020%208144%209872>* >>>>>>> >>>>>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang <y...@kryptoncloud.com> >>>>>>> wrote: >>>>>>> >>>>>>>> writes 30k/second is the main thing. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle < >>>>>>>> daeme...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Assuming you meant 100k, that likely for something with 16mb of >>>>>>>>> storage (probably way small) where the data is more that 64k hence >>>>>>>>> will not >>>>>>>>> fit into the row cache. >>>>>>>>> >>>>>>>>> >>>>>>>>> *.......* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 >>>>>>>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872 >>>>>>>>> <%28%2B44%29%20%280%29%2020%208144%209872>* >>>>>>>>> >>>>>>>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang <y...@kryptoncloud.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and >>>>>>>>>> 600GB ssd EBS). >>>>>>>>>> I can reach a cluster wide write requests of 30k/second and read >>>>>>>>>> request about 100/second. The cluster OS load constantly above 10. >>>>>>>>>> Are >>>>>>>>>> those normal? >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> Yuan >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> -- >>>> ———————— >>>> Ben Slater >>>> Chief Product Officer >>>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >>>> +61 437 929 798 >>>> >>> >>> >> >
