Re: read request is slow

[Dieudonné Madishon NGAYA]

Hi,
Below some different tools to monitor cassandra:
1) Nodetool
Nodetool has many options
2)Jconsole
3) Opscenter
4) tools like top, htop, vmstats, sar, dstat etc ... are also very usefull.

Best regards
_

[image:
https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]

      


*Dieudonne Madishon NGAYA*
Datastax, Cassandra Architect
*P: *7048580065
*w: *www.dmnbigdata.com
*E: *dmng...@dmnbigdata.com
*Private E: *dmng...@gmail.com
*A: *Charlotte,NC,28273, USA


On Sun, Mar 17, 2019 at 12:48 AM Sundaramoorthy, Natarajan <
natarajan_sundaramoor...@optum.com> wrote:

> Also would like to know  what monitoring should I setup so that if it
> happens again I can provide more information. Thanks
>
>
>
>
>
> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
> *Sent:* Saturday, March 16, 2019 5:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> I'm guessing you're getting 100MB from the comments in the config, which
> suggest 100MB per core.  This advice is pretty outdated and should be
> updated.
>
>
>
> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
> suggest reading up on how GC works, I linked to a post in an earlier email.
>
>
>
> These are the flags you'd need to set in your jvm.options, or
> jvm-server.options depending on the version you're using:
>
>
>
> -Xmx8G
>
> -Xms8G
>
> -Xmn4G
>
>
>
> 1 core is probably going to be a problem, Cassandra creates a lot of
> threads and relies on doing work concurrently.  I wouldn't use less than 8
> cores in a production environment.
>
>
>
> On Sun, Mar 17, 2019 at 3:12 AM Dieudonné Madishon NGAYA <
> dmng...@gmail.com> wrote:
>
> Starting point for me: max_heap_size to 8gb and heap_newsize to 100mb.
> Then restart node by node then watch system.log to see if you are seeing G.C
>
>
>
> On Sat, Mar 16, 2019 at 9:56 AM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
> So you guys are suggesting
>
>
>
> MAX_HEAP_SIZE  by 8/12/16GB
>
>
>
> And
>
>
>
> HEAP_NEWSIZE to 100 MB
>
>
>
> And
>
>
>
> heap with 50% of that as a starting point? Hw do I do this?
>
>
>
> Thanks
>
>
>
>
>
> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
> *Sent:* Saturday, March 16, 2019 12:15 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> I agreed with jon haddad , your MAX_HEAP_SIZE is very small. you have lot
> of RAM (256 GB), you can start your  MAX_HEAP_SIZE  by 8GB and increase if
> necessary.
>
> Since you have only 1 physical core if i understood , you can set your 
> HEAP_NEWSIZE
> to 100 MB
>
>
>
> Best regards
>
> _
>
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>
>
>
>
>
>
>
> On Sat, Mar 16, 2019 at 1:07 AM Jon Haddad  wrote:
>
> I can't say I've ever used 100MB new gen with Cassandra, but in my
> experience I've found small new gen to be incredibly harmful for
> performance.  It doesn't surprise me at all that you'd hit some serious GC
> issues.  My guess is you're filling up the new gen very quickly and
> promoting everything in very quick cycles, leading to memory fragmentation
> and soon after full GCs.  2GB is a tiny heap and I would never, under any
> circumstances, run a 2GB heap in a production environment.  I'd only use
> under 8 GB in a circle CI free tier for integration tests.
>
>
>
> I suggest you use a minimum of 8, preferably 12-16GB of total heap with
> 50% of that as a starting point.  There's a bunch of posts floating around
> on the topic, here's one I wrote:
> http://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>
>
>
> Jon
>
>
>
> On Sat, Mar 16, 2019 at 5:49 PM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
> Here you go. Thanks
>
> - name: MAX_HEAP_SIZE
>
>   value: 2048M
>
> - name: MY_POD_NAMESPACE
>
>   valueFrom:
>
> fieldRef:
>
>   apiVersion: v1
>
>   fieldPath: metadata.namespace
>
> - name: HEAP_NEWSIZE
>
>   value: 100M
>
>
>
>
>
>
>
> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
> *Sent:* Friday, March 15, 2019 11:18 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: read request is slow
>
>
>
> Is it possible to have thes

Re: read request is slow

[Jeff Jirsa]

Also https://github.com/aragozin/jvm-tools

Especially
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/docs/TTOP.md



On Sun, Mar 17, 2019 at 9:04 AM Dieudonné Madishon NGAYA 
wrote:

> Hi,
> Below some different tools to monitor cassandra:
> 1) Nodetool
> Nodetool has many options
> 2)Jconsole
> 3) Opscenter
> 4) tools like top, htop, vmstats, sar, dstat etc ... are also very usefull.
>
> Best regards
> _
>
> [image:
> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
> 
>   
> 
>
> *Dieudonne Madishon NGAYA*
> Datastax, Cassandra Architect
> *P: *7048580065
> *w: *www.dmnbigdata.com
> *E: *dmng...@dmnbigdata.com
> *Private E: *dmng...@gmail.com
> *A: *Charlotte,NC,28273, USA
>
>
> On Sun, Mar 17, 2019 at 12:48 AM Sundaramoorthy, Natarajan <
> natarajan_sundaramoor...@optum.com> wrote:
>
>> Also would like to know  what monitoring should I setup so that if it
>> happens again I can provide more information. Thanks
>>
>>
>>
>>
>>
>> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
>> *Sent:* Saturday, March 16, 2019 5:25 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: read request is slow
>>
>>
>>
>> I'm guessing you're getting 100MB from the comments in the config, which
>> suggest 100MB per core.  This advice is pretty outdated and should be
>> updated.
>>
>>
>>
>> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
>> suggest reading up on how GC works, I linked to a post in an earlier email.
>>
>>
>>
>> These are the flags you'd need to set in your jvm.options, or
>> jvm-server.options depending on the version you're using:
>>
>>
>>
>> -Xmx8G
>>
>> -Xms8G
>>
>> -Xmn4G
>>
>>
>>
>> 1 core is probably going to be a problem, Cassandra creates a lot of
>> threads and relies on doing work concurrently.  I wouldn't use less than 8
>> cores in a production environment.
>>
>>
>>
>> On Sun, Mar 17, 2019 at 3:12 AM Dieudonné Madishon NGAYA <
>> dmng...@gmail.com> wrote:
>>
>> Starting point for me: max_heap_size to 8gb and heap_newsize to 100mb.
>> Then restart node by node then watch system.log to see if you are seeing G.C
>>
>>
>>
>> On Sat, Mar 16, 2019 at 9:56 AM Sundaramoorthy, Natarajan <
>> natarajan_sundaramoor...@optum.com> wrote:
>>
>> So you guys are suggesting
>>
>>
>>
>> MAX_HEAP_SIZE  by 8/12/16GB
>>
>>
>>
>> And
>>
>>
>>
>> HEAP_NEWSIZE to 100 MB
>>
>>
>>
>> And
>>
>>
>>
>> heap with 50% of that as a starting point? Hw do I do this?
>>
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
>> *Sent:* Saturday, March 16, 2019 12:15 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: read request is slow
>>
>>
>>
>> I agreed with jon haddad , your MAX_HEAP_SIZE is very small. you have lot
>> of RAM (256 GB), you can start your  MAX_HEAP_SIZE  by 8GB and increase if
>> necessary.
>>
>> Since you have only 1 physical core if i understood , you can set your 
>> HEAP_NEWSIZE
>> to 100 MB
>>
>>
>>
>> Best regards
>>
>> _
>>
>>
>> [image:
>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>> 
>>   
>> 
>>
>> *Dieudonne Madishon NGAYA*
>> Datastax, Cassandra Architect
>> *P: *7048580065
>> *w: *www.dmnbigdata.com
>> *E: *dmng...@dmnbigdata.com
>> *Private E: *dmng...@gmail.com
>> *A: *Charlotte,NC,28273, USA
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Mar 16, 2019 at 1:07 AM Jon Haddad  wrote:
>>
>> I can't say I've ever used 100MB new gen with Cassandra, but in my
>> experience I've found small new gen to be incredibly harmful for
>> performance.  It doesn't surprise me at all that you'd hit some serious GC
>> issues.  My guess is you're filling up the new gen very quickly and
>> promoting everything in very quick cycles, leading to memory fragmentation
>> and soon after full GCs.  2GB is a tiny heap and I would never, under any
>> circumstances, run a 2GB heap in a production environment.  I'd only use
>> under 8 GB in a circle CI free tier for integration tests.
>>
>>
>>
>> I suggest you use a minimum of 8, preferably 12-16GB of total heap with
>> 50% of that as a starting point.  There's a bunch of posts floating around
>> on the topic, here's one I wrote:
>> http://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>>
>>
>>
>> Jon
>>
>>
>>
>> On Sat, Mar 16, 2019 at 5:49 PM Sundaramoorthy, Natarajan <
>> natarajan_sundaramoor...@optum.com> wrote:
>>
>> Here you go. Thanks
>>
>> - name: MAX_HEAP_SIZE
>>
>>   value: 2048M
>>
>> - name: MY_POD_NAMESPACE
>>
>>   valueFrom:

Re: read request is slow

[Dieudonné Madishon NGAYA]

For your information,since cassandra 3.0, it includes ttop and other
options inside sjk
nodetool sjk
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSjk.html

Best regards
_

[image:
https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]

      


*Dieudonne Madishon NGAYA*
Datastax, Cassandra Architect
*P: *7048580065
*w: *www.dmnbigdata.com
*E: *dmng...@dmnbigdata.com
*Private E: *dmng...@gmail.com
*A: *Charlotte,NC,28273, USA


On Sun, Mar 17, 2019 at 12:39 PM Jeff Jirsa  wrote:

> Also https://github.com/aragozin/jvm-tools
>
> Especially
> https://github.com/aragozin/jvm-tools/blob/master/sjk-core/docs/TTOP.md
>
>
>
> On Sun, Mar 17, 2019 at 9:04 AM Dieudonné Madishon NGAYA <
> dmng...@gmail.com> wrote:
>
>> Hi,
>> Below some different tools to monitor cassandra:
>> 1) Nodetool
>> Nodetool has many options
>> 2)Jconsole
>> 3) Opscenter
>> 4) tools like top, htop, vmstats, sar, dstat etc ... are also very
>> usefull.
>>
>> Best regards
>> _
>>
>> [image:
>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>> 
>>   
>> 
>>
>> *Dieudonne Madishon NGAYA*
>> Datastax, Cassandra Architect
>> *P: *7048580065
>> *w: *www.dmnbigdata.com
>> *E: *dmng...@dmnbigdata.com
>> *Private E: *dmng...@gmail.com
>> *A: *Charlotte,NC,28273, USA
>>
>>
>> On Sun, Mar 17, 2019 at 12:48 AM Sundaramoorthy, Natarajan <
>> natarajan_sundaramoor...@optum.com> wrote:
>>
>>> Also would like to know  what monitoring should I setup so that if it
>>> happens again I can provide more information. Thanks
>>>
>>>
>>>
>>>
>>>
>>> *From:* Jon Haddad [mailto:j...@jonhaddad.com]
>>> *Sent:* Saturday, March 16, 2019 5:25 PM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: read request is slow
>>>
>>>
>>>
>>> I'm guessing you're getting 100MB from the comments in the config, which
>>> suggest 100MB per core.  This advice is pretty outdated and should be
>>> updated.
>>>
>>>
>>>
>>> I'd use 8GB total heap and 4GB new gen as a starting point.  I really
>>> suggest reading up on how GC works, I linked to a post in an earlier email.
>>>
>>>
>>>
>>> These are the flags you'd need to set in your jvm.options, or
>>> jvm-server.options depending on the version you're using:
>>>
>>>
>>>
>>> -Xmx8G
>>>
>>> -Xms8G
>>>
>>> -Xmn4G
>>>
>>>
>>>
>>> 1 core is probably going to be a problem, Cassandra creates a lot of
>>> threads and relies on doing work concurrently.  I wouldn't use less than 8
>>> cores in a production environment.
>>>
>>>
>>>
>>> On Sun, Mar 17, 2019 at 3:12 AM Dieudonné Madishon NGAYA <
>>> dmng...@gmail.com> wrote:
>>>
>>> Starting point for me: max_heap_size to 8gb and heap_newsize to 100mb.
>>> Then restart node by node then watch system.log to see if you are seeing G.C
>>>
>>>
>>>
>>> On Sat, Mar 16, 2019 at 9:56 AM Sundaramoorthy, Natarajan <
>>> natarajan_sundaramoor...@optum.com> wrote:
>>>
>>> So you guys are suggesting
>>>
>>>
>>>
>>> MAX_HEAP_SIZE  by 8/12/16GB
>>>
>>>
>>>
>>> And
>>>
>>>
>>>
>>> HEAP_NEWSIZE to 100 MB
>>>
>>>
>>>
>>> And
>>>
>>>
>>>
>>> heap with 50% of that as a starting point? Hw do I do this?
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com]
>>> *Sent:* Saturday, March 16, 2019 12:15 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: read request is slow
>>>
>>>
>>>
>>> I agreed with jon haddad , your MAX_HEAP_SIZE is very small. you have
>>> lot of RAM (256 GB), you can start your  MAX_HEAP_SIZE  by 8GB and increase
>>> if necessary.
>>>
>>> Since you have only 1 physical core if i understood , you can set your 
>>> HEAP_NEWSIZE
>>> to 100 MB
>>>
>>>
>>>
>>> Best regards
>>>
>>> _
>>>
>>>
>>> [image:
>>> https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour]
>>> 
>>>   
>>> 
>>>
>>> *Dieudonne Madishon NGAYA*
>>> Datastax, Cassandra Architect
>>> *P: *7048580065
>>> *w: *www.dmnbigdata.com
>>> *E: *dmng...@dmnbigdata.com
>>> *Private E: *dmng...@gmail.com
>>> *A: *Charlotte,NC,28273, USA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Mar 16, 2019 at 1:07 AM Jon Haddad  wrote:
>>>
>>> I can't say I've ever used 100MB new gen with Cassandra, but in my
>>> experience I've found small new gen to be incredibly harmful for
>>> performance.  It doesn't surprise me at all t