Re: Cheat Sheet for Unix based OS, Performance troubleshooting

2019-07-30 Thread Julien Laurenceau
Hi, My personnal favorites when Prometheus is not there and/or insufficient are: - iostat -> log all disk metrics volume by volume - nmon -> logs all metrics, but cannot see volume by volume IOPS - nmonvisualizer Regards JL # Monitor ## install dstat nmon iostat ioperf sar ## install nmonvisua

Assassinate or decommission?

2019-07-30 Thread Rahul Reddy
Hello, While removing old data center any specific reason to use assassinate instead of decommission ? https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsDecomissionDC.html

Re: Assassinate or decommission?

2019-07-30 Thread Rhys Campbell
The advice is to only use assassinate when all else fails. Decommission will make sure any data that needs to be streamed elsewhere will be. Generally decommission > removenode > assassinate is the recommended attempt order https://thelastpickle.com/blog/2018/09/18/assassinate.html Rahul Reddy

Re: Assassinate or decommission?

2019-07-30 Thread Rahul Reddy
Thanks Rhys, I was always using nodetool decommission when I do single node sofar. Not sure why datastax doc mention to use first attempt as assassinate. Have you seen any issues like gossip information stayed for longer time using decommission for removing DC? On Tue, Jul 30, 2019, 5:43 AM Rhys

Re: Assassinate or decommission?

2019-07-30 Thread Rhys Campbell
Are you sure it says to use assassinate as the first resort? Definately not the case Rahul Reddy schrieb am Di., 30. Juli 2019, 12:05: > Thanks Rhys, > > I was always using nodetool decommission when I do single node sofar. Not > sure why datastax doc mention to use first attempt as assassinate.

Re: RMI TCP Connection threads

2019-07-30 Thread Vlad
Restarting Cassandra helped. On Monday, July 29, 2019, 9:05:06 AM GMT+3, Dinesh Joshi wrote: Try obtaining a thread dump. It will help debug. Anything that goes via JMX such as nodetool could be responsible for it. Dinesh On Jul 28, 2019, at 10:57 PM, Vlad wrote: Hi, suddenly I not

Re: Assassinate or decommission?

2019-07-30 Thread Oleksandr Shulgin
On Tue, Jul 30, 2019 at 12:11 PM Rhys Campbell wrote: > Are you sure it says to use assassinate as the first resort? Definately > not the case > It does. I think the reason is that it says that you should run a full repair first, and before that--stop writing to the DC being decommissioned. Th

Re: RMI TCP Connection threads

2019-07-30 Thread Oleksandr Shulgin
On Tue, Jul 30, 2019 at 12:34 PM Vlad wrote: > Restarting Cassandra helped. > But for how long?.. -- Alex

Cassandra node down metric

2019-07-30 Thread Rahul Reddy
Hello, I'm using jmx metric node org_apache_cassandra_net_failuredetector_downendpointcount to monitor number of Cassandra nodes down. For any reason (aws schedule retirement) we decommission Cassandra node this metric shows the node down for 72 hours until the gossip clearead. We want keep trac

Repair / compaction for 6 nodes, 2 DC cluster

2019-07-30 Thread Martin Xue
Hello, Good day. This is Martin. Can someone help me with the following query regarding Cassandra repair and compaction? Currently we have a large keyspace (keyspace_event) with 1TB of data (in /var/lib/cassandra/data/keyspace_event); There is a cluster with Datacenter 1 contains 3 nodes, Data c