Just did some more debugging it looks like the "nodetool compactionstats" which is hung/taking time during this period causing the delay in metrics. I still puzzled why the nodetool compactionstats commands takes longer on all the nodes at the same time, when one node is being restarted
$ time nodetool compactionstats > pending tasks: 0 > > real 1m17.559s > user 0m2.340s > sys 0m0.248s On Tue, Jun 2, 2020 at 10:25 AM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Also during this time, I am losing metrics for all the nodes in the > cluster (metrics agent is timing out collecting within 10s) and recovers > once the node starts the CQL port. Is there any known issue which could > cause this? In my case the delay between Gossip settle and CQL port open is > 3 minutes, metrics were lost for all the nodes during the 3 minute period. > > On Tue, Jun 2, 2020 at 7:55 AM Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> Thank you, >> >> Does that mean there is no way to improve this delay? And i have to live >> with it since i have more tables? >> >> On Tuesday, June 2, 2020, Durity, Sean R <sean_r_dur...@homedepot.com> >> wrote: >> >>> As I understand it, Cassandra clusters should be limited to a number of >>> tables in the low hundreds (under 200), at most. What you are seeing is the >>> carving up of memtables for each of those 3,000. I try to limit my clusters >>> to roughly 100 tables. >>> >>> >>> >>> >>> >>> Sean Durity >>> >>> >>> >>> *From:* Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com> >>> *Sent:* Tuesday, June 2, 2020 10:48 AM >>> *To:* user@cassandra.apache.org >>> *Subject:* [EXTERNAL] Re: Cassandra Bootstrap Sequence >>> >>> >>> >>> 3000 tables >>> >>> On Tuesday, June 2, 2020, Durity, Sean R <sean_r_dur...@homedepot.com> >>> wrote: >>> >>> How many total tables in the cluster? >>> >>> >>> >>> >>> >>> Sean Durity >>> >>> >>> >>> *From:* Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com> >>> *Sent:* Monday, June 1, 2020 8:36 PM >>> *To:* user@cassandra.apache.org >>> *Subject:* [EXTERNAL] Re: Cassandra Bootstrap Sequence >>> >>> >>> >>> Thanks Erick, >>> >>> >>> >>> I see below tasks are being run mostly. I didn't quite understand what >>> exactly these scheduled tasks are for? Is there a way to reduce the boot-up >>> time or do I have to live with this delay? >>> >>> >>> >>> $ zgrep "CompactionStrategyManager.java:380 - Recreating compaction >>> strategy" debug.log* | wc -l >>> 3249 >>> $ zgrep "DiskBoundaryManager.java:53 - Refreshing disk boundary cache >>> for" debug.log* | wc -l >>> 6293 >>> $ zgrep "DiskBoundaryManager.java:92 - Got local ranges" debug.log* | >>> wc -l >>> 6308 >>> $ zgrep "DiskBoundaryManager.java:56 - Updating boundaries from >>> DiskBoundaries" debug.log* | wc -l >>> 3249 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Jun 1, 2020 at 5:01 PM Erick Ramirez <erick.rami...@datastax.com> >>> wrote: >>> >>> There's quite a lot of steps that takes place during the startup >>> sequence between these 2 lines: >>> >>> >>> >>> >>> *INFO [main] 2020-05-31 23:51:15,555 Gossiper.java:1723 - No gossip >>> backlog; proceeding *INFO [main] 2020-05-31 23:54:06,867 >>> NativeTransportService.java:70 - Netty using native Epoll event loop >>> >>> >>> >>> For the most part, it's taken up by CompactionStrategyManager and >>> DiskBoundaryManager. If you check debug.log, you'll see that it's >>> mostly updating disk boundaries. The length of time it takes is >>> proportional to the number of tables in the cluster. >>> >>> >>> >>> Have a look at this section [1] of CassandraDaemon if you're interested >>> in the details of the startup sequence. Cheers! >>> >>> >>> >>> [1] >>> https://github.com/apache/cassandra/blob/cassandra-3.11.3/src/java/org/apache/cassandra/service/CassandraDaemon.java#L399-L435 >>> [github.com] >>> <https://urldefense.com/v3/__https:/github.com/apache/cassandra/blob/cassandra-3.11.3/src/java/org/apache/cassandra/service/CassandraDaemon.java*L399-L435__;Iw!!M-nmYVHPHQ!dt_R3xGLIK4vc3FdekacgZnl6PDJVAqW_c-yBaIAmQsoVKp7SoW7VeM3gc7VSLx2KgcKBSE$> >>> >>> >>> ------------------------------ >>> >>> >>> The information in this Internet Email is confidential and may be >>> legally privileged. It is intended solely for the addressee. Access to this >>> Email by anyone else is unauthorized. If you are not the intended >>> recipient, any disclosure, copying, distribution or any action taken or >>> omitted to be taken in reliance on it, is prohibited and may be unlawful. >>> When addressed to our clients any opinions or advice contained in this >>> Email are subject to the terms and conditions expressed in any applicable >>> governing The Home Depot terms of business or client engagement letter. The >>> Home Depot disclaims all responsibility and liability for the accuracy and >>> content of this attachment and for any damages or losses arising from any >>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >>> items of a destructive nature, which may be contained in this attachment >>> and shall not be liable for direct, indirect, consequential or special >>> damages in connection with this e-mail message or its attachment. >>> >>> >>> ------------------------------ >>> >>> The information in this Internet Email is confidential and may be >>> legally privileged. It is intended solely for the addressee. Access to this >>> Email by anyone else is unauthorized. If you are not the intended >>> recipient, any disclosure, copying, distribution or any action taken or >>> omitted to be taken in reliance on it, is prohibited and may be unlawful. >>> When addressed to our clients any opinions or advice contained in this >>> Email are subject to the terms and conditions expressed in any applicable >>> governing The Home Depot terms of business or client engagement letter. The >>> Home Depot disclaims all responsibility and liability for the accuracy and >>> content of this attachment and for any damages or losses arising from any >>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other >>> items of a destructive nature, which may be contained in this attachment >>> and shall not be liable for direct, indirect, consequential or special >>> damages in connection with this e-mail message or its attachment. >>> >>
