Let me be more specific - run the async java profiler and generate a flame graph to determine where CPU time is spent.
On Wed, Apr 17, 2019 at 11:36 AM Jon Haddad <j...@jonhaddad.com> wrote: > > Run the async java profiler on the node to determine what it's doing: > https://github.com/jvm-profiling-tools/async-profiler > > On Wed, Apr 17, 2019 at 11:31 AM Carl Mueller > <carl.muel...@smartthings.com.invalid> wrote: > > > > No, we just did the package upgrade 2.1.9 --> 2.2.13 > > > > It definitely feels like some indexes are being recalculated or the entire > > sstables are being scanned due to suspected corruption. > > > > > > On Wed, Apr 17, 2019 at 12:32 PM Jeff Jirsa <jji...@gmail.com> wrote: > >> > >> There was a time when changing some of the parameters (especially bloom > >> filter FP ratio) would cause the bloom filters to be rebuilt on startup if > >> the sstables didnt match what was in the schema, leading to a delay like > >> that and similar logs. Any chance you changed the schema on that table > >> since the last time you restarted it? > >> > >> > >> > >> On Wed, Apr 17, 2019 at 10:30 AM Carl Mueller > >> <carl.muel...@smartthings.com.invalid> wrote: > >>> > >>> Oh, the table in question is SizeTiered, had about 10 sstables total, it > >>> was JBOD across two data directories. > >>> > >>> On Wed, Apr 17, 2019 at 12:26 PM Carl Mueller > >>> <carl.muel...@smartthings.com> wrote: > >>>> > >>>> We are doing a ton of upgrades to get out of 2.1.x. We've done probably > >>>> 20-30 clusters so far and have not encountered anything like this yet. > >>>> > >>>> After upgrade of a node, the restart takes a long time. like 10 minutes > >>>> long. ALmost all of our other nodes took less than 2 minutes to upgrade > >>>> (aside from sstableupgrades). > >>>> > >>>> The startup stalls on a particular table, it is the largest table at > >>>> about 300GB, but we have upgraded other clusters with about that much > >>>> data without this 8-10 minute delay. We have the ability to roll back > >>>> the node, and the restart as a 2.1.x node is normal with no delays. > >>>> > >>>> Alas this is a prod cluster so we are going to try to sstable load the > >>>> data on a lower environment and try to replicate the delay. If we can, > >>>> we will turn on debug logging. > >>>> > >>>> This occurred on the first node we tried to upgrade. It is possible it > >>>> is limited to only this node, but we are gunshy to play around with > >>>> upgrades in prod. > >>>> > >>>> We have an automated upgrading program that flushes, snapshots, shuts > >>>> down gossip, drains before upgrade, suppressed autostart on upgrade, and > >>>> has worked about as flawlessly as one could hope for so far for 2.1->2.2 > >>>> and 2.2-> 3.11 upgrades. > >>>> > >>>> INFO [main] 2019-04-16 17:22:17,004 ColumnFamilyStore.java:389 - > >>>> Initializing zzzz.access_token > >>>> INFO [main] 2019-04-16 17:22:17,096 ColumnFamilyStore.java:389 - > >>>> Initializing zzzz.refresh_token > >>>> INFO [main] 2019-04-16 17:28:52,929 ColumnFamilyStore.java:389 - > >>>> Initializing zzzz.userid > >>>> INFO [main] 2019-04-16 17:28:52,930 ColumnFamilyStore.java:389 - > >>>> Initializing zzzz.access_token_by_auth > >>>> > >>>> You can see the 6:30 delay in the startup log above. All the other > >>>> keyspace/tables initialize in under a second. > >>>> > >>>> --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
