+1 And IMHO, the explicit cleaner is not such a good idea for distributed reg. impls that employ their own strategies for cache invalidation.
D. On Oct 17, 2016, 04:12 -0400, Tom Poage <[email protected]>, wrote: > Disabling the registry cleaner brought load average on our (4) servers down > to 0.01-0.20 (from 4.0-15.0). > > cas.properties: > > ticket.registry.cleaner.startdelay=-1 > > (value could have been zero, but -1 seemed more mnemonic of the intent) > > Tom. > > > On Oct 14, 2016, at 1:28 PM, Tom Poage <[email protected] > > (mailto:[email protected])> wrote: > > Looks like we found the source of the load issue. > > > > Best we can tell, somewhere about 4.2.5 the RegistryCleaner embedded in the > > DefaultTicketRegistry was refactored into a TicketRegistryCleaner that’s > > now automatically picked up and started for all registry types (*). This > > cleaner walks the entire cache map, by default every two minutes, and by > > chunks with an exclusive lock. Multiply that by 50k entries and several > > servers all competing to do the same thing and it’s no wonder there’s some > > load. :-) > > > > Question now is whether to disable on all nodes, or enable on only one in > > the cluster. Caches like Hazelcast and Ehcache have a time-to-live eviction > > policy, so it seems to me the registry cleaner is unnecessary for this type > > of cache. > > > > The CAS code suggests the cleaner can be disabled, albeit somewhat > > indirectly, by setting the “ticket.registry.cleaner.startdelay” property to > > less than or equal to zero. > > > > Tom. > > > > * > > https://github.com/apereo/cas/commit/c1cbde11c5722e1930357d3dc3bdb6d4cffa8214 > > > > From: Misagh Moayyed <[email protected] (mailto:[email protected])> > > Date: Friday, October 14, 2016 at 10:43 AM > > To: Tom Poage <[email protected] (mailto:[email protected])>, CAS > > Community <[email protected] (mailto:[email protected])> > > Subject: Re: [cas-user] Server load w/ 4.2.6 > > > > > > You can exclude the hazelcast dependency from the relevant module, and > > provide your own exact version. > > > > -- > > Misagh > > > > From: Tom Poage <[email protected]> (mailto:[email protected]) > > Reply: Tom Poage <[email protected]> (mailto:[email protected]) > > Date: October 14, 2016 at 8:44:49 PM > > To: CAS Community <[email protected]> (mailto:[email protected]) > > Subject: Re: [cas-user] Server load w/ 4.2.6 > > > > > > > Disabling the fourth node doesn't change anything. > > > > > > Profiling shows the highest CPU/time is spent in Hazelcast. Whether this > > > is a result of the updated Hazelcast version or the new synchronous CAS > > > code remains to be seen. > > > > > > Is it oossible to downgrade Hazelcast version (say, to 3.6) on CAS 4.2.6, > > > i.e. were any new Hazelcast version-specific changes made between roughly > > > 4.2.[12] and 4.2.6? > > > > > > Thanks. > > > Tom. > > > > > > > On Oct 13, 2016, at 2:18 PM, Tom Poage <[email protected] > > > > (mailto:[email protected])> wrote: > > > > br/>> Afternoon, < > > > > br/>> On moving from 4.2.1 to 4.2.6, our apparent syystem load > > > > increased dramatically. > > > > br/>> Run queue went from as high as 4 to nearly 30, with (Linux) load > > > > average jumping from a max of 0.2 to about 15 for a user base (TGT > > > > count) of 46k. > > > > br/>> A code diff doesn’t seem to show much, exxcept perhaps for the > > > > addition of a synchronous ticketTransactionManager. The only other > > > > likely candidate is either the bump in Hazelcast version, or that we > > > > went from 3 to 4 (single CPU) VMs in the cluster (point-to-point > > > > instead of multicast). CPU increased from a high of about 20% (usually > > > > 5-8%) to the 50% range. This is on all nodes. Ironically, response time > > > > doesn’t seem all that bad, though is a bit sluggish. > > > > br/>> Anyone else experience something similar?? > > > > br/>> Thanks!! > > > > Tom. > > > > br/>> -- br/>> CAS gitter chatroom: https://gitter.im/apereo/cas < > > > > CAS mailing list guidelines: > > > > https://apereo.github.io/cas/Mailing-Lists.html > > > > CAS documentation website: https://apereo.github.io/cas > > > > CAS project website: https://github.com/apereo/cas > > > > --- br/>> You received this message because you are ssubscribed to the > > > > Google Groups "CAS Community" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [email protected] > > > > (mailto:[email protected]). > > > > To post to this group, send email to [email protected] > > > > (mailto:[email protected]). > > > > Visit this group at > > > > https://groups.google.com/a/apereo.org/group/cas-user/. > > > > To view this discussion on the web visit > > > > https://groups.google.com/a/apereo.org/d/msgid/cas-user/F67D31AA-2CFC-4DDA-8C5D-922E0B87798F%40ucdavis.edu. > > > > For more options, visit https://groups.google.com/a/apereo.org/d/optout. > > > > > > -- br/>CAS gitter chatroom: https://gitter.im/apereo/cas < > > > CAS mailing list guidelines: > > > https://apereo.github.io/cas/Mailing-Lists.html > > > CAS documentation website: https://apereo.github.io/cas > > > CAS project website: https://github.com/apereo/cas > > > --- br/>You received this message because you are subscribed to the > > > Google Groups "CAS Community" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [email protected] > > > (mailto:[email protected]). > > > To post to this group, send email to [email protected] > > > (mailto:[email protected]). > > > Visit this group at > > > https://groups.google.com/a/apereo.org/group/cas-user/. > > > To view this discussion on the web visit > > > https://groups.google.com/a/apereo.org/d/msgid/cas-user/4FAA8E69-3C2E-46E4-9BB5-9C48E5D40A31%40ucdavis.edu. > > > For more options, visit https://groups.google.com/a/apereo.org/d/optout. > > > > -- > CAS gitter chatroom: https://gitter.im/apereo/cas > CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html > CAS documentation website: https://apereo.github.io/cas > CAS project website: https://github.com/apereo/cas > --- > You received this message because you are subscribed to the Google Groups > "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > (mailto:[email protected]). > To post to this group, send email to [email protected] > (mailto:[email protected]). > Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/B234A09E-75FE-45D7-998B-F634BF40EBAE%40ucdavis.edu > > (https://groups.google.com/a/apereo.org/d/msgid/cas-user/B234A09E-75FE-45D7-998B-F634BF40EBAE%40ucdavis.edu?utm_medium=email&utm_source=footer). > For more options, visit https://groups.google.com/a/apereo.org/d/optout. -- CAS gitter chatroom: https://gitter.im/apereo/cas CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html CAS documentation website: https://apereo.github.io/cas CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/cbd12630-2870-4637-8b2e-95155d9c93d0%40Spark. For more options, visit https://groups.google.com/a/apereo.org/d/optout.
