Hi Matt, Thank you for the quick reply and information - I had not considered file handles as a culprit. If the issue recurs, I will dig into that.
We did find some anomalies on the virtual machine where the mod_auth_cas behavior manifested (outdated VMware tools and really old VM hardware version). Not sure if those were related but we've since updated both. Will keep this list posted with our findings and experience going forward. Thank you again for your time, suggestion, and expertise - it is appreciated! -Neil On Monday, May 22, 2017 at 10:07:44 AM UTC-6, matt wrote: > > Hi Neil, > > Without the logs, it is difficult to tell. It /could/ be related to time > drift, but I'd find it unlikely that that would prevent writing to disk. > > More likely, I'd investigate number of open file handles. Did some httpd > sub-process (e.g., a CGI or PHP) possibly create an egregious number of > handles? This would likely show in error messages printed to the logs. > lsof could also be your friend here. > > Matt > > > On May 19, 2017 11:15 AM, "Neil Sabol" <[email protected] <javascript:>> > wrote: > > Hello CAS Community, > > I hope this message finds you all well. > > As time permits, I am hoping to pick your brains about a mysterious issue > we experienced recently with mod_auth_cas (suspect it was not mod_auth_cas > itself but something related). > > We have been running mod_auth_cas (version 1.1) in production for a long > time without incident. Yesterday, we began to experience a strange behavior > on one of our production servers: > > > - mod_auth_cas stopped creating cookies in the defined CASCookiePath > (no users were able to login to the application - all requests for > CAS-protected resources resulted in a redirect back to the CAS login page > and a 401 error upon return to the application) > > - Debug logs did not reveal anything interesting - the only related > entries I noticed were the following > > > *[debug] mod_auth_cas.c(930): [client X.X.X.X] Cache entry > 'ae0aa61bf431d62b9e4be00089e87df8' could not be opened, referer: > http://something.unm.edu <http://something.unm.edu> [debug] > mod_auth_cas.c(1676): [client X.X.X.X] Cookie > 'ae0aa61bf431d62b9e4be00089e87df8' is corrupt or invalid, referer: > http://something.unm.edu <http://something.unm.edu>* > > - Permissions, file system status, etc. were all good - from all > appearances, mod_auth_cas was not attempting to create cookies in the > CASCookiePath (confirmed apache could write to the path, etc.) > > - The CASCookiePath directory contained only a .metadata file about > 2-3 hours after this issue started occurring > > > We ended up using the IT hammer to restore the affected VM from snapshot, > so I no longer have the specific logs or state of the system available. The > restore did the trick (mod_auth_cas resumed normal operation and began > creating cookies in the CASCookiePath), but I am concerned this issue may > recur. > > The only possible explanation for this that I can think of (in hindsight) > is time drift between the application server/clients/cas server. Does that > sound possible? If yes, would something like that be logged with debug > logging enabled? > > If you have any insight or guidance into what could cause this sort of > situation with mod_auth_cas, please let me know. > > Thank you in advance for your time and expertise! > -Neil > > -- > - CAS gitter chatroom: https://gitter.im/apereo/cas > - CAS mailing list guidelines: > https://apereo.github.io/cas/Mailing-Lists.html > - CAS documentation website: https://apereo.github.io/cas > - CAS project website: https://github.com/apereo/cas > --- > You received this message because you are subscribed to the Google Groups > "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/9b0635b6-657c-4b2e-a091-3acd4b0fec1c%40apereo.org > > <https://groups.google.com/a/apereo.org/d/msgid/cas-user/9b0635b6-657c-4b2e-a091-3acd4b0fec1c%40apereo.org?utm_medium=email&utm_source=footer> > . > > > -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html - CAS documentation website: https://apereo.github.io/cas - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/ffd07c74-3657-4072-961b-94539cfcd01c%40apereo.org.
