On Mon, Jan 16, 2012 at 6:38 AM, Florian Haas <flor...@hastexo.com> wrote: > On Sun, Jan 15, 2012 at 9:27 PM, Andrew Beekhof <and...@beekhof.net> wrote: >> On Thu, Jan 12, 2012 at 11:01 PM, Florian Haas <flor...@hastexo.com> wrote: >>> On Thu, Jan 5, 2012 at 10:15 PM, Florian Haas <flor...@hastexo.com> wrote: >>>> Florian Haas (2): >>>> extra: add rsyslog configuration snippet >>>> extra: add logrotate configuration snippet >>>> >>>> configure.ac | 4 +++ >>>> extra/Makefile.am | 2 +- >>>> extra/logrotate/Makefile.am | 5 ++++ >>>> extra/logrotate/pacemaker.conf.in | 7 ++++++ >>>> extra/rsyslog/Makefile.am | 5 ++++ >>>> extra/rsyslog/pacemaker.conf.in | 39 >>>> +++++++++++++++++++++++++++++++++++++ >>>> 6 files changed, 61 insertions(+), 1 deletions(-) >>>> create mode 100644 extra/logrotate/Makefile.am >>>> create mode 100644 extra/logrotate/pacemaker.conf.in >>>> create mode 100644 extra/rsyslog/Makefile.am >>>> create mode 100644 extra/rsyslog/pacemaker.conf.in >>> >>> Any takers on these? >> >> Sorry, I was off working on the new fencing logic and then corosync >> 2.0 support (when cman and all the plugins, including ours, go away). >> >> So a couple of comments... >> >> I fully agree that the state of our logging needs work and I can >> understand people wanting to keep the vast majority of our logs out of >> syslog. >> I'm less thrilled about one-file-per-subsystem, the cluster will often >> do a lot within a single second and splitting everything up really >> hurts the ability to correlate messages. >> I'd also suggest that /some/ information not coming directly from the >> RAs is still appropriate for syslog (such as "I'm going to move A from >> B to C" or "I'm about to turn of node D"), so the nuclear option isn't >> really thrilling me. > > So everything that is logged by the RAs with ocf_log, as I wrote in > the original post, _is_ still going to whatever the default syslog > destination may be. The rsyslog config doesn't change that at all.
By "Nuclear", I meant nothing at all from Pacemaker. If thats what you want, there's a far easier way to achieve this and keep usable logs around for debugging, set facility to none and add a logfile. > (Stuff that the RAs simply barf out to stdout/err would go to the lrmd > log.) I maintain that this is the stuff that is also most useful to > people. And with just that information in the syslog, you usually get > a pretty clear idea of what the heck the cluster is doing on a node, > and in what order, in about 20 lines of logs close together -- rather > than intermingled with potentially hundreds of lines of other > cluster-related log output. Did I not just finish agreeing that "hundreds of lines of other cluster-related log[s]" was a problem? I just don't think your knee-jerk "everything must go" approach is the answer. > > And disabling the "nuclear option" is a simple means of adding a "#" > before "& ~" in the config file. You can ship it that way by default > if you think that's more appropriate. That way, people would get the > split-out logs _plus_ everything in one file, which IMHO is sometimes > very useful for pengine or lrmd troubleshooting/debugging. I, > personally, just don't want Pacemaker to flood my /var/log/messages, Did you see me arguing against that? > so I'd definitely leave the "& ~" in there, but that may be personal > preference. I wonder what others think. > >> In addition to the above distractions, I've been coming up to speed on >> libqb's logging which is opening up a lot of new doors and should >> hopefully help solve the underlying log issues. >> For starters it lets syslog/stderr/logfile all log at different levels >> of verbosity (and formats), it also supports blackboxes of which a >> dump can be triggered in response to an error condition or manually by >> the admin. >> >> The plan is something along the lines of: syslog gets NOTICE and >> above, anything else (depending on debug level and trace options) goes >> to /var/log/(cluster/?)pacemaker or whatever was configured in >> corosync. >> However, before I can enact that there will need to be an audit of the >> messages currently going to INFO (674 entries) and NOTICE(160 entries) >> with some getting bumped up, others down (possibly even to debug). >> I'd certainly be interested in feedback as to which logs should and >> should not make it. > > Yes, even so, I (again, this is personal preference) would definitely > not want pengine logging (which even if half its INFO messages get > demoted to DEBUG, would still be pretty verbose) in my default > messages file. Sigh, please take time out from preaching to actually read the replies. You might learn something. Your precious default messages file wouldn't be getting /any/ INFO logs from pacemaker. And I'm guessing your complaints are based on 1.0 logging too right? Because for a long time now, the PE in 1.1 has only logged NOTICE and above by default, which means about 1 additional line per resource + errors/warnings (and that will soon drop to 1 per resource changing state). >> If you want to get analytical about it, there is also an awk script >> that I use when looking at what we log. >> I'd be interested in some numbers from the field. > > Thanks; I can look at that after LCA. > > Cheers, > Florian > > -- > Need help with High Availability? > http://www.hastexo.com/now > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
