On Sep 17, 2014, at 8:43 PM, Jay Faulkner <j...@jvf.cc> wrote: > Comments inline. > >> -----Original Message----- >> From: Monty Taylor [mailto:mord...@inaugust.com] >> Sent: Wednesday, September 17, 2014 7:34 PM >> To: openstack-dev@lists.openstack.org >> Subject: Re: [openstack-dev] Log Rationalization -- Bring it on! >> >> On 09/17/2014 04:42 PM, Rochelle.RochelleGrober wrote: >>> TL;DR: I consider the poor state of log consistency a major >>> impediment for more widespread adoption of OpenStack and would like to >>> volunteer to own this cross-functional process to begin to unify and >>> standardize logging messages and attributes for Kilo while dealing >>> with the most egregious issues as the community identifies them. >>> >> >> I fully support this, and I, for one, welcome our new log-standardization >> overlords. >> > > Something that could be interesting is to see if we can emit metrics > everytime a loggable event happens. There's already a spec+code being drafted > for Ironic in Kilo (https://review.openstack.org/#/c/100729/ > &https://review.openstack.org/#/c/103202/) that we're using downstream to > emit metrics from Ironic.
You may be interested to see how Swift has integrated StatsD events into a log adapter. https://github.com/openstack/swift/blob/master/swift/common/utils.py#L1197 See also the StatsdClient class in that same file. --John > > If we have good organization of logging events, and levels, perhaps there's > possibly a way to make it easy for metrics to be emitted at that time as well. > > - > Jay Faulkner > >> >>> >>> Recap from some mail threads: >>> >>> >>> >>> From Sean Dague on Kilo cycle goals: >>> >>> 2. Consistency in southbound interfaces (Logging first) >>> >>> >>> >>> Logging and notifications are south bound interfaces from OpenStack >>> providing information to people, or machines, about what is going on. >>> >>> There is also a 3rd proposed south bound with osprofiler. >>> >>> >>> >>> For Kilo: I think it's reasonable to complete the logging standards >>> and implement them. I expect notifications (which haven't quite kicked >>> off) are going to take 2 cycles. >>> >>> >>> >>> I'd honestly *really* love to see a unification path for all the the >>> southbound parts, logging, osprofiler, notifications, because there is >>> quite a bit of overlap in the instrumentation/annotation inside the >>> main code for all of these. >>> >>> >>> And from Doug Hellmann: 1. Sean has done a lot of analysis and started >>> a spec on standardizing logging guidelines where he is gathering input >>> from developers, deployers, and operators [1]. >>> Because it is far enough for us to see real progress, it's a good >>> place for us to start experimenting with how to drive cross-project >>> initiatives involving code and policy changes from outside of a single >>> project. We have a couple of potentially related specs in Oslo as part >>> of the oslo.log graduation work [2] [3], but I think most of the work >>> will be within the applications. >>> >>> [1] https://review.openstack.org/#/c/91446/ [2] >>> https://blueprints.launchpad.net/oslo.log/+spec/app-agnostic-logging-p >>> arameters >>> >>> >> [3] https://blueprints.launchpad.net/oslo.log/+spec/remove-context- >> adapter >>> >>> >>> >>> And from James Blair: >>> >>> 1) Improve log correlation and utility >>> >>> >>> >>> If we're going to improve the stability of OpenStack, we have to be >>> able to understand what's going on when it breaks. That's both true >>> as developers when we're trying to diagnose a failure in an >>> integration test, and it's true for operators who are all too often >>> diagnosing the same failure in a real deployment. Consistency in >>> logging across projects as well as a cross-project request token would >>> go a long way toward this. >>> >>> While I am not currently managing an OpenStack deployment, writing >>> tests or code, or debugging the stack, I have spent many years doing >>> just that. Through QA, Ops and Customer support, I have come to revel >>> in good logging and log messages and curse the holes and vagaries in >>> many systems. >>> >>> Defining/refining logs to be useful and usable is a cross-functional >>> effort that needs to include: >>> >>> · Operators >>> >>> · QA >>> >>> · End Users >>> >>> · Community managers >>> >>> · Tech Pubs >>> >>> · Translators >>> >>> · Developers >>> >>> · TC (which provides the forum and impetus for all the >>> projects to cooperate on this) >>> >>> At the moment, I think this effort may best work under the auspices of >>> Oslo (oslo.log), I'd love to hear other proposals. >>> >>> Here is the beginnings of my proposal of how to attack and subdue the >>> painful state of logs: >>> >>> >>> · Post this email to the MLs (dev, ops, enduser) to get >>> feedback, garner support and participants in the process (Done;-) >>> >>> · In parallel: >>> >>> o Collect up problems, issues, ideas, solutions on an etherpad >>> https://etherpad.openstack.org/p/Log-Rationalization where anyone in >>> the communities can post. >>> >>> o Categorize reported Log issues into classes (already identified >>> classes): >>> >>> § Format Consistency across projects >>> >>> § Log level definition and categorization across classes >>> >>> § Time syncing entries across tens of logfiles >>> >>> § Relevancy/usefulness of information provided within messages >>> >>> § Etc (missing a lot here, but I'm sure folks will speak up) >>> >>> o Analyze existing log message formats, standards across integrated >>> projects >>> >>> o File bugs where issues identified are actual project bugs >>> >>> o Build a session outline for F2F working session at the Paris >>> Design Summit >>> >>> · At the Paris Design Summit, use a session and/or pod >>> discussions to set priorities, recruit contributors, start and/or >>> flesh out specs and blueprints >>> >>> · Proceed according to priorities, specs, blueprints, >>> contributions and changes as needed as the work progresses. >>> >>> · Keep an active and open rapport and reporting process for >>> the user community to comment and participate in the processes. >>> Measures of success: >>> >>> · Log messages provide consistency of format enough for >>> productive mining through operator writable scripts >>> >>> · Problem debugging is simplified through the ability to >>> trust timestamps across all OpenStack logs (and use scripts to get to >>> the time you want in any/all of the logfiles) >>> >>> · Standards for format, content, levels and translations have >>> been proposed and agreed to be adopted across all OpenStack integrated >>> projects >>> >>> · The user communities demonstrate an increased level of >>> trust and decreased level of frustration with OpenStack logging >>> (surveys, bug reports, other measures?) >>> >>> · The log team can disband >>> >>> I expect that getting the logs in very good shape will take more than >>> just the Kilo timeframe, but once momentum has built, which should >>> happened during Kilo, the process should move very quickly. A lot of >>> this could be handled through "while you're in there" or "low hanging >>> fruit" once the standards are established. The bigger win will be if >>> we can ensure what we define/design is extensible over the longer life >>> of OpenStack. >>> >>> --Rocky >>> >>> >>> >>> >>> _______________________________________________ OpenStack- >> dev mailing >>> list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
