On Wed, Oct 23, 2013 at 1:20 PM, Sean Dague <s...@dague.net> wrote: > One of the efforts that we're working on from the QA team is tooling that > ensures we aren't stack tracing into our test logs during normal tempest > runs. Random stack traces are scary to cloud admins consuming OpenStack > logs, and exceptions in the logs should really be exceptional events (and > indicative of a failing system), not something that we do by default. Our > intent is to gate code on clean logs (no stacktraces) eventually (i.e. if > you try to land a patch that causes stack traces in OpenStack, that becomes > a failing condition), and we've got an incremental white list based > approach that should let us make forward progress on that. But on that > thread - http://lists.openstack.org/**pipermail/openstack-dev/2013-** > October/017012.html<http://lists.openstack.org/pipermail/openstack-dev/2013-October/017012.html>we > exposed another issue... across projects, OpenStack is very inconsistent > with logging. > > First... baseline, these are the logging levels that we have in OpenStack > today (with numeric values, higher = worse): > > CRITICAL = 50 > FATAL = CRITICAL > ERROR = 40 > WARNING = 30 > WARN = WARNING > AUDIT = 21 # invented for oslo-logging > INFO = 20 > DEBUG = 10 > NOTSET = 0 > > We also have TRACE, which isn't a level per say, it happens at another > level. However TRACE is typically an ERROR in the way we use it. > > > Some examples of oddities in the current system (all from a single > devstack/tempest run): > > Example 1: > ============== > > n-conductor log in tempest/devstack - http://logs.openstack.org/70/** > 52870/3/check/check-tempest-**devstack-vm-full/f46b756/logs/** > screen-n-cond.txt.gz<http://logs.openstack.org/70/52870/3/check/check-tempest-devstack-vm-full/f46b756/logs/screen-n-cond.txt.gz> > > Total log lines: 84076 > Total non DEBUG lines: 61 > > Question: do we need more than 1 level of DEBUG? 3 orders of magnitude > information change between INFO -> DEBUG seems too steep a cliff. > > Example 2: > ============== > > ceilometer-collector - http://logs.openstack.org/70/** > 52870/3/check/check-tempest-**devstack-vm-full/f46b756/logs/** > screen-ceilometer-collector.**txt.gz<http://logs.openstack.org/70/52870/3/check/check-tempest-devstack-vm-full/f46b756/logs/screen-ceilometer-collector.txt.gz> > > AUDIT log level being used as DEBUG level (even though it's higher > than INFO). > > 2013-10-23 12:24:20.093 26234 AUDIT ceilometer.pipeline [-] Flush pipeline > meter_pipeline > 2013-10-23 12:24:20.093 26234 AUDIT ceilometer.pipeline [-] Flush pipeline > cpu_pipeline > 2013-10-23 12:24:20.094 26234 AUDIT ceilometer.pipeline [-] Flush pipeline > meter_pipeline > 2013-10-23 12:24:20.094 26234 AUDIT ceilometer.pipeline [-] Flush pipeline > cpu_pipeline > > (this is every second, for most seconds, for the entire run) > > Example 3: > =============== > > cinder-api - http://logs.openstack.org/70/** > 52870/3/check/check-tempest-**devstack-vm-full/f46b756/logs/** > screen-c-api.txt.gz?level=**ERROR<http://logs.openstack.org/70/52870/3/check/check-tempest-devstack-vm-full/f46b756/logs/screen-c-api.txt.gz?level=ERROR> > ERROR level being used for 404s of volumes > > Example 4: > =============== > glance-api - http://logs.openstack.org/70/**52870/3/check/check-tempest-** > devstack-vm-full/f46b756/logs/**screen-g-api.txt.gz<http://logs.openstack.org/70/52870/3/check/check-tempest-devstack-vm-full/f46b756/logs/screen-g-api.txt.gz> > > 2013-10-23 12:23:27.436 23731 ERROR glance.store.sheepdog [-] Error in > store configuration: Unexpected error while running command.Command: > collieExit code: 127Stdout: ''Stderr: '/bin/sh: 1: collie: not found\n' > 2013-10-23 12:23:27.436 23731 WARNING glance.store.base [-] Failed to > configure store correctly: Store sheepdog could not be configured > correctly. Reason: Error in store configuration: Unexpected error while > running command.Command: collieExit code: 127Stdout: ''Stderr: '/bin/sh: 1: > collie: not found\n' Disabling add method. > > part of every single Tempest / Devstack run, even though we aren't trying > to configure sheepdog in the gate > > > I think we can, and should do better, and started trying to brain dump > into this etherpad - https://etherpad.openstack.**org/p/icehouse-logging-* > *harmonization<https://etherpad.openstack.org/p/icehouse-logging-harmonization>(examples > included). > > This is one of those topics that I think our current 6 track summit model > doesn't make easy address, as we really need general concensus across any > project that's using oslo-logging, so I believe mailing list is the better > option, at least for now. > > > Goals - Short Term > =============== > As much feedback as possible from both core projects and openstack > deployers about the kinds of things that they believe we should be logging, > and the kinds of levels they think those things should land at. >
Deprecation warnings! Based on the approach we're taking in the patch below, we'll be able to notate how imminently a feature is facing deprecation. Right now, they're just landing in WARNING, but I think we'll surely a desire to silence, prioritize or redirect those messages using different log levels (for example, based on how imminently a feature is facing deprecation). https://review.openstack.org/#/c/50486/ We're about to officially deprecate the v2.0 API in keystone, which tempest is obviously testing and existing clients are still using, so we'll be producing a LOT of such warnings very soon as a kick in the pants to move clients to v3 (and ensure v3 has parity with v2). > Determining how crazy it is to try to harmonize this across services. > > Figure out who else wants to help. Where help means: > * helping figure out what's already concensus in services > * helping figure out things that are really aberrant from that concensus > * helping build concensus with various core teams on a common > * helping with contributions to projects that are interested in > contributions to move them closer to the concensus > > Determining if everyone just hates the idea, and I should give up now. :) > (That is a valid response to this RFC, feel free to put that out there). > > > Goals - Long Term > =============== > A set of guidelines on logging standards so that OpenStack as a whole > feels more whole when it comes to dealing with the log data. > > These are going to be guidelines, not rules. Some projects are always > going to have unique needs. But I suspect a lot of developers would be > happy with a cheet sheet that helps them figure out what log level > different things are presented at. > > > Comments / rotten fruit are welcomed. Also welcomed are any additional POV > on this issue. > > -Sean > > -- > Sean Dague > http://dague.net > > ______________________________**_________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.**org <OpenStack-dev@lists.openstack.org> > http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> > -- -Dolph
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
