Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

2018-01-25 Thread Clay Gerrard
On Thu, Jan 25, 2018 at 7:01 PM, Matt Riedemann wrote: > Is ThreadSafeSysLogHandler something that could live in oslo.log so we > don't have to whack this mole everywhere at random times? That might make sense, unless we can get eventlet's monkey patching of the logging module to do something s

Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

2018-01-25 Thread Matt Riedemann
On 1/25/2018 6:41 PM, Clay Gerrard wrote: Does it help that swift also had to fix this? https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415 The interesting/useful bit is where we replace our primary loghandlers createLock method to use o

Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

2018-01-25 Thread Clay Gerrard
Does it help that swift also had to fix this? https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415 The interesting/useful bit is where we replace our primary loghandlers createLock method to use one of these [Green|OS]-thread-safe PipeMutex

[openstack-dev] Swift-backed volume backups are still breaking the gate

2018-01-25 Thread Matt Riedemann
We thought things were fixed with [1] but it turns out that swiftclient logs requests and responses at DEBUG level, so we're still switching thread context during a backup write and failing the backup operation, causing copious amounts of pain in the gate and piling up the rechecks. I've got a