Re: [openstack-dev] [all] gate debugging

On Aug 28, 2014, at 2:16 PM, Sean Dague wrote: > On 08/28/2014 02:07 PM, Joe Gordon wrote: >> >> >> >> On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague > > wrote: >> >>On 08/28/2014 12:48 PM, Doug Hellmann wrote: >>> >>> On Aug 27, 2014, at 5:56 PM, Sean Dague >

Re: [openstack-dev] [all] gate debugging

On Aug 28, 2014, at 2:15 PM, Sean Dague wrote: > On 08/28/2014 01:48 PM, Doug Hellmann wrote: >> >> On Aug 28, 2014, at 1:17 PM, Sean Dague wrote: >> >>> On 08/28/2014 12:48 PM, Doug Hellmann wrote: On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: > On 08/27/2014 05:27 PM,

Re: [openstack-dev] [all] gate debugging

On Thu, Aug 28, 2014 at 5:17 AM, Thierry Carrez wrote: > David Kranz wrote: > > On 08/27/2014 03:43 PM, Sean Dague wrote: > >> On 08/27/2014 03:33 PM, David Kranz wrote: > >>> Race conditions are what makes debugging very hard. I think we are in > >>> the process of experimenting with such an ide

Re: [openstack-dev] [all] gate debugging

On 08/28/2014 02:07 PM, Joe Gordon wrote: > > > > On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague > wrote: > > On 08/28/2014 12:48 PM, Doug Hellmann wrote: > > > > On Aug 27, 2014, at 5:56 PM, Sean Dague > wrote: > > > >> On 0

Re: [openstack-dev] [all] gate debugging

On 08/28/2014 01:48 PM, Doug Hellmann wrote: > > On Aug 28, 2014, at 1:17 PM, Sean Dague wrote: > >> On 08/28/2014 12:48 PM, Doug Hellmann wrote: >>> >>> On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: >>> On 08/27/2014 05:27 PM, Doug Hellmann wrote: > > On Aug 27, 2014, at 2:54 PM,

Re: [openstack-dev] [all] gate debugging

On Thu, Aug 28, 2014 at 12:44 PM, Doug Hellmann wrote: > I usually use the functions for editing ENABLED_SERVICES. Is it still > common to edit the variable directly? > > Not generally. It was looking at it in log files to see what was/was not enabled where I started to think about that. The de

Re: [openstack-dev] [all] gate debugging

On Thu, Aug 28, 2014 at 10:17 AM, Sean Dague wrote: > On 08/28/2014 12:48 PM, Doug Hellmann wrote: > > > > On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: > > > >> On 08/27/2014 05:27 PM, Doug Hellmann wrote: > >>> > >>> On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > >>> > Note: thread in

Re: [openstack-dev] [all] gate debugging

On Aug 28, 2014, at 1:17 PM, Sean Dague wrote: > On 08/28/2014 12:48 PM, Doug Hellmann wrote: >> >> On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: >> >>> On 08/27/2014 05:27 PM, Doug Hellmann wrote: On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > Note: thread intentiona

Re: [openstack-dev] [all] gate debugging

On Aug 28, 2014, at 1:00 PM, Dean Troyer wrote: > On Thu, Aug 28, 2014 at 11:48 AM, Doug Hellmann wrote: > In my case, a neutron call failed. Most of the other services seem to have a > *-api.log file, but neutron doesn’t. It took a little while to find the > API-related messages in screen-q-

Re: [openstack-dev] [all] gate debugging

On 08/28/2014 12:48 PM, Doug Hellmann wrote: > > On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: > >> On 08/27/2014 05:27 PM, Doug Hellmann wrote: >>> >>> On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: >>> Note: thread intentionally broken, this is really a different topic. On 08/

Re: [openstack-dev] [all] gate debugging

On Thu, Aug 28, 2014 at 11:48 AM, Doug Hellmann wrote: > In my case, a neutron call failed. Most of the other services seem to have > a *-api.log file, but neutron doesn’t. It took a little while to find the > API-related messages in screen-q-svc.txt (I’m glad I’ve been around long > enough to kn

Re: [openstack-dev] [all] gate debugging

On Aug 27, 2014, at 5:56 PM, Sean Dague wrote: > On 08/27/2014 05:27 PM, Doug Hellmann wrote: >> >> On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: >> >>> Note: thread intentionally broken, this is really a different topic. >>> >>> On 08/27/2014 02:30 PM, Doug Hellmann wrote:> On Aug 27,

Re: [openstack-dev] [all] gate debugging

David Kranz wrote: > On 08/27/2014 03:43 PM, Sean Dague wrote: >> On 08/27/2014 03:33 PM, David Kranz wrote: >>> Race conditions are what makes debugging very hard. I think we are in >>> the process of experimenting with such an idea: asymetric gating by >>> moving functional tests to projects, mak

Re: [openstack-dev] [all] gate debugging

As it has been pointed out previously in this thread debugging gate failures is mostly about chasing race conditions, which in some cases involve the most disparate interactions between Openstack services [1]. Finding the root cause of these races is a mix of knowledge, pragmatism, and luck. Havin

Re: [openstack-dev] [all] gate debugging

On Wed, Aug 27, 2014 at 05:47:09PM -0400, Doug Hellmann wrote: > > On Aug 27, 2014, at 5:27 PM, Doug Hellmann wrote: > > > > > On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > > > >> Note: thread intentionally broken, this is really a different topic. > >> > >> On 08/27/2014 02:30 PM, Doug H

Re: [openstack-dev] [all] gate debugging

On 08/27/2014 05:27 PM, Doug Hellmann wrote: > > On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > >> Note: thread intentionally broken, this is really a different topic. >> >> On 08/27/2014 02:30 PM, Doug Hellmann wrote:> >>> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: >>> On Wed, 27 Au

Re: [openstack-dev] [all] gate debugging

On Aug 27, 2014, at 5:27 PM, Doug Hellmann wrote: > > On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > >> Note: thread intentionally broken, this is really a different topic. >> >> On 08/27/2014 02:30 PM, Doug Hellmann wrote:> >>> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: >>> On

Re: [openstack-dev] [all] gate debugging

On Aug 27, 2014, at 2:54 PM, Sean Dague wrote: > Note: thread intentionally broken, this is really a different topic. > > On 08/27/2014 02:30 PM, Doug Hellmann wrote:> >> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: >> >>> On Wed, 27 Aug 2014, Doug Hellmann wrote: >>> I have found it

Re: [openstack-dev] [all] gate debugging

On 08/27/2014 03:43 PM, Sean Dague wrote: > On 08/27/2014 03:33 PM, David Kranz wrote: >> On 08/27/2014 02:54 PM, Sean Dague wrote: >>> Note: thread intentionally broken, this is really a different topic. >>> >>> On 08/27/2014 02:30 PM, Doug Hellmann wrote:> On Aug 27, 2014, at 1:30 PM, Chris

Re: [openstack-dev] [all] gate debugging

On 08/27/2014 03:43 PM, Sean Dague wrote: On 08/27/2014 03:33 PM, David Kranz wrote: On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote:> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: O

Re: [openstack-dev] [all] gate debugging

On 08/27/2014 03:33 PM, David Kranz wrote: > On 08/27/2014 02:54 PM, Sean Dague wrote: >> Note: thread intentionally broken, this is really a different topic. >> >> On 08/27/2014 02:30 PM, Doug Hellmann wrote:> >>> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: >>> On Wed, 27 Aug 2014, Doug H

Re: [openstack-dev] [all] gate debugging

On 08/27/2014 02:54 PM, Sean Dague wrote: Note: thread intentionally broken, this is really a different topic. On 08/27/2014 02:30 PM, Doug Hellmann wrote:> On Aug 27, 2014, at 1:30 PM, Chris Dent wrote: On Wed, 27 Aug 2014, Doug Hellmann wrote: I have found it immensely helpful, for examp

Re: [openstack-dev] [all] gate debugging

On 2014-08-27 14:54:55 -0400 (-0400), Sean Dague wrote: [...] > I think we break down on communication when we get into a > conversation of "I want to learn gate debugging" because I don't > quite know what that means, or where the starting point of > understanding is. So those intentions are well