Re: [openstack-dev] [Neutron] Issue with pymysql

On 8 July 2015 at 14:30, Salvatore Orlando wrote: > I agree and I would make the switch as soon as possible. The graphite > graph you posted showed that since 6/28 the difference in failure rate is > such that isn't even statistically significant. However, spikes in failure > rates of the unstabl

Re: [openstack-dev] [Neutron] Issue with pymysql

I agree and I would make the switch as soon as possible. The graphite graph you posted showed that since 6/28 the difference in failure rate is such that isn't even statistically significant. However, spikes in failure rates of the unstable job also suggest that you're starting to chase a moving ta

Re: [openstack-dev] [Neutron] Issue with pymysql

Hi, Another brief update on the matter: Failure rate trends [1] are showing that unstable (w/ multiple API workers + pymysql driver) and stable configurations (w/o) are virtually aligned and I am proposing that it is time to drop the unstable infra configuration [2,3] that allowed the team to tri

Re: [openstack-dev] [Neutron] Issue with pymysql

Hi, A brief update on the issue that sparked this thread: A little over a week ago, bug [1] was filed. The gist of that was that the switch to pymysql unveiled a number of latent race conditions that made Neutron unstable. To try and nip these in the bud, the Neutron team filed a number of patch

Re: [openstack-dev] [Neutron] Issue with pymysql

Sean, Thanks for quick fix/revert https://review.openstack.org/#/c/191010/ This unblocked Rally gates... Best regards, Boris Pavlovic On Fri, Jun 12, 2015 at 8:56 PM, Clint Byrum wrote: > Excerpts from Mike Bayer's message of 2015-06-12 09:42:42 -0700: > > > > On 6/12/15 11:37 AM, Mike Bayer w

Re: [openstack-dev] [Neutron] Issue with pymysql

Excerpts from Mike Bayer's message of 2015-06-12 09:42:42 -0700: > > On 6/12/15 11:37 AM, Mike Bayer wrote: > > > > > > On 6/11/15 9:32 PM, Eugene Nikanorov wrote: > >> Hi neutrons, > >> > >> I'd like to draw your attention to an issue discovered by rally gate job: > >> http://logs.openstack.org/9

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/12/15 11:37 AM, Mike Bayer wrote: On 6/11/15 9:32 PM, Eugene Nikanorov wrote: Hi neutrons, I'd like to draw your attention to an issue discovered by rally gate job: http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE I

Re: [openstack-dev] [Neutron] Issue with pymysql

This is exactly what I did in [1]. Cheers, Armando [1] https://review.openstack.org/#/q/status:open+branch:master+topic:neutron-unstable,n,z On 12 June 2015 at 09:00, Clint Byrum wrote: > Excerpts from Chris Dent's message of 2015-06-12 03:47:02 -0700: > > On Fri, 12 Jun 2015, Joe Gordon wrote

Re: [openstack-dev] [Neutron] Issue with pymysql

Excerpts from Chris Dent's message of 2015-06-12 03:47:02 -0700: > On Fri, 12 Jun 2015, Joe Gordon wrote: > > > Glad to see us catch these issues early. > > Yes! CI is doing exactly the job it is supposed to be doing here. It > is finding bugs in code. When that happens we should fix the bugs, >

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/12/15 6:42 AM, Sean Dague wrote: On 06/12/2015 06:31 AM, Joe Gordon wrote: On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague mailto:s...@dague.net>> wrote: On 06/12/2015 01:17 AM, Salvatore Orlando wrote: > It is however interesting that both "lock wait timeouts" and "missing >

Re: [openstack-dev] [Neutron] Issue with pymysql

On 6/11/15 9:32 PM, Eugene Nikanorov wrote: Hi neutrons, I'd like to draw your attention to an issue discovered by rally gate job: http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE I don't have bandwidth to take a deep look

Re: [openstack-dev] [Neutron] Issue with pymysql

On Fri, 12 Jun 2015, Joe Gordon wrote: Glad to see us catch these issues early. Yes! CI is doing exactly the job it is supposed to be doing here. It is finding bugs in code. When that happens we should fix the bugs, not revert. Even if it stalls other stuff. -- Chris Dent tw:@anticdent freeno

Re: [openstack-dev] [Neutron] Issue with pymysql

On 06/12/2015 06:31 AM, Joe Gordon wrote: > > > On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague > wrote: > > On 06/12/2015 01:17 AM, Salvatore Orlando wrote: > > It is however interesting that both "lock wait timeouts" and "missing > > savepoint" errors occur in o

Re: [openstack-dev] [Neutron] Issue with pymysql

On Fri, Jun 12, 2015 at 7:13 PM, Sean Dague wrote: > On 06/12/2015 01:17 AM, Salvatore Orlando wrote: > > It is however interesting that both "lock wait timeouts" and "missing > > savepoint" errors occur in operations pertaining the same table - > > securitygroups in this case. > > I wonder if th

Re: [openstack-dev] [Neutron] Issue with pymysql

On 06/12/2015 01:17 AM, Salvatore Orlando wrote: > It is however interesting that both "lock wait timeouts" and "missing > savepoint" errors occur in operations pertaining the same table - > securitygroups in this case. > I wonder if the switch to pymysl has not actually uncovered some other > bug

Re: [openstack-dev] [Neutron] Issue with pymysql

It is however interesting that both "lock wait timeouts" and "missing savepoint" errors occur in operations pertaining the same table - securitygroups in this case. I wonder if the switch to pymysl has not actually uncovered some other bug in Neutron. I have no opposition to a revert, but since th

[openstack-dev] [Neutron] Issue with pymysql

Hi neutrons, I'd like to draw your attention to an issue discovered by rally gate job: http://logs.openstack.org/96/190796/4/check/gate-rally-dsvm-neutron-rally/7a18e43/logs/screen-q-svc.txt.gz?level=TRACE I don't have bandwidth to take a deep look at it, but first impression is that it is some i