On 12/20/2016 06:50 PM, Cathy Zhang wrote:
Hi Bernard,
Thanks for the email. I will take a look at this. Xiaodong has been working on
tempest test scripts.
I will work with Xiaodong on this issue.
I've added a comment to the issue which refers to upstream SQLAlchemy
issue https://bitbucket.org/zzzeek/sqlalchemy/issues/3803 as a potential
contributor, though looking at the logs linked from the issue it appears
that database deadlocks are also occurring which may also be a precursor
here. There are many improvements in SQLAlchemy 1.1 such that the
"rollback()" state should not be as susceptible to a corrupted database
connection as seems to be the case here.
Cathy
-----Original Message-----
From: Bernard Cafarelli [mailto:bcafa...@redhat.com]
Sent: Tuesday, December 20, 2016 3:00 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [networking-sfc] Intermittent database transaction
issues, affecting the tempest gate
Hi everyone,
we have an open bug (thanks Igor for the report) on DB transaction issues:
https://bugs.launchpad.net/networking-sfc/+bug/1630503
The thing is, I am seeing quite a few tempest gate failures that follow the
same pattern: at some point in the test suite, the service gets warnings/errors
from the DB layer (reentrant call, closed transaction, nested rollback, …), and
all following tests fail.
This affects both master and stable/newton branches (not many changes for now
in the DB parts between these branches)
Some examples:
* https://review.openstack.org/#/c/400396/ failed with console log
http://logs.openstack.org/96/400396/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/c27920b/console.html#_2016-12-16_12_44_47_564544
and service log
http://logs.openstack.org/96/400396/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/c27920b/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-16_12_44_32_301
* https://review.openstack.org/#/c/405391/ failed,
http://logs.openstack.org/91/405391/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/7e2b1de/console.html.gz#_2016-12-16_13_05_17_384323
and
http://logs.openstack.org/91/405391/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/7e2b1de/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-16_13_04_11_840
* another on master branch: https://review.openstack.org/#/c/411194/
with
http://logs.openstack.org/94/411194/1/gate/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/90633de/console.html.gz#_2016-12-15_22_36_15_216260
and
http://logs.openstack.org/94/411194/1/gate/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/90633de/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-15_22_35_53_310
I took a look at the errors, but only found old-and-apparently-fixed pymysql
bugs, and suggestions like:
*
http://docs.sqlalchemy.org/en/latest/faq/sessions.html#this-session-s-transaction-has-been-rolled-back-due-to-a-previous-exception-during-flush-or-similar
* https://review.openstack.org/#/c/230481/
Not really my forte, so if someone could take a look at these logs and fix the
problem, it would be great! Especially with the upcoming multinode tempest gate
Thanks,
--
Bernard Cafarelli
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
