Root cause of the instance boot failures is app armor on the neutron-
gateway blocking neutron agents from creating temporary directories:
1vlRG/" pid=1412869 comm="neutron-dhcp-ag" requested_mask="c" denied_mask="c"
fsuid=115 ouid=115
[76035.437502] audit: type=1400 audit(1524677252.781:36019): apparmor="DENIED"
operation="mkdir" profile="/usr/bin/neutron-dhcp-agent" name="/tmp/tmp4AIVtB/"
pid=1412869 comm="neutron-dhcp-ag" requested_mask="c" denied_mask="c" fsuid=115
ouid=115
Both the dhcp-agent and the l3-agent both show the problem.
Assigning this bug to neutron-gateway for the app armor bug
A secondary issue that is as yet not root caused is DBConnection errors
from all the API charms connecting to percona cluster. After changing
the neutron-gateway aa-profile-mode to complain we saw these errors much
less frequently but they did not go away entirely.
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
[req-dcf87632-8b6e-4071-a336-64b1442dc7fe - - - - -] Database connection was
found disconnected; reconnecting: DBConnectionError:
(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during
query') [SQL: u'SELECT 1']
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines Traceback (most
recent call last):
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 73, in
_connect_ping_listener
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
connection.scalar(select([1]))
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 877, in
scalar
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines return
self.execute(object, *multiparams, **params).scalar()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in
execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines return
meth(self, multiparams, params)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in
_execute_on_connection
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines return
connection._execute_clauseelement(self, multiparams, params)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in
_execute_clauseelement
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
compiled_sql, distilled_params
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in
_execute_context
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines context)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1398, in
_handle_dbapi_exception
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
util.raise_from_cause(newraise, exc_info)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in
raise_from_cause
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in
_execute_context
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines context)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in
do_execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
cursor.execute(statement, parameters)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 165, in execute
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines result =
self._query(query)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 321, in _query
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
conn.query(q)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 860, in query
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1061, in
_read_query_result
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
result.read()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1349, in read
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
first_packet = self.connection._read_packet()
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 991, in
_read_packet
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
packet_header = self._read_bytes(4)
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines File
"/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1037, in
_read_bytes
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to
MySQL server during query') [SQL: u'SELECT 1']
2018-04-25 17:49:33.562 617800 ERROR oslo_db.sqlalchemy.engines
Next steps:
Please redeploy with neutron-gatway aa-profile-mode=complain while we fix the
apparmor profile bug. The current deploy has been tainted by all of our
debugging attempts. Though we are able to launch instances it no longer
represents a valid test.
** Changed in: charm-neutron-gateway
Status: Invalid => Triaged
** Changed in: charm-neutron-gateway
Importance: Undecided => Critical
** Changed in: charm-neutron-gateway
Assignee: (unassigned) => David Ames (thedac)
** Changed in: charm-neutron-gateway
Milestone: None => 18.05
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1761536
Title:
Nova compute manager failed to create virtual interface
Status in OpenStack neutron-gateway charm:
Triaged
Status in OpenStack neutron-openvswitch charm:
New
Status in OpenStack Compute (nova):
Invalid
Bug description:
Rally test scenario:
NovaServers.boot_server_associate_and_dissociate_floating_ip fails.
All 5 nova-compute-kvm instances timeout:
--------------------------------------------------------------------------------
Task 2ccf3cf6-c252-4e0f-8fdd-ca58ad819aff has 5 error(s)
--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server
s_rally_504bd98b_fLz3akho:23cbd6ad-67f7-4e0f-9095-390f50897b62 to
become ('ACTIVE') current status BUILD
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line
71, in _run_scenario_once
getattr(scenario_inst, method_name)(**scenario_kwargs)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/servers.py",
line 1116, in run
server = self._boot_server(image, flavor, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/rally/task/atomic.py", line
91, in func_atomic_actions
f = func(self, *args, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py",
line 86, in _boot_server
check_interval=CONF.openstack.nova_server_boot_poll_interval
File "/usr/local/lib/python2.7/dist-packages/rally/task/utils.py", line
252, in wait_for_status
timeout=timeout)
TimeoutException: Rally tired waiting 300.00 seconds for Server
s_rally_504bd98b_fLz3akho:23cbd6ad-67f7-4e0f-9095-390f50897b62 to become
('ACTIVE') current status BUILD
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-gateway/+bug/1761536/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
