Public bug reported:
Recently, nothing changed on the test case, but we got this failure:
https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html
LOG:
Traceback (most recent call last):
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
utils.wait_until_true(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py",
line 79, in wait_until_true
while not predicate():
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 153, in no_fip_pf_connectivity
return not fip_pf_connectivity(6)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 146, in fip_pf_connectivity
self.check_servers_hostnames(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py",
line 537, in check_servers_hostnames
ssh_client.get_hostname())
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 292, in get_hostname
return self.exec_command('hostname')
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 333, in wrapped_f
return self(f, *args, **kw)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 423, in __call__
do = self.iter(retry_state=retry_state)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 360, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in
__get_result
raise self._exception
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 426, in __call__
result = fn(*args, **kwargs)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 171, in exec_command
return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in exec_command
channel.exec_command(cmd)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 72, in _check
return func(self, *args, **kwds)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 257, in exec_command
self._wait_for_event()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 1219, in _wait_for_event
self.event.wait()
File "/usr/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py",
line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
Some search in the neutron LP bug list, there are some old bugs related to this
project "paramiko":
https://bugs.launchpad.net/neutron/+bug/1892861
https://bugs.launchpad.net/neutron/+bug/1571486
https://bugs.launchpad.net/neutron/+bug/1253896
The issue seems to be _wait_for_event is waiting for something, and
finally get timeout.
After some searching in the repo of paramiko in github, I saw this issue:
https://github.com/paramiko/paramiko/pull/1248
With a commit for this:
https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
But it is not included in any release version.
Some thoughts:
1. if it is possible to not use paramiko?
2. run (ssh) commands directly by some oslo libs?
3. ping paramiko maintainers to release once to see if the test success rate
can get better
** Affects: neutron
Importance: Undecided
Status: New
** Affects: tempest
Importance: Undecided
Status: New
** Also affects: tempest
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926109
Title:
SSH timeout (wait timeout) due to potential paramiko issue
Status in neutron:
New
Status in tempest:
New
Bug description:
Recently, nothing changed on the test case, but we got this failure:
https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html
LOG:
Traceback (most recent call last):
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
utils.wait_until_true(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py",
line 79, in wait_until_true
while not predicate():
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 153, in no_fip_pf_connectivity
return not fip_pf_connectivity(6)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py",
line 146, in fip_pf_connectivity
self.check_servers_hostnames(
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py",
line 537, in check_servers_hostnames
ssh_client.get_hostname())
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 292, in get_hostname
return self.exec_command('hostname')
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 333, in wrapped_f
return self(f, *args, **kw)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 423, in __call__
do = self.iter(retry_state=retry_state)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 360, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in
__get_result
raise self._exception
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py",
line 426, in __call__
result = fn(*args, **kwargs)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py",
line 171, in exec_command
return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in
exec_command
channel.exec_command(cmd)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 72, in _check
return func(self, *args, **kwds)
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 257, in exec_command
self._wait_for_event()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py",
line 1219, in _wait_for_event
self.event.wait()
File "/usr/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File
"/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py",
line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
Some search in the neutron LP bug list, there are some old bugs related to
this project "paramiko":
https://bugs.launchpad.net/neutron/+bug/1892861
https://bugs.launchpad.net/neutron/+bug/1571486
https://bugs.launchpad.net/neutron/+bug/1253896
The issue seems to be _wait_for_event is waiting for something, and
finally get timeout.
After some searching in the repo of paramiko in github, I saw this issue:
https://github.com/paramiko/paramiko/pull/1248
With a commit for this:
https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
But it is not included in any release version.
Some thoughts:
1. if it is possible to not use paramiko?
2. run (ssh) commands directly by some oslo libs?
3. ping paramiko maintainers to release once to see if the test success rate
can get better
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926109/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
