The above issue is caused since kolla-ansible and ceph-ansible both use
the same router_id('51') for keepalived configuration. A different
router id for kolla-ansible can be set with
'keepalived_virtual_router_id' option in globals.yml.
This issue can be marked as solved now.
** No longer affects: glance
** Changed in: kolla
Status: New => Incomplete
** Changed in: kolla
Status: Incomplete => Opinion
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1954430
Title:
Openstack image command-line operations fail for large images in Kolla
Victoria
Status in kolla:
Opinion
Bug description:
**Bug Report**
What happened:
Openstack image upload and save commands fail with the following
'Connection reset by peer' error, for only large images.
# openstack image save windows-server2016 --file /tmp/t --debug
GET call to image for
http://172.16.201.254:9292/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file
used request id req-d15c2664-b776-4486-8f12-fb0eb2bdfef8
Unable to download image: ("Connection broken: ConnectionResetError(104,
'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by
peer'))
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line
438, in _error_catcher
yield
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line
519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/lib64/python3.6/http/client.py", line 463, in read
n = self.readinto(b)
File "/usr/lib64/python3.6/http/client.py", line 507, in readinto
n = self.fp.readinto(b)
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 758,
in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line
576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line
541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line
455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken:
ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py",
line 57, in download
chunk_size=chunk_size):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 761,
in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken:
ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cliff/app.py", line 407, in
run_subcommand
result = cmd.run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/osc_lib/command/command.py",
line 39, in run
return super(Command, self).run(parsed_args)
File "/usr/local/lib/python3.6/site-packages/cliff/command.py", line 186,
in run
return_code = self.take_action(parsed_args) or 0
File
"/usr/local/lib/python3.6/site-packages/openstackclient/image/v2/image.py",
line 847, in take_action
image_client.download_image(image.id, stream=True, output=output_file)
File "/usr/local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py",
line 451, in download_image
self, stream=stream, output=output, chunk_size=chunk_size)
File "/usr/local/lib/python3.6/site-packages/openstack/image/_download.py",
line 65, in download
"Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection
broken: ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
clean_up SaveImage: Unable to download image: ("Connection broken:
ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
END return value: 1
The haproxy logs show a 'CD--' status, which suggest that the client
aborted the connection.
Dec 10 11:46:58 node003 haproxy[43]: 172.16.201.1:51626
[10/Dec/2021:11:46:47.358] glance_api_front glance_api_back/node001
0/0/0/219/11319 200 1132157897 - - CD-- 297/3/0/0/0 0/0 "GET
/v2/images/d2eb0d3c-b455-4c18-9cbe-4f6941044d7d/file HTTP/1.1"
The glance-api server logs the following 'Connection reset by peer'
error, same as openstack cmdline client.
node001: 2021-12-10 11:52:15.247 58 INFO eventlet.wsgi.server
[req-4af3af95-6363-4857-ad82-7a30c784bc40 a8814484a71c4c5193282bcfd5e6ff3f
fea7b584dcc343c19c65498e099ffd80 - default default] Traceback (most recent call
last):
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line
591, in handle_one_response
node001: write(b''.join(towrite))
node001: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line
537, in write
node001: wfile.writelines(towrite)
node001: File "/usr/lib64/python3.6/socket.py", line 604, in write
node001: return self._sock.send(b)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py",
line 397, in send
node001: return self._send_loop(self.fd.send, data, flags)
node001: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py",
line 384, in _send_loop
node001: return send_method(data, *args)
node001: ConnectionResetError: [Errno 104] Connection reset by peer
Can this issue be resolved by modifying the default haproxy configuration?
What you expected to happen:
The 'openstack image create' and 'openstack image save' commands
should execute succesfully.
How to reproduce it (minimal and precise):
Try to upload a 40GB+ image with 'openstack image create' command, in
a Kolla Victoria setup, in High availability mode, with Haproxy and
virtual IPs.
**Environment**:
* OS : CentOS Linux 8
* Kernel : 4.18.0-305.3.1.el8.x86_64
* Docker version : 20.10.8(Docker Engine - Community)
* Kolla-ansible version : 11.1.0
* Openstack version : Victoria
To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla/+bug/1954430/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
