Public bug reported: We discovered that if the nova metadata service is down, then the neutron metadata service starts printing stack traces with a 500 HTTP code to the user.
Demo on a newly installed devstack $ systemctl stop devstack@n-api-meta.service Then inside a VM: $ curl http://169.254.169.254/latest/meta-data/hostname <html> <head> <title>500 Internal Server Error</title> </head> <body> <h1>500 Internal Server Error</h1> An unknown error has occurred. Please try your request again.<br /><br /> </body> </html>$ Stack trace: ERROR neutron.agent.metadata.agent Traceback (most recent call last): ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 85, in __call__ ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 249, in _proxy_request ERROR neutron.agent.metadata.agent resp = requests.request(method=req.method, url=url, ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request ERROR neutron.agent.metadata.agent return session.request(method=method, url=url, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request ERROR neutron.agent.metadata.agent resp = self.send(prep, **send_kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send ERROR neutron.agent.metadata.agent r = adapter.send(request, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send ERROR neutron.agent.metadata.agent raise ConnectionError(e, request=request) ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.136.16.184', port=8775): Max retries exceeded with url: /latest/meta-data/hostname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ce93f38b0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')) ERROR neutron.agent.metadata.agent INFO eventlet.wsgi.server [-] ::ffff:192.168.100.14,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1392403 Also, in our installation the nova service is behind nginx. And if we stop nova metadata service we also get 500 http code but with another traceback: 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent [-] Unexpected error.: Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 93, in __call__ 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 288, in _proxy_request 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent resp.status_code) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent 2024-03-25 20:27:01.988 24 INFO eventlet.wsgi.server [-] 10.197.115.207,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1369441 It seems to me that it is also better to handle nginx-like gateway errors a bit more correctly. These 500 HTTP codes worries us because we are trying to create an alert system one of the criteria for which is 500 codes. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2059032 Title: Neutron metadata service returns http code 500 if nova metadata service is down Status in neutron: New Bug description: We discovered that if the nova metadata service is down, then the neutron metadata service starts printing stack traces with a 500 HTTP code to the user. Demo on a newly installed devstack $ systemctl stop devstack@n-api-meta.service Then inside a VM: $ curl http://169.254.169.254/latest/meta-data/hostname <html> <head> <title>500 Internal Server Error</title> </head> <body> <h1>500 Internal Server Error</h1> An unknown error has occurred. Please try your request again.<br /><br /> </body> </html>$ Stack trace: ERROR neutron.agent.metadata.agent Traceback (most recent call last): ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 85, in __call__ ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 249, in _proxy_request ERROR neutron.agent.metadata.agent resp = requests.request(method=req.method, url=url, ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request ERROR neutron.agent.metadata.agent return session.request(method=method, url=url, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request ERROR neutron.agent.metadata.agent resp = self.send(prep, **send_kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send ERROR neutron.agent.metadata.agent r = adapter.send(request, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send ERROR neutron.agent.metadata.agent raise ConnectionError(e, request=request) ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.136.16.184', port=8775): Max retries exceeded with url: /latest/meta-data/hostname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ce93f38b0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')) ERROR neutron.agent.metadata.agent INFO eventlet.wsgi.server [-] ::ffff:192.168.100.14,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1392403 Also, in our installation the nova service is behind nginx. And if we stop nova metadata service we also get 500 http code but with another traceback: 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent [-] Unexpected error.: Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 93, in __call__ 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 288, in _proxy_request 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent resp.status_code) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent 2024-03-25 20:27:01.988 24 INFO eventlet.wsgi.server [-] 10.197.115.207,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1369441 It seems to me that it is also better to handle nginx-like gateway errors a bit more correctly. These 500 HTTP codes worries us because we are trying to create an alert system one of the criteria for which is 500 codes. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2059032/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
