Public bug reported: I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04. I recognized in the logs that Nova sometimes fails during vm creation with the following exception:
2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last): 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise_exc=False) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return self.request(url, 'GET', **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return wrapped(*args, **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity resp = send(**kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise exceptions.SSLError(msg) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Keystone's public endpoint should only visible to external clients. All internal OpenStack services should use the internalURL for authentication purposes. I think my configuration is correct. The "auth_url" point to Keystone's internal URL, whereas "auth_uri" points to Keystone's public endpoint. The strange thing is, that sometimes after a service restart, Nova uses the Keystone's internal URL and sometimes the Keystone's public URL. I want to avoid https based communication for the internal cloud services. $ openstack endpoint list | grep keystone | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone | identity | True | internal | http://os-identity.mycompany.com:5000/v3 | | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone | identity | True | public | https://os-cloud.mycompany.com:5000/v3 | | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone | identity | True | admin | http://os-identity.mycompany.com:35357/v3 | ################ nova.conf ################ ... [keystone_authtoken] auth_type = password auth_uri = http://os-cloud.mycompany.com:5000 auth_url = http://os-identity:35357 memcached_servers = os-memcache:11211 password = novapass project_domain_name = default project_name = service user_domain_name = default username = nova ... Using the option "insecure = True" is a workaround to avoid that Nova sometimes fails when the service uses Keystone's public https endpoint. Can someone please have a look? ** Affects: nova Importance: Undecided Status: New ** Description changed: I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04. I recognized in the logs that Nova sometimes fails during vm creation with the following exception: - 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.materna.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) + 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last): 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise_exc=False) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return self.request(url, 'GET', **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return wrapped(*args, **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity resp = send(**kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise exceptions.SSLError(msg) - 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.materna.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) - 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity + 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) + 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Keystone's public endpoint should only visible to external clients. All internal OpenStack services should use the internalURL for authentication purposes. I think my configuration is correct. The "auth_url" point to Keystone's internal URL, whereas "auth_uri" points to Keystone's public endpoint. The strange thing is, that sometimes after a service restart, Nova uses the Keystone's internal URL and sometimes the Keystone's public URL. I want to avoid https based communication for the internal cloud services. $ openstack endpoint list | grep keystone - | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone | identity | True | internal | http://os-identity.materna.com:5000/v3 | - | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone | identity | True | public | https://os-cloud.materna.com:5000/v3 | - | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone | identity | True | admin | http://os-identity.materna.com:35357/v3 | + | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone | identity | True | internal | http://os-identity.mycompany.com:5000/v3 | + | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone | identity | True | public | https://os-cloud.mycompany.com:5000/v3 | + | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone | identity | True | admin | http://os-identity.mycompany.com:35357/v3 | ################ nova.conf ################ ... [keystone_authtoken] auth_type = password - auth_uri = http://os-cloud.materna.com:5000 + auth_uri = http://os-cloud.mycompany.com:5000 auth_url = http://os-identity:35357 memcached_servers = os-memcache:11211 password = novapass project_domain_name = default project_name = service user_domain_name = default username = nova ... Using the option "insecure = True" is a workaround to avoid that Nova sometimes fails when the service uses Keystone's public https endpoint. Can someone please have a look? -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1716344 Title: Nova-API sometimes uses Keystone's public endpoint Status in OpenStack Compute (nova): New Bug description: I have setup a fresh HA deployment of OpenStack Pike on Ubuntu 16.04. I recognized in the logs that Nova sometimes fails during vm creation with the following exception: 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity [req-6efab9e1-78f5-4e85-8247-686ff4f3568c dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] Unable to contact keystone to verify project_id: SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Traceback (most recent call last): 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/nova/api/openstack/identity.py", line 42, in verify_project_id 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise_exc=False) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 845, in get 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return self.request(url, 'GET', **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity return wrapped(*args, **kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity resp = send(**kwargs) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 765, in _send_request 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity raise exceptions.SSLError(msg) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity SSLError: SSL exception connecting to https://os-cloud.mycompany.com:5000/v3/projects/ed2d2efd86dd40e7a45491d8502318d3: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",) 2017-09-11 09:31:28.909 5604 ERROR nova.api.openstack.identity Keystone's public endpoint should only visible to external clients. All internal OpenStack services should use the internalURL for authentication purposes. I think my configuration is correct. The "auth_url" point to Keystone's internal URL, whereas "auth_uri" points to Keystone's public endpoint. The strange thing is, that sometimes after a service restart, Nova uses the Keystone's internal URL and sometimes the Keystone's public URL. I want to avoid https based communication for the internal cloud services. $ openstack endpoint list | grep keystone | 00a22bfee72141ddadacd0e357161075 | RegionOne | keystone | identity | True | internal | http://os-identity.mycompany.com:5000/v3 | | 7178e534cb4e4c5e9a67066ff3e9c433 | RegionOne | keystone | identity | True | public | https://os-cloud.mycompany.com:5000/v3 | | f5ed3bba70274d7fa03f2ceaab96c3c9 | RegionOne | keystone | identity | True | admin | http://os-identity.mycompany.com:35357/v3 | ################ nova.conf ################ ... [keystone_authtoken] auth_type = password auth_uri = http://os-cloud.mycompany.com:5000 auth_url = http://os-identity:35357 memcached_servers = os-memcache:11211 password = novapass project_domain_name = default project_name = service user_domain_name = default username = nova ... Using the option "insecure = True" is a workaround to avoid that Nova sometimes fails when the service uses Keystone's public https endpoint. Can someone please have a look? To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1716344/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
