Sometimes you just got to start with a clean white board. With a complete stack rebuild nova seems to be working just fine. Some changes with the rebuild :-
- Provisioned glance, cinder and ceph , nova on same. As this scales out compute will have its own dedicated stack. - glance, cinder and nova haproxy services are now able to use default ports. Don’t think this was an issue earlier, but… - separate stack for mariadb cluster and identity. - Ceph same as before on its dedicated backend cluster. Facing some lock timeout issues with memcached for apache 2 authentication . Changed driver to use sql backend for now. Next on to neutron and layer3 integration. Hopefully this helps someone else. Thanks Rajiv Duggal rdug...@dex.com<mailto:rdug...@dex.com> From: Rajiv Duggal Sent: Monday, January 11, 2016 12:02 PM To: openstack <openstack@lists.openstack.org> Subject: [Openstack] [NOVA] nova image-list error class 'glanceclient.exc.CommunicationError'> (HTTP 500) I am a new bee to the this list. After a lot of attempts to get this working wanted to try out the mailing list for ideas. Please review this email in detail before suggesting to check password or other basic stuff. My conclusion is that my environment on liberty is running glance v2. Nova is making calls to v2 of nova api at port 8794. This returns a error. Not sure if it is calling v1 of glance or not. Glance v1 api is erroring out. Looking for some pointers to below. How do I add more debug to ensure nova is using glance api v2. I have seen some articles that state nova uses v1 api of glance, but not sure if that applies to liberty release. If yes then how do I get v1 glance api working on the box. See below for all the debug and issue details. Any help or response is greatly appreciated. Getting API error with nova image-list. Glance image-list works. Openstack image list works. Nova service-list works. cinder list works. Glance v2 api works and is able to create images. Cinder is able to create volumes from the images. All configuration files are enabled to debug and verbose . Nothing showing in the debug logs that would add more information. Glance is using version 2. IF I try version 1 api glance will error out. My openstack release is Liberty. Does glance api v1 works with liberty. How do I enable this. MY storage backend is ceph. RBD is able to communicate to all volumes and user ids correctly. Keyrings are ok and keystone does not show any errors. Controller box has haproxy,mariabd,glance and cinder and nova is installed on its own box with neutron. Password in the mysql database and config files are correct. Since haproxy shares the node with cinder and glance end points ports # are non default and pointing to haproxy service end points. Cinder and glance itself is running on the default ports. # openstack image list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | | de815da9-135a-4f8d-99ff-9e954672d775 | cirros | # glance image-list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | de815da9-135a-4f8d-99ff-9e954672d775 | cirros | | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | +--------------------------------------+--------+ # nova service-list +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+ | 3 | nova-cert | ostackn1 | internal | enabled | up | 2016-01-11T19:29:52.000000 | - | | 5 | nova-consoleauth | ostackn1 | internal | enabled | up | 2016-01-11T19:29:56.000000 | - | | 7 | nova-conductor | ostackn1 | internal | enabled | up | 2016-01-11T19:29:49.000000 | - | | 27 | nova-scheduler | ostackn1 | internal | enabled | up | 2016-01-11T19:29:51.000000 | - | | 29 | nova-compute | ostackn1 | nova | enabled | up | 2016-01-11T19:29:54.000000 | - | | 31 | nova-network | ostackn1 | internal | enabled | up | 2016-01-11T19:29:49.000000 | - | +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+ root@ostackn1:~# nova --debug image-list DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:35358/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54 DEBUG (connectionpool:385) "GET /v2.0 HTTP/1.1" 200 336 DEBUG (session:215) RESP: [200] Content-Length: 336 Vary: X-Auth-Token X-Distribution: Ubuntu Date: Mon, 11 Jan 2016 19:26:51 GMT Content-Type: application/json X-Openstack-Request-Id: req-e5ff4279-4632-4ae2-9f4f-af6cfdcb6727 RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.1.4.54:35358/v2.0/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}} DEBUG (v2:86) Making authentication request to http://10.1.4.54:35358/v2.0/tokens DEBUG (connectionpool:385) "POST /v2.0/tokens HTTP/1.1" 200 3066 DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8" INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54 DEBUG (connectionpool:385) "GET /v2/ HTTP/1.1" 200 374 DEBUG (session:215) RESP: [200] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 374 Content-Type: application/json X-Compute-Request-Id: req-0a444962-ebec-4994-bd82-d00bacbfca60 RESP BODY: {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.1.4.54:8794/v2/";, "rel": "self"}, {"href": "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}} DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8" DEBUG (connectionpool:385) "GET /v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail HTTP/1.1" 500 209 DEBUG (session:215) RESP: [500] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 209 Content-Type: application/json; charset=UTF-8 X-Compute-Request-Id: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84 RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'glanceclient.exc.CommunicationError'>", "code": 500}} DEBUG (shell:905) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 903, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 830, in main args.func(self.cs, args) File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 1115, in do_image_list image_list = cs.images.list(limit=limit) File "/usr/lib/python2.7/dist-packages/novaclient/v2/images.py", line 74, in list return self._list('/images%s%s' % (detail, query), 'images') File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list _resp, body = self.api.client.get(url) File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 93, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84) ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84) root@ostackc1:~# glance --os-image-api-version 2 image-list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | de815da9-135a-4f8d-99ff-9e954672d775 | cirros | | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | +--------------------------------------+--------+ root@ostackc1:~# glance --debug --os-image-api-version 1 image-list curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}318d44da7749a1b808ca32024fedc494a0f76133' -H 'Content-Type: application/octet-stream' http://10.1.4.54:9293/v1/images/detail?sort_key=name&sort_dir=asc&limit=20 Request returned failure status 500. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/glanceclient/shell.py", line 700, in main args.func(client, args) File "/usr/lib/python2.7/dist-packages/glanceclient/v1/shell.py", line 123, in do_image_list utils.print_list(images, columns) File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 183, in print_list for o in objs: File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list for image in paginate(params, return_request_id): File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate images, resp = self._list(url, "images") File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list resp, body = self.client.get(url) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 280, in get return self._request('GET', url, **kwargs) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in _request resp, body_iter = self._handle_response(resp) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 93, in _handle_response raise exc.from_response(resp, resp.content) HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+ | 045de6640536445da86969241286fb01 | regionOne | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)s<http://10.1.4.54:8081/v1/AUTH_%25(tenant_id)s> | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)s<http://10.1.4.54:8081/v1/AUTH_%25(tenant_id)s> | http://10.1.4.54:8081 | f3f2704480ea4b02bff61741e3ba734c | | 0c399f7fd9f9456ea158679ac2d9e800 | regionOne | http://10.1.4.54:8794/v2/%(tenant_id)s<http://10.1.4.54:8794/v2/%25(tenant_id)s> | http://10.1.4.54:8794/v2/%(tenant_id)s<http://10.1.4.54:8794/v2/%25(tenant_id)s> | http://10.1.4.54:8794/v2/%(tenant_id)s<http://10.1.4.54:8794/v2/%25(tenant_id)s> | 6365cc8ba6c24c5fa624fc587e9d6614 | | 1a8e073f90bf4450b0bb5fbc6d356b4a | regionOne | http://10.1.4.54:8777/v1/%(tenant_id)s<http://10.1.4.54:8777/v1/%25(tenant_id)s> | http://10.1.4.54:8777/v1/%(tenant_id)s<http://10.1.4.54:8777/v1/%25(tenant_id)s> | http://10.1.4.54:8777/v1/%(tenant_id)s<http://10.1.4.54:8777/v1/%25(tenant_id)s> | 8ff95f752a104f0f8f8e65ab02a550a2 | | 33f8ec041a2c444d975317a9c74f5b7e | regionOne | http://10.1.4.54:9293 | http://10.1.4.54:9293 | http://10.1.4.54:9293 | 622081c852b4498cb71b6b94651f8577 | | e6887352979c41b7aad8a0a77d67d80b | regionOne | http://10.1.4.54:5001/v2.0 | http://10.1.4.54:5001/v2.0 | http://10.1.4.54:35358/v2.0 | 2aeac7db1c154706ab2560d978073232 | | ec164aed45c04499900098fa21ce9aac | regionOne | http://10.1.4.54:8777/v2/%(tenant_id)s<http://10.1.4.54:8777/v2/%25(tenant_id)s> | http://10.1.4.54:8777/v2/%(tenant_id)s<http://10.1.4.54:8777/v2/%25(tenant_id)s> | http://10.1.4.54:8777/v2/%(tenant_id)s<http://10.1.4.54:8777/v2/%25(tenant_id)s> | d857ec11a2414b95a5dc5b7f10cb7300 | | f34cea0c5db04f17bde7b37e48d7372f | regionOne | http://10.1.4.54:9697 | http://10.1.4.54:9697 | http://10.1.4.54:9697 | eee2db07934f4353b13211375a5df6a5 | | ff54d6bd114f4e58bb1f26cf63f1d8af | regionOne | http://10.1.4.54:8793/services/Cloud | http://10.1.4.54:8793/services/Cloud | http://10.1.4.54:8793/services/Cloud | 2ad2865f6ef747a39112b2b58be13a1f | +----------------------------------+-----------+---------------------------------------------+------------------------------------------- Best Regards, Rajiv Duggal Office: 805.388.1711 x 528 Mobile: 805.312.3499 Email: rdug...@dexsystems.com|<mailto:rdug...@dexsystems.com|> www.dex.com<http://www.dex.com>
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
