Sleeping few hours, putting debugs on, leaving half of api's &
schedulers of. Everything works.
Taking debugs off, restarting all cinder services, on all nodes. Still
works. So this far, problem solved, but, still, I would be happy to know
better how to debug what is broken. We still have some rabbitmq queues
which have plenty of messages, but nobody seems to be consuming them.
--Tavasti
On 29/12/15 02:51, Remo Mattei wrote:
Did you try to enable debugging?
Inviato da iPhone
Il giorno 28 dic 2015, alle ore 15:53, Markku Tavasti
<markku.tava...@cybercom.com> ha scritto:
Hi!
I would need some help for debugging problems on cinder volume creation &
deletion.
Quite many of operations end up to left running forever.
Our config is Openstack Kilo running in RHEL7. We are using RH RDO packaged
version.
We have 4 master nodes, running most of the services loadbalanced with haproxy.
Cinder volume is having 2 availability zones.
Cinder has been working fine, but after we had rabbitmq cluster problems
(needed reboot), cinder has not been working properly. Many volume create
operations are left running as 'creating', and deletions are left 'deleting'.
With rabbitmqctl list_queues I can see, that there are some queues
accumulating, but why? Some cinder-scheduler_fanout_LONGHEXHERE and some
reply_LONGHEXHERE.
And in cinder logs, no errors, just info messages, for example for
unsuccessfull creation:
api.log:2015-12-29 01:05:15.881 2567 INFO cinder.api.openstack.wsgi
[req-5bfdeeb8-d47d-471d-b8db-6a526c7a00cf tavasti
91ac58eb50db4384b9afeb5d8c622f8d - - -] POST http://volume-api.domain.com
/v2/91ac58eb50db4384b9afeb5d8c622f8d/volumes api.log:2015-12-29 01:05:15.882
2567 INFO cinder.api.v2.volumes [req-5bfdeeb8-d47d-471d-b8db-6a526c7a00cf
tavasti 91ac58eb50db4384b9afeb5d8c622f8d - - -] Create volume of 1 GB
After those, I cannot see anything relevant, on any of the cinder logs on any
nodes. Specially, on cinder scheduler logs, there is silence, even with
successfull volume creations.
Any ideas how to track what is broken?
--Tavasti, Openstack admin
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
!DSPAM:1,5681cef5149381170821010!
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
