Dear Openstack community,
I had to reboot my server and now I can't start one of the vms (danrod-server)
which boots from volume.
vm details
[root@openstack ~(keystone_admin)]# openstack server show danrod-server
+--------------------------------------+----------------------------------------------------------+
| Field | Value
|
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | openstack.localdomain
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstack.localdomain
|
| OS-EXT-SRV-ATTR:instance_name | instance-00000069
|
| OS-EXT-STS:power_state | NOSTATE
|
| OS-EXT-STS:task_state | None
|
| OS-EXT-STS:vm_state | stopped
|
| OS-SRV-USG:launched_at | 2018-05-29T11:22:17.000000
|
| OS-SRV-USG:terminated_at | None
|
| accessIPv4 |
|
| accessIPv6 |
|
| addresses | privatenetwork=192.168.1.105,
129.94.14.238 |
| config_drive |
|
| created | 2018-05-29T11:22:09Z
|
| flavor | xlarge.cpu.xxxlarge.ram (6)
|
| hostId |
ecef276660cd714fe626073a18c11fe1c00bec91c15516178fb6ac28 |
| id | 1e59f329-072e-48ae-abf1-266eba437508
|
| image |
|
| key_name | None
|
| name | danrod-server
|
| os-extended-volumes:volumes_attached | [{u'id':
u'f1ac2e94-b0ed-4089-898f-5b6467fb51e3'}] |
| project_id | d58cf22d960e4de49b71658aee642e94
|
| properties |
|
| security_groups | [{u'name': u'admin'}, {u'name':
u'R-Studio Server'}] |
| status | SHUTOFF
|
| updated | 2018-10-08T02:52:41Z
|
| user_id | c412f34c353244eabecd4b6dc4d36392
|
+--------------------------------------+----------------------------------------------------------+
List of volumes
[root@openstack ~(keystone_admin)]# openstack volume list --all
+--------------------------------------+--------------+--------+------+----------------------------------------+
| ID | Display Name | Status | Size |
Attached to |
+--------------------------------------+--------------+--------+------+----------------------------------------+
| f1ac2e94-b0ed-4089-898f-5b6467fb51e3 | | in-use | 700 |
Attached to danrod-server on /dev/vda |
+--------------------------------------+--------------+--------+------+----------------------------------------+
nova-compute.log
2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] Multipath discovery for iSCSI not
enabled.
2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] Trying to connect to iSCSI portal
129.94.14.254:3260
2018-10-08 13:51:46.504 4015 WARNING os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at:
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
Will rescan & retry. Try number: 0.
2018-10-08 13:51:47.510 4015 WARNING os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at:
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
Will rescan & retry. Try number: 1.
2018-10-08 13:51:51.519 4015 WARNING os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at:
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
Will rescan & retry. Try number: 2.
2018-10-08 13:52:00.527 4015 WARNING os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at:
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
Will rescan & retry. Try number: 3.
2018-10-08 13:52:16.535 4015 WARNING os_brick.initiator.connectors.iscsi
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at:
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
Will rescan & retry. Try number: 4.
2018-10-08 13:52:23.357 4015 INFO nova.compute.resource_tracker
[req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Auditing locally available
compute resources for node openstack.localdomain
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker
[req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Total usable vcpus: 56,
total allocated vcpus: 52
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker
[req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Final resource view:
name=openstack.localdomain phys_ram=524173MB used_ram=454608MB phys_disk=9312GB
used_disk=1540GB total_vcpus=56 used_vcpus=52 pci_stats=[]
2018-10-08 13:52:23.733 4015 INFO nova.compute.resource_tracker
[req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Compute_service record
updated for openstack.localdomain:openstack.localdomain
2018-10-08 13:52:41.665 4015 INFO nova.compute.manager
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] [instance:
1e59f329-072e-48ae-abf1-266eba437508] Successfully reverted task state from
powering-on on failure for instance.
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
[req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392
d58cf22d960e4de49b71658aee642e94 - - -] Exception during message handling
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in
_process_incoming
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150,
in dispatch
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121,
in _do_dispatch
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in
wrapped
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server function_name,
call_dict, binary)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in
wrapped
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in
decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
LOG.warning(msg, e, instance=instance)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in
decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in
decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in
decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self.force_reraise()
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in
decorated_function
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2475, in
start_instance
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self._power_on(context, instance)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2445, in
_power_on
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2434, in
power_on
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self._hard_reboot(context, instance, network_info, block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2307, in
_hard_reboot
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
block_device_info=block_device_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4658, in
_get_guest_xml
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server context)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4483, in
_get_guest_config
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server flavor,
guest.os_type)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3523, in
_get_guest_storage_config
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
self._connect_volume(connection_info, info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1099, in
_connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
vol_driver.connect_volume(connection_info, disk_info)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py", line 64,
in connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server device_info =
self.connector.connect_volume(connection_info['data'])
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in
trace_logging_wrapper
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in
inner
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line
404, in connect_volume
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server raise
exception.VolumeDeviceNotFound(device=host_devices)
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
VolumeDeviceNotFound: Volume device not found at
[u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0'].
2018-10-08 13:52:41.672 4015 ERROR oslo_messaging.rpc.server
Cinder logs
2018-10-08 08:52:58.368 7069 ERROR oslo.messaging._drivers.impl_rabbit [-]
[3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server 129.94.14.254:5672 closed
the connection. Check login credentials: Socket closed
2018-10-08 08:52:59.395 7069 ERROR oslo.messaging._drivers.impl_rabbit [-]
[3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server on 129.94.14.254:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port:
60428
2018-10-08 08:53:01.404 7069 ERROR oslo.messaging._drivers.impl_rabbit [-]
[3dce0b7f-7eea-45e9-9fa5-220e783c0401] AMQP server on 129.94.14.254:5672 is
unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port:
60428
2018-10-08 08:53:04.131 7069 ERROR cinder.service
[req-f6138d6c-a09f-4ed4-881f-ed095d172461 - - - - -] model server went away
2018-10-08 08:53:04.131 7069 ERROR cinder.service Traceback (most recent call
last):
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/service.py", line 445, in report_state
2018-10-08 08:53:04.131 7069 ERROR cinder.service service_ref =
objects.Service.get_by_id(ctxt, self.service_id)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 287, in
get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service orm_obj =
db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/db/api.py", line 1624, in get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service return
IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in
wrapper
2018-10-08 08:53:04.131 7069 ERROR cinder.service return f(*args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6204, in
get_by_id
2018-10-08 08:53:04.131 7069 ERROR cinder.service return
_GET_METHODS[model](context, id, *args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 178, in
wrapper
2018-10-08 08:53:04.131 7069 ERROR cinder.service return f(*args, **kwargs)
2018-10-08 08:53:04.131 7069 ERROR cinder.service File
"/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 483, in
service_get
2018-10-08 08:53:04.131 7069 ERROR cinder.service service = None if not
query else query.first()
...
However the connection to the rabbitmq server is fine?
[root@openstack ~(keystone_admin)]# sudo ss -nolpt | grep 5672
LISTEN 0 128 *:25672 *:*
users:(("beam.smp",pid=2465,fd=43))
LISTEN 0 128 :::5672 :::*
users:(("beam.smp",pid=2465,fd=52))
[root@openstack ~(keystone_admin)]# telnet 129.94.14.254 5672
Trying 129.94.14.254...
Connected to 129.94.14.254.
Escape character is '^]'.
Any advice in regards how to fix this issue and be able to bring the vm back
online would be highly appreciated?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E:
manuel...@garvan.org.au<mailto:manuel...@garvan.org.au>
NOTICE
Please consider the environment before printing this email. This message and
any attachments are intended for the addressee named and may contain legally
privileged/confidential/copyright information. If you are not the intended
recipient, you should not read, use, disclose, copy or distribute this
communication. If you have received this message in error please notify us at
once by return email and then delete both messages. We accept no liability for
the distribution of viruses or similar in electronic communications. This
notice should not be removed.
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
