Hi, I have created a development setup with Devstack on 2 machines, where one is a controller + compute node (IP: 172.23.29.96) and other is a storage node(IP: 172.23.29.118). I am running all the services on the controller, except the c-vol service which runs on the storage node.
Whenever I try to create a new instance, the volume gets created but
the instance is stuck with "Spawning" status, and in some time gets
errored out. This volume is then not attached to the VM. These volumes
are LVM based, SCSI backed volumes.
During this time, I can see the following 2 errors of interest in
stack traces/logs. It would be really helpful if anyone here can take
a look and point me in the right direction.
I've also attached the stack traces in case the formatting gets messed up.
====== START OF STACK TRACE 1 =========
DEBUG oslo.privsep.daemon [-] privsep: Exception during
request[140390878014992]: Unexpected error while running command.
Command: iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e -p
172.23.29.118:3260 --login
Exit code: 8
Stdout: u'Logging in to [iface: default, target:
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal:
172.23.29.118,3260] (multiple)\n'
Stderr: u'iscsiadm: Could not login to [iface: default, target:
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal:
172.23.29.118,3260].\nis\
csiadm: initiator reported error (8 - connection timed out)\niscsiadm: Could
not log into all portals\n' {{(pid=14097) loop
/usr/lib/python2.7/site-packages/oslo_privs\
ep/daemon.py:449}}
ERROR oslo.privsep.daemon Traceback (most recent call last):
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 445, in loop
ERROR oslo.privsep.daemon reply = self._process_cmd(*msg)
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 428, in
_process_cmd
ERROR oslo.privsep.daemon ret = func(*f_args, **f_kwargs)
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 209, in
_wrap
ERROR oslo.privsep.daemon return func(*args, **kwargs)
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 194,
in execute_root
ERROR oslo.privsep.daemon return custom_execute(*cmd, shell=False,
run_as_root=False, **kwargs)
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 143,
in custom_execute
ERROR oslo.privsep.daemon on_completion=on_completion, *cmd, **kwargs)
ERROR oslo.privsep.daemon File
"/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424,
in execute
ERROR oslo.privsep.daemon cmd=sanitized_cmd)
ERROR oslo.privsep.daemon ProcessExecutionError: Unexpected error while running
command.
ERROR oslo.privsep.daemon Command: iscsiadm -m node -T
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e -p
172.23.29.118:3260 --login
ERROR oslo.privsep.daemon Exit code: 8
ERROR oslo.privsep.daemon Stdout: u'Logging in to [iface: default, target:
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal:
172.23.29.118\
,3260] (multiple)\n'
ERROR oslo.privsep.daemon Stderr: u'iscsiadm: Could not login to [iface:
default, target:
iqn.2010-10.org.openstack:volume-9ae59d83-ec09-4fd5-aa2c-5049a29bed5e, portal\
: 172.23.29.118,3260].\niscsiadm: initiator reported error (8 - connection
timed out)\niscsiadm: Could not log into all portals\n'
ERROR oslo.privsep.daemon
====== END OF STACK TRACE 1 ===========
====== START OF STACK TRACE 2 ===========
[instance: d2a44cc2-c367-4d6e-b572-0d174e44d817] Instance failed to spawn:
VolumeDeviceNotFound: Volume device not found at .
Error: Traceback (most recent call last):
Error: File "/opt/stack/nova/nova/compute/manager.py", line 2357, in
_build_resources
Error: yield resources
Error: File "/opt/stack/nova/nova/compute/manager.py", line 2121, in
_build_and_run_instance
Error: block_device_info=block_device_info)
Error: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3075, in
spawn
Error: mdevs=mdevs)
Error: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5430, in
_get_guest_xml
Error: context, mdevs)
Error: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5216, in
_get_guest_config
Error: flavor, guest.os_type)
Error: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4030, in
_get_guest_storage_config
Error: self._connect_volume(context, connection_info, instance)
Error: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1222, in
_connect_volume
Error: vol_driver.connect_volume(connection_info, instance)
Error: File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 64, in
connect_volume
Error: device_info = self.connector.connect_volume(connection_info['data'])
Error: File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150,
in trace_logging_wrapper
Error: result = f(*args, **kwargs)
Error: File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in
inner
Error: return f(*args, **kwargs)
Error: File
"/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line
516, in connect_volume
Error: self._cleanup_connection(connection_properties, force=True)
Error: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
220, in __exit__
Error: self.force_reraise()
Error: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
196, in force_reraise
Error: six.reraise(self.type_, self.value, self.tb)
Error: File
"/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line
510, in connect_volume
Error: return self._connect_single_volume(connection_properties)
Error: File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61,
in _wrapper
Error: return r.call(f, *args, **kwargs)
Error: File "/usr/lib/python2.7/site-packages/retrying.py", line 212, in call
Error: raise attempt.get()
Error: File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
Error: six.reraise(self.value[0], self.value[1], self.value[2])
Error: File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call
Error: attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
Error: File
"/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line
585, in _connect_single_volume
Error: raise exception.VolumeDeviceNotFound(device='')
Error: VolumeDeviceNotFound: Volume device not found at .
====== END OF STACK TRACE 2 ===========
--
Tushar Tyagi
t...@fastmail.com
stack_trace_1.log
Description: Binary data
stack_trace_2.log
Description: Binary data
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
