On 03/03/2014 11:47 AM, Sandro Bonazzola wrote:
Il 03/03/2014 11:33, René Koch ha scritto:
Hi,
I have some issues with hosted engine (oVirt 3.4 prerelease repo on CentOS 6.5).
My setups is the following:
2 hosts (will be 4 in the future) with 4 GlusterFS shares:
- engine (for hosted engine)
- iso (for ISO domain)
- ovirt (oVirt storage domain)
I had a split-brain situation today (after rebooting both nodes) on
hosted-engine.lockspace file on engine GlusterFS volume which I resolved.
How did you solved it? By switching to NFS only?
I removed the file on host1 (directly on the brick) and ran "gluster
volume heal engine full", which synced the file from host2 to host1.
hosted engine used engine share via NFS (TCP) as glusterfs isn't supported for
oVirt hosted engine, yet. I'll switch to GlusterFS as soon as oVirt
will support it (I hope this will be soon as RHEV 3.3 is already supporting
GlusterFS for hosted engine).
First of all ovirt-ha-agent fails to start on both nodes:
# service ovirt-ha-agent start
Starting ovirt-ha-agent: [ OK ]
# service ovirt-ha-agent status
ovirt-ha-agent dead but subsys locked
MainThread::INFO::2014-03-03
11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
ovirt-hosted-engine-ha agent 1.1.0 started
MainThread::INFO::2014-03-03
11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
Found
certificate common name: 10.0.200.101
MainThread::CRITICAL::2014-03-03
11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could
not start ha-agent
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
97, in run
self._run_agent()
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
154, in _run_agent
hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 152, in __init__
"STOP_VM": self._stop_engine_vm
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py",
line 56, in __init__
logger, actions)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
line 93, in __init__
self._logger = FSMLoggerAdapter(logger, self)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
line 16, in __init__
super(FSMLoggerAdapter, self).__init__(logger, None)
TypeError: super() argument 1 must be type, not classobj
If I want to start my hosted engine, I receive the following error in vdsm
logs, which makes absolutly no sense to me, as there is plenty of disk
space available:
Thread-62::DEBUG::2014-03-03
11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown libvirterror:
ecode: 38 edom: 42 level: 2 message: Failed
to acquire lock: No space left on device
seems like a vdsm failure in starting monitor the hosted engine storage domain.
Can you attach vdsm logs?
Logs are quite big for an email (6.8MB).
I attached the last entries which show the information for vm-start.
Thread-62::DEBUG::2014-03-03
11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released
Thread-62::ERROR::2014-03-03
11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/vm.py", line 3159, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line
92, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: Failed to acquire lock: No space left on device
Thread-62::DEBUG::2014-03-03 11:24:46,286::vm::2720::vm.Vm::(setDownStatus)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: Failed
to acquire lock: No space left on device
# df -h | grep engine
ovirt-host01:/engine 281G 21G 261G 8%
/rhev/data-center/mnt/ovirt-host01:_engine
# sudo -u vdsm dd if=/dev/zero
of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test
bs=512 count=100
100+0 records in
100+0 records out
51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s
Could you give me some information on how to fix the ovirt-ha-agent and then
hosted-engine storage issue? Thanks a lot.
Btw, I had some issues during installation which I will explain in separate
emails.
libvirtEventLoop::DEBUG::2014-03-03 11:18:58,466::vm::5111::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::event Started detail 0 opaque None
libvirtEventLoop::DEBUG::2014-03-03 11:18:58,468::vm::5111::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::event Suspended detail 0 opaque None
Thread-55::DEBUG::2014-03-03 11:18:58,503::vm::4967::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Looking for drive with attributes {'name': u'hdc', 'bootOrder': '', 'boot': [], 'readonly': True, 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'path': '', 'type': u'cdrom'}
Thread-55::DEBUG::2014-03-03 11:18:58,503::vm::4988::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Matched {'name': (u'hdc', u'hdc'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, True), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}), 'path': ('', ''), 'type': (u'cdrom', u'cdrom')}
Thread-55::DEBUG::2014-03-03 11:18:58,503::vm::5007::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Matched {'name': (u'hdc', u'hdc'), 'bootOrder': ('', None), 'boot': ([], None), 'readonly': (True, 'True'), 'address': ({u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}), 'path': ('', ''), 'type': (u'cdrom', 'disk')}
Thread-55::DEBUG::2014-03-03 11:18:58,503::vm::4967::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Looking for drive with attributes {'name': u'vda', 'bootOrder': u'1', 'boot': [<DOM Element: boot at 0x7f31f40f75f0>], 'readonly': False, 'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'path': u'/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', 'type': u'disk'}
Thread-55::DEBUG::2014-03-03 11:18:58,504::vm::4988::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Matched {'name': (u'vda', u'vda'), 'bootOrder': (u'1', u'1'), 'boot': ([<DOM Element: boot at 0x7f31f40f75f0>], None), 'readonly': (False, False), 'address': ({u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}), 'path': (u'/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', '/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf'), 'type': (u'disk', u'disk')}
Thread-55::DEBUG::2014-03-03 11:18:58,504::vm::5007::vm.Vm::(_getUnderlyingDriveInfo) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Matched {'name': (u'vda', u'vda'), 'bootOrder': (u'1', u'1'), 'boot': ([<DOM Element: boot at 0x7f31f40f75f0>], None), 'readonly': (False, 'False'), 'address': ({u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}), 'path': (u'/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', '/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf'), 'type': (u'disk', 'disk')}
Thread-55::DEBUG::2014-03-03 11:18:58,546::sampling::285::vm.Vm::(start) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Start statistics collection
Thread-57::DEBUG::2014-03-03 11:18:58,547::sampling::314::vm.Vm::(run) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Stats thread started
Thread-57::INFO::2014-03-03 11:18:58,549::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:18:58,552::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
Thread-55::DEBUG::2014-03-03 11:18:58,559::vmChannels::194::vds::(register) Add fileno 57 to listener's channels.
Thread-55::WARNING::2014-03-03 11:18:58,561::vm::3650::vm.Vm::(_readPauseCode) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_readPauseCode unsupported by libvirt vm
Thread-55::DEBUG::2014-03-03 11:18:58,570::vm::2252::vm.Vm::(_startUnderlyingVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released
VM Channels Listener::DEBUG::2014-03-03 11:18:58,760::vmChannels::102::vds::(_do_add_channels) fileno 57 was added to unconnected channels.
VM Channels Listener::DEBUG::2014-03-03 11:18:58,760::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:18:58,760::guestIF::147::vm.Vm::(_connect) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Attempting connection to /var/lib/libvirt/qemu/channels/f26dd37e-13b5-430c-b2f2-ecd098b82a91.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2014-03-03 11:18:58,760::guestIF::150::vm.Vm::(_connect) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Connected to /var/lib/libvirt/qemu/channels/f26dd37e-13b5-430c-b2f2-ecd098b82a91.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2014-03-03 11:18:58,782::vmChannels::144::vds::(_handle_unconnected) Connecting to fileno 57 succeeded.
VM Channels Listener::DEBUG::2014-03-03 11:19:28,817::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-57::INFO::2014-03-03 11:19:58,627::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:19:58,631::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
VM Channels Listener::DEBUG::2014-03-03 11:19:58,851::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:20:28,885::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-57::INFO::2014-03-03 11:20:58,680::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:20:58,685::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
VM Channels Listener::DEBUG::2014-03-03 11:20:58,921::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:21:28,954::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-57::INFO::2014-03-03 11:21:58,739::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:21:58,744::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
VM Channels Listener::DEBUG::2014-03-03 11:21:58,989::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:22:29,022::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-57::INFO::2014-03-03 11:22:58,790::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:22:58,795::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
VM Channels Listener::DEBUG::2014-03-03 11:22:59,056::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:23:29,090::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-57::INFO::2014-03-03 11:23:58,841::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-57::INFO::2014-03-03 11:23:58,846::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
VM Channels Listener::DEBUG::2014-03-03 11:23:59,124::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
VM Channels Listener::DEBUG::2014-03-03 11:24:29,158::vmChannels::91::vds::(_handle_timeouts) Timeout on fileno 57.
Thread-59::DEBUG::2014-03-03 11:24:42,871::BindingXMLRPC::975::vds::(wrapper) client [127.0.0.1]::call vmDestroy with ('f26dd37e-13b5-430c-b2f2-ecd098b82a91',) {}
Thread-59::INFO::2014-03-03 11:24:42,872::API::327::vds::(destroy) vmContainerLock acquired by vm f26dd37e-13b5-430c-b2f2-ecd098b82a91
Thread-59::DEBUG::2014-03-03 11:24:42,872::vm::4610::vm.Vm::(destroy) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::destroy Called
Thread-59::INFO::2014-03-03 11:24:42,872::vm::4556::vm.Vm::(releaseVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Release VM resources
Thread-59::DEBUG::2014-03-03 11:24:42,883::sampling::292::vm.Vm::(stop) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Stop statistics collection
Thread-59::DEBUG::2014-03-03 11:24:42,883::vmChannels::205::vds::(unregister) Delete fileno 57 from listener.
Thread-57::DEBUG::2014-03-03 11:24:42,883::sampling::323::vm.Vm::(run) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Stats thread finished
VM Channels Listener::DEBUG::2014-03-03 11:24:43,174::vmChannels::112::vds::(_do_del_channels) fileno 57 was removed from listener.
libvirtEventLoop::DEBUG::2014-03-03 11:24:43,300::vm::5111::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::event Stopped detail 1 opaque None
libvirtEventLoop::INFO::2014-03-03 11:24:43,300::vm::2313::vm.Vm::(_onQemuDeath) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::underlying process disconnected
Thread-59::INFO::2014-03-03 11:24:43,301::logUtils::44::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID=None)
Thread-59::INFO::2014-03-03 11:24:43,302::logUtils::47::dispatcher::(wrapper) Run and protect: teardownImage, Return response: None
Thread-59::INFO::2014-03-03 11:24:43,303::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='f26dd37e-13b5-430c-b2f2-ecd098b82a91')
Thread-59::INFO::2014-03-03 11:24:43,310::logUtils::47::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None
Thread-59::DEBUG::2014-03-03 11:24:43,311::vm::4604::vm.Vm::(deleteVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Total desktops after destroy of f26dd37e-13b5-430c-b2f2-ecd098b82a91 is 0
Thread-59::DEBUG::2014-03-03 11:24:43,311::BindingXMLRPC::982::vds::(wrapper) return vmDestroy with {'status': {'message': 'Machine destroyed', 'code': 0}}
libvirtEventLoop::DEBUG::2014-03-03 11:24:43,312::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'f26dd37e-13b5-430c-b2f2-ecd098b82a91'
libvirtEventLoop::DEBUG::2014-03-03 11:24:43,313::vm::2720::vm.Vm::(setDownStatus) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: Admin shut down
libvirtEventLoop::DEBUG::2014-03-03 11:24:43,314::sampling::292::vm.Vm::(stop) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Stop statistics collection
libvirtEventLoop::DEBUG::2014-03-03 11:24:43,315::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Domain not found: no domain with matching uuid 'f26dd37e-13b5-430c-b2f2-ecd098b82a91'
Thread-61::DEBUG::2014-03-03 11:24:45,265::BindingXMLRPC::975::vds::(wrapper) client [127.0.0.1]::call vmCreate with ({'emulatedMachine': 'pc', 'vmId': 'f26dd37e-13b5-430c-b2f2-ecd098b82a91', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': '7952662c-6f5b-456a-9e98-fc5149cf7334', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'bootOrder': '1', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '92df64df-4298-40d9-b1be-1b340dc56baf', 'imageID': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', 'specParams': {}, 'readonly': 'false', 'domainID': '2851af27-8744-445d-9fb1-a0d083c8dc82', 'optional': 'false', 'deviceId': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:72:0a:2b', 'linkActive': 'true', 'network': 'ovirtmgmt', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '5de38462-b692-45c4-a2e5-2b065356c4ad', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '2c8f0407-dbd4-4c2c-abc5-e05e177dac09', 'alias': 'console0'}], 'smp': '2', 'memSize': '4096', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'vmName': 'HostedEngine', 'display': 'vnc'},) {}
Thread-61::INFO::2014-03-03 11:24:45,270::API::668::vds::(_getNetworkIp) network None: using 0
Thread-61::INFO::2014-03-03 11:24:45,270::clientIF::366::vds::(createVm) vmContainerLock acquired by vm f26dd37e-13b5-430c-b2f2-ecd098b82a91
Thread-62::DEBUG::2014-03-03 11:24:45,272::vm::2231::vm.Vm::(_startUnderlyingVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Start
Thread-61::DEBUG::2014-03-03 11:24:45,272::clientIF::379::vds::(createVm) Total desktops after creation of f26dd37e-13b5-430c-b2f2-ecd098b82a91 is 1
Thread-62::DEBUG::2014-03-03 11:24:45,273::vm::2235::vm.Vm::(_startUnderlyingVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations acquired
Thread-61::DEBUG::2014-03-03 11:24:45,273::BindingXMLRPC::982::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'nicModel': 'rtl8139,pv', 'vmId': 'f26dd37e-13b5-430c-b2f2-ecd098b82a91', 'emulatedMachine': 'pc', 'displayIp': '0', 'pid': '0', 'devices': [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': '7952662c-6f5b-456a-9e98-fc5149cf7334', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'bootOrder': '1', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '92df64df-4298-40d9-b1be-1b340dc56baf', 'imageID': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', 'specParams': {}, 'readonly': 'false', 'domainID': '2851af27-8744-445d-9fb1-a0d083c8dc82', 'optional': 'false', 'deviceId': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:72:0a:2b', 'linkActive': 'true', 'network': 'ovirtmgmt', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '5de38462-b692-45c4-a2e5-2b065356c4ad', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '2c8f0407-dbd4-4c2c-abc5-e05e177dac09', 'alias': 'console0'}], 'smp': '2', 'vmType': 'kvm', 'displaySecurePort': '-1', 'memSize': '4096', 'displayPort': '-1', 'cpuType': 'Westmere', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'clientIp': '', 'vmName': 'HostedEngine', 'display': 'vnc'}}
Thread-62::INFO::2014-03-03 11:24:45,274::vm::3086::vm.Vm::(_run) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::VM wrapper has started
Thread-62::INFO::2014-03-03 11:24:45,275::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', volUUID='92df64df-4298-40d9-b1be-1b340dc56baf', options=None)
Thread-62::INFO::2014-03-03 11:24:45,277::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3499315200', 'apparentsize': '26843545600'}
Thread-62::INFO::2014-03-03 11:24:45,278::clientIF::324::vds::(prepareVolumePath) prepared volume path:
Thread-62::INFO::2014-03-03 11:24:45,278::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='2851af27-8744-445d-9fb1-a0d083c8dc82', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc', leafUUID='92df64df-4298-40d9-b1be-1b340dc56baf')
Thread-62::WARNING::2014-03-03 11:24:45,281::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82 already exists
Thread-62::DEBUG::2014-03-03 11:24:45,281::fileSD::446::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc
Thread-62::DEBUG::2014-03-03 11:24:45,282::fileVolume::529::Storage.Volume::(validateVolumePath) validate path for 92df64df-4298-40d9-b1be-1b340dc56baf
Thread-62::INFO::2014-03-03 11:24:45,284::logUtils::47::dispatcher::(wrapper) Run and protect: prepareImage, Return response: {'info': {'domainID': '2851af27-8744-445d-9fb1-a0d083c8dc82', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', 'volumeID': '92df64df-4298-40d9-b1be-1b340dc56baf', 'leasePath': '/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf.lease', 'imageID': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc'}, 'path': '/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', 'imgVolumesInfo': [{'domainID': '2851af27-8744-445d-9fb1-a0d083c8dc82', 'volType': 'path', 'leaseOffset': 0, 'path': '/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf', 'volumeID': '92df64df-4298-40d9-b1be-1b340dc56baf', 'leasePath': '/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf.lease', 'imageID': '0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc'}]}
Thread-62::INFO::2014-03-03 11:24:45,284::clientIF::324::vds::(prepareVolumePath) prepared volume path: /var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf
Thread-62::DEBUG::2014-03-03 11:24:45,294::utils::556::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_hostedengine' (cwd None)
Thread-62::DEBUG::2014-03-03 11:24:45,354::utils::576::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-62::INFO::2014-03-03 11:24:45,355::hooks::100::root::(_runHooksDir)
Thread-62::DEBUG::2014-03-03 11:24:45,355::vm::3125::vm.Vm::(_run) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::<?xml version="1.0" encoding="utf-8"?><domain type="kvm">
<name>HostedEngine</name>
<uuid>f26dd37e-13b5-430c-b2f2-ecd098b82a91</uuid>
<memory>4194304</memory>
<currentMemory>4194304</currentMemory>
<vcpu current="2">2</vcpu>
<memtune>
<min_guarantee>0</min_guarantee>
</memtune>
<devices>
<channel type="unix">
<target name="com.redhat.rhevm.vdsm" type="virtio"/>
<source mode="bind" path="/var/lib/libvirt/qemu/channels/f26dd37e-13b5-430c-b2f2-ecd098b82a91.com.redhat.rhevm.vdsm"/>
</channel>
<channel type="unix">
<target name="org.qemu.guest_agent.0" type="virtio"/>
<source mode="bind" path="/var/lib/libvirt/qemu/channels/f26dd37e-13b5-430c-b2f2-ecd098b82a91.org.qemu.guest_agent.0"/>
</channel>
<input bus="ps2" type="mouse"/>
<graphics autoport="yes" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"/>
<console type="pty">
<target port="0" type="virtio"/>
</console>
<memballoon model="none"/>
<controller model="virtio-scsi" type="scsi"/>
<interface type="bridge">
<address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/>
<mac address="00:16:3e:72:0a:2b"/>
<model type="virtio"/>
<source bridge="ovirtmgmt"/>
<filterref filter="vdsm-no-mac-spoofing"/>
<link state="up"/>
</interface>
<disk device="cdrom" snapshot="no" type="file">
<address bus="1" controller="0" target="0" type="drive" unit="0"/>
<source file="" startupPolicy="optional"/>
<target bus="ide" dev="hdc"/>
<readonly/>
<serial/>
</disk>
<disk device="disk" snapshot="no" type="file">
<address bus="0x00" domain="0x0000" function="0x0" slot="0x06" type="pci"/>
<source file="/var/run/vdsm/storage/2851af27-8744-445d-9fb1-a0d083c8dc82/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf"/>
<target bus="virtio" dev="vda"/>
<serial>0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc</serial>
<boot order="1"/>
<driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/>
</disk>
<lease>
<key>92df64df-4298-40d9-b1be-1b340dc56baf</key>
<lockspace>2851af27-8744-445d-9fb1-a0d083c8dc82</lockspace>
<target offset="0" path="/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/92df64df-4298-40d9-b1be-1b340dc56baf.lease"/>
</lease>
</devices>
<os>
<type arch="x86_64" machine="pc">hvm</type>
<smbios mode="sysinfo"/>
</os>
<sysinfo type="smbios">
<system>
<entry name="manufacturer">oVirt</entry>
<entry name="product">oVirt Node</entry>
<entry name="version">6-5.el6.centos.11.2</entry>
<entry name="serial">00000000-0000-0000-0000-00259075876C</entry>
<entry name="uuid">f26dd37e-13b5-430c-b2f2-ecd098b82a91</entry>
</system>
</sysinfo>
<clock adjustment="0" offset="variable">
<timer name="rtc" tickpolicy="catchup"/>
</clock>
<features>
<acpi/>
</features>
<cpu match="exact">
<model>Westmere</model>
</cpu>
<on_poweroff>destroy</on_poweroff><on_reboot>destroy</on_reboot><on_crash>destroy</on_crash></domain>
Thread-62::DEBUG::2014-03-03 11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space left on device
Thread-62::DEBUG::2014-03-03 11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released
Thread-62::ERROR::2014-03-03 11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/vm.py", line 3159, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: Failed to acquire lock: No space left on device
Thread-62::DEBUG::2014-03-03 11:24:46,286::vm::2720::vm.Vm::(setDownStatus) vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: Failed to acquire lock: No space left on device
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
