See answers below. The only log requested that changed when trying to activate
is the vdsm.log, it is attached.
-----Original Message-----
From: Nir Soffer [mailto:[email protected]]
Sent: August 24, 2014 6:15 AM
To: Chris @ VeeroTech.net
Cc: [email protected]
Subject: Re: [ovirt-users] Master Storage goes Inactive - Failed to activate
Storage Domain
----- Original Message -----
> From: "Chris @ VeeroTech.net" <[email protected]>
> To: "Nir Soffer" <[email protected]>
> Cc: [email protected]
> Sent: Friday, August 22, 2014 4:22:46 AM
> Subject: RE: [ovirt-users] Master Storage goes Inactive - Failed to activate
> Storage Domain
>
> I have verified that the storage service (gluster) is functioning
> correctly by mounting it on another server and writing/reading.
Testing on another host only prove that your gluster service is accessible from
the other host. Please try to access gluster from the host that vdsm is running
on.
I have tested and this functions as it should.
> The setup is basic,
> two identical servers set up as hosts, four drives on each server.
> First drive is currently for the OS, second and third drive are set up
> to use Gluster distributed replicate. Fourth is not used.
Having only two hosts with gluster replication may be unreliable. Please
consult gluster folks how to setup gluster in a reliable way.
I am only testing Ovirt with Gluster, this is not a production system. I have
other servers lined up should the testing prove successful.
> Please let me know what
> other information you require, and thank you for the help.
Please share with us these logs, for the timeframe of this issue:
/var/log/messages
/var/log/sanlock.log
/var/log/vdsm/vdsm.log
>
> Chris
>
>
> -----Original Message-----
> From: Nir Soffer [mailto:[email protected]]
> Sent: Thursday, August 21, 2014 6:54 PM
> To: Chris @ VeeroTech.net
> Cc: [email protected]
> Subject: Re: [ovirt-users] Master Storage goes Inactive - Failed to
> activate Storage Domain
>
> ----- Original Message -----
> > From: "Chris @ VeeroTech.net" <[email protected]>
> > To: [email protected]
> > Sent: Monday, August 18, 2014 2:40:22 AM
> > Subject: [ovirt-users] Master Storage goes Inactive - Failed to activate
> > Storage Domain
> >
> >
> >
> > Hello All,
> >
> >
> >
> > I am having an issue on my test setup, the master storage goes
> > inactive. It is set up as to use Gluster and the volume is up and
> > running without issues.
> > Here is a snippet of /var/log/ovirt-engine/engine.log (these are
> > just the
> > ERRORs):
> >
> ...
> >
> > Here is the vdsm.log on one of the hosts:
> >
> >
> >
> > [root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
> >
> ...
>
> > Thread-36::WARNING::2014-08-17
> > 19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::
> > (_initMailbox) HSM_MailboxMonitor - Could not initialize mailbox,
> > will not accept requests until init succeeds
>
> Your host cannot write to the storage.
>
> We need more information about your setup.
>
> Can you check that the storage service is accessible?
>
>
[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterTasksList
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24
10:06:19,055::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:19,103::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000232199 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:19,103::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:21,105::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:21,152::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000259325 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:21,153::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:23,155::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:23,203::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000237248 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:23,203::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-13::DEBUG::2014-08-24
10:06:24,829::task::595::Storage.TaskManager.Task::(_updateState)
Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state init -> state
preparing
Thread-13::INFO::2014-08-24 10:06:24,830::logUtils::44::dispatcher::(wrapper)
Run and protect: repoStats(options=None)
Thread-13::INFO::2014-08-24 10:06:24,830::logUtils::47::dispatcher::(wrapper)
Run and protect: repoStats, Return response:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24
10:06:24,830::task::1191::Storage.TaskManager.Task::(prepare)
Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::finished:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '7.8', 'valid': False}}
Thread-13::DEBUG::2014-08-24
10:06:24,830::task::595::Storage.TaskManager.Task::(_updateState)
Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::moving from state preparing ->
state finished
Thread-13::DEBUG::2014-08-24
10:06:24,831::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24
10:06:24,831::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24
10:06:24,831::task::993::Storage.TaskManager.Task::(_decref)
Task=`3cc3ea32-1a3f-42e0-9482-189437782cfd`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24
10:06:25,205::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:25,253::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000253969 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:25,253::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-340::DEBUG::2014-08-24
10:06:26,992::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24
10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24
10:06:26,993::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24
10:06:26,993::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24 10:06:26,993::iscsi::403::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24
10:06:26,994::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24
10:06:27,042::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-340::ERROR::2014-08-24
10:06:27,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring
information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
self.refreshStorage()
File "/usr/share/vdsm/storage/misc.py", line 760, in helper
return sm(*args, **kwargs)
File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
self.__lastResult = self.__func(*args, **kwargs)
File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
multipath.rescan()
File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
supervdsm.getProxy().hbaRescan()
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in hbaRescan
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24
10:06:27,255::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:27,302::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000244622 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:27,303::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:29,305::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:29,352::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000261808 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:29,352::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:31,355::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:31,402::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000251987 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:31,402::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:33,405::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:33,453::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000202061 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:33,454::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:35,456::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:35,503::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000280783 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:35,503::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-340::DEBUG::2014-08-24
10:06:37,058::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24
10:06:37,059::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24
10:06:37,060::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24
10:06:37,060::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24 10:06:37,060::iscsi::403::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24
10:06:37,061::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24
10:06:37,114::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-340::ERROR::2014-08-24
10:06:37,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring
information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
self.refreshStorage()
File "/usr/share/vdsm/storage/misc.py", line 760, in helper
return sm(*args, **kwargs)
File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
self.__lastResult = self.__func(*args, **kwargs)
File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
multipath.rescan()
File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
supervdsm.getProxy().hbaRescan()
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in hbaRescan
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24
10:06:37,506::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:37,556::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000253556 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:37,556::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:39,559::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:39,606::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000225893 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:39,606::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-13::DEBUG::2014-08-24
10:06:40,511::task::595::Storage.TaskManager.Task::(_updateState)
Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state init -> state
preparing
Thread-13::INFO::2014-08-24 10:06:40,511::logUtils::44::dispatcher::(wrapper)
Run and protect: repoStats(options=None)
Thread-13::INFO::2014-08-24 10:06:40,512::logUtils::47::dispatcher::(wrapper)
Run and protect: repoStats, Return response:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24
10:06:40,512::task::1191::Storage.TaskManager.Task::(prepare)
Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::finished:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}}
Thread-13::DEBUG::2014-08-24
10:06:40,512::task::595::Storage.TaskManager.Task::(_updateState)
Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::moving from state preparing ->
state finished
Thread-13::DEBUG::2014-08-24
10:06:40,512::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24
10:06:40,512::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24
10:06:40,512::task::993::Storage.TaskManager.Task::(_decref)
Task=`48a7069a-a873-4931-be41-36b0ddfbfbca`::ref 0 aborting False
Thread-349::DEBUG::2014-08-24
10:06:41,609::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:41,655::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000266326 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:41,656::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:43,658::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:43,705::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000228889 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:43,706::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:45,155::BindingXMLRPC::318::vds::(wrapper)
client [74.63.225.147] flowID [5f151d27]
Thread-13::DEBUG::2014-08-24
10:06:45,155::task::595::Storage.TaskManager.Task::(_updateState)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state init -> state
preparing
Thread-13::INFO::2014-08-24 10:06:45,155::logUtils::44::dispatcher::(wrapper)
Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06',
options=None)
Thread-13::DEBUG::2014-08-24
10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24
10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-13::DEBUG::2014-08-24
10:06:45,156::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24
10:06:45,156::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-13::DEBUG::2014-08-24 10:06:45,156::iscsi::403::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24
10:06:45,157::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24
10:06:45,204::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-13::ERROR::2014-08-24
10:06:45,205::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based metadata
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
status = self._getSpmStatusInfo(pool)
File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
(pool.spmRole,) + pool.getSpmStatus()))
File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
return self._backend.getSpmStatus()
File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus
lVer, spmId = self.masterDomain.inquireClusterLock()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
self.refreshStorage()
File "/usr/share/vdsm/storage/misc.py", line 760, in helper
return sm(*args, **kwargs)
File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
self.__lastResult = self.__func(*args, **kwargs)
File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
multipath.rescan()
File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
supervdsm.getProxy().hbaRescan()
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in hbaRescan
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24
10:06:45,206::task::866::Storage.TaskManager.Task::(_setError)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: ("Can't read
LV based metadata",)
Thread-13::DEBUG::2014-08-24
10:06:45,206::task::885::Storage.TaskManager.Task::(_run)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._run:
5e508d32-1d1f-4b8f-96be-7c8435476763 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',)
{} failed - stopping task
Thread-13::DEBUG::2014-08-24
10:06:45,206::task::1217::Storage.TaskManager.Task::(stop)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::stopping in state preparing (force
False)
Thread-13::DEBUG::2014-08-24
10:06:45,206::task::993::Storage.TaskManager.Task::(_decref)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 1 aborting True
Thread-13::INFO::2014-08-24
10:06:45,206::task::1171::Storage.TaskManager.Task::(prepare)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::aborting: Task is aborted: 'Error
validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24
10:06:45,206::task::1176::Storage.TaskManager.Task::(prepare)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Prepare: aborted: Error validating
master storage domain
Thread-13::DEBUG::2014-08-24
10:06:45,207::task::993::Storage.TaskManager.Task::(_decref)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24
10:06:45,207::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::Task._doAbort: force False
Thread-13::DEBUG::2014-08-24
10:06:45,207::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24
10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state preparing ->
state aborting
Thread-13::DEBUG::2014-08-24
10:06:45,207::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::_aborting: recover policy none
Thread-13::DEBUG::2014-08-24
10:06:45,207::task::595::Storage.TaskManager.Task::(_updateState)
Task=`5e508d32-1d1f-4b8f-96be-7c8435476763`::moving from state aborting ->
state failed
Thread-13::DEBUG::2014-08-24
10:06:45,208::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24
10:06:45,208::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24
10:06:45,208::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Error validating master storage domain: ("Can\'t read LV based
metadata",)', 'code': 355}}
Thread-349::DEBUG::2014-08-24
10:06:45,708::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:45,756::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000255964 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:45,756::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-13::DEBUG::2014-08-24 10:06:45,883::BindingXMLRPC::318::vds::(wrapper)
client [74.63.225.147] flowID [852653f]
Thread-13::DEBUG::2014-08-24
10:06:45,883::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state init -> state
preparing
Thread-13::INFO::2014-08-24 10:06:45,884::logUtils::44::dispatcher::(wrapper)
Run and protect: connectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '',
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type':
'glusterfs', 'password': '******', 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)
Thread-13::DEBUG::2014-08-24
10:06:45,889::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath:
glusterSD/*
Thread-15::DEBUG::2014-08-24
10:06:45,890::__init__::225::IOProcess::(_processLogs) Marshaling
message...DEBUGDEBUGDEBUGDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Message size is 119
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Marshaling
message...DEBUGDEBUGDEBUGDEBUGDEBUG|Queuing request...
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Queuing request in the
thread pool...
Thread-15::DEBUG::2014-08-24
10:06:45,891::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-24
10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Finding callback
'glob'...
Thread-15::DEBUG::2014-08-24
10:06:45,892::__init__::225::IOProcess::(_processLogs) (89757) Got request for
method 'glob'
Thread-15::DEBUG::2014-08-24
10:06:45,893::__init__::225::IOProcess::(_processLogs) Marshaling
message...DEBUGDEBUGDEBUGDEBUGDEBUG|(89757) Building response
Thread-15::DEBUG::2014-08-24
10:06:45,893::__init__::225::IOProcess::(_processLogs) (89757) Queuing response
Thread-15::DEBUG::2014-08-24
10:06:45,893::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-15::DEBUG::2014-08-24
10:06:45,894::__init__::225::IOProcess::(_processLogs) Sending response sized
169
Thread-13::DEBUG::2014-08-24
10:06:45,895::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids:
(u'2ed32e34-2483-4262-a57b-72bf70f3e989',)
Thread-13::DEBUG::2014-08-24
10:06:45,895::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs:
{2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}
Thread-13::INFO::2014-08-24 10:06:45,895::logUtils::47::dispatcher::(wrapper)
Run and protect: connectStorageServer, Return response: {'statuslist':
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24
10:06:45,896::task::1191::Storage.TaskManager.Task::(prepare)
Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::finished: {'statuslist':
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-13::DEBUG::2014-08-24
10:06:45,896::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::moving from state preparing ->
state finished
Thread-13::DEBUG::2014-08-24
10:06:45,896::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24
10:06:45,896::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24
10:06:45,896::task::993::Storage.TaskManager.Task::(_decref)
Task=`2c3ab811-cf17-41b8-8d09-848dd95cf15f`::ref 0 aborting False
Thread-13::DEBUG::2014-08-24 10:06:45,986::BindingXMLRPC::318::vds::(wrapper)
client [74.63.225.147] flowID [7b161bce]
Thread-13::DEBUG::2014-08-24
10:06:45,987::task::595::Storage.TaskManager.Task::(_updateState)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state init -> state
preparing
Thread-13::INFO::2014-08-24 10:06:45,987::logUtils::44::dispatcher::(wrapper)
Run and protect: getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06',
options=None)
Thread-13::DEBUG::2014-08-24
10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-08-24
10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-13::DEBUG::2014-08-24
10:06:45,988::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-08-24
10:06:45,988::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-13::DEBUG::2014-08-24 10:06:45,988::iscsi::403::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-08-24
10:06:45,989::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-13::DEBUG::2014-08-24
10:06:46,038::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-13::ERROR::2014-08-24
10:06:46,039::hsm::614::Storage.HSM::(getSpmStatus) Can't read LV based metadata
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
status = self._getSpmStatusInfo(pool)
File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
(pool.spmRole,) + pool.getSpmStatus()))
File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
return self._backend.getSpmStatus()
File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus
lVer, spmId = self.masterDomain.inquireClusterLock()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
self.refreshStorage()
File "/usr/share/vdsm/storage/misc.py", line 760, in helper
return sm(*args, **kwargs)
File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
self.__lastResult = self.__func(*args, **kwargs)
File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
multipath.rescan()
File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
supervdsm.getProxy().hbaRescan()
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in hbaRescan
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-13::ERROR::2014-08-24
10:06:46,039::task::866::Storage.TaskManager.Task::(_setError)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 615, in getSpmStatus
raise se.StorageDomainMasterError("Can't read LV based metadata")
StorageDomainMasterError: Error validating master storage domain: ("Can't read
LV based metadata",)
Thread-13::DEBUG::2014-08-24
10:06:46,040::task::885::Storage.TaskManager.Task::(_run)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._run:
59772503-6431-42a9-b6b0-8e7f4b95a7f5 ('83ff2f2d-3c8e-4865-8656-c01d867cff06',)
{} failed - stopping task
Thread-13::DEBUG::2014-08-24
10:06:46,040::task::1217::Storage.TaskManager.Task::(stop)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::stopping in state preparing (force
False)
Thread-13::DEBUG::2014-08-24
10:06:46,040::task::993::Storage.TaskManager.Task::(_decref)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 1 aborting True
Thread-13::INFO::2014-08-24
10:06:46,040::task::1171::Storage.TaskManager.Task::(prepare)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::aborting: Task is aborted: 'Error
validating master storage domain' - code 355
Thread-13::DEBUG::2014-08-24
10:06:46,040::task::1176::Storage.TaskManager.Task::(prepare)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Prepare: aborted: Error validating
master storage domain
Thread-13::DEBUG::2014-08-24
10:06:46,041::task::993::Storage.TaskManager.Task::(_decref)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::ref 0 aborting True
Thread-13::DEBUG::2014-08-24
10:06:46,041::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::Task._doAbort: force False
Thread-13::DEBUG::2014-08-24
10:06:46,041::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::DEBUG::2014-08-24
10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state preparing ->
state aborting
Thread-13::DEBUG::2014-08-24
10:06:46,041::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::_aborting: recover policy none
Thread-13::DEBUG::2014-08-24
10:06:46,041::task::595::Storage.TaskManager.Task::(_updateState)
Task=`59772503-6431-42a9-b6b0-8e7f4b95a7f5`::moving from state aborting ->
state failed
Thread-13::DEBUG::2014-08-24
10:06:46,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-08-24
10:06:46,042::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-13::ERROR::2014-08-24
10:06:46,042::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Error validating master storage domain: ("Can\'t read LV based
metadata",)', 'code': 355}}
Thread-340::DEBUG::2014-08-24
10:06:47,118::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-340::DEBUG::2014-08-24
10:06:47,118::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24
10:06:47,119::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-340::DEBUG::2014-08-24
10:06:47,119::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling
method
Thread-340::DEBUG::2014-08-24 10:06:47,119::iscsi::403::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-340::DEBUG::2014-08-24
10:06:47,119::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-340::DEBUG::2014-08-24
10:06:47,168::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-340::ERROR::2014-08-24
10:06:47,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989 monitoring
information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
self.refreshStorage()
File "/usr/share/vdsm/storage/misc.py", line 760, in helper
return sm(*args, **kwargs)
File "/usr/share/vdsm/storage/misc.py", line 745, in __call__
self.__lastResult = self.__func(*args, **kwargs)
File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
multipath.rescan()
File "/usr/share/vdsm/storage/multipath.py", line 107, in rescan
supervdsm.getProxy().hbaRescan()
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in hbaRescan
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in
_callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Thread-349::DEBUG::2014-08-24
10:06:47,759::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:47,807::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000229862 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:47,808::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:49,810::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:49,857::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000265654 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:49,857::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:51,859::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:51,907::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000233321 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:51,907::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
Thread-349::DEBUG::2014-08-24
10:06:53,909::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) /bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox
iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-349::DEBUG::2014-08-24
10:06:53,957::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) FAILED:
<err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outbox':
Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000235232 s, 0.0 kB/s\n"; <rc> = 1
Thread-349::WARNING::2014-08-24
10:06:53,958::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMailbox)
HSM_MailboxMonitor - Could not initialize mailbox, will not accept requests
until init succeeds
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
