Hi Brent,
From your logs, I could able to see that gluster mount has become
read-only.
This happens only two circumstances,
1. Client-side quorum is enabled on that volume and its not met
2. Client-side quorum is enabled on that volume and the first brick of
the replica pair went down.
I think you would have met the second scenario.
Client-side quorum is enabled on gluster volume to prevent network
partition problems which would lead to split-brain scenarios.
If you want to restore your storage domain, you can disable client-side
quorum momentarily using RHEVM UI
or using gluster cli. ( # gluster volume reset <vol-name> quorum-type )
Hope that helps.
-- Satheesaran S
On 12/19/2014 01:09 PM, Sahina Bose wrote:
[+Sas - thanks for the link to virt-store usecase article inline]
On 12/18/2014 06:56 PM, Brent Hartzell wrote:
Hello,
I had actually gotten this sorted out, somewhat. If I disable server
quorum
on the volume, the storage domain will activate. The volume is/was
optimized
for virt store via oVirt. The brick in question was not the first brick
added to the volume through oVirt however, it appears that it may
have been
the first brick in the replica being used, but I'm not certain how to
find
this out.
The recommended setting is to have both client and server side quorum
turned on. But turning on server-side quorum with a 2-way replica
volume would mean that your volume goes offline when one of the bricks
goes down.
"gluster volume info" command will give you information about the
volume topology. So will the bricks sub-tab for Volume in oVirt. The
order in which the bricks are listed, is the order of the replica sets.
Disabling quorum allowed me to get the VM's affected back online
however, is
this the recommended procedure? I tried to use replace-brick with
another
node but it failed because the failed brick was not available. Would we
leave quorum disabled until that brick gets replaced? IE - rebuild the
server with the same hostname/IP file structure and rebalance the
cluster?
http://www.gluster.org/community/documentation/index.php/Virt-store-usecase
- for recommendations on volume tunables.
You could add another brick to your volume to make it a replica 3 and
then turn on quorum?
For help on recovering your volume, I suggest you write to
[email protected]
////
While that happened, I read somewhere about this happening with a
replica 2
- I've created a new volume with replica 3 and plan to test this
again. Is
there any info you can point me to for how to handle this when it
happens or
what the correct procedure is when a "first" brick fails?
-----Original Message-----
From: Sahina Bose [mailto:[email protected]]
Sent: Thursday, December 18, 2014 3:51 AM
To: Vered Volansky; Brent Hartzell
Cc: [email protected]
Subject: Re: [ovirt-users] Cannot activate storage domain
On 12/18/2014 01:35 PM, Vered Volansky wrote:
Adding Sahina.
----- Original Message -----
From: "Brent Hartzell" <[email protected]>
To: [email protected]
Sent: Thursday, December 18, 2014 3:38:11 AM
Subject: [ovirt-users] Cannot activate storage domain
Have the following:
6 hosts - virt + Gluster shared
Gluster volume is distributed-replicate - replica 2
Shutting down servers one at a time all work except for 1 brick. If
we shut down one specific brick (1 brick per host) - we're unable to
activate the storage domain. VM's that were actively running from
other bricks continue to run. Whatever was running form that specific
brick fails to run, gets paused etc.
Error log shows the entry below. I'm not certain what it's saying is
read only.nothing is read only that I can find.
2014-12-17 19:57:13,362 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-47) [4e9290a2] Command
SpmStatusVDSCommand(HostName = U23.domainame.net, HostId =
0db58e46-68a3-4ba0-a8aa-094893c045a1, storagePoolId =
7ccd6ea9-7d80-4170-afa1-64c10c185aa6) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 30] Read-only file system, code = 100
2014-12-17 19:57:13,363 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(DefaultQuartzScheduler_Worker-47) [4e9290a2]
hostFromVds::selectedVds - U23.domainname.net, spmStatus returned
null!
According to Ovirt/Gluster, if a brick goes down, the VM should be
able to be restarted from another brick without issue. This does not
appear to be the case. If we take other bricks offline, it appears to
work as expected.
Something with this specific brick cases everything to break which
then makes any VM's that were running from the brick unable to start.
Do you have the recommended options for using volume as virt store
turned
on? Is client-side quorum turned on for the volume? Is the brick that
causes
the issue, the first brick in the replica set?
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
