Hello,
some envir. answers :
*****************************************************
OS = RHEL - 7 - 2.151
kernel = 3.10.0 - 327.10.1.el7.x86_64
KVM = 2.3.0 - 31.el7_2.7.1
libvirt = libvirt-1.2.17-13.el7_2.3
vdsm = vdsm-4.17.23.2-0.el7
glusterfs = glusterfs-3.7.9-1.el7
ovirt = 3.5.6.2-1
*****************************************************
# gluster peer status
Number of Peers: 4
Hostname: 1hp2
Uuid: 8e87cf18-8958-41b7-8d24-7ee420a1ef9f
State: Peer in Cluster (Connected)
Hostname: 2hp2
Uuid: b1d987d8-0b42-4ce4-b85f-83b4072e0990
State: Peer in Cluster (Connected)
Hostname: 2hp1
Uuid: a1cbe1a8-88ad-4e89-8a0e-d2bb2b6786d8
State: Peer in Cluster (Connected)
Hostname: kvmarbiter
Uuid: bb1d63f1-7757-4c07-b70d-aa2f68449e21
State: Peer in Cluster (Connected)
*****************************************************
== "C" ==
Volume Name: 12HP12-D2R3A1P2
Type: Distributed-Replicate
Volume ID: 3c22d3dc-7c6e-4e37-9e0b-78410873ed6d
Status: Started
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: 1hp1:/STORAGES/P2/GFS
Brick2: 1hp2:/STORAGES/P2/GFS
Brick3: kvmarbiter:/STORAGES/P2-1/GFS (arbiter)
Brick4: 2hp1:/STORAGES/P2/GFS
Brick5: 2hp2:/STORAGES/P2/GFS
Brick6: kvmarbiter:/STORAGES/P2-2/GFS (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
*****************************************************
== "A" ==
Volume Name: 1HP12-R3A1P1
Type: Replicate
Volume ID: e4121610-6128-4ecc-86d3-1429ab3b8356
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: 1hp1:/STORAGES/P1/GFS
Brick2: 1hp2:/STORAGES/P1/GFS
Brick3: kvmarbiter:/STORAGES/P1-1/GFS (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
cluster.data-self-heal-algorithm: full
performance.write-behind: on
performance.low-prio-threads: 32
performance.write-behind-window-size: 128MB
network.ping-timeout: 10
*****************************************************
== "B" ==
Volume Name: 2HP12-R3A1P1
Type: Replicate
Volume ID: d3d260cd-455f-42d6-9580-d88ae6df0519
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: 2hp1:/STORAGES/P1/GFS
Brick2: 2hp2:/STORAGES/P1/GFS
Brick3: kvmarbiter:/STORAGES/P1-2/GFS (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
cluster.data-self-heal-algorithm: full
performance.write-behind: on
performance.low-prio-threads: 32
performance.write-behind-window-size: 128MB
network.ping-timeout: 10
The oVirt volumes(storages) have the same name as gluster volumes ( eg:
"B" = 2HP12-R3A1P1( ovirt storage ) = 2HP12-R3A1P1( gluster volume name ) )
In the test the master volume was "A" = 1HP12-R3A1P1
regs. Pavel
PS: logs will follow as webstore pointer ... this takes some time
On 31.3.2016 14:30, Yaniv Kaul wrote:
Hi Pavel,
Thanks for the report. Can you begin with a more accurate description
of your environment?
Begin with host, oVirt and Gluster versions. Then continue with the
exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the
mapping between domains and volumes?).
Are there any logs you can share with us?
I'm sure with more information, we'd be happy to look at the issue.
Y.
On Thu, Mar 31, 2016 at 3:09 PM, p...@email.cz <mailto:p...@email.cz>
<p...@email.cz <mailto:p...@email.cz>> wrote:
Hello,
we tried the following test - with unwanted results
input:
5 node gluster
A = replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 )
B = replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 )
C = distributed replica 3 arbiter 1 ( node1+node2, node3+node4,
each arbiter on node 5)
node 5 has only arbiter replica ( 4x )
TEST:
1) directly reboot one node - OK ( is not important which ( data
node or arbiter node ))
2) directly reboot two nodes - OK ( if nodes are not from the
same replica )
3) directly reboot three nodes - yes, this is the main problem
and a questions ....
- rebooted all three nodes from replica "B" ( not so
possible, but who knows ... )
- all VMs with data on this replica was paused ( no data
access ) - OK
- all VMs running on replica "B" nodes lost ( started
manually, later )( datas on other replicas ) - acceptable
BUT
- !!! all oVIrt domains went down !! - master domain is on
replica "A" which lost only one member from three !!!
so we are not expecting that all domain will go down,
especially master with 2 live members.
Results:
- the whole cluster unreachable until at all domains up -
depent of all nodes up !!!
- all paused VMs started back - OK
- rest of all VMs rebooted and runnig - OK
Questions:
1) why all domains down if master domain ( on replica "A" )
has two runnig members ( 2 of 3 ) ??
2) how to fix that colaps without waiting to all nodes up ? (
in worste case if node has HW error eg. ) ??
3) which oVirt cluster policy can prevent that situation ??
( if any )
regs.
Pavel
_______________________________________________
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
