Hi guys, we rebooted a standby node of a healthy cluster and suddenly all the
resources on the primary cluster restarted. What's up with that? Before
rebooting the standby node, we did the normal stuff to verify that all was well.
crm_mon showed all nodes online, in their expected roles, with correct quorum
votes
cat /proc/drbd showed correct dbbd status
corosync-cfgtool -s showed all rings active without faults
When we rebooted the standby node (ha08c), crm_mon on the primary node (ha08a)
showed that all the resources stopped and then restarted, resulting in brief
loss of availability to customers.
Following is what crm_mon showed before server ha08c was rebooted and after it
came back up. Following that is our crm configuration.
============
Last updated: Mon May 7 11:13:32 2012
Stack: openais
Current DC: ha08a.mycharts.md - partition with quorum
Version: 1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe
3 Nodes configured, 3 expected votes
4 Resources configured.
============
Online: [ ha08a.mycharts.md ha08b.mycharts.md ha08c.mycharts.md ]
Master/Slave Set: ms_drbd0
Masters: [ ha08a.mycharts.md ]
Slaves: [ ha08c.mycharts.md ]
Master/Slave Set: ms_drbd1
Masters: [ ha08b.mycharts.md ]
Slaves: [ ha08c.mycharts.md ]
Resource Group: g_clust06
p_fs_clust06 (ocf::heartbeat:Filesystem): Started
ha08a.mycharts.md
p_vip_clust06 (ocf::heartbeat:IPaddr2): Started
ha08a.mycharts.md
p_mysql_371 (lsb:mysql_371): Started ha08a.mycharts.md
p_mysql_372 (lsb:mysql_372): Started ha08a.mycharts.md
p_mysql_373 (lsb:mysql_373): Started ha08a.mycharts.md
p_mysql_374 (lsb:mysql_374): Started ha08a.mycharts.md
p_mysql_375 (lsb:mysql_375): Started ha08a.mycharts.md
p_mysql_376 (lsb:mysql_376): Started ha08a.mycharts.md
p_mysql_047 (lsb:mysql_047): Started ha08a.mycharts.md
p_mysql_100 (lsb:mysql_100): Started ha08a.mycharts.md
p_mysql_379 (lsb:mysql_379): Started ha08a.mycharts.md
p_mysql_377 (lsb:mysql_377): Started ha08a.mycharts.md
p_mysql_378 (lsb:mysql_378): Started ha08a.mycharts.md
p_mysql_380 (lsb:mysql_380): Started ha08a.mycharts.md
p_mysql_381 (lsb:mysql_381): Started ha08a.mycharts.md
p_mysql_382 (lsb:mysql_382): Started ha08a.mycharts.md
p_mysql_383 (lsb:mysql_383): Started ha08a.mycharts.md
p_mysql_384 (lsb:mysql_384): Started ha08a.mycharts.md
p_mysql_385 (lsb:mysql_385): Started ha08a.mycharts.md
p_mysql_386 (lsb:mysql_386): Started ha08a.mycharts.md
p_mysql_387 (lsb:mysql_387): Started ha08a.mycharts.md
p_mysql_002 (lsb:mysql_002): Started ha08a.mycharts.md
p_mysql_035 (lsb:mysql_035): Started ha08a.mycharts.md
p_mysql_049 (lsb:mysql_049): Started ha08a.mycharts.md
p_mysql_097 (lsb:mysql_097): Started ha08a.mycharts.md
p_mysql_024 (lsb:mysql_024): Started ha08a.mycharts.md
p_mysql_077 (lsb:mysql_077): Started ha08a.mycharts.md
p_mysql_084 (lsb:mysql_084): Started ha08a.mycharts.md
p_mysql_113 (lsb:mysql_113): Started ha08a.mycharts.md
p_mysql_116 (lsb:mysql_116): Started ha08a.mycharts.md
p_mysql_388 (lsb:mysql_388): Started ha08a.mycharts.md
p_mysql_389 (lsb:mysql_389): Started ha08a.mycharts.md
p_mysql_390 (lsb:mysql_390): Started ha08a.mycharts.md
p_mysql_391 (lsb:mysql_391): Started ha08a.mycharts.md
p_mysql_392 (lsb:mysql_392): Started ha08a.mycharts.md
p_mysql_393 (lsb:mysql_393): Started ha08a.mycharts.md
p_mysql_394 (lsb:mysql_394): Started ha08a.mycharts.md
p_mysql_395 (lsb:mysql_395): Started ha08a.mycharts.md
p_mysql_396 (lsb:mysql_396): Started ha08a.mycharts.md
p_mysql_397 (lsb:mysql_397): Started ha08a.mycharts.md
p_mysql_398 (lsb:mysql_398): Started ha08a.mycharts.md
p_mysql_399 (lsb:mysql_399): Started ha08a.mycharts.md
p_mysql_400 (lsb:mysql_400): Started ha08a.mycharts.md
p_mysql_401 (lsb:mysql_401): Started ha08a.mycharts.md
p_mysql_402 (lsb:mysql_402): Started ha08a.mycharts.md
p_mysql_403 (lsb:mysql_403): Started ha08a.mycharts.md
p_mysql_404 (lsb:mysql_404): Started ha08a.mycharts.md
p_mysql_405 (lsb:mysql_405): Started ha08a.mycharts.md
p_mysql_104 (lsb:mysql_104): Started ha08a.mycharts.md
p_mysql_406 (lsb:mysql_406): Started ha08a.mycharts.md
p_mysql_407 (lsb:mysql_407): Started ha08a.mycharts.md
p_mysql_408 (lsb:mysql_408): Started ha08a.mycharts.md
p_mysql_409 (lsb:mysql_409): Started ha08a.mycharts.md
p_mysql_410 (lsb:mysql_410): Started ha08a.mycharts.md
p_mysql_017 (lsb:mysql_017): Started ha08a.mycharts.md
p_mysql_089 (lsb:mysql_089): Started ha08a.mycharts.md
p_mysql_090 (lsb:mysql_090): Started ha08a.mycharts.md
p_mysql_109 (lsb:mysql_109): Started ha08a.mycharts.md
p_mysql_411 (lsb:mysql_411): Started ha08a.mycharts.md
p_mysql_412 (lsb:mysql_412): Started ha08a.mycharts.md
p_mysql_413 (lsb:mysql_413): Started ha08a.mycharts.md
p_mysql_414 (lsb:mysql_414): Started ha08a.mycharts.md
p_mysql_415 (lsb:mysql_415): Started ha08a.mycharts.md
p_mysql_416 (lsb:mysql_416): Started ha08a.mycharts.md
p_mysql_417 (lsb:mysql_417): Started ha08a.mycharts.md
p_mysql_020 (lsb:mysql_020): Started ha08a.mycharts.md
p_mysql_076 (lsb:mysql_076): Started ha08a.mycharts.md
p_mysql_101 (lsb:mysql_101): Started ha08a.mycharts.md
p_mysql_051 (lsb:mysql_051): Started ha08a.mycharts.md
Resource Group: g_clust07
p_fs_clust07 (ocf::heartbeat:Filesystem): Started
ha08b.mycharts.md
p_vip_clust07 (ocf::heartbeat:IPaddr2): Started
ha08b.mycharts.md
p_mysql_418 (lsb:mysql_418): Started ha08b.mycharts.md
p_mysql_419 (lsb:mysql_419): Started ha08b.mycharts.md
p_mysql_420 (lsb:mysql_420): Started ha08b.mycharts.md
p_mysql_421 (lsb:mysql_421): Started ha08b.mycharts.md
p_mysql_422 (lsb:mysql_422): Started ha08b.mycharts.md
p_mysql_423 (lsb:mysql_423): Started ha08b.mycharts.md
p_mysql_424 (lsb:mysql_424): Started ha08b.mycharts.md
p_mysql_425 (lsb:mysql_425): Started ha08b.mycharts.md
p_mysql_426 (lsb:mysql_426): Started ha08b.mycharts.md
p_mysql_427 (lsb:mysql_427): Started ha08b.mycharts.md
p_mysql_056 (lsb:mysql_056): Started ha08b.mycharts.md
p_mysql_133 (lsb:mysql_133): Started ha08b.mycharts.md
p_mysql_428 (lsb:mysql_428): Started ha08b.mycharts.md
p_mysql_429 (lsb:mysql_429): Started ha08b.mycharts.md
p_mysql_430 (lsb:mysql_430): Started ha08b.mycharts.md
p_mysql_431 (lsb:mysql_431): Started ha08b.mycharts.md
p_mysql_432 (lsb:mysql_432): Started ha08b.mycharts.md
p_mysql_433 (lsb:mysql_433): Started ha08b.mycharts.md
p_mysql_434 (lsb:mysql_434): Started ha08b.mycharts.md
p_mysql_435 (lsb:mysql_435): Started ha08b.mycharts.md
p_mysql_436 (lsb:mysql_436): Started ha08b.mycharts.md
p_mysql_437 (lsb:mysql_437): Started ha08b.mycharts.md
p_mysql_438 (lsb:mysql_438): Started ha08b.mycharts.md
p_mysql_439 (lsb:mysql_439): Started ha08b.mycharts.md
p_mysql_440 (lsb:mysql_440): Started ha08b.mycharts.md
p_mysql_441 (lsb:mysql_441): Started ha08b.mycharts.md
p_mysql_442 (lsb:mysql_442): Started ha08b.mycharts.md
p_mysql_443 (lsb:mysql_443): Started ha08b.mycharts.md
p_mysql_444 (lsb:mysql_444): Started ha08b.mycharts.md
p_mysql_445 (lsb:mysql_445): Started ha08b.mycharts.md
p_mysql_446 (lsb:mysql_446): Started ha08b.mycharts.md
p_mysql_447 (lsb:mysql_447): Started ha08b.mycharts.md
p_mysql_448 (lsb:mysql_448): Started ha08b.mycharts.md
p_mysql_449 (lsb:mysql_449): Started ha08b.mycharts.md
p_mysql_450 (lsb:mysql_450): Started ha08b.mycharts.md
p_mysql_019 (lsb:mysql_019): Started ha08b.mycharts.md
p_mysql_120 (lsb:mysql_120): Started ha08b.mycharts.md
p_mysql_451 (lsb:mysql_451): Started ha08b.mycharts.md
p_mysql_452 (lsb:mysql_452): Started ha08b.mycharts.md
p_mysql_453 (lsb:mysql_453): Started ha08b.mycharts.md
p_mysql_454 (lsb:mysql_454): Started ha08b.mycharts.md
p_mysql_455 (lsb:mysql_455): Started ha08b.mycharts.md
p_mysql_456 (lsb:mysql_456): Started ha08b.mycharts.md
p_mysql_457 (lsb:mysql_457): Started ha08b.mycharts.md
p_mysql_458 (lsb:mysql_458): Started ha08b.mycharts.md
p_mysql_459 (lsb:mysql_459): Started ha08b.mycharts.md
p_mysql_460 (lsb:mysql_460): Started ha08b.mycharts.md
Here is our crm config...
^[[?1034hnode ha08a.mycharts.md \
attributes standby="off"
node ha08b.mycharts.md \
attributes standby="off"
node ha08c.mycharts.md \
attributes standby="off"
primitive p_drbd0 ocf:linbit:drbd \
params drbd_resource="ha01_mysql" \
op monitor interval="30s" role="Slave" \
op monitor interval="15s" role="Master"
primitive p_drbd1 ocf:linbit:drbd \
params drbd_resource="ha02_mysql" \
op monitor interval="30s" role="Slave" \
op monitor interval="15s" role="Master"
primitive p_fs_clust06 ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/ha01_mysql" fstype="ext3"
options="noatime"
primitive p_fs_clust07 ocf:heartbeat:Filesystem \
params device="/dev/drbd1" directory="/ha02_mysql" fstype="ext3"
options="noatime"
primitive p_mysql_002 lsb:mysql_002
primitive p_mysql_017 lsb:mysql_017
primitive p_mysql_019 lsb:mysql_019
primitive p_mysql_020 lsb:mysql_020
primitive p_mysql_024 lsb:mysql_024
primitive p_mysql_035 lsb:mysql_035
primitive p_mysql_047 lsb:mysql_047
primitive p_mysql_049 lsb:mysql_049
primitive p_mysql_051 lsb:mysql_051
primitive p_mysql_056 lsb:mysql_056
primitive p_mysql_076 lsb:mysql_076
primitive p_mysql_077 lsb:mysql_077
primitive p_mysql_084 lsb:mysql_084
primitive p_mysql_089 lsb:mysql_089
primitive p_mysql_090 lsb:mysql_090
primitive p_mysql_097 lsb:mysql_097
primitive p_mysql_100 lsb:mysql_100
primitive p_mysql_101 lsb:mysql_101
primitive p_mysql_104 lsb:mysql_104
primitive p_mysql_109 lsb:mysql_109
primitive p_mysql_113 lsb:mysql_113
primitive p_mysql_116 lsb:mysql_116
primitive p_mysql_120 lsb:mysql_120
primitive p_mysql_133 lsb:mysql_133
primitive p_mysql_371 lsb:mysql_371
primitive p_mysql_372 lsb:mysql_372
primitive p_mysql_373 lsb:mysql_373
primitive p_mysql_374 lsb:mysql_374
primitive p_mysql_375 lsb:mysql_375
primitive p_mysql_376 lsb:mysql_376
primitive p_mysql_377 lsb:mysql_377
primitive p_mysql_378 lsb:mysql_378
primitive p_mysql_379 lsb:mysql_379
primitive p_mysql_380 lsb:mysql_380
primitive p_mysql_381 lsb:mysql_381
primitive p_mysql_382 lsb:mysql_382
primitive p_mysql_383 lsb:mysql_383
primitive p_mysql_384 lsb:mysql_384
primitive p_mysql_385 lsb:mysql_385
primitive p_mysql_386 lsb:mysql_386
primitive p_mysql_387 lsb:mysql_387
primitive p_mysql_388 lsb:mysql_388
primitive p_mysql_389 lsb:mysql_389
primitive p_mysql_390 lsb:mysql_390
primitive p_mysql_390 lsb:mysql_390
primitive p_mysql_391 lsb:mysql_391
primitive p_mysql_392 lsb:mysql_392
primitive p_mysql_393 lsb:mysql_393
primitive p_mysql_394 lsb:mysql_394
primitive p_mysql_395 lsb:mysql_395
primitive p_mysql_396 lsb:mysql_396
primitive p_mysql_397 lsb:mysql_397
primitive p_mysql_398 lsb:mysql_398
primitive p_mysql_399 lsb:mysql_399
primitive p_mysql_400 lsb:mysql_400
primitive p_mysql_401 lsb:mysql_401
primitive p_mysql_402 lsb:mysql_402
primitive p_mysql_403 lsb:mysql_403
primitive p_mysql_404 lsb:mysql_404
primitive p_mysql_405 lsb:mysql_405
primitive p_mysql_406 lsb:mysql_406
primitive p_mysql_407 lsb:mysql_407
primitive p_mysql_408 lsb:mysql_408
primitive p_mysql_409 lsb:mysql_409
primitive p_mysql_410 lsb:mysql_410
primitive p_mysql_411 lsb:mysql_411
primitive p_mysql_412 lsb:mysql_412
primitive p_mysql_413 lsb:mysql_413
primitive p_mysql_414 lsb:mysql_414
primitive p_mysql_415 lsb:mysql_415
primitive p_mysql_416 lsb:mysql_416
primitive p_mysql_417 lsb:mysql_417
primitive p_mysql_418 lsb:mysql_418
primitive p_mysql_419 lsb:mysql_419
primitive p_mysql_420 lsb:mysql_420
primitive p_mysql_421 lsb:mysql_421
primitive p_mysql_422 lsb:mysql_422
primitive p_mysql_423 lsb:mysql_423
primitive p_mysql_424 lsb:mysql_424
primitive p_mysql_425 lsb:mysql_425
primitive p_mysql_426 lsb:mysql_426
primitive p_mysql_427 lsb:mysql_427
primitive p_mysql_428 lsb:mysql_428
primitive p_mysql_429 lsb:mysql_429
primitive p_mysql_430 lsb:mysql_430
primitive p_mysql_431 lsb:mysql_431
primitive p_mysql_432 lsb:mysql_432
primitive p_mysql_433 lsb:mysql_433
primitive p_mysql_434 lsb:mysql_434
primitive p_mysql_435 lsb:mysql_435
primitive p_mysql_436 lsb:mysql_436
primitive p_mysql_437 lsb:mysql_437
primitive p_mysql_438 lsb:mysql_438
primitive p_mysql_439 lsb:mysql_439
primitive p_mysql_440 lsb:mysql_440
primitive p_mysql_441 lsb:mysql_441
primitive p_mysql_442 lsb:mysql_442
primitive p_mysql_443 lsb:mysql_443
primitive p_mysql_444 lsb:mysql_444
primitive p_mysql_445 lsb:mysql_445
primitive p_mysql_446 lsb:mysql_446
primitive p_mysql_447 lsb:mysql_447
primitive p_mysql_448 lsb:mysql_448
primitive p_mysql_449 lsb:mysql_449
primitive p_mysql_450 lsb:mysql_450
primitive p_mysql_451 lsb:mysql_451
primitive p_mysql_452 lsb:mysql_452
primitive p_mysql_453 lsb:mysql_453
primitive p_mysql_454 lsb:mysql_454
primitive p_mysql_455 lsb:mysql_455
primitive p_mysql_456 lsb:mysql_456
primitive p_mysql_457 lsb:mysql_457
primitive p_mysql_458 lsb:mysql_458
primitive p_mysql_459 lsb:mysql_459
primitive p_mysql_460 lsb:mysql_460
primitive p_vip_clust06 ocf:heartbeat:IPaddr2 \
params ip="192.168.10.208" cidr_netmask="32" \
op monitor interval="15s"
primitive p_vip_clust07 ocf:heartbeat:IPaddr2 \
params ip="192.168.10.209" cidr_netmask="32" \
op monitor interval="15s"
group g_clust06 p_fs_clust06 p_vip_clust06 p_mysql_371 p_mysql_372 p_mysql_373
p_mysql_374 p_mysql_375 p_mysql_376 p_mysql_047 p_mysql_100 p_mysql_379
p_mysql_377 p_mysql_378 p_mysql_380 p_mysql_381 p_mysql_382 p_mysql_383
p_mysql_384 p_mysql_385 p_mysql_386 p_mysql_387 p_mysql_002 p_mysql_035
p_mysql_049 p_mysql_097 p_mysql_024 p_mysql_077 p_mysql_084 p_mysql_113
p_mysql_116 p_mysql_388 p_mysql_389 p_mysql_390 p_mysql_391 p_mysql_392
p_mysql_393 p_mysql_394 p_mysql_395 p_mysql_396 p_mysql_397 p_mysql_398
p_mysql_399 p_mysql_400 p_mysql_401 p_mysql_402 p_mysql_403 p_mysql_404
p_mysql_405 p_mysql_104 p_mysql_406 p_mysql_407 p_mysql_408 p_mysql_409
p_mysql_410 p_mysql_017 p_mysql_089 p_mysql_090 p_mysql_109 p_mysql_411
p_mysql_412 p_mysql_413 p_mysql_414 p_mysql_415 p_mysql_416 p_mysql_417
p_mysql_020 p_mysql_076 p_mysql_101 p_mysql_051 \
meta target-role="Started"
group g_clust07 p_fs_clust07 p_vip_clust07 p_mysql_418 p_mysql_419 p_mysql_420
p_mysql_421 p_mysql_422 p_mysql_423 p_mysql_424 p_mysql_425 p_mysql_426
p_mysql_427 p_mysql_056 p_mysql_133 p_mysql_428 p_mysql_429 p_mysql_430
p_mysql_431 p_mysql_432 p_mysql_433 p_mysql_434 p_mysql_435 p_mysql_436
p_mysql_437 p_mysql_438 p_mysql_439 p_mysql_440 p_mysql_441 p_mysql_442
p_mysql_443 p_mysql_444 p_mysql_445 p_mysql_446 p_mysql_447 p_mysql_448
p_mysql_449 p_mysql_450 p_mysql_019 p_mysql_120 p_mysql_451 p_mysql_452
p_mysql_453 p_mysql_454 p_mysql_455 p_mysql_456 p_mysql_457 p_mysql_458
p_mysql_459 p_mysql_460
ms ms_drbd0 p_drbd0 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true" is-managed="true"
ms ms_drbd1 p_drbd1 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true" is-managed="true"
location cli-prefer-g_clust06 g_clust06 \
rule $id="cli-prefer-rule-g_clust06" inf: #uname eq ha08a.mycharts.md
location loc1 g_clust06 50: ha08a.mycharts.md
location loc3 g_clust07 50: ha08b.mycharts.md
location loc_ms_drbd0 ms_drbd0 \
rule $id="loc_ms_drbd0-rule" -inf: #uname ne ha08a.mycharts.md and
#uname ne ha08c.mycharts.md
location loc_ms_drbd1 ms_drbd1 \
rule $id="loc_ms_drbd1-rule" -inf: #uname ne ha08b.mycharts.md and
#uname ne ha08c.mycharts.md
colocation c_clust06 inf: g_clust06 ms_drbd0:Master
colocation c_clust07 inf: g_clust07 ms_drbd1:Master
order o1 inf: ms_drbd0:promote g_clust06:start
order o2 inf: ms_drbd1:promote g_clust07:start
property $id="cib-bootstrap-options" \
dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \
cluster-infrastructure="openais" \
expected-quorum-votes="3" \
stonith-enabled="false" \
no-quorum-policy="ignore" \
symmetric-cluster="true" \
last-lrm-refresh="1320604660" \
maintenance-mode="false"
rsc_defaults $id="rsc-options" \
resource-stickiness="400"
--
Eric Robinson
Disclaimer - May 7, 2012
This email and any files transmitted with it are confidential and intended
solely for [email protected]. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physicians' Managed Care or Physician Select Management.
Warning: Although Physicians' Managed Care or Physician Select Management has
taken reasonable precautions to ensure no viruses are present in this email,
the company cannot accept responsibility for any loss or damage arising from
the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
