Here is a shot at that problem: I see you are using lsb to start the mysql (would be better to use the mysql-ra as that is more reliable to detect running/failing mysql). I guess the init-script of mysql returns some bogus return-code on nodes where mysql is not running. Then it looks to pacemaker as if mysql is started where it shouldn't be (and active more often then it should be) and to get back to a sane and known state, pacemaker stops all mysql (and all dependent resources) and start them again. [*] Try: a) fix the init-skript or b) use ocf:heartbeat:mysql instead of lsb:mysql.
[*] I have the same with lsb:bind while all the other used lsb-resources are just fine. And I am fine with a bind restarting every now and then... Please excuse the top-post. Have fun, Arnold On 07.05.2012 20:27, Robinson, Eric wrote: > Hi guys, we rebooted a standby node of a healthy cluster and suddenly all the > resources on the primary cluster restarted. What's up with that? Before > rebooting the standby node, we did the normal stuff to verify that all was > well. > > crm_mon showed all nodes online, in their expected roles, with correct quorum > votes > cat /proc/drbd showed correct dbbd status > corosync-cfgtool -s showed all rings active without faults > > When we rebooted the standby node (ha08c), crm_mon on the primary node > (ha08a) showed that all the resources stopped and then restarted, resulting > in brief loss of availability to customers. > > Following is what crm_mon showed before server ha08c was rebooted and after > it came back up. Following that is our crm configuration. > > ============ > Last updated: Mon May 7 11:13:32 2012 > Stack: openais > Current DC: ha08a.mycharts.md - partition with quorum > Version: 1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe > 3 Nodes configured, 3 expected votes > 4 Resources configured. > ============ > > Online: [ ha08a.mycharts.md ha08b.mycharts.md ha08c.mycharts.md ] > > Master/Slave Set: ms_drbd0 > Masters: [ ha08a.mycharts.md ] > Slaves: [ ha08c.mycharts.md ] > Master/Slave Set: ms_drbd1 > Masters: [ ha08b.mycharts.md ] > Slaves: [ ha08c.mycharts.md ] > Resource Group: g_clust06 > p_fs_clust06 (ocf::heartbeat:Filesystem): Started > ha08a.mycharts.md > p_vip_clust06 (ocf::heartbeat:IPaddr2): Started > ha08a.mycharts.md > p_mysql_371 (lsb:mysql_371): Started ha08a.mycharts.md > p_mysql_372 (lsb:mysql_372): Started ha08a.mycharts.md > p_mysql_373 (lsb:mysql_373): Started ha08a.mycharts.md > p_mysql_374 (lsb:mysql_374): Started ha08a.mycharts.md > p_mysql_375 (lsb:mysql_375): Started ha08a.mycharts.md > p_mysql_376 (lsb:mysql_376): Started ha08a.mycharts.md > p_mysql_047 (lsb:mysql_047): Started ha08a.mycharts.md > p_mysql_100 (lsb:mysql_100): Started ha08a.mycharts.md > p_mysql_379 (lsb:mysql_379): Started ha08a.mycharts.md > p_mysql_377 (lsb:mysql_377): Started ha08a.mycharts.md > p_mysql_378 (lsb:mysql_378): Started ha08a.mycharts.md > p_mysql_380 (lsb:mysql_380): Started ha08a.mycharts.md > p_mysql_381 (lsb:mysql_381): Started ha08a.mycharts.md > p_mysql_382 (lsb:mysql_382): Started ha08a.mycharts.md > p_mysql_383 (lsb:mysql_383): Started ha08a.mycharts.md > p_mysql_384 (lsb:mysql_384): Started ha08a.mycharts.md > p_mysql_385 (lsb:mysql_385): Started ha08a.mycharts.md > p_mysql_386 (lsb:mysql_386): Started ha08a.mycharts.md > p_mysql_387 (lsb:mysql_387): Started ha08a.mycharts.md > p_mysql_002 (lsb:mysql_002): Started ha08a.mycharts.md > p_mysql_035 (lsb:mysql_035): Started ha08a.mycharts.md > p_mysql_049 (lsb:mysql_049): Started ha08a.mycharts.md > p_mysql_097 (lsb:mysql_097): Started ha08a.mycharts.md > p_mysql_024 (lsb:mysql_024): Started ha08a.mycharts.md > p_mysql_077 (lsb:mysql_077): Started ha08a.mycharts.md > p_mysql_084 (lsb:mysql_084): Started ha08a.mycharts.md > p_mysql_113 (lsb:mysql_113): Started ha08a.mycharts.md > p_mysql_116 (lsb:mysql_116): Started ha08a.mycharts.md > p_mysql_388 (lsb:mysql_388): Started ha08a.mycharts.md > p_mysql_389 (lsb:mysql_389): Started ha08a.mycharts.md > p_mysql_390 (lsb:mysql_390): Started ha08a.mycharts.md > p_mysql_391 (lsb:mysql_391): Started ha08a.mycharts.md > p_mysql_392 (lsb:mysql_392): Started ha08a.mycharts.md > p_mysql_393 (lsb:mysql_393): Started ha08a.mycharts.md > p_mysql_394 (lsb:mysql_394): Started ha08a.mycharts.md > p_mysql_395 (lsb:mysql_395): Started ha08a.mycharts.md > p_mysql_396 (lsb:mysql_396): Started ha08a.mycharts.md > p_mysql_397 (lsb:mysql_397): Started ha08a.mycharts.md > p_mysql_398 (lsb:mysql_398): Started ha08a.mycharts.md > p_mysql_399 (lsb:mysql_399): Started ha08a.mycharts.md > p_mysql_400 (lsb:mysql_400): Started ha08a.mycharts.md > p_mysql_401 (lsb:mysql_401): Started ha08a.mycharts.md > p_mysql_402 (lsb:mysql_402): Started ha08a.mycharts.md > p_mysql_403 (lsb:mysql_403): Started ha08a.mycharts.md > p_mysql_404 (lsb:mysql_404): Started ha08a.mycharts.md > p_mysql_405 (lsb:mysql_405): Started ha08a.mycharts.md > p_mysql_104 (lsb:mysql_104): Started ha08a.mycharts.md > p_mysql_406 (lsb:mysql_406): Started ha08a.mycharts.md > p_mysql_407 (lsb:mysql_407): Started ha08a.mycharts.md > p_mysql_408 (lsb:mysql_408): Started ha08a.mycharts.md > p_mysql_409 (lsb:mysql_409): Started ha08a.mycharts.md > p_mysql_410 (lsb:mysql_410): Started ha08a.mycharts.md > p_mysql_017 (lsb:mysql_017): Started ha08a.mycharts.md > p_mysql_089 (lsb:mysql_089): Started ha08a.mycharts.md > p_mysql_090 (lsb:mysql_090): Started ha08a.mycharts.md > p_mysql_109 (lsb:mysql_109): Started ha08a.mycharts.md > p_mysql_411 (lsb:mysql_411): Started ha08a.mycharts.md > p_mysql_412 (lsb:mysql_412): Started ha08a.mycharts.md > p_mysql_413 (lsb:mysql_413): Started ha08a.mycharts.md > p_mysql_414 (lsb:mysql_414): Started ha08a.mycharts.md > p_mysql_415 (lsb:mysql_415): Started ha08a.mycharts.md > p_mysql_416 (lsb:mysql_416): Started ha08a.mycharts.md > p_mysql_417 (lsb:mysql_417): Started ha08a.mycharts.md > p_mysql_020 (lsb:mysql_020): Started ha08a.mycharts.md > p_mysql_076 (lsb:mysql_076): Started ha08a.mycharts.md > p_mysql_101 (lsb:mysql_101): Started ha08a.mycharts.md > p_mysql_051 (lsb:mysql_051): Started ha08a.mycharts.md > Resource Group: g_clust07 > p_fs_clust07 (ocf::heartbeat:Filesystem): Started > ha08b.mycharts.md > p_vip_clust07 (ocf::heartbeat:IPaddr2): Started > ha08b.mycharts.md > p_mysql_418 (lsb:mysql_418): Started ha08b.mycharts.md > p_mysql_419 (lsb:mysql_419): Started ha08b.mycharts.md > p_mysql_420 (lsb:mysql_420): Started ha08b.mycharts.md > p_mysql_421 (lsb:mysql_421): Started ha08b.mycharts.md > p_mysql_422 (lsb:mysql_422): Started ha08b.mycharts.md > p_mysql_423 (lsb:mysql_423): Started ha08b.mycharts.md > p_mysql_424 (lsb:mysql_424): Started ha08b.mycharts.md > p_mysql_425 (lsb:mysql_425): Started ha08b.mycharts.md > p_mysql_426 (lsb:mysql_426): Started ha08b.mycharts.md > p_mysql_427 (lsb:mysql_427): Started ha08b.mycharts.md > p_mysql_056 (lsb:mysql_056): Started ha08b.mycharts.md > p_mysql_133 (lsb:mysql_133): Started ha08b.mycharts.md > p_mysql_428 (lsb:mysql_428): Started ha08b.mycharts.md > p_mysql_429 (lsb:mysql_429): Started ha08b.mycharts.md > p_mysql_430 (lsb:mysql_430): Started ha08b.mycharts.md > p_mysql_431 (lsb:mysql_431): Started ha08b.mycharts.md > p_mysql_432 (lsb:mysql_432): Started ha08b.mycharts.md > p_mysql_433 (lsb:mysql_433): Started ha08b.mycharts.md > p_mysql_434 (lsb:mysql_434): Started ha08b.mycharts.md > p_mysql_435 (lsb:mysql_435): Started ha08b.mycharts.md > p_mysql_436 (lsb:mysql_436): Started ha08b.mycharts.md > p_mysql_437 (lsb:mysql_437): Started ha08b.mycharts.md > p_mysql_438 (lsb:mysql_438): Started ha08b.mycharts.md > p_mysql_439 (lsb:mysql_439): Started ha08b.mycharts.md > p_mysql_440 (lsb:mysql_440): Started ha08b.mycharts.md > p_mysql_441 (lsb:mysql_441): Started ha08b.mycharts.md > p_mysql_442 (lsb:mysql_442): Started ha08b.mycharts.md > p_mysql_443 (lsb:mysql_443): Started ha08b.mycharts.md > p_mysql_444 (lsb:mysql_444): Started ha08b.mycharts.md > p_mysql_445 (lsb:mysql_445): Started ha08b.mycharts.md > p_mysql_446 (lsb:mysql_446): Started ha08b.mycharts.md > p_mysql_447 (lsb:mysql_447): Started ha08b.mycharts.md > p_mysql_448 (lsb:mysql_448): Started ha08b.mycharts.md > p_mysql_449 (lsb:mysql_449): Started ha08b.mycharts.md > p_mysql_450 (lsb:mysql_450): Started ha08b.mycharts.md > p_mysql_019 (lsb:mysql_019): Started ha08b.mycharts.md > p_mysql_120 (lsb:mysql_120): Started ha08b.mycharts.md > p_mysql_451 (lsb:mysql_451): Started ha08b.mycharts.md > p_mysql_452 (lsb:mysql_452): Started ha08b.mycharts.md > p_mysql_453 (lsb:mysql_453): Started ha08b.mycharts.md > p_mysql_454 (lsb:mysql_454): Started ha08b.mycharts.md > p_mysql_455 (lsb:mysql_455): Started ha08b.mycharts.md > p_mysql_456 (lsb:mysql_456): Started ha08b.mycharts.md > p_mysql_457 (lsb:mysql_457): Started ha08b.mycharts.md > p_mysql_458 (lsb:mysql_458): Started ha08b.mycharts.md > p_mysql_459 (lsb:mysql_459): Started ha08b.mycharts.md > p_mysql_460 (lsb:mysql_460): Started ha08b.mycharts.md > > Here is our crm config... > > ^[[?1034hnode ha08a.mycharts.md \ > attributes standby="off" > node ha08b.mycharts.md \ > attributes standby="off" > node ha08c.mycharts.md \ > attributes standby="off" > primitive p_drbd0 ocf:linbit:drbd \ > params drbd_resource="ha01_mysql" \ > op monitor interval="30s" role="Slave" \ > op monitor interval="15s" role="Master" > primitive p_drbd1 ocf:linbit:drbd \ > params drbd_resource="ha02_mysql" \ > op monitor interval="30s" role="Slave" \ > op monitor interval="15s" role="Master" > primitive p_fs_clust06 ocf:heartbeat:Filesystem \ > params device="/dev/drbd0" directory="/ha01_mysql" fstype="ext3" > options="noatime" > primitive p_fs_clust07 ocf:heartbeat:Filesystem \ > params device="/dev/drbd1" directory="/ha02_mysql" fstype="ext3" > options="noatime" > primitive p_mysql_002 lsb:mysql_002 > primitive p_mysql_017 lsb:mysql_017 > primitive p_mysql_019 lsb:mysql_019 > primitive p_mysql_020 lsb:mysql_020 > primitive p_mysql_024 lsb:mysql_024 > primitive p_mysql_035 lsb:mysql_035 > primitive p_mysql_047 lsb:mysql_047 > primitive p_mysql_049 lsb:mysql_049 > primitive p_mysql_051 lsb:mysql_051 > primitive p_mysql_056 lsb:mysql_056 > primitive p_mysql_076 lsb:mysql_076 > primitive p_mysql_077 lsb:mysql_077 > primitive p_mysql_084 lsb:mysql_084 > primitive p_mysql_089 lsb:mysql_089 > primitive p_mysql_090 lsb:mysql_090 > primitive p_mysql_097 lsb:mysql_097 > primitive p_mysql_100 lsb:mysql_100 > primitive p_mysql_101 lsb:mysql_101 > primitive p_mysql_104 lsb:mysql_104 > primitive p_mysql_109 lsb:mysql_109 > primitive p_mysql_113 lsb:mysql_113 > primitive p_mysql_116 lsb:mysql_116 > primitive p_mysql_120 lsb:mysql_120 > primitive p_mysql_133 lsb:mysql_133 > primitive p_mysql_371 lsb:mysql_371 > primitive p_mysql_372 lsb:mysql_372 > primitive p_mysql_373 lsb:mysql_373 > primitive p_mysql_374 lsb:mysql_374 > primitive p_mysql_375 lsb:mysql_375 > primitive p_mysql_376 lsb:mysql_376 > primitive p_mysql_377 lsb:mysql_377 > primitive p_mysql_378 lsb:mysql_378 > primitive p_mysql_379 lsb:mysql_379 > primitive p_mysql_380 lsb:mysql_380 > primitive p_mysql_381 lsb:mysql_381 > primitive p_mysql_382 lsb:mysql_382 > primitive p_mysql_383 lsb:mysql_383 > primitive p_mysql_384 lsb:mysql_384 > primitive p_mysql_385 lsb:mysql_385 > primitive p_mysql_386 lsb:mysql_386 > primitive p_mysql_387 lsb:mysql_387 > primitive p_mysql_388 lsb:mysql_388 > primitive p_mysql_389 lsb:mysql_389 > primitive p_mysql_390 lsb:mysql_390 > primitive p_mysql_390 lsb:mysql_390 > primitive p_mysql_391 lsb:mysql_391 > primitive p_mysql_392 lsb:mysql_392 > primitive p_mysql_393 lsb:mysql_393 > primitive p_mysql_394 lsb:mysql_394 > primitive p_mysql_395 lsb:mysql_395 > primitive p_mysql_396 lsb:mysql_396 > primitive p_mysql_397 lsb:mysql_397 > primitive p_mysql_398 lsb:mysql_398 > primitive p_mysql_399 lsb:mysql_399 > primitive p_mysql_400 lsb:mysql_400 > primitive p_mysql_401 lsb:mysql_401 > primitive p_mysql_402 lsb:mysql_402 > primitive p_mysql_403 lsb:mysql_403 > primitive p_mysql_404 lsb:mysql_404 > primitive p_mysql_405 lsb:mysql_405 > primitive p_mysql_406 lsb:mysql_406 > primitive p_mysql_407 lsb:mysql_407 > primitive p_mysql_408 lsb:mysql_408 > primitive p_mysql_409 lsb:mysql_409 > primitive p_mysql_410 lsb:mysql_410 > primitive p_mysql_411 lsb:mysql_411 > primitive p_mysql_412 lsb:mysql_412 > primitive p_mysql_413 lsb:mysql_413 > primitive p_mysql_414 lsb:mysql_414 > primitive p_mysql_415 lsb:mysql_415 > primitive p_mysql_416 lsb:mysql_416 > primitive p_mysql_417 lsb:mysql_417 > primitive p_mysql_418 lsb:mysql_418 > primitive p_mysql_419 lsb:mysql_419 > primitive p_mysql_420 lsb:mysql_420 > primitive p_mysql_421 lsb:mysql_421 > primitive p_mysql_422 lsb:mysql_422 > primitive p_mysql_423 lsb:mysql_423 > primitive p_mysql_424 lsb:mysql_424 > primitive p_mysql_425 lsb:mysql_425 > primitive p_mysql_426 lsb:mysql_426 > primitive p_mysql_427 lsb:mysql_427 > primitive p_mysql_428 lsb:mysql_428 > primitive p_mysql_429 lsb:mysql_429 > primitive p_mysql_430 lsb:mysql_430 > primitive p_mysql_431 lsb:mysql_431 > primitive p_mysql_432 lsb:mysql_432 > primitive p_mysql_433 lsb:mysql_433 > primitive p_mysql_434 lsb:mysql_434 > primitive p_mysql_435 lsb:mysql_435 > primitive p_mysql_436 lsb:mysql_436 > primitive p_mysql_437 lsb:mysql_437 > primitive p_mysql_438 lsb:mysql_438 > primitive p_mysql_439 lsb:mysql_439 > primitive p_mysql_440 lsb:mysql_440 > primitive p_mysql_441 lsb:mysql_441 > primitive p_mysql_442 lsb:mysql_442 > primitive p_mysql_443 lsb:mysql_443 > primitive p_mysql_444 lsb:mysql_444 > primitive p_mysql_445 lsb:mysql_445 > primitive p_mysql_446 lsb:mysql_446 > primitive p_mysql_447 lsb:mysql_447 > primitive p_mysql_448 lsb:mysql_448 > primitive p_mysql_449 lsb:mysql_449 > primitive p_mysql_450 lsb:mysql_450 > primitive p_mysql_451 lsb:mysql_451 > primitive p_mysql_452 lsb:mysql_452 > primitive p_mysql_453 lsb:mysql_453 > primitive p_mysql_454 lsb:mysql_454 > primitive p_mysql_455 lsb:mysql_455 > primitive p_mysql_456 lsb:mysql_456 > primitive p_mysql_457 lsb:mysql_457 > primitive p_mysql_458 lsb:mysql_458 > primitive p_mysql_459 lsb:mysql_459 > primitive p_mysql_460 lsb:mysql_460 > primitive p_vip_clust06 ocf:heartbeat:IPaddr2 \ > params ip="192.168.10.208" cidr_netmask="32" \ > op monitor interval="15s" > primitive p_vip_clust07 ocf:heartbeat:IPaddr2 \ > params ip="192.168.10.209" cidr_netmask="32" \ > op monitor interval="15s" > group g_clust06 p_fs_clust06 p_vip_clust06 p_mysql_371 p_mysql_372 > p_mysql_373 p_mysql_374 p_mysql_375 p_mysql_376 p_mysql_047 p_mysql_100 > p_mysql_379 p_mysql_377 p_mysql_378 p_mysql_380 p_mysql_381 p_mysql_382 > p_mysql_383 p_mysql_384 p_mysql_385 p_mysql_386 p_mysql_387 p_mysql_002 > p_mysql_035 p_mysql_049 p_mysql_097 p_mysql_024 p_mysql_077 p_mysql_084 > p_mysql_113 p_mysql_116 p_mysql_388 p_mysql_389 p_mysql_390 p_mysql_391 > p_mysql_392 p_mysql_393 p_mysql_394 p_mysql_395 p_mysql_396 p_mysql_397 > p_mysql_398 p_mysql_399 p_mysql_400 p_mysql_401 p_mysql_402 p_mysql_403 > p_mysql_404 p_mysql_405 p_mysql_104 p_mysql_406 p_mysql_407 p_mysql_408 > p_mysql_409 p_mysql_410 p_mysql_017 p_mysql_089 p_mysql_090 p_mysql_109 > p_mysql_411 p_mysql_412 p_mysql_413 p_mysql_414 p_mysql_415 p_mysql_416 > p_mysql_417 p_mysql_020 p_mysql_076 p_mysql_101 p_mysql_051 \ > meta target-role="Started" > group g_clust07 p_fs_clust07 p_vip_clust07 p_mysql_418 p_mysql_419 > p_mysql_420 p_mysql_421 p_mysql_422 p_mysql_423 p_mysql_424 p_mysql_425 > p_mysql_426 p_mysql_427 p_mysql_056 p_mysql_133 p_mysql_428 p_mysql_429 > p_mysql_430 p_mysql_431 p_mysql_432 p_mysql_433 p_mysql_434 p_mysql_435 > p_mysql_436 p_mysql_437 p_mysql_438 p_mysql_439 p_mysql_440 p_mysql_441 > p_mysql_442 p_mysql_443 p_mysql_444 p_mysql_445 p_mysql_446 p_mysql_447 > p_mysql_448 p_mysql_449 p_mysql_450 p_mysql_019 p_mysql_120 p_mysql_451 > p_mysql_452 p_mysql_453 p_mysql_454 p_mysql_455 p_mysql_456 p_mysql_457 > p_mysql_458 p_mysql_459 p_mysql_460 > ms ms_drbd0 p_drbd0 \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" is-managed="true" > ms ms_drbd1 p_drbd1 \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" is-managed="true" > location cli-prefer-g_clust06 g_clust06 \ > rule $id="cli-prefer-rule-g_clust06" inf: #uname eq ha08a.mycharts.md > location loc1 g_clust06 50: ha08a.mycharts.md > location loc3 g_clust07 50: ha08b.mycharts.md > location loc_ms_drbd0 ms_drbd0 \ > rule $id="loc_ms_drbd0-rule" -inf: #uname ne ha08a.mycharts.md and > #uname ne ha08c.mycharts.md > location loc_ms_drbd1 ms_drbd1 \ > rule $id="loc_ms_drbd1-rule" -inf: #uname ne ha08b.mycharts.md and > #uname ne ha08c.mycharts.md > colocation c_clust06 inf: g_clust06 ms_drbd0:Master > colocation c_clust07 inf: g_clust07 ms_drbd1:Master > order o1 inf: ms_drbd0:promote g_clust06:start > order o2 inf: ms_drbd1:promote g_clust07:start > property $id="cib-bootstrap-options" \ > dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="3" \ > stonith-enabled="false" \ > no-quorum-policy="ignore" \ > symmetric-cluster="true" \ > last-lrm-refresh="1320604660" \ > maintenance-mode="false" > rsc_defaults $id="rsc-options" \ > resource-stickiness="400" > > > -- > Eric Robinson > > > > > > > > Disclaimer - May 7, 2012 > This email and any files transmitted with it are confidential and intended > solely for [email protected]. If you are not the named addressee > you should not disseminate, distribute, copy or alter this email. Any views > or opinions presented in this email are solely those of the author and might > not represent those of Physicians' Managed Care or Physician Select > Management. Warning: Although Physicians' Managed Care or Physician Select > Management has taken reasonable precautions to ensure no viruses are present > in this email, the company cannot accept responsibility for any loss or > damage arising from the use of this email or attachments. > This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems -- Dieses Email wurde elektronisch erstellt und ist ohne handschriftliche Unterschrift gültig.
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
