May 21 10:48:07 wimuds1 logger: rc.primary is running...
May 21 10:50:07 wimuds1 logger: rc.primary is running...
May 21 10:50:56 wimuds1 crm_standby: [27610]: info: Invoked: crm_standby -G 
May 21 10:51:00 wimuds1 crm_standby: [27969]: info: Invoked: crm_standby -G 
May 21 10:52:05 wimuds1 crm_standby: [1231]: info: Invoked: crm_standby -G 
May 21 10:52:07 wimuds1 logger: rc.primary is running...
May 21 10:52:53 wimuds1 crm_standby: [6302]: info: Invoked: crm_standby -G 
May 21 10:52:55 wimuds1 crm_standby: [6347]: info: Invoked: crm_standby -G 
May 21 10:52:57 wimuds1 crm_standby: [6414]: info: Invoked: crm_standby -G 
May 21 10:52:58 wimuds1 crm_standby: [6541]: info: Invoked: crm_standby -G 
May 21 10:52:59 wimuds1 crm_standby: [6576]: info: Invoked: crm_standby -G 
May 21 10:53:01 wimuds1 crm_standby: [6642]: info: Invoked: crm_standby -G 
May 21 10:53:02 wimuds1 crm_standby: [6708]: info: Invoked: crm_standby -G 
May 21 10:53:04 wimuds1 crm_standby: [6720]: info: Invoked: crm_standby -v true 
May 21 10:53:04 wimuds1 crm_standby: [6723]: info: Invoked: crm_standby -G 
May 21 10:53:04 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=32:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=rc.primary_6_stop_0 )
May 21 10:53:04 wimuds1 lrmd: [4421]: info: rsc:rc.primary_6: stop
May 21 10:53:04 wimuds1 lrmd: [6788]: WARN: For LSB init script, no additional parameters are needed.
May 21 10:53:04 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation rc.primary_6_monitor_120000 (call=26, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:04 wimuds1 lrmd: [4421]: info: RA output: (rc.primary_6:stop:stdout) Removing /PRIMARY file, Switching to Standby. 
May 21 10:53:04 wimuds1 logger: Removing /PRIMARY file, Switching to Standby.
May 21 10:53:04 wimuds1 lrmd: [4421]: info: RA output: (rc.primary_6:stop:stdout) Stopping rc.primary applications... 
May 21 10:53:04 wimuds1 lrmd: [4421]: info: RA output: (rc.primary_6:stop:stdout) Finished stopping rc.primary applications. 
May 21 10:53:04 wimuds1 logger: Finished stopping rc.primary applications.
May 21 10:53:04 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation rc.primary_6_stop_0 (call=27, rc=0, cib-update=36, confirmed=true) complete ok
May 21 10:53:05 wimuds1 cib: [6816]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-16.raw
May 21 10:53:05 wimuds1 cib: [6816]: info: write_cib_contents: Wrote version 1.118.0 of the CIB to disk (digest: c6ea0d670cf0a8fc38c53f3504b8f1e1)
May 21 10:53:05 wimuds1 cib: [6816]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.HJ7u94 (digest: /var/lib/heartbeat/crm/cib.GSnlTU)
May 21 10:53:05 wimuds1 crm_standby: [6822]: info: Invoked: crm_standby -G 
May 21 10:53:05 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=29:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=mysqld_5_stop_0 )
May 21 10:53:05 wimuds1 lrmd: [4421]: info: rsc:mysqld_5: stop
May 21 10:53:05 wimuds1 lrmd: [6855]: WARN: For LSB init script, no additional parameters are needed.
May 21 10:53:05 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation mysqld_5_monitor_180000 (call=24, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:10 wimuds1 lrmd: [4421]: info: RA output: (mysqld_5:stop:stdout) Stopping MySQL:  
May 21 10:53:10 wimuds1 lrmd: [4421]: info: RA output: (mysqld_5:stop:stdout) [
May 21 10:53:10 wimuds1 lrmd: [4421]: info: RA output: (mysqld_5:stop:stdout)   OK  ]
May 21 10:53:10 wimuds1 lrmd: [4421]: info: RA output: (mysqld_5:stop:stdout) 
 
May 21 10:53:10 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation mysqld_5_stop_0 (call=28, rc=0, cib-update=37, confirmed=true) complete ok
May 21 10:53:10 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=26:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=httpd_4_stop_0 )
May 21 10:53:10 wimuds1 lrmd: [4421]: info: rsc:httpd_4: stop
May 21 10:53:10 wimuds1 lrmd: [7238]: WARN: For LSB init script, no additional parameters are needed.
May 21 10:53:10 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation httpd_4_monitor_120000 (call=22, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:10 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout) Stopping httpd: 
May 21 10:53:12 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout) [
May 21 10:53:12 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout)   OK  
May 21 10:53:12 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout) ]
May 21 10:53:12 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout) 
May 21 10:53:12 wimuds1 lrmd: [4421]: info: RA output: (httpd_4:stop:stdout)  
May 21 10:53:12 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation httpd_4_stop_0 (call=29, rc=0, cib-update=38, confirmed=true) complete ok
May 21 10:53:12 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=23:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=Filesystem_3_stop_0 )
May 21 10:53:12 wimuds1 lrmd: [4421]: info: rsc:Filesystem_3: stop
May 21 10:53:12 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation Filesystem_3_monitor_120000 (call=20, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:13 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) umount: /drbd: device is busy umount: /drbd: device is busy 
May 21 10:53:13 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) /drbd:      
May 21 10:53:13 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stdout)  13716 15483
May 21 10:53:13 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr)         cc 
May 21 10:53:14 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) umount: /drbd: device is busy umount: /drbd: device is busy 
May 21 10:53:14 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) /drbd:           
May 21 10:53:14 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stdout)  13716
May 21 10:53:14 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr)    c 
May 21 10:53:15 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) umount: /drbd: device is busy umount: /drbd: device is busy 
May 21 10:53:15 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) /drbd:       
May 21 10:53:15 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stdout)  13716
May 21 10:53:15 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr)        c 
May 21 10:53:16 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) umount: /drbd: device is busy umount: /drbd: device is busy 
May 21 10:53:16 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr) /drbd:      
May 21 10:53:16 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stdout)  13716
May 21 10:53:16 wimuds1 lrmd: [4421]: info: RA output: (Filesystem_3:stop:stderr)         c 
May 21 10:53:17 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation Filesystem_3_stop_0 (call=30, rc=0, cib-update=39, confirmed=true) complete ok
May 21 10:53:19 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=20:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=drbddisk_2_stop_0 )
May 21 10:53:19 wimuds1 lrmd: [4421]: info: rsc:drbddisk_2: stop
May 21 10:53:19 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation drbddisk_2_monitor_120000 (call=18, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:19 wimuds1 kernel: drbd0: role( Primary -> Secondary ) 
May 21 10:53:19 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation drbddisk_2_stop_0 (call=31, rc=0, cib-update=40, confirmed=true) complete ok
May 21 10:53:20 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=17:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=IPaddr2_2_stop_0 )
May 21 10:53:20 wimuds1 lrmd: [4421]: info: rsc:IPaddr2_2: stop
May 21 10:53:20 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation IPaddr2_2_monitor_5000 (call=16, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:20 wimuds1 kernel: nx_inetaddr_event[638]: Ignoring early callback event 2 on eth2:wimuds-dig
May 21 10:53:20 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation IPaddr2_2_stop_0 (call=32, rc=0, cib-update=41, confirmed=true) complete ok
May 21 10:53:20 wimuds1 avahi-daemon[7021]: Withdrawing address record for 192.168.73.18 on eth2.
May 21 10:53:21 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=14:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=IPaddr2_1_stop_0 )
May 21 10:53:21 wimuds1 lrmd: [4421]: info: rsc:IPaddr2_1: stop
May 21 10:53:21 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation IPaddr2_1_monitor_5000 (call=14, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:21 wimuds1 avahi-daemon[7021]: Withdrawing address record for 165.115.204.215 on eth1.
May 21 10:53:21 wimuds1 kernel: nx_inetaddr_event[638]: Ignoring early callback event 2 on eth1:wimuds
May 21 10:53:21 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation IPaddr2_1_stop_0 (call=33, rc=0, cib-update=42, confirmed=true) complete ok
May 21 10:53:22 wimuds1 kernel: drbd0: peer( Secondary -> Primary ) 
May 21 10:53:22 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=45:101:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=CL_stonith_node02:0_stop_0 )
May 21 10:53:22 wimuds1 lrmd: [4421]: info: rsc:CL_stonith_node02:0: stop
May 21 10:53:22 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation CL_stonith_node02:0_monitor_120000 (call=12, rc=-2, cib-update=0, confirmed=true) Cancelled unknown exec error
May 21 10:53:22 wimuds1 lrmd: [8480]: info: Try to stop STONITH resource <rsc_id=CL_stonith_node02:0> : Device=external/riloe-iders
May 21 10:53:23 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation CL_stonith_node02:0_stop_0 (call=34, rc=0, cib-update=43, confirmed=true) complete ok
May 21 10:53:28 wimuds1 cib: [4420]: info: cib_stats: Processed 58 operations (172.00us average, 0% utilization) in the last 10min
May 21 10:53:56 wimuds1 crm_standby: [9237]: info: Invoked: crm_standby -G 
May 21 10:54:36 wimuds1 crm_standby: [10002]: info: Invoked: crm_standby -v false 
May 21 10:54:37 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=37:102:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=CL_stonith_node02:0_start_0 )
May 21 10:54:37 wimuds1 lrmd: [4421]: info: rsc:CL_stonith_node02:0: start
May 21 10:54:37 wimuds1 lrmd: [10003]: info: Try to start STONITH resource <rsc_id=CL_stonith_node02:0> : Device=external/riloe-iders
May 21 10:54:37 wimuds1 cib: [10005]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-17.raw
May 21 10:54:37 wimuds1 cib: [10005]: info: write_cib_contents: Wrote version 1.119.0 of the CIB to disk (digest: edfda9493d8a1d922a82bfed6bf4fbbe)
May 21 10:54:37 wimuds1 cib: [10005]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.7siqIO (digest: /var/lib/heartbeat/crm/cib.qkLs5N)
May 21 10:54:43 wimuds1 stonithd: [4422]: info: CL_stonith_node02:0 stonith resource started
May 21 10:54:43 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation CL_stonith_node02:0_start_0 (call=35, rc=0, cib-update=44, confirmed=true) complete ok
May 21 10:54:45 wimuds1 crmd: [4424]: info: do_lrm_rsc_op: Performing key=38:102:0:bab4f3f4-86e8-476e-a68d-35217001c11a op=CL_stonith_node02:0_monitor_120000 )
May 21 10:54:51 wimuds1 crmd: [4424]: info: process_lrm_event: LRM operation CL_stonith_node02:0_monitor_120000 (call=36, rc=0, cib-update=45, confirmed=false) complete ok
May 21 10:55:07 wimuds1 crm_standby: [10766]: info: Invoked: crm_standby -G 
May 21 10:56:02 wimuds1 crm_standby: [12279]: info: Invoked: crm_standby -G 
May 21 10:57:42 wimuds1 crm_standby: [14548]: info: Invoked: crm_standby -G 
May 21 11:00:33 wimuds1 logger: Mark:HB_REPORT:1242925231
May 21 11:00:43 wimuds1 crmadmin: [19168]: info: Invoked: crmadmin -D 
May 21 11:00:44 wimuds1 cibadmin: [19170]: info: Invoked: cibadmin -Ql -o nodes 
