pengine[11911]: 2008/06/17_10:38:33 info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
stonithd[11751]: 2008/06/17_10:41:33 ERROR: Failed to STONITH the node x3650a: optype=RESET, op_result=TIMEOUT
tengine[11910]: 2008/06/17_10:41:33 info: tengine_stonith_callback: call=12047, optype=1, node_name=x3650a, result=2, node_list=, action=18:5:2241a831-4622-4b30-be50-0d3144af21af
crmd[11753]: 2008/06/17_10:41:33 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
tengine[11910]: 2008/06/17_10:41:33 ERROR: tengine_stonith_callback: Stonith of x3650a failed (2)... aborting transition.
crmd[11753]: 2008/06/17_10:41:33 info: do_state_transition: All 1 cluster nodes are eligible to run resources.
tengine[11910]: 2008/06/17_10:41:33 info: update_abort_priority: Abort priority upgraded to 1000000
tengine[11910]: 2008/06/17_10:41:33 info: update_abort_priority: Abort action 0 superceeded by 2
tengine[11910]: 2008/06/17_10:41:33 info: run_graph: ====================================================
tengine[11910]: 2008/06/17_10:41:33 notice: run_graph: Transition 5: (Complete=4, Pending=0, Fired=0, Skipped=9, Incomplete=0)
stonithd[11751]: 2008/06/17_10:41:33 WARN: Managed external_r_stonith-node01_1 process 12047 killed by signal 9 [SIGKILL - Kill, unblockable].
stonithd[11751]: 2008/06/17_10:41:33 WARN: child exits, but not tracked.
pengine[11911]: 2008/06/17_10:41:33 notice: unpack_config: On loss of CCM Quorum: Ignore
pengine[11911]: 2008/06/17_10:41:33 WARN: determine_online_status_fencing: Node x3650a (05e25c65-839e-48a8-9c3b-b94f63ea4b55) is un-expectedly down
pengine[11911]: 2008/06/17_10:41:33 info: determine_online_status_fencing: 	ha_state=dead, ccm_state=false, crm_state=online, join_state=down, expected=member
pengine[11911]: 2008/06/17_10:41:33 WARN: determine_online_status: Node x3650a is unclean
pengine[11911]: 2008/06/17_10:41:33 info: determine_online_status: Node x3650b is shutting down
pengine[11911]: 2008/06/17_10:41:33 notice: group_print: Resource Group: grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: native_print:     prmFsPostgreSQLDB1	(heartbeat::ocf:Filesystem):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 notice: native_print:     prmFsPostgreSQLDB2	(heartbeat::ocf:Filesystem):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 notice: native_print:     prmFsPostgreSQLDB3	(heartbeat::ocf:Filesystem):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 notice: native_print:     prmIpPostgreSQLDB	(heartbeat::ocf:IPaddr):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 notice: native_print:     prmApPostgreSQLDB	(heartbeat::ocf:pgsql):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 notice: native_print: r_stonith-node01	(stonith:external/ibmrsa-telnet):	Started x3650b
pengine[11911]: 2008/06/17_10:41:33 notice: native_print: r_stonith-node02	(stonith:external/ibmrsa-telnet):	Started x3650a
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource prmFsPostgreSQLDB1 cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource prmFsPostgreSQLDB2 cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource prmFsPostgreSQLDB3 cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource prmIpPostgreSQLDB cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource prmApPostgreSQLDB cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB1_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB2_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB3_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmIpPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmApPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource r_stonith-node01 cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650b	Stop r_stonith-node01
pengine[11911]: 2008/06/17_10:41:33 WARN: native_color: Resource r_stonith-node02 cannot run anywhere
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action r_stonith-node02_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 info: stage6: Scheduling Node x3650b for shutdown
pengine[11911]: 2008/06/17_10:41:33 WARN: stage6: Scheduling Node x3650a for STONITH
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: prmFsPostgreSQLDB1_stop_0 is implicit after x3650a is fenced
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: Re-creating actions for grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB1
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB2_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB3_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmIpPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmApPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: prmFsPostgreSQLDB2_stop_0 is implicit after x3650a is fenced
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: Re-creating actions for grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB1
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB2
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmFsPostgreSQLDB3_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmIpPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmApPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: prmFsPostgreSQLDB3_stop_0 is implicit after x3650a is fenced
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: Re-creating actions for grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB1
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB2
crmd[11753]: 2008/06/17_10:41:33 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB3
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmIpPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmApPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: prmIpPostgreSQLDB_stop_0 is implicit after x3650a is fenced
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: Re-creating actions for grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB1
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB2
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB3
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmIpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Action prmApPostgreSQLDB_stop_0 on x3650a is unrunnable (offline)
pengine[11911]: 2008/06/17_10:41:33 WARN: custom_action: Marking node x3650a unclean
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: prmApPostgreSQLDB_stop_0 is implicit after x3650a is fenced
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: Re-creating actions for grpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB1
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB2
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmFsPostgreSQLDB3
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmIpPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 notice: StopRsc:   x3650a	Stop prmApPostgreSQLDB
pengine[11911]: 2008/06/17_10:41:33 info: native_stop_constraints: r_stonith-node02_stop_0 is implicit after x3650a is fenced
tengine[11910]: 2008/06/17_10:41:33 info: unpack_graph: Unpacked transition 6: 13 actions in 13 synapses
tengine[11910]: 2008/06/17_10:41:33 info: te_pseudo_action: Pseudo action 12 fired and confirmed
tengine[11910]: 2008/06/17_10:41:33 info: te_pseudo_action: Pseudo action 15 fired and confirmed
tengine[11910]: 2008/06/17_10:41:33 info: te_pseudo_action: Pseudo action 16 fired and confirmed
tengine[11910]: 2008/06/17_10:41:33 info: te_fence_node: Executing reboot fencing operation (18) on x3650a (timeout=180000)
stonithd[11751]: 2008/06/17_10:41:33 info: client tengine [pid: 11910] want a STONITH operation RESET to node x3650a.
pengine[11911]: 2008/06/17_10:41:33 WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/heartbeat/pengine/pe-warn-120.bz2
stonithd[11751]: 2008/06/17_10:41:33 info: stonith_operate_locally::2375: sending fencing op (RESET) for x3650a to device external (rsc_id=r_stonith-node01, pid=12064)
pengine[11911]: 2008/06/17_10:41:33 info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11739 exited. 7 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11741 exited. 6 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11743 exited. 5 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11738 exited. 4 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11740 exited. 3 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11742 exited. 2 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: Core process 11744 exited. 1 remaining
heartbeat[11732]: 2008/06/17_10:41:37 info: x3650b Heartbeat shutdown complete.
ccm[11748]: 2008/06/17_10:41:37 ERROR: Lost connection to heartbeat service. Need to bail out.
crmd[11753]: 2008/06/17_10:41:37 info: mem_handle_func:IPC broken, ccm is dead before the client!
cib[11749]: 2008/06/17_10:41:37 info: mem_handle_func:IPC broken, ccm is dead before the client!
crmd[11753]: 2008/06/17_10:41:37 ERROR: ccm_dispatch: CCM connection appears to have failed: rc=-1.
cib[11749]: 2008/06/17_10:41:37 ERROR: cib_ccm_dispatch: CCM connection appears to have failed: rc=-1.
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_log: [[FSA]] Input I_ERROR from ccm_dispatch() received in state (S_TRANSITION_ENGINE)
cib[11749]: 2008/06/17_10:41:37 ERROR: cib_ccm_dispatch: Exiting to recover from CCM connection failure
pengine[11911]: 2008/06/17_10:41:37 info: pengine_shutdown: Exiting PEngine (SIGTERM)
crmd[11753]: 2008/06/17_10:41:37 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_RECOVERY [ input=I_ERROR cause=C_CCM_CALLBACK origin=ccm_dispatch ]
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported
tengine[11910]: 2008/06/17_10:41:37 info: update_abort_priority: Abort priority upgraded to 1000000
crmd[11753]: 2008/06/17_10:41:37 WARN: do_election_vote: Not voting in election, we're in state S_RECOVERY
tengine[11910]: 2008/06/17_10:41:37 info: update_abort_priority: Abort action 0 superceeded by 3
crmd[11753]: 2008/06/17_10:41:37 info: do_dc_release: DC role released
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to pengine: [11911]
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to tengine: [11910]
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_log: [[FSA]] Input I_TERMINATE from do_recover() received in state (S_RECOVERY)
crmd[11753]: 2008/06/17_10:41:37 info: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]
attrd[11752]: 2008/06/17_10:41:37 ERROR: cib_native_msgready: Message pending on command channel [11749]
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Terminating the pengine
attrd[11752]: 2008/06/17_10:41:37 ERROR: crm_log_message_adv: #========= cib:cmd message start ==========#
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to pengine: [11911]
attrd[11752]: 2008/06/17_10:41:37 ERROR: MSG: No message to dump
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Terminating the tengine
attrd[11752]: 2008/06/17_10:41:37 info: cib_native_msgready: Lost connection to the CIB service [11749].
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to tengine: [11910]
attrd[11752]: 2008/06/17_10:41:37 CRIT: cib_native_dispatch: Lost connection to the CIB service [11749/callback].
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Waiting for subsystems to exit
attrd[11752]: 2008/06/17_10:41:37 ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated...
crmd[11753]: 2008/06/17_10:41:37 WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: All subsystems stopped, continuing
crmd[11753]: 2008/06/17_10:41:37 WARN: do_log: [[FSA]] Input I_PENDING from do_election_vote() received in state (S_TERMINATE)
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Terminating the pengine
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to pengine: [11911]
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Terminating the tengine
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to tengine: [11910]
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Waiting for subsystems to exit
crmd[11753]: 2008/06/17_10:41:37 WARN: register_fsa_input_adv: do_shutdown stalled the FSA with pending inputs
tengine[11910]: 2008/06/17_10:41:37 ERROR: subsystem_msg_dispatch: The server 11753 has left us: Shutting down...NOW
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: All subsystems stopped, continuing
crmd[11753]: 2008/06/17_10:41:37 info: crmdManagedChildDied: Process pengine:[11911] exited (signal=0, exitcode=0)
crmd[11753]: 2008/06/17_10:41:37 WARN: do_log: [[FSA]] Input I_RELEASE_SUCCESS from do_dc_release() received in state (S_TERMINATE)
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Terminating the tengine
crmd[11753]: 2008/06/17_10:41:37 info: stop_subsystem: Sent -TERM to tengine: [11910]
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: Waiting for subsystems to exit
crmd[11753]: 2008/06/17_10:41:37 info: do_shutdown: All subsystems stopped, continuing
crmd[11753]: 2008/06/17_10:41:37 ERROR: cib_native_msgready: Message pending on command channel [11749]
crmd[11753]: 2008/06/17_10:41:37 ERROR: crm_log_message_adv: #========= cib:cmd message start ==========#
crmd[11753]: 2008/06/17_10:41:37 ERROR: MSG: No message to dump
crmd[11753]: 2008/06/17_10:41:37 info: cib_native_msgready: Lost connection to the CIB service [11749].
crmd[11753]: 2008/06/17_10:41:37 CRIT: cib_native_dispatch: Lost connection to the CIB service [11749/callback].
crmd[11753]: 2008/06/17_10:41:37 ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated...
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_log: [[FSA]] Input I_ERROR from crmd_cib_connection_destroy() received in state (S_TERMINATE)
crmd[11753]: 2008/06/17_10:41:37 ERROR: verify_stopped: 1 pending LRM operations at shutdown
crmd[11753]: 2008/06/17_10:41:37 ERROR: ghash_print_pending: Pending action: r_stonith-node01:10
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_exit: Performing A_EXIT_1 - forcefully exiting the CRMd
crmd[11753]: 2008/06/17_10:41:37 ERROR: do_exit: Could not recover from internal error
crmd[11753]: 2008/06/17_10:41:37 info: do_exit: [crmd] stopped (2)
stonithd[11751]: 2008/06/17_10:41:37 ERROR: Disconnected with heartbeat daemon
stonithd[11751]: 2008/06/17_10:41:37 notice: /usr/lib64/heartbeat/stonithd normally quit.
