Re: [Pacemaker] Node stuck in pending state

2014-04-10 Thread Andrew Beekhof
On 11 Apr 2014, at 12:29 am, Brian J. Murrell wrote: > On Thu, 2014-04-10 at 10:04 +1000, Andrew Beekhof wrote: >> >> Brian: the detective work above is highly appreciated > > NP. I feel like I am getting better at reading these logs and can > provide some more detailed dissection of them.

Re: [Pacemaker] Node stuck in pending state

2014-04-10 Thread Brian J. Murrell
On Thu, 2014-04-10 at 10:04 +1000, Andrew Beekhof wrote: > > Brian: the detective work above is highly appreciated NP. I feel like I am getting better at reading these logs and can provide some more detailed dissection of them. And am happy to do so to help get to the bottom of things. :-) >

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Andrew Beekhof
On 10 Apr 2014, at 4:49 am, Brian J. Murrell wrote: > On Tue, 2014-04-08 at 17:29 -0400, Digimer wrote: >> Looks like your fencing (stonith) failed. > > Where? If I'm reading the logs correctly, it looks like stonith worked. > Here's the stonith: > > Apr 8 09:53:21 lotus-4vm6 stonith-ng[249

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Brian J. Murrell
On Tue, 2014-04-08 at 17:29 -0400, Digimer wrote: > Looks like your fencing (stonith) failed. Where? If I'm reading the logs correctly, it looks like stonith worked. Here's the stonith: Apr 8 09:53:21 lotus-4vm6 stonith-ng[2492]: notice: log_operation: Operation 'reboot' [3306] (call 2 from

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Campbell, Gene
This is giving me a lot to go on, thanks for the responses from everyone. Gene On 4/9/14, 10:32 AM, "Digimer" wrote: >When a node enters an unknown state (from the perspective of the rest of >the cluster), it is extremely unsafe to assume what state it is in. The >only safe option is to block

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Digimer
I know what you're talking about. I'm often on the receiving side of unhelpful responses when I ask for help (outside of clustering, usually). It can be extremely annoying and having people speak up can really help improve the friendliness of a community. Only thing I'd suggest is to do so in a

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Michael Monette
Sorry if I was rude towards you. I just feel like I see this sort of thing a lot(other lists) and I guess this time I wanted to just say something.. I wish I could have helped him myself. Mike On April 9, 2014 1:34:42 PM EDT, Digimer wrote: >On 09/04/14 01:09 PM, Michael Monette wrote: >> May

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Digimer
On 09/04/14 01:09 PM, Michael Monette wrote: Maybe he's looking for a reason as to why his stonith is failing. You are basically just repeating to him his stonith is failing...and he already knows because it says so like 20 times in the logs he posted. You got too caught up on giving him how to p

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Digimer
When a node enters an unknown state (from the perspective of the rest of the cluster), it is extremely unsafe to assume what state it is in. The only safe option is to block and call a fence to put the lost node into a known state. Only when the fence action confirms that the lost node was succ

Re: [Pacemaker] Node stuck in pending state

2014-04-09 Thread Campbell, Gene
Thanks for the response. I hope you don¹t mind a couple questions along the way to understanding this issue. We have storage attached to vm5 Power is cut to vm5 Failover to vm6 happens and storage is made available there vm5 reboots Can you tell Where fencing is happening in this picture? Will

Re: [Pacemaker] Node stuck in pending state

2014-04-08 Thread Digimer
Looks like your fencing (stonith) failed. On 08/04/14 05:25 PM, Campbell, Gene wrote: Hello fine folks in Pacemaker land. Hopefully you could share your insight into this little problem for us. We have a intermittent problem with failover. two node cluster first node power is cut failover b

[Pacemaker] Node stuck in pending state

2014-04-08 Thread Campbell, Gene
Hello fine folks in Pacemaker land. Hopefully you could share your insight into this little problem for us. We have a intermittent problem with failover. two node cluster first node power is cut failover begins to second node first node reboots crm_mon -1 on the rebooted node is PENDING (neve