(12.12.06 12:18), Andrew Beekhof wrote:
On Wed, Dec 5, 2012 at 8:32 PM, Kazunori INOUE
<inouek...@intellilink.co.jp> wrote:
(12.12.05 02:02), David Vossel wrote:
----- Original Message -----
From: "Kazunori INOUE" <inouek...@intellilink.co.jp>
To: "The Pacemaker cluster resource manager"
<pacemaker@oss.clusterlabs.org>
Sent: Monday, December 3, 2012 11:41:56 PM
Subject: Re: [Pacemaker] node status does not change even if pacemakerd
dies
(12.12.03 20:24), Andrew Beekhof wrote:
On Mon, Dec 3, 2012 at 8:15 PM, Kazunori INOUE
<inouek...@intellilink.co.jp> wrote:
(12.11.30 23:52), David Vossel wrote:
----- Original Message -----
From: "Kazunori INOUE" <inouek...@intellilink.co.jp>
To: "pacemaker@oss" <pacemaker@oss.clusterlabs.org>
Sent: Friday, November 30, 2012 2:38:50 AM
Subject: [Pacemaker] node status does not change even if
pacemakerd dies
Hi,
I am testing the latest version.
- ClusterLabs/pacemaker 9c13d14640(Nov 27, 2012)
- corosync 92e0f9c7bb(Nov 07, 2012)
- libqb 30a7871646(Nov 29, 2012)
Although I killed pacemakerd, node status did not change.
[dev1 ~]$ pkill -9 pacemakerd
[dev1 ~]$ crm_mon
:
Stack: corosync
Current DC: dev2 (2472913088) - partition with quorum
Version: 1.1.8-9c13d14
2 Nodes configured, unknown expected votes
0 Resources configured.
Online: [ dev1 dev2 ]
[dev1 ~]$ ps -ef|egrep 'corosync|pacemaker'
root 11990 1 1 16:05 ? 00:00:00 corosync
496 12010 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/cib
root 12011 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/stonithd
root 12012 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/lrmd
496 12013 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/attrd
496 12014 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/pengine
496 12015 1 0 16:05 ? 00:00:00
/usr/libexec/pacemaker/crmd
We want the node status to change to
OFFLINE(stonith-enabled=false),
UNCLEAN(stonith-enabled=true).
That is, we want the function of this deleted code.
https://github.com/ClusterLabs/pacemaker/commit/dfdfb6c9087e644cb898143e198b240eb9a928b4
How are you launching pacemakerd? The systemd service script
relaunches
pacemakerd on failure and pacemakerd has the ability to attach to
all the
old processes if they are still around as if nothing happened.
-- Vossel
Hi David,
We are using RHEL6 and use it for a while after this.
Therefore, I start it by the following commands.
$ /etc/init.d/pacemakerd start
or
$ service pacemaker start
Ok.
Are you using the pacemaker plugin?
When using cman or corosync 2.0, pacemakerd isn't strictly needed
for
normal operation.
Its only there to shutdown and/or respawn failed components.
We are using corosync 2.1,
so service does not stop normally after pacemakerd died.
$ pkill -9 pacemakerd
$ service pacemaker stop
$ echo $?
0
$ ps -ef|egrep 'corosync|pacemaker'
root 3807 1 0 13:10 ? 00:00:00 corosync
496 3827 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/cib
root 3828 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/stonithd
root 3829 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/lrmd
496 3830 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/attrd
496 3831 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/pengine
496 3832 1 0 13:10 ? 00:00:00
/usr/libexec/pacemaker/crmd
Ah yes, that is a problem.
Having pacemaker still running when the init script says it is down...
that is bad. Perhaps we should just make the init script smart enough to
check to make sure all the pacemaker components are down after pacemakerd is
down.
The argument of whether or not the failure of pacemakerd is something that
the cluster should be alerted to is something i'm not sure about. With the
corosync 2.0 stack, pacemakerd really doesn't do anything except launch
processes/relaunch processes. A cluster can be completely functional
without a pacemakerd instance running anywhere. If any of the actual
pacemaker components on a node fail, the logic that causes that node to get
fenced has nothing to do with pacemakerd.
-- Vossel
Hi,
I think that "relaunch processes" of pacemakerd is a very useful function,
so I want to avoid management of a resource in the node in which pacemakerd
does not exist.
You do understand that the node will be fenced if any of those
processes fail right?
Its not like a node could end up in a bad state if pacemakerd isn't
around to respawn things.
The "relaunch processes" is there in attempt to recover before anyone
else notices.
So essentially what you're asking for, is to fence the node and
migrate all the resources so that in the future IF another process
dies, we MIGHT not have to fence the node and migrate all the
resources.
Which doesn't really make much sense IMHO.
Also, I don't think I've ever seen pacemakerd actually crash while the
cluster is running.
All the "interesting" stuff it does happens before the child processes
are launched. It really doesn't do anything that would make it likely
to crash.
Hi,
I recognize that pacemakerd is much less likely to crash.
However, a possibility of being killed by OOM_Killer etc. is not 0%.
So I think that a user gets confused. since behavior at the time of process
death differs even if pacemakerd is running.
case A)
When pacemakerd and other processes (crmd etc.) are the parent-child relation.
$ service corosync start ; service pacemaker start
$ ps -ef|egrep 'corosync|pacemaker|UID'
UID PID PPID C STIME TTY TIME CMD
root 4347 1 0 10:53 ? 00:00:00 corosync
root 4365 1 0 10:53 ? 00:00:00 pacemakerd
496 4367 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/cib
root 4368 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 4369 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 4370 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 4371 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/pengine
496 4372 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/crmd
For example, crmd died.
However, since it is relaunched, the state of the cluster is not affected.
$ pkill -9 crmd
$ ps -ef|egrep 'corosync|pacemaker'
root 4347 1 0 10:53 ? 00:00:01 corosync
root 4365 1 0 10:53 ? 00:00:00 pacemakerd
496 4367 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/cib
root 4368 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 4369 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 4370 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 4371 4365 0 10:53 ? 00:00:00 /usr/libexec/pacemaker/pengine
496 4428 4365 0 10:56 ? 00:00:00 /usr/libexec/pacemaker/crmd
$ crm_mon -1
Last updated: Wed Dec 12 10:56:20 2012
Last change: Wed Dec 12 10:53:40 2012 via crmd on dev2
Stack: corosync
Current DC: dev2 (2472913088) - partition with quorum
Version: 1.1.8-3035414
2 Nodes configured, unknown expected votes
0 Resources configured.
Online: [ dev1 dev2 ]
case B)
When pacemakerd and other processes are NOT the parent-child relation.
Although pacemakerd was killed, it assumed the state where it was respawned by
Upstart.
$ service corosync start ; service pacemaker start
$ pkill -9 pacemakerd
$ ps -ef|egrep 'corosync|pacemaker|UID'
UID PID PPID C STIME TTY TIME CMD
root 21091 1 1 14:52 ? 00:00:00 corosync
496 21099 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/cib
root 21100 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 21101 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 21102 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 21103 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/pengine
496 21104 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/crmd
root 21128 1 1 14:53 ? 00:00:00 /usr/sbin/pacemakerd
In this case, the node will be set to UNCLEAN if crmd dies.
That is, the node will be fenced if there is stonith resource.
$ pkill -9 crmd
$ crm_mon -1
Last updated: Wed Dec 12 14:53:48 2012
Last change: Wed Dec 12 14:53:10 2012 via crmd on dev2
Stack: corosync
Current DC: dev2 (2472913088) - partition with quorum
Version: 1.1.8-3035414
2 Nodes configured, unknown expected votes
0 Resources configured.
Node dev1 (2506467520): UNCLEAN (online)
Online: [ dev2 ]
How about making behavior selectable with an option?
When pacemakerd dies,
mode A) which behaves in an existing way. (default)
mode B) which makes the node UNCLEAN.
Best Regards,
Kazunori INOUE
Making stop work when there is no pacemakerd process is a different
matter. We can make that work.
Though the best solution is to relaunch pacemakerd, if it is difficult,
I think that a shortcut method is to make a node unclean.
And now, I tried Upstart a little bit.
1) started the corosync and pacemaker.
$ cat /etc/init/pacemaker.conf
respawn
script
[ -f /etc/sysconfig/pacemaker ] && {
. /etc/sysconfig/pacemaker
}
exec /usr/sbin/pacemakerd
end script
$ service co start
Starting Corosync Cluster Engine (corosync): [ OK ]
$ initctl start pacemaker
pacemaker start/running, process 4702
$ ps -ef|egrep 'corosync|pacemaker'
root 4695 1 0 17:21 ? 00:00:00 corosync
root 4702 1 0 17:21 ? 00:00:00 /usr/sbin/pacemakerd
496 4703 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/cib
root 4704 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 4705 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 4706 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 4707 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/pengine
496 4708 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/crmd
2) killed pacemakerd.
$ pkill -9 pacemakerd
$ ps -ef|egrep 'corosync|pacemaker'
root 4695 1 0 17:21 ? 00:00:01 corosync
496 4703 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/cib
root 4704 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 4705 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 4706 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 4707 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/pengine
496 4708 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/crmd
root 4760 1 1 17:24 ? 00:00:00 /usr/sbin/pacemakerd
3) then I stopped pacemakerd. however, some processes did not stop.
$ initctl stop pacemaker
pacemaker stop/waiting
$ ps -ef|egrep 'corosync|pacemaker'
root 4695 1 0 17:21 ? 00:00:01 corosync
496 4703 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/cib
root 4704 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/stonithd
root 4705 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd
496 4706 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/attrd
496 4707 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/pengine
Best Regards,
Kazunori INOUE
This isnt the case when the plugin is in use though, but then I'd
also
have expected most of the processes to die also.
Since node status will also change if such a result is brought,
we desire to become so.
----
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)
$ ./configure --sysconfdir=/etc --localstatedir=/var
--without-cman
--without-heartbeat
-snip-
pacemaker configuration:
Version = 1.1.8 (Build: 9c13d14)
Features = generated-manpages agent-manpages
ascii-docs
publican-docs ncurses libqb-logging libqb-ipc lha-fencing
corosync-native
snmp
$ cat config.log
-snip-
6000 | #define BUILD_VERSION "9c13d14"
6001 | /* end confdefs.h. */
6002 | #include <gio/gio.h>
6003 |
6004 | int
6005 | main ()
6006 | {
6007 | if (sizeof (GDBusProxy))
6008 | return 0;
6009 | ;
6010 | return 0;
6011 | }
6012 configure:32411: result: no
6013 configure:32417: WARNING: Unable to support systemd/upstart.
You need
to use glib >= 2.26
-snip-
6286 | #define BUILD_VERSION "9c13d14"
6287 | #define SUPPORT_UPSTART 0
6288 | #define SUPPORT_SYSTEMD 0
Best Regards,
Kazunori INOUE
related bugzilla:
http://bugs.clusterlabs.org/show_bug.cgi?id=5064
Best Regards,
Kazunori INOUE
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
