Hi, On Tue, Jan 18, 2011 at 08:34:57AM +0530, akshay punja wrote: > Please let me know if any one has solved this issue. CCM exiting with return > code 100 and system rebooting
Either bad installation or some kind of security mechanism preventing heartbeat/ccm from operating normally. For instance, this looks suspicious: Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler parameters.: Operation not permitted Thanks, Dejan > On Mon, Jan 17, 2011 at 1:29 PM, akshay punja <akshay.pu...@gmail.com>wrote: > > > Hi All, > > > > We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with > > heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment. > > > > Node : we are using two node in a cluster and hosting a bunch of > > application on the HA. > > > > We are seeing a strange rebooting of one of the nodes *Managed > > /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could > > be possible issue and how could we fix it. > > * > > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes > > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: false > > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is > > disabled --enabling logging daemon is recommended > > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: > > ************************** > > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration validated. > > Starting heartbeat 3.0.2 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version 3.0.2 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation: > > 1293182645 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write > > socket priority set to IPTOS_LOWDELAY on eth0 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send > > socket to device: eth0 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound > > receive socket to device: eth0 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on > > port 694 interface eth0 to 172.21.52.135 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: > > G_main_add_TriggerHandler: Added signal manual handler > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: > > G_main_add_TriggerHandler: Added signal manual handler > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: > > G_main_add_SignalHandler: Added signal handler for signal 17 > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler > > parameters.: Operation not permitted > > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set to: > > 'up' > > Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set scheduler > > parameters.: Operation not permitted > > Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set scheduler > > parameters.: Operation not permitted > > Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set scheduler > > parameters.: Operation not permitted > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating > > status to active > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set to: > > 'active' > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/ccm" (100,101) > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/cib" (100,101) > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/lrmd -r" (0,0) > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/stonithd" (0,0) > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/attrd" (100,101) > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client > > "/usr/lib/heartbeat/crmd" (100,101) > > Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting > > "/usr/lib/heartbeat/ccm" as uid 100 gid 101 (pid 19576) > > Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting > > "/usr/lib/heartbeat/cib" as uid 100 gid 101 (pid 19577) > > Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting > > "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 19578) > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: > > Added signal handler for signal 15 > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: > > Added signal handler for signal 17 > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: > > Added signal handler for signal 10 > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: > > Added signal handler for signal 12 > > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started. > > Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting > > "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 19579) > > Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting > > "/usr/lib/heartbeat/attrd" as uid 100 gid 101 (pid 19580) > > *Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed > > /usr/lib/heartbeat/ccm process 19576 exited with return code 100. > > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system. > > Reason: /usr/lib/heartbeat/ccm* > > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler: > > Added signal handler for signal 10 > > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler: > > Added signal handler for signal 12 > > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect: > > Connecting to Heartbeat > > Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting > > "/usr/lib/heartbeat/crmd" as uid 100 gid 101 (pid 19581) > > Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS > > FAILED: /sbin/reboot -nf returned 0 > > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: register_heartbeat_conn: > > Cannot sign on with heartbeat: > > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to > > cluster > > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: > > /usr/lib/heartbeat/stonithd abnormally abort. > > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown: > > Master Control process died. > > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with > > SIGTERM > > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with > > SIGTERM > > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with > > SIGTERM > > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP > > dead): Killing ourselves.* > > > > Regards, > > Akshay > > > > > > * > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
