Any update on this ?
On Sat, Jul 12, 2014 at 11:55 PM, K Mehta <kiranmehta1...@gmail.com> wrote: > Andrew, > Attached is the report. > > Regards, > Kiran > > > On Fri, Jul 11, 2014 at 4:26 AM, Andrew Beekhof <and...@beekhof.net> > wrote: > >> Can you run crm_report for the period covered by your test and attach the >> result please? >> >> On 10 Jul 2014, at 4:32 pm, K Mehta <kiranmehta1...@gmail.com> wrote: >> >> > Didnt see any buffer size suggesstion in syslog >> > Changed buffer size from 20k to 200k and rebooted both systems >> > >> > Did the following after reboot: >> > >> > [root@vsanqa11 kiran]# cat /etc/sysconfig/pacemaker | grep PCMK_ipc >> > # PCMK_ipc_type=shared-mem|socket|posix|sysv >> > export PCMK_ipc_buffer=204800 >> > [root@vsanqa11 kiran]# echo $PCMK_ipc_buffer >> > 204800 >> > >> > [root@vsanqa12 ~]# cat /etc/sysconfig/pacemaker | grep PCMK_ipc >> > # PCMK_ipc_type=shared-mem|socket|posix|sysv >> > export PCMK_ipc_buffer=204800 >> > [root@vsanqa12 ~]# echo $PCMK_ipc_buffer >> > 204800 >> > >> > >> > >> > Same error seen on creation of master >> > Jul 9 23:20:11 vsanqa11 cibadmin[9347]: notice: crm_log_args: >> Invoked: /usr/sbin/cibadmin -c -R --xml-text <crm_config>#012 >> <cluster_property_set id="cib-bootstrap-options">#012 <nvpair >> id="cib-bootstrap-options-dc-version" name="dc-version" >> value="1.1.10-14.el6_5.3-368c726"/>#012 <nvpair >> id="cib-bootstrap-options-cluster-infrastructure" >> name="cluster-infrastructure" value="cman"/>#012 #012 <nvpair >> id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" >> value="ignore"/>#012 <nvpair id="cib-bootstrap-options-stonith-enabled" >> name="stoni >> > Jul 9 23:20:12 vsanqa11 cibadmin[9352]: notice: crm_log_args: >> Invoked: /usr/sbin/cibadmin -c -R --xml-text <crm_config>#012 >> <cluster_property_set id="cib-bootstrap-options">#012 <nvpair >> id="cib-bootstrap-options-dc-version" name="dc-version" >> value="1.1.10-14.el6_5.3-368c726"/>#012 <nvpair >> id="cib-bootstrap-options-cluster-infrastructure" >> name="cluster-infrastructure" value="cman"/>#012 #012 <nvpair >> id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" >> value="false"/>#012 <nvpair id="cib-bootstrap-options-no-quorum-policy" >> name="no-quor >> > Jul 9 23:20:12 vsanqa11 cibadmin[9466]: notice: crm_log_args: >> Invoked: /usr/sbin/cibadmin -o resources -C -X <primitive class="ocf" >> id="vha-256ac0ea-c7bd-4c86-89e3-8b934c6b7281" provider="heartbeat" >> type="vgc-cm-agent.ocf"><instance_attributes >> id="vha-256ac0ea-c7bd-4c86-89e3-8b934c6b7281-instance_attributes"><nvpair >> id="vha-256ac0ea-c7bd-4c86-89e3-8b934c6b7281-instance_attributes-cluster_uuid" >> name="cluster_uuid" >> value="256ac0ea-c7bd-4c86-89e3-8b934c6b7281"/></instance_attributes><operations><op >> id="vha-256ac0ea-c7bd-4c86-89e3-8b934c6b7281-mon >> > Error: unable to locate command: /usr/sbin/cibadmin >> > >> > >> > >> > >> > >> > On Thu, Jul 10, 2014 at 3:40 AM, Andrew Beekhof <and...@beekhof.net> >> wrote: >> > >> > On 9 Jul 2014, at 6:49 pm, K Mehta <kiranmehta1...@gmail.com> wrote: >> > >> > > Hi, >> > > >> > > [root@vsanqa11 ~]# rpm -qa | grep pcs ; rpm -qa | grep pace ; rpm >> -qa | grep libqb; rpm -qa | grep coro; rpm -qa | grep cman >> > > pcs-0.9.90-2.el6.centos.2.noarch >> > > pacemaker-cli-1.1.10-14.el6_5.3.x86_64 >> > > pacemaker-libs-1.1.10-14.el6_5.3.x86_64 >> > > pacemaker-1.1.10-14.el6_5.3.x86_64 >> > > pacemaker-cluster-libs-1.1.10-14.el6_5.3.x86_64 >> > > libqb-devel-0.16.0-2.el6.x86_64 >> > > libqb-0.16.0-2.el6.x86_64 >> > > corosynclib-1.4.1-17.el6_5.1.x86_64 >> > > corosync-1.4.1-17.el6_5.1.x86_64 >> > > cman-3.0.12.1-59.el6_5.2.x86_64 >> > > [root@vsanqa11 ~]# uname -a >> > > Linux vsanqa11 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC >> 2012 x86_64 x86_64 x86_64 GNU/Linux >> > > [root@vsanqa11 ~]# cat /etc/redhat-release >> > > CentOS release 6.3 (Final) >> > > >> > > >> > > Created 27 resources >> > > >> > > [root@vsanqa11 ~]# pcs status >> > > Cluster name: vsanqa11_12 >> > > Last updated: Wed Jul 9 01:30:05 2014 >> > > Last change: Wed Jul 9 01:24:12 2014 via cibadmin on vsanqa11 >> > > Stack: cman >> > > Current DC: vsanqa11 - partition with quorum >> > > Version: 1.1.10-14.el6_5.3-368c726 >> > > 2 Nodes configured >> > > 54 Resources configured >> > > >> > > >> > > Online: [ vsanqa11 vsanqa12 ] >> > > >> > > Full list of resources: >> > > >> > > Master/Slave Set: ms-0feba438-0f16-4de5-bf18-9d576cf4dd26 >> [vha-0feba438-0f16-4de5-bf18-9d576cf4dd26] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-329ad1bd-2f9c-483d-a052-270731aefd70 >> [vha-329ad1bd-2f9c-483d-a052-270731aefd70] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-b5b9e8dc-87c9-4229-b425-870d1bc1f107 >> [vha-b5b9e8dc-87c9-4229-b425-870d1bc1f107] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-56112cc8-4c56-4454-84ea-ba9b7b82dde7 >> [vha-56112cc8-4c56-4454-84ea-ba9b7b82dde7] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-20fd5ed6-8e72-4a6b-9b1a-73bd96e1f253 >> [vha-20fd5ed6-8e72-4a6b-9b1a-73bd96e1f253] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-77af4d7e-d78c-4799-a24a-7536d2cccc25 >> [vha-77af4d7e-d78c-4799-a24a-7536d2cccc25] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-913f6e7e-932f-4c3f-9e7f-a70c4153b4c7 >> [vha-913f6e7e-932f-4c3f-9e7f-a70c4153b4c7] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-3d3ae48c-1955-4c64-b951-f2a2621a70b7 >> [vha-3d3ae48c-1955-4c64-b951-f2a2621a70b7] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-a8101da1-636f-483b-90a9-9a18fd5a5793 >> [vha-a8101da1-636f-483b-90a9-9a18fd5a5793] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-290e0fdd-19c6-4b40-8046-0fa2c94a7320 >> [vha-290e0fdd-19c6-4b40-8046-0fa2c94a7320] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-8341368b-6b6a-4bf0-bb0a-38b32ffec2f4 >> [vha-8341368b-6b6a-4bf0-bb0a-38b32ffec2f4] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-6695f479-0d21-4388-9540-a440c32e0944 >> [vha-6695f479-0d21-4388-9540-a440c32e0944] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-85fe517e-9a2b-4ac1-b6e1-e4da57b49969 >> [vha-85fe517e-9a2b-4ac1-b6e1-e4da57b49969] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-5b9adf5c-3bb5-4353-8c5f-a3e1508f7c4a >> [vha-5b9adf5c-3bb5-4353-8c5f-a3e1508f7c4a] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-57b4b7a1-d3bb-4b94-b8dc-8a84e4a3de03 >> [vha-57b4b7a1-d3bb-4b94-b8dc-8a84e4a3de03] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-b4fbac14-ef19-4861-93e5-14574e101484 >> [vha-b4fbac14-ef19-4861-93e5-14574e101484] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-ec4836d3-b768-4abd-83bf-3a61143346ce >> [vha-ec4836d3-b768-4abd-83bf-3a61143346ce] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-28d6db1a-bdba-4fc0-b74c-ef74548cf714 >> [vha-28d6db1a-bdba-4fc0-b74c-ef74548cf714] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-c6cfd538-17f4-4c4b-b259-731e2cac75f3 >> [vha-c6cfd538-17f4-4c4b-b259-731e2cac75f3] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-06d19988-7549-4882-b780-fd598714ec7f >> [vha-06d19988-7549-4882-b780-fd598714ec7f] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-ec105b52-42b1-42f5-931f-249fbc2f16c4 >> [vha-ec105b52-42b1-42f5-931f-249fbc2f16c4] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-6cc9a97b-9714-4cab-bf52-103c46b8593f >> [vha-6cc9a97b-9714-4cab-bf52-103c46b8593f] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-b80b4a69-67e2-43bd-8eba-f6c18f78d706 >> [vha-b80b4a69-67e2-43bd-8eba-f6c18f78d706] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-24817d3c-98e0-4408-9bce-67d8bb2495db >> [vha-24817d3c-98e0-4408-9bce-67d8bb2495db] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-a6d08422-8592-42cf-995a-8c72c6368cd2 >> [vha-a6d08422-8592-42cf-995a-8c72c6368cd2] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-100f3ec9-03aa-476c-9569-60463cc2ac8e >> [vha-100f3ec9-03aa-476c-9569-60463cc2ac8e] >> > > Masters: [ vsanqa12 ] >> > > Slaves: [ vsanqa11 ] >> > > Master/Slave Set: ms-7ec2decb-5426-4ecf-a60f-9996ea0ca445 >> [vha-7ec2decb-5426-4ecf-a60f-9996ea0ca445] >> > > Masters: [ vsanqa11 ] >> > > Slaves: [ vsanqa12 ] >> > > >> > > >> > > [root@vsanqa11 ~]# pcs resource create >> vha-3de5ab16-9917-4b90-93d2-7b04fc71879c ocf:heartbeat:vgc-cm-agent.ocf >> cluster_uuid=3de5ab16-9917-4b90-93d2-7b04fc71879c op monitor role="Master" >> interval=30s timeout=100s op monitor role="Master" interval=30s timeout=100s >> > > >> > > >> > > pcs status output includes >> > > vha-3de5ab16-9917-4b90-93d2-7b04fc71879c >> (ocf::heartbeat:vgc-cm-agent.ocf): Started vsanqa11 >> > > >> > > >> > > [root@vsanqa11 ~]# pcs resource master >> ms-3de5ab16-9917-4b90-93d2-7b04fc71879c >> vha-3de5ab16-9917-4b90-93d2-7b04fc71879c meta clone-max=2 >> globally-unique=false target-role=started >> > > Error: unable to locate command: /usr/sbin/cibadmin >> > >> > An odd error, but I'm pretty sure you're hitting IPC limits. >> > Check out /etc/sysconfig/pacemaker and look for 'ipc' (then make the >> number much bigger :) >> > The logs will probably indicate a good value to choose. >> > >> > Two extra notes >> > - since you're not using systemd, prefix the line with 'export ' >> > - you'll need to set the same values in your shell environment >> > >> > >> > > >> > > >> > > Deleted one of the resources that was created successfully and reran >> above command to create master resource >> ms-3de5ab16-9917-4b90-93d2-7b04fc71879c and it worked. >> > > >> > > Tried this multiple times and each time saw the same behavior >> > > >> > > Attached is the debug output for the above command. This output >> contains string "unable to locate command: /usr/sbin/cibadmin" >> > > >> > > Regards, >> > > kiran >> > > >> > > >> > > <pcs_debug>_______________________________________________ >> > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> > > >> > > Project Home: http://www.clusterlabs.org >> > > Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> > > Bugs: http://bugs.clusterlabs.org >> > >> > >> > _______________________________________________ >> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> > >> > Project Home: http://www.clusterlabs.org >> > Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> > Bugs: http://bugs.clusterlabs.org >> > >> > >> > _______________________________________________ >> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> > >> > Project Home: http://www.clusterlabs.org >> > Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> > Bugs: http://bugs.clusterlabs.org >> >> >> _______________________________________________ >> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> >> >
_______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
