On 13 Jun 2014, at 7:51 pm, Johan Huysmans <johan.huysm...@inuits.be> wrote:
> I tested with a custom build libqb rpm > (libqb-0.17.0-1.15.f8b4.dirty.el6.x86_64) however that didn't solved the > issue. > So for now I will add the export of the PCMK_ipc_buffer to my bash_profile. Did you rebuild pacemaker once you had the new libqb stuff installed? The relevant code can't be compiled in unless it has the new libqb API at build time. > > thx > Johan > > > On 13-06-14 11:23, Johan Huysmans wrote: >> I exported the PCMK_ipc_buffer in my console before running the crm_mon and >> everything worked. >> But I see this more as a workaround (could set this in my bash_profile). >> >> I currently have libqb-0.16.0-2.el6.x86_64 installed, >> which version should fix this ? >> >> Thanks! >> Johan H. >> >> >> On 13-06-14 10:41, Andrew Beekhof wrote: >>> On 13 Jun 2014, at 5:35 pm, Johan Huysmans <johan.huysm...@inuits.be> >>> wrote: >>> >>> >>>> Hi, >>>> >>>> The PCMK_ipc_buffer was already set to 10000000 (10M). >>>> >>> Try setting that in your environment too (otherwise the CLI tools wont know >>> about the new value). >>> Or grab a newer libqb which _should_ do the right thing anyway >>> >>> >>>> For testing I increased to buffer to 10000000 (100M), without results. >>>> >>>> I decreased the buffer to the default 20480 (20K) as it then shows the >>>> suggested value. >>>> If I leave it running for a couple of minutes I got these suggested values: >>>> Jun 13 06:33:53 [62206] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (3854984 bytes >>>> suggested) >>>> Jun 13 06:33:54 [62206] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (7709968 bytes >>>> suggested) >>>> Jun 13 06:33:58 [62206] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (15419936 bytes >>>> suggested) >>>> Jun 13 06:35:55 [62206] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (30839872 bytes >>>> suggested) >>>> Jun 13 06:37:56 [62206] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (61679744 bytes >>>> suggested) >>>> Jun 13 07:16:47 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (3952852 bytes >>>> suggested) >>>> Jun 13 07:16:47 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (7905704 bytes >>>> suggested) >>>> Jun 13 07:18:03 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (15811408 bytes >>>> suggested) >>>> Jun 13 07:18:43 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (31622816 bytes >>>> suggested) >>>> Jun 13 07:18:48 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (63245632 bytes >>>> suggested) >>>> Jun 13 07:20:49 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (126491264 bytes >>>> suggested) >>>> Jun 13 07:22:50 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (252982528 bytes >>>> suggested) >>>> Jun 13 07:22:52 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (505965056 bytes >>>> suggested) >>>> Jun 13 07:23:57 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (1011930112 bytes >>>> suggested) >>>> Jun 13 07:24:51 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (2023860224 bytes >>>> suggested) >>>> Jun 13 07:26:52 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (-247246848 bytes >>>> suggested) >>>> Jun 13 07:27:22 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (-494493696 bytes >>>> suggested) >>>> Jun 13 07:29:22 [44112] SRV-5-1 cib: error: crm_ipc_prepare: >>>> Could not compress the message into less than the configured ipc limit >>>> (20480 bytes).Set PCMK_ipc_buffer to a higher value (-988987392 bytes >>>> suggested) >>>> >>>> There is definitely something wrong. Is it the printing of the suggested >>>> value or is it something else ? >>>> >>>> If I check the cib.xml files in /var/lib/pacemaker/cib/ all files are a >>>> bit smaller then 300K. >>>> >>>> Changing these buffers did not solve my problem not getting results from >>>> crm_mon. >>>> >>>> Gr. >>>> Johan >>>> >>>> >>>> On 13-06-14 01:13, Andrew Beekhof wrote: >>>> >>>>> On 12 Jun 2014, at 10:53 pm, Johan Huysmans <johan.huysm...@inuits.be> >>>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>>> Hi All, >>>>>> >>>>>> I deployed Pacemaker 1.1.12-rc2 on our platform to test the cib changes. >>>>>> This was needed on our setup as it contains 6 nodes, 150 resources and >>>>>> the cib process was using lots of cpu. >>>>>> >>>>>> With a limited set of resources (6 nodes, 30 resources) everything >>>>>> worked as expected, including crm_mon. >>>>>> When loading the complete set of resources we lost the crm_mon >>>>>> functionality on all nodes. >>>>>> The cluster is running as expected (running all resources) however we >>>>>> don't have any visibility. >>>>>> >>>>>> I noticed that operations performing changes did actually work like (crm >>>>>> resource stop <resourcename>), >>>>>> but crm resource status didn't work (using crmsh-2.0+git46-1.1.x86_64). >>>>>> >>>>>> I noticed that /dev/shm/qb-cib_ro* files are created, and lsof shows >>>>>> that they are both opened by crm_mon and cib. >>>>>> >>>>>> >>>>>> When executing "crm_mon -1" I get following messages in >>>>>> /var/log/messages (and /var/log/pacemaker.log) >>>>>> Jun 12 12:47:38 [8062] SRV-5-1 cib: notice: crm_ipcs_sendv: >>>>>> Response 2 to 0x1810370[17836] (1091618 bytes) failed: Resource >>>>>> temporarily unavailable (-11) >>>>>> Jun 12 12:47:38 [8062] SRV-5-1 cib: warning: do_local_notify: >>>>>> Sync reply to crm_mon failed: No message of desired type >>>>>> >>>>>> >>>>>> Restarting the pacemaker and cman service of 1 node didn't solve it. >>>>>> >>>>>> >>>>>> What is causing this problem and how can I resolve it ? >>>>>> >>>>>> >>>>> Almost certainly you're hitting IPC limits associated with large clusters. >>>>> >>>>> You should be able to tune: >>>>> >>>>> # PCMK_ipc_buffer=20480 >>>>> >>>>> In /etc/sysconfig/pacemaker and then restart the cluster. >>>>> >>>>> Note also: >>>>> >>>>> # For non-systemd based systems, prefix 'export' to each enabled line >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Pacemaker mailing list: >>>>> >>>>> Pacemaker@oss.clusterlabs.org >>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>>> >>>>> >>>>> >>>>> Project Home: >>>>> >>>>> http://www.clusterlabs.org >>>>> >>>>> >>>>> Getting started: >>>>> >>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>> >>>>> >>>>> Bugs: >>>>> >>>>> http://bugs.clusterlabs.org >>>> _______________________________________________ >>>> Pacemaker mailing list: >>>> Pacemaker@oss.clusterlabs.org >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>>> >>>> >>>> Project Home: >>>> http://www.clusterlabs.org >>>> >>>> Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> >>>> Bugs: >>>> http://bugs.clusterlabs.org >>> >>> >>> _______________________________________________ >>> Pacemaker mailing list: >>> Pacemaker@oss.clusterlabs.org >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >>> >>> >>> Project Home: >>> http://www.clusterlabs.org >>> >>> Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> >>> Bugs: >>> http://bugs.clusterlabs.org >> >> >> >> _______________________________________________ >> Pacemaker mailing list: >> Pacemaker@oss.clusterlabs.org >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker >> >> >> Project Home: >> http://www.clusterlabs.org >> >> Getting started: >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> >> Bugs: >> http://bugs.clusterlabs.org > > _______________________________________________ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
