In case anyone is interested, we figured out what was causing this issue. Being that we run a Ruby on Rails application and naturally Puppet runs on Ruby as well, we actually load TC Malloc memory allocator for our application. By running one of the "applying configuration" processes in gstack, we were able to discover that it was getting stuck on libtcmalloc.so.
The version of TC Malloc we were running that caused this issue was 4.2.0 but since upgrading to latest 4.2.6 we have not experienced another occurrence of this issue. On Monday, June 15, 2015 at 10:12:03 PM UTC-4, Franck wrote: > > We've been experiencing a lot of "Command exceeded timeouts" on basic > shell commands using the "exec" type for tasks that should execute fairly > fast: > > Jun 15 15:45:44 host1 puppet-agent[57648]: > (/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime && /bin/ln -s > /usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout > Jun 10 21:15:24 host1 puppet-agent[57081]: > (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/ > vmware-uninstall-tools.pl]/onlyif) Check "/usr/bin/test -f /usr/bin/ > vmware-uninstall-tools.pl" exceeded timeout > Jun 10 23:56:02 host1 puppet-agent[40286]: > (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y > open-vm-tools.x86_64]/unless) Check "/bin/rpm -q open-vm-tools" exceeded > timeout > > All these commands can be run locally to the host and return fairly > quickly, but when puppet executes them they time out. Extending the timeout > is an option but ridiculous since default is 300 seconds and none of these > commands should take 5 minutes or more to return. > > Some of the things observed is that this only affects CentOS 6.x hosts as > we also have Ubuntu 14.x hosts and they do not experience these problems. > Also, we've played around with different versions of the puppet agent > along with different versions of Ruby and none of them had any effect as > this condition persists regardless. Also, this does not seem to affect all > of our CentOS 6.x hosts but only certain ones -- randomly. Running puppet > agent in debug mode does not seem to uncover what's going on as it just > hangs when it gets to the "exec". > > It's very annoying and actually dangerous in some cases as the puppet > agent will continue spawning multiple "applying configuration" processes > which will cause hosts to swap memory as each takes up more and more memory > and in some instances will hose them entirely. > We've had to remove these manifests that cause these conditions in the > interim but right now we have a lot of hosts we need to manage with puppet > so we need to be able to use this. > > Basic info on the hosts in question: > > - Puppet: 3.7.5 > - Ruby: 2.1.2 > - CentOS 6.6 > > Anyone have any ideas as to what could be causing this? > > Thanks. > Franck > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/64526946-314f-46c2-8ba5-d7fbf1ee09e8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
