There are two primary bottlenecks for pluginsync:
* the md5 sum it needs to do on both ends to compare and see if a
resync is needed
* the transfer itself
If you run puppet a second time, with pluginsync enabled (but this
time nothing should be transferred) - how much time does it take
compared to the first run? I'm only curious as it might rule out
transfer versus the 'md5' operation. *shrug*. Alas though, I haven't
seen this specific problem myself - are you running the puppetmaster
using webrick or some sort of application server like 'passenger' or
'unicorn'? Its a long-shot, but webrick is single-threaded which has
been known to be a problem - alas it shouldn't be a problem during a
sequential pluginsync :-(.
On Mon, Jan 7, 2013 at 6:28 PM, Kirk Steffensen
<k...@steffensenfamily.com> wrote:
> Ken,
>
> Thanks. I agree with your gut feeling, but after running the first round of
> tests you suggested, I don't think that's it. Bandwidth is good and there
> are no netstat errors. (Test results at the end of this email.)
>
> Actually, I just realized that the Puppet client on the Ubuntu box is
> running Puppet 2.7.11, which does not have pluginsync set to true by default
> (The CentOS boxes have Puppet 3.0.2, which has pluginsync set to true by
> default.) I watched the Ubuntu box start up, and it didn't give me a "Info:
> Retrieving plugin" notice like the CentOS box does. When I changed
> pluginsync to true on the Ubuntu box, it also took many minutes to sync the
> stdlib module. (I didn't time it during this run, but it was about 10
> minutes. I'd be happy to actually time it, if it would help troubleshoot.)
>
> While I didn't time the entire stdlib sync on the Ubuntu box, I did time
> several of the individual files. It's taking about 10 seconds per file to
> copy it over to the client from the Puppet Master. Since there are 90 files
> in stdlib, it is taking many minutes to sync the plugin, even though there
> are only 852KB of total data. Each file is resulting in a Puppet notice
> like the following output:
>
> notice:
> /File[/var/lib/puppet/lib/puppet/parser/functions/str2bool.rb]/ensure:
> defined content as '{md5}ab045013031d01a0e9335af92580dde6'
>
> For my short-term needs, my solution is to change pluginsync to false on the
> CentOS boxes, since I'm only using the stdlib module to get the anchor
> definition, and that doesn't need to be on the client side. But this will
> not be a long-term solution, as I will probably end up developing modules
> that need pluginsync to be true.
>
> Is anyone else who is using stdlib with pluginsync = true seeing this same
> 13-14 minute sync time during the first puppet run on a machine?
>
> Here are the results of the tests:
>
> iperf on CentOS: 298 Mbits/sec
> iperf on Ubuntu: 2.55 Gbits/sec
>
> $ netstat -in [on CentOS - no errors]
> Kernel Interface table
> Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR
> Flg
> eth0 1500 0 4813366 0 0 0 3076279 0 0
> 0 BMRU
> lo 16436 0 1140661 0 0 0 1140661 0 0
> 0 LRU
> vboxnet0 1500 0 0 0 0 0 866 0 0
> 0 BMRU
>
> The VBoxGuestAdditions are added in the basebox generation (using veewee,
> another great tool!) with the following in postinstall.sh. This is
> resulting in the most current (and more importantly, the matching version to
> the host):
>
> ***********
> # Install vagrant keys
> mkdir /home/vagrant/.ssh
> chmod 700 /home/vagrant/.ssh
> cd /home/vagrant/.ssh
> wget --no-check-certificate
> 'https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub' -O
> authorized_keys
> chown -R vagrant /home/vagrant/.ssh
>
> # Install the virtualbox guest additions
> VBOX_VERSION=$(cat /home/vagrant/.vbox_version)
> cd /tmp
> wget
> http://download.virtualbox.org/virtualbox/$VBOX_VERSION/VBoxGuestAdditions_$VBOX_VERSION.iso
> mount -o loop VBoxGuestAdditions_$VBOX_VERSION.iso /mnt
> sh /mnt/VBoxLinuxAdditions.run
> umount /mnt
>
> rm VBoxGuestAdditions_$VBOX_VERSION.iso
> ***********
>
> Thanks,
> Kirk
>
>
> On Monday, January 7, 2013 12:20:34 PM UTC-5, Ken Barber wrote:
>>
>> My immediate gut feeling on this would be network not Puppet being the
>> issue actually, especially if another client is having success at
>> doing the sync.
>>
>> Its a virt so it could be hypervisor drivers or some other issue, its
>> an old version of the kernel as well - its more likely to happen -
>> although I haven't seen it myself lately :-). Run something like
>> 'iperf' on both ends to confirm your getting the right kind of network
>> performance, compare that performance to your Ubuntu target ... and
>> check out netstat -in for errors etc. etc.
>>
>> Check to make sure the right VirtualBox hypervisor drivers were
>> installed when building your Centos vagrant box btw
>> (VBoxGuestAdditions) ... I'm not sure of its origin but this can be an
>> important step for network performance.
>>
>>
>> On Mon, Jan 7, 2013 at 4:50 PM, Kirk Steffensen
>> <ki...@steffensenfamily.com> wrote:
>> > Hi,
>> >
>> > I have a fresh CentOS 5.8 Vagrant VM that I'm using to emulate a
>> > customer's
>> > server. During the first Puppet run, it takes 13 minutes and 48 seconds
>> > to
>> > sync the Puppet Labs stdlib module. On a similar Ubuntu 12.04.1 Vagrant
>> > VM,
>> > Puppet starts up, and almost instantly goes from plugin sync to facter
>> > load.
>> >
>> > Is this load time for stdlib on a RHEL variant normal? It's only 580K
>> > of
>> > data to transfer. It seems like it should be almost instantaneous, even
>> > on
>> > an older kernel like CentOS 5.8.
>> >
>> > I've posted the relevant part of the Puppet run's output to this gist:
>> > https://gist.github.com/4475110
>> >
>> > If anyone has any insight or any troubleshooting steps to try, I'd
>> > appreciate it.
>> >
>> > Thanks,
>> > Kirk
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Puppet Users" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/puppet-users/-/jH71g1du7icJ.
>> > To post to this group, send email to puppet...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > puppet-users...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/gdGOknbXSW0J.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
