Hi,
I must admit I'd really like to like puppet, so I'm a bit agitated by
this issue, and won't rest till I know the cause :) .
Sorry for the information overload, hope you don't mind.
First of all, I've followed Martin's suggestion and simplified my
config.
This is my configuration in it's entirety:
nodes.pp:
node 'puptest.domain.com' {
include onlyone
}
onlyone.pp:
class onlyone {
$pack_list = ["large-package"]
package { $pack_list:
ensure => installed
}
$username = "someuser"
user { "$username":
comment => "Some user",
home => "/home/$username",
managehome => "true",
shell => '/bin/bash'
}
file { "/data/installed/by/large-package":
owner => "$username",
group => "$username",
recurse => "true",
require => [ User["$username"], Package["large-package"] ],
ensure => "directory",
checksum => undef
}
}
No effect though, the issue still occurs after all files have been
chown'd.
I've also tried attaching gdb to the puppetd process and breaking
during high CPU usage.
If I do a backtrace then, most of the backtrace contains ?? -> can't
derive.
If I only leave the methods gdb can determine, this is the stacktrace:
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
ruby_xmalloc () from /usr/lib/libruby1.8.so.1.8
rb_yield_values () from /usr/lib/libruby1.8.so.1.8
rb_ary_each () from /usr/lib/libruby1.8.so.1.8
rb_funcall2 () from /usr/lib/libruby1.8.so.1.8
rb_iterate () from /usr/lib/libruby1.8.so.1.8
rb_block_call () from /usr/lib/libruby1.8.so.1.8
No idea if this is of any use ...
I've also tried using --debug, but no debug statements are printed
either during the entire time puppetd is hogging the CPU.
Will have a second look though, maybe something interesting is printed
earlier...
On Mar 12, 1:58 pm, Martin Wheldon <mwhel...@googlemail.com> wrote:
> Hi,
>
> Is this directory the only resource you are managing or are there others.
> If not then try disabling all other resources and see if you still get
> the same issue?
>
> Best Regards
>
> Martin Wheldon
>
> On Fri, Mar 12, 2010 at 12:55 PM, DieterVDW <dieter...@gmail.com> wrote:
> > Also, whatever Puppetd is doing then, it seems to be unnecessary.
>
> > If I execute puppetd -tv and wait until all files have been chown'd,
> > and then hit CTRL-C and run puppetd again,
> > the second time, puppetd finishes in about a minute.
> > The total process of running puppetd twice takes about 4 minutes .
>
> > So by manipulating the execution of puppetd using a CTRL-C keystroke I
> > can reach the intended end state in 4 minutes.
> > If I leave puppetd to do it by itself, it takes 30+ minutes of 100%
> > CPU usage...
>
> > On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote:
> >> Aren't the time values in seconds?
> >> Because I can't match the "Total: 54.78" line to the "notice: Finished
> >> catalog run in 1877.06 seconds" .
>
> >> Also, the strange thing is that the actual changing of owner and group
> >> happens pretty fast.
> >> I see these messages passing during +- 20s for all files:
> >> notice: /File[/some/file]/owner: owner changed 'wronguser' to
> >> 'rightuser'
> >> notice: /File[/some/file]/group: group changed 'wronggroup' to
> >> 'rightgroup'
>
> >> It's AFTER this has happened that puppetd stalls for 30+ minutes.
> >> So it must be some kind of wrapup action... ?
>
> >> D
>
> >> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
>
> >> > So, it looks like 30 minutes is being taken up in the Package type and
> >> > 19 in the File type.
>
> >> > The Package type probably won't eat up much CPU at all as it's just
> >> > fetching and installing packages (unless they are some huge packages).
>
> >> > You seem to be trying to recursively manage *something* using the File
> >> > type.
>
> >> > What happens is that Puppet creates an actual file object for each
> >> > file and directory in the recursed path. Then, it takes the graph and
> >> > applies the changes to each file that has been seen. I'm not certain,
> >> > but I think that it builds this tree each time it runs to ensure that
> >> > nothing has changed under the path.
>
> >> > This processing has to happen on the client since the server has no
> >> > previous knowledge of the client's file structure.
>
> >> > That said, for deep directory/many file file statements, just spawn an
> >> > exec until the internal Puppet logic can be optimized.
>
> >> > Trevor
>
> >> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote:
> >> > > Some more information from the (undocumented) --summarize option I
> >> > > just discovered:
>
> >> > > Changes:
> >> > > Total: 4271
> >> > > Resources:
> >> > > Applied: 4271
> >> > > Out of sync: 2138
> >> > > Scheduled: 4435
> >> > > Total: 115
> >> > > Time:
> >> > > Config retrieval: 1.36
> >> > > Exec: 0.77
> >> > > File: 19.23
> >> > > Filebucket: 0.00
> >> > > Host: 0.00
> >> > > Package: 31.99
> >> > > Schedule: 0.00
> >> > > Service: 1.42
> >> > > User: 0.01
> >> > > Total: 54.78
> >> > > warning: Value of 'preferred_serialization_format' (pson) is invalid
> >> > > for report, using default (marshal)
> >> > > notice: Finished catalog run in 1877.06 seconds
>
> >> > > It seems to me the cause of the delays is not recorded in the time
> >> > > overview?
>
> >> > > D
>
> >> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote:
> >> > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
>
> >> > >> > Puppet doesn't handle a folder with lots of files well. It handles
> >> > >> > large files even worse. The standard advice is "Try putting the
> >> > >> > files in a package and distributing them using apt." Another
> >> > >> > common answer is to try combining exec and rsync. I ended up using
> >> > >> > apt. Here are the tutorials I used:
>
> >> > >> The problem is, I -am- using apt!
> >> > >> Those files are downloaded and installed using apt, I just want puppet
> >> > >> to make sure they are owned by a certain user and group.
> >> > >> That's the only thing puppet needs to do.
>
> >> > >> If I do the same in bash:
> >> > >> find directory/ | while read file ; do chown user:group "$file" ; done
>
> >> > >> real 0m28.119s
> >> > >> user 0m4.064s
> >> > >> sys 0m12.725s
>
> >> > >> I can live with overhead from slow ruby, etc etc ...
> >> > >> But apparently Puppet is 60x slower than bash for doing the same
> >> > >> action!
>
> >> > >> During my investigation for this problem I've seen a lot of people
> >> > >> saying things like:
> >> > >> "Puppet doesn't handle a folder with lots of files well"
> >> > >> 60x times slower isn't really "not handling well". It's crap.
> >> > >> I'm a bit baffled by the defeatism I see in the Puppet community
> >> > >> concerning Puppet CPU usage.
>
> >> > >> I can't really believe people would call what I am experiencing "not
> >> > >> handling well", so I suppose I'm having another, worse, issue
> >> > >> concerning my setup?
>
> >> > >> Also the usual suspect for puppet not handling large file collections
> >> > >> well seems to be the checksumming.
> >> > >> But with that turned of, what's keeping puppet busy?
>
> >> > >> Puppet eats up 30 minutes of CPU time, I see two options:
> >> > >> - Puppet is doing something necessary for it's functioning during that
> >> > >> time, in which case somebody should know what it is doing. Anybody?
> >> > >> - Or the code is gravely flawed and needs to be fixed.
>
> >> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't
> >> > >> production ready?
> >> > >> I really can't have this on production machines...
>
> >> > > --
> >> > > You received this message because you are subscribed to the Google
> >> > > Groups "Puppet Users" group.
> >> > > To post to this group, send email to puppet-us...@googlegroups.com.
> >> > > To unsubscribe from this group, send email to
> >> > > puppet-users+unsubscr...@googlegroups.com.
> >> > > For more options, visit this group
> >> > > athttp://groups.google.com/group/puppet-users?hl=en.
>
> >> > --
> >> > Trevor Vaughan
> >> > Vice President, Onyx Point, Inc
> >> > (410) 541-6699
> >> > tvaug...@onyxpoint.com
>
> >> > -- This account not approved for unencrypted proprietary information --
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-us...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
