Aren't the time values in seconds? Because I can't match the "Total: 54.78" line to the "notice: Finished catalog run in 1877.06 seconds" .
Also, the strange thing is that the actual changing of owner and group happens pretty fast. I see these messages passing during +- 20s for all files: notice: /File[/some/file]/owner: owner changed 'wronguser' to 'rightuser' notice: /File[/some/file]/group: group changed 'wronggroup' to 'rightgroup' It's AFTER this has happened that puppetd stalls for 30+ minutes. So it must be some kind of wrapup action... ? D On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote: > So, it looks like 30 minutes is being taken up in the Package type and > 19 in the File type. > > The Package type probably won't eat up much CPU at all as it's just > fetching and installing packages (unless they are some huge packages). > > You seem to be trying to recursively manage *something* using the File type. > > What happens is that Puppet creates an actual file object for each > file and directory in the recursed path. Then, it takes the graph and > applies the changes to each file that has been seen. I'm not certain, > but I think that it builds this tree each time it runs to ensure that > nothing has changed under the path. > > This processing has to happen on the client since the server has no > previous knowledge of the client's file structure. > > That said, for deep directory/many file file statements, just spawn an > exec until the internal Puppet logic can be optimized. > > Trevor > > > > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > > Some more information from the (undocumented) --summarize option I > > just discovered: > > > Changes: > > Total: 4271 > > Resources: > > Applied: 4271 > > Out of sync: 2138 > > Scheduled: 4435 > > Total: 115 > > Time: > > Config retrieval: 1.36 > > Exec: 0.77 > > File: 19.23 > > Filebucket: 0.00 > > Host: 0.00 > > Package: 31.99 > > Schedule: 0.00 > > Service: 1.42 > > User: 0.01 > > Total: 54.78 > > warning: Value of 'preferred_serialization_format' (pson) is invalid > > for report, using default (marshal) > > notice: Finished catalog run in 1877.06 seconds > > > It seems to me the cause of the delays is not recorded in the time > > overview? > > > D > > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: > > >> > Puppet doesn't handle a folder with lots of files well. It handles > >> > large files even worse. The standard advice is "Try putting the files > >> > in a package and distributing them using apt." Another common answer is > >> > to try combining exec and rsync. I ended up using apt. Here are the > >> > tutorials I used: > > >> The problem is, I -am- using apt! > >> Those files are downloaded and installed using apt, I just want puppet > >> to make sure they are owned by a certain user and group. > >> That's the only thing puppet needs to do. > > >> If I do the same in bash: > >> find directory/ | while read file ; do chown user:group "$file" ; done > > >> real 0m28.119s > >> user 0m4.064s > >> sys 0m12.725s > > >> I can live with overhead from slow ruby, etc etc ... > >> But apparently Puppet is 60x slower than bash for doing the same > >> action! > > >> During my investigation for this problem I've seen a lot of people > >> saying things like: > >> "Puppet doesn't handle a folder with lots of files well" > >> 60x times slower isn't really "not handling well". It's crap. > >> I'm a bit baffled by the defeatism I see in the Puppet community > >> concerning Puppet CPU usage. > > >> I can't really believe people would call what I am experiencing "not > >> handling well", so I suppose I'm having another, worse, issue > >> concerning my setup? > > >> Also the usual suspect for puppet not handling large file collections > >> well seems to be the checksumming. > >> But with that turned of, what's keeping puppet busy? > > >> Puppet eats up 30 minutes of CPU time, I see two options: > >> - Puppet is doing something necessary for it's functioning during that > >> time, in which case somebody should know what it is doing. Anybody? > >> - Or the code is gravely flawed and needs to be fixed. > > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't > >> production ready? > >> I really can't have this on production machines... > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To post to this group, send email to puppet-us...@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/puppet-users?hl=en. > > -- > Trevor Vaughan > Vice President, Onyx Point, Inc > (410) 541-6699 > tvaug...@onyxpoint.com > > -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
