So, it looks like 30 minutes is being taken up in the Package type and 19 in the File type.
The Package type probably won't eat up much CPU at all as it's just fetching and installing packages (unless they are some huge packages). You seem to be trying to recursively manage *something* using the File type. What happens is that Puppet creates an actual file object for each file and directory in the recursed path. Then, it takes the graph and applies the changes to each file that has been seen. I'm not certain, but I think that it builds this tree each time it runs to ensure that nothing has changed under the path. This processing has to happen on the client since the server has no previous knowledge of the client's file structure. That said, for deep directory/many file file statements, just spawn an exec until the internal Puppet logic can be optimized. Trevor On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > Some more information from the (undocumented) --summarize option I > just discovered: > > Changes: > Total: 4271 > Resources: > Applied: 4271 > Out of sync: 2138 > Scheduled: 4435 > Total: 115 > Time: > Config retrieval: 1.36 > Exec: 0.77 > File: 19.23 > Filebucket: 0.00 > Host: 0.00 > Package: 31.99 > Schedule: 0.00 > Service: 1.42 > User: 0.01 > Total: 54.78 > warning: Value of 'preferred_serialization_format' (pson) is invalid > for report, using default (marshal) > notice: Finished catalog run in 1877.06 seconds > > It seems to me the cause of the delays is not recorded in the time > overview? > > D > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: >> >> > Puppet doesn't handle a folder with lots of files well. It handles large >> > files even worse. The standard advice is "Try putting the files in a >> > package and distributing them using apt." Another common answer is to try >> > combining exec and rsync. I ended up using apt. Here are the tutorials I >> > used: >> >> The problem is, I -am- using apt! >> Those files are downloaded and installed using apt, I just want puppet >> to make sure they are owned by a certain user and group. >> That's the only thing puppet needs to do. >> >> If I do the same in bash: >> find directory/ | while read file ; do chown user:group "$file" ; done >> >> real 0m28.119s >> user 0m4.064s >> sys 0m12.725s >> >> I can live with overhead from slow ruby, etc etc ... >> But apparently Puppet is 60x slower than bash for doing the same >> action! >> >> During my investigation for this problem I've seen a lot of people >> saying things like: >> "Puppet doesn't handle a folder with lots of files well" >> 60x times slower isn't really "not handling well". It's crap. >> I'm a bit baffled by the defeatism I see in the Puppet community >> concerning Puppet CPU usage. >> >> I can't really believe people would call what I am experiencing "not >> handling well", so I suppose I'm having another, worse, issue >> concerning my setup? >> >> Also the usual suspect for puppet not handling large file collections >> well seems to be the checksumming. >> But with that turned of, what's keeping puppet busy? >> >> Puppet eats up 30 minutes of CPU time, I see two options: >> - Puppet is doing something necessary for it's functioning during that >> time, in which case somebody should know what it is doing. Anybody? >> - Or the code is gravely flawed and needs to be fixed. >> >> Is this a stroke of bad luck, or should I conclude that Puppet isn't >> production ready? >> I really can't have this on production machines... > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-us...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaug...@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
