Also, whatever Puppetd is doing then, it seems to be unnecessary. If I execute puppetd -tv and wait until all files have been chown'd, and then hit CTRL-C and run puppetd again, the second time, puppetd finishes in about a minute. The total process of running puppetd twice takes about 4 minutes .
So by manipulating the execution of puppetd using a CTRL-C keystroke I can reach the intended end state in 4 minutes. If I leave puppetd to do it by itself, it takes 30+ minutes of 100% CPU usage... On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote: > Aren't the time values in seconds? > Because I can't match the "Total: 54.78" line to the "notice: Finished > catalog run in 1877.06 seconds" . > > Also, the strange thing is that the actual changing of owner and group > happens pretty fast. > I see these messages passing during +- 20s for all files: > notice: /File[/some/file]/owner: owner changed 'wronguser' to > 'rightuser' > notice: /File[/some/file]/group: group changed 'wronggroup' to > 'rightgroup' > > It's AFTER this has happened that puppetd stalls for 30+ minutes. > So it must be some kind of wrapup action... ? > > D > > On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote: > > > So, it looks like 30 minutes is being taken up in the Package type and > > 19 in the File type. > > > The Package type probably won't eat up much CPU at all as it's just > > fetching and installing packages (unless they are some huge packages). > > > You seem to be trying to recursively manage *something* using the File type. > > > What happens is that Puppet creates an actual file object for each > > file and directory in the recursed path. Then, it takes the graph and > > applies the changes to each file that has been seen. I'm not certain, > > but I think that it builds this tree each time it runs to ensure that > > nothing has changed under the path. > > > This processing has to happen on the client since the server has no > > previous knowledge of the client's file structure. > > > That said, for deep directory/many file file statements, just spawn an > > exec until the internal Puppet logic can be optimized. > > > Trevor > > > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > > > Some more information from the (undocumented) --summarize option I > > > just discovered: > > > > Changes: > > > Total: 4271 > > > Resources: > > > Applied: 4271 > > > Out of sync: 2138 > > > Scheduled: 4435 > > > Total: 115 > > > Time: > > > Config retrieval: 1.36 > > > Exec: 0.77 > > > File: 19.23 > > > Filebucket: 0.00 > > > Host: 0.00 > > > Package: 31.99 > > > Schedule: 0.00 > > > Service: 1.42 > > > User: 0.01 > > > Total: 54.78 > > > warning: Value of 'preferred_serialization_format' (pson) is invalid > > > for report, using default (marshal) > > > notice: Finished catalog run in 1877.06 seconds > > > > It seems to me the cause of the delays is not recorded in the time > > > overview? > > > > D > > > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: > > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: > > > >> > Puppet doesn't handle a folder with lots of files well. It handles > > >> > large files even worse. The standard advice is "Try putting the files > > >> > in a package and distributing them using apt." Another common answer > > >> > is to try combining exec and rsync. I ended up using apt. Here are > > >> > the tutorials I used: > > > >> The problem is, I -am- using apt! > > >> Those files are downloaded and installed using apt, I just want puppet > > >> to make sure they are owned by a certain user and group. > > >> That's the only thing puppet needs to do. > > > >> If I do the same in bash: > > >> find directory/ | while read file ; do chown user:group "$file" ; done > > > >> real 0m28.119s > > >> user 0m4.064s > > >> sys 0m12.725s > > > >> I can live with overhead from slow ruby, etc etc ... > > >> But apparently Puppet is 60x slower than bash for doing the same > > >> action! > > > >> During my investigation for this problem I've seen a lot of people > > >> saying things like: > > >> "Puppet doesn't handle a folder with lots of files well" > > >> 60x times slower isn't really "not handling well". It's crap. > > >> I'm a bit baffled by the defeatism I see in the Puppet community > > >> concerning Puppet CPU usage. > > > >> I can't really believe people would call what I am experiencing "not > > >> handling well", so I suppose I'm having another, worse, issue > > >> concerning my setup? > > > >> Also the usual suspect for puppet not handling large file collections > > >> well seems to be the checksumming. > > >> But with that turned of, what's keeping puppet busy? > > > >> Puppet eats up 30 minutes of CPU time, I see two options: > > >> - Puppet is doing something necessary for it's functioning during that > > >> time, in which case somebody should know what it is doing. Anybody? > > >> - Or the code is gravely flawed and needs to be fixed. > > > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't > > >> production ready? > > >> I really can't have this on production machines... > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Puppet Users" group. > > > To post to this group, send email to puppet-us...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > puppet-users+unsubscr...@googlegroups.com. > > > For more options, visit this group > > > athttp://groups.google.com/group/puppet-users?hl=en. > > > -- > > Trevor Vaughan > > Vice President, Onyx Point, Inc > > (410) 541-6699 > > tvaug...@onyxpoint.com > > > -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
