Yep, you're right, seconds not minutes. That's what I get for answering e-mails pre-caffeine.
Did you try spitting out a --debug to a file and then post-analyzing that? Trevor On Fri, Mar 12, 2010 at 7:55 AM, DieterVDW <dieter...@gmail.com> wrote: > Also, whatever Puppetd is doing then, it seems to be unnecessary. > > If I execute puppetd -tv and wait until all files have been chown'd, > and then hit CTRL-C and run puppetd again, > the second time, puppetd finishes in about a minute. > The total process of running puppetd twice takes about 4 minutes . > > So by manipulating the execution of puppetd using a CTRL-C keystroke I > can reach the intended end state in 4 minutes. > If I leave puppetd to do it by itself, it takes 30+ minutes of 100% > CPU usage... > > On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote: >> Aren't the time values in seconds? >> Because I can't match the "Total: 54.78" line to the "notice: Finished >> catalog run in 1877.06 seconds" . >> >> Also, the strange thing is that the actual changing of owner and group >> happens pretty fast. >> I see these messages passing during +- 20s for all files: >> notice: /File[/some/file]/owner: owner changed 'wronguser' to >> 'rightuser' >> notice: /File[/some/file]/group: group changed 'wronggroup' to >> 'rightgroup' >> >> It's AFTER this has happened that puppetd stalls for 30+ minutes. >> So it must be some kind of wrapup action... ? >> >> D >> >> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote: >> >> > So, it looks like 30 minutes is being taken up in the Package type and >> > 19 in the File type. >> >> > The Package type probably won't eat up much CPU at all as it's just >> > fetching and installing packages (unless they are some huge packages). >> >> > You seem to be trying to recursively manage *something* using the File >> > type. >> >> > What happens is that Puppet creates an actual file object for each >> > file and directory in the recursed path. Then, it takes the graph and >> > applies the changes to each file that has been seen. I'm not certain, >> > but I think that it builds this tree each time it runs to ensure that >> > nothing has changed under the path. >> >> > This processing has to happen on the client since the server has no >> > previous knowledge of the client's file structure. >> >> > That said, for deep directory/many file file statements, just spawn an >> > exec until the internal Puppet logic can be optimized. >> >> > Trevor >> >> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: >> > > Some more information from the (undocumented) --summarize option I >> > > just discovered: >> >> > > Changes: >> > > Total: 4271 >> > > Resources: >> > > Applied: 4271 >> > > Out of sync: 2138 >> > > Scheduled: 4435 >> > > Total: 115 >> > > Time: >> > > Config retrieval: 1.36 >> > > Exec: 0.77 >> > > File: 19.23 >> > > Filebucket: 0.00 >> > > Host: 0.00 >> > > Package: 31.99 >> > > Schedule: 0.00 >> > > Service: 1.42 >> > > User: 0.01 >> > > Total: 54.78 >> > > warning: Value of 'preferred_serialization_format' (pson) is invalid >> > > for report, using default (marshal) >> > > notice: Finished catalog run in 1877.06 seconds >> >> > > It seems to me the cause of the delays is not recorded in the time >> > > overview? >> >> > > D >> >> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: >> > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: >> >> > >> > Puppet doesn't handle a folder with lots of files well. It handles >> > >> > large files even worse. The standard advice is "Try putting the >> > >> > files in a package and distributing them using apt." Another common >> > >> > answer is to try combining exec and rsync. I ended up using apt. >> > >> > Here are the tutorials I used: >> >> > >> The problem is, I -am- using apt! >> > >> Those files are downloaded and installed using apt, I just want puppet >> > >> to make sure they are owned by a certain user and group. >> > >> That's the only thing puppet needs to do. >> >> > >> If I do the same in bash: >> > >> find directory/ | while read file ; do chown user:group "$file" ; done >> >> > >> real 0m28.119s >> > >> user 0m4.064s >> > >> sys 0m12.725s >> >> > >> I can live with overhead from slow ruby, etc etc ... >> > >> But apparently Puppet is 60x slower than bash for doing the same >> > >> action! >> >> > >> During my investigation for this problem I've seen a lot of people >> > >> saying things like: >> > >> "Puppet doesn't handle a folder with lots of files well" >> > >> 60x times slower isn't really "not handling well". It's crap. >> > >> I'm a bit baffled by the defeatism I see in the Puppet community >> > >> concerning Puppet CPU usage. >> >> > >> I can't really believe people would call what I am experiencing "not >> > >> handling well", so I suppose I'm having another, worse, issue >> > >> concerning my setup? >> >> > >> Also the usual suspect for puppet not handling large file collections >> > >> well seems to be the checksumming. >> > >> But with that turned of, what's keeping puppet busy? >> >> > >> Puppet eats up 30 minutes of CPU time, I see two options: >> > >> - Puppet is doing something necessary for it's functioning during that >> > >> time, in which case somebody should know what it is doing. Anybody? >> > >> - Or the code is gravely flawed and needs to be fixed. >> >> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't >> > >> production ready? >> > >> I really can't have this on production machines... >> >> > > -- >> > > You received this message because you are subscribed to the Google >> > > Groups "Puppet Users" group. >> > > To post to this group, send email to puppet-us...@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > puppet-users+unsubscr...@googlegroups.com. >> > > For more options, visit this group >> > > athttp://groups.google.com/group/puppet-users?hl=en. >> >> > -- >> > Trevor Vaughan >> > Vice President, Onyx Point, Inc >> > (410) 541-6699 >> > tvaug...@onyxpoint.com >> >> > -- This account not approved for unencrypted proprietary information -- > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-us...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaug...@onyxpoint.com -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
