On Mon, Oct 22, 2012 at 11:17:52PM +0200, Brice Figureau wrote: > On 23/10/12 01:39, Nikola Petrov wrote: > > On Mon, Oct 22, 2012 at 12:09:45PM -0700, Bostjan Skufca wrote: > >> Hi there, > >> > >> I'm running into slow catalog runs because of many files that are managed. > >> I was thinking about some optimizations of this functionality. > >> > >> 1: On puppetmaster: > >> For files with "source => 'puppet:///modules...' puppetmaster should > >> already calculate md5 and send it with the catalog. > >> > >> 2: On managed node: > >> As md5s for files are already there once catalog is received, there is no > >> need for x https calls (x is the number of files managed with source=> > >> parameter) > >> > >> 3. Puppetmaster md5 cache > >> This would of course put some strain on puppetmaster, which would then > >> benefit from some sort of file md5 cache: > >> - when md5 is calculated, put in into cache, key is filename. Also add > >> file > >> mtime and time of cache insert. > >> - on each catalog request, for each file in the catalog check if mtime has > >> changed, and if so, recalculate md5 hash, else just retrieve md5 hash from > >> cache > >> - some sort of stale cache entries removal, based on cache insert time, > >> maybe at the end of each puppet catalog compilation, maybe controlled with > >> probability 1:100 or something > >> > >> Do you have any comments about these optimizations? They will be greatly > >> appreciated... really :) > >> > >> b. > >> > > Hi, > > > > When using puppet I found that it is a far better idea to serf files > > with something else. You will be far better with something else for this > > job like sftp or ssh. My conclusions just came from the fact that we > > were trying to import a big dump(2GB seems ok to me but who knows) and > > puppet just died because they are not streaming the file but it is > > *fully* loaded into memory. > > This assertion is not true anymore since at least 2.6.0. > Also, for big files you can activate http compression on the client, > this might help (or not, YMMV).
Nice to hear that. It is true that the version with which I tested this was somewhat old. Best, Nikola -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
