On Tuesday, April 29, 2014 2:53:18 PM UTC-5, Ádám Sándor wrote:
>
> I managed to solve the problem :) First we added more memory to the
> machine. With 1GB of memory the script ran further but still failed. In
> this case it became apparent that it's doing something with file
> permissions. It was setting group 70 (don't even know what kind of group
> that is) to some scripts it was supposed to copy for execution. It was
> doing this very slowly - tens of seconds for each file - and eating more
> and more memory in the process, until it reached 1GB and the vserver killed
> it.
>
>
When you say you fixed the problem, do you mean that the new failure occurs
while applying a different resource?
> The fix was very simple, but I'm still intrigued as to what went wrong,
> especially since it was such a hard-to-debug issue. I changed this resource:
> file { 'migration-scripts':
> path => $scriptPath,
> ensure => directory,
> source => 'puppet:///modules/ks/migration',
> recurse => true,
> owner => 'root'
> }
>
> to this:
> file { 'migration-scripts':
> path => $scriptPath,
> ensure => directory,
> source => 'puppet:///modules/ks/migration',
> recurse => true
> }
>
> If someone can give an explanation I would be very happy to read it. Also
> because I only got 1GB of memory on the server temporarily and I'm
> suspecting another similar problem in my manifest since with 512GB memory
> the execution didn't get to the part where I now spotted the group
> permission setting.
>
>
As Felix said, this is not generally expected or documented behavior. The
best we can do is guess, and we're working with incomplete information when
it comes to that. I observe, however, that recursive File resources can be
a source of trouble. They do work, but they require memory on the agent
proportional to the total number of files, plus a bit. Also, if your
recursive File is following links ('links' parameter) then it is possible
to give it a logically infinite directory tree to attempt to sync. You
don't specify 'links' in your example, and I don't recall offhand what the
default behavior is in that regard, but it's something to watch out for.
In addition, File resources in general are expensive when the managed
content is large and you use the default checksum algorithm. The choice of
algorithm is a trade-off between expense of the test and likelihood of a
wrong result, and Puppet defaults to the least likelihood of a wrong result.
If you are managing large collections of files or file collections having
large aggregate size, then you would be better off avoiding recursive File
resources. The usual recommendation is to bundle up the files into a
package (e.g. an RPM or DEB), put that in a local repository, and manage it
via a Package resource.
As for the group assigned to newly-created files, it is probably the
(numeric) group the source files have on the master. If you don't specify
a group for a source'd file, and that file does not initially exist on the
target node, then the group assigned to it at creation time is the one the
source file has. Similarly for the file owner. If that's indeed what you
are seeing then that aspect of the observed behavior is as expected. (In
contrast, if you don't specify the group (or owner), and the file does
already exist, then Puppet does not modify its group (or owner).)
I cannot explain why declining to manage the file owner makes a significant
difference in memory consumption, but it's reasonable to suppose that the
large memory consumption is a factor in the poor performance. The
'physical' memory assigned to your virtual machine is not necessarily
matched with real physical memory on the underlying vserver host.
Therefore, processes in the VM that consume a lot of memory may cause a lot
of page faults on the host, with the attending performance impact. The
negative performance effects of such page faulting might even be magnified
in the VM.
John
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/569c0062-4372-4428-a865-51e0d587b127%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
