Just noticed something else interesting. When running puppet apply or
puppet agent and it doesn't hang, puppet only spawns 1 process and there is
no puppet process left running afterwards, it seems to clean itself up and
die off. When puppet runs and hangs, it spawns 2 processes each looking
identical with the same name and even though the output from puppet apply
or agent says it finished running (after letting it timeout on the hang) it
leaves 1 process behind still running. I attached strace to this running
process and the output looks like so:
pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 263395792}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 248281920}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 170310944}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 199996576}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 244956432}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 141958480}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 22737840}) =
-1 ETIMEDOUT (Connection timed out)
[pid 61271] futex(0x7f5e2454b6a0, FUTEX_WAIT_PRIVATE, 2, {0, 78315952}) =
-1 ETIMEDOUT (Connection timed out)
and as best as I can tell it would just keep repeating that indefinitely.
This seems quite odd to me and has to be related in some way but I'm not
sure how.
On Tuesday, June 23, 2015 at 3:45:20 PM UTC-6, Dan Finn wrote:
>
> Hello,
>>
>
> I am Franck's coworker. I spent some time looking into this today and
> have some more info but unfortunately am still really lost as to what is
> going on or what is causing this.
>
> Here is the output captured from a puppet agent -t --debug run where a
> node hangs:
>
>
> https://github.com/danfinn/errors/blob/master/puppet_debug_output_during_hang.txt
>
> And here is the output from strace captured during the same host hanging
> at the same point in the manifest (hangs at line 3441):
>
>
> https://github.com/danfinn/errors/blob/master/puppet_hang_strace_no_follow_forks.txt
>
>
> I understand that this strace would be more helpful with -ff enabled but
> unfortunately I couldn't get puppet to run like that, I'm not sure why but
> it generated a bunch of errors and never ran the manifest.
>
> One thing I noticed is that while puppet is hanging running /usr/bin/test,
> or at least it is telling me that it's running that and waiting, that
> process does not show up in the command list. Also, running that identical
> /usr/bin/test command at the same time that puppet is hanging on it works
> fine, it returns very quickly.
>
> I was able to get this test system to hang both via puppet agent and
> running the manifest locally with puppet apply.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/5949b475-91fb-463a-9ed8-4e4e672aaee6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
