On Thu, 2010-08-26 at 03:46 -0700, kit wrote:
> [snip]

> debug: catalog supports formats: b64_zlib_yaml dot marshal pson raw
> yaml; using pson
> info: Caching catalog for kitbackup002.mid.dmz

It just downloaded the catalog.

> debug: Puppet::Type::Package::ProviderUp2date: file /usr/sbin/up2date-
> nox does not exist
> debug: Puppet::Type::Package::ProviderYum: file yum does not exist
> debug: Puppet::Type::Package::ProviderAptrpm: file apt-get does not
> exist
> debug: Puppet::Type::Package::ProviderApt: file /usr/bin/apt-get does
> not exist
> debug: Puppet::Type::Package::ProviderFink: file /sw/bin/fink does not
> exist
> debug: Puppet::Type::Package::ProviderNim: file /usr/sbin/nimclient
> does not exist
> debug: Puppet::Type::Package::ProviderSun: file /usr/bin/pkginfo does
> not exist
> debug: Puppet::Type::Package::ProviderDpkg: file /usr/bin/dpkg does
> not exist
> debug: Puppet::Type::Package::ProviderUrpmi: file urpmi does not exist
> debug: Puppet::Type::Package::ProviderZypper: file /usr/bin/zypper
> does not exist
> debug: Puppet::Type::Package::ProviderAptitude: file /usr/bin/aptitude
> does not exist
> debug: Puppet::Type::Package::ProviderRpm: file rpm does not exist
> debug: Puppet::Type::Package::ProviderAix: file /usr/bin/lslpp does
> not exist
> debug: Puppet::Type::Package::ProviderPortage: file /usr/bin/eix does
> not exist
> debug: Puppet::Type::Package::ProviderPkg: file /usr/bin/pkg does not
> exist
> debug: Puppet::Type::Package::ProviderSunfreeware: file pkg-get does
> not exist
> debug: Puppet::Type::Package::ProviderHpux: file /usr/sbin/swlist does
> not exist
> debug: Puppet::Type::Package::ProviderRug: file /usr/bin/rug does not
> exist
> debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
> yaml; using pson

At this stage, it tries to fetch file metadata for a sourced file.

> From gdb:
> [Switching to Thread 0x28a79580 (LWP 100120)]
> 0x281b81a7 in __error () from /lib/libthr.so.3
> (gdb) set $ary = (int)backtrace(-1)
> (gdb) set $count = *($ary+8)
> (gdb) set $index = 0
> (gdb) while $index < $count
>  >x/1s *((int)rb_ary_entry($ary, $index)+12)
>  >set $index = $index + 1
>  >end
> 0x29f020c0:      "/usr/local/lib/ruby/1.8/timeout.rb:59:in `start'"
> 0x29f02100:      "/usr/local/lib/ruby/1.8/timeout.rb:58:in `timeout'"
> 0x29f02140:      "/usr/local/lib/ruby/1.8/timeout.rb:93:in `timeout'"
> 0x29f02180:      "/usr/local/lib/ruby/1.8/net/protocol.rb:134:in
> `rbuf_fill'"
> 0x29f021c0:      "/usr/local/lib/ruby/1.8/net/protocol.rb:116:in
> `readuntil'"
> 0x29f02200:      "/usr/local/lib/ruby/1.8/net/protocol.rb:126:in
> `readline'"
> 0x29f02240:      "/usr/local/lib/ruby/1.8/net/http.rb:2024:in
> `read_status_line'"
> 0x29f02280:      "/usr/local/lib/ruby/1.8/net/http.rb:2013:in
> `read_new'"
> 0x29f022c0:      "/usr/local/lib/ruby/1.8/net/http.rb:1050:in
> `request'"
> 0x29f02300:      "/usr/local/lib/ruby/1.8/net/http.rb:1037:in
> `request'"
> 0x29f02340:      "/usr/local/lib/ruby/1.8/net/http.rb:543:in `start'"
> 0x29f02380:      "/usr/local/lib/ruby/1.8/net/http.rb:1035:in
> `request'"
> 0x29f023c0:      "/usr/local/lib/ruby/1.8/net/http.rb:772:in `get'"
> 0x29f03060:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/indirector/
> rest.rb:71:in `find'"
> 0x29f04040:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/indirector/
> indirection.rb:193:in `find'"
> 0x29f030b0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> indirector.rb:50:in `find'"
> 0x29f040a0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/file/
> source.rb:141:in `init_metadata'"
> 0x29f03100:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/file/
> source.rb:139:in `each'"
> 0x29f04100:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/file/
> source.rb:139:in `init_metadata'"
> 0x29f03150:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/util/
> cacher.rb:106:in `send'"
> 0x29f031a0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/util/
> cacher.rb:106:in `cached_value'"
> 0x29f02080:      "/usr/local/lib/ruby/1.8/monitor.rb:242:in
> `synchronize'"
> 0x29f031f0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/util/
> cacher.rb:98:in `cached_value'"
> 0x29f03240:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/util/
> cacher.rb:48:in `metadata'"
> 0x29f04160:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/file/
> source.rb:99:in `copy_source_values'"
> 0x29f03290:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/
> file.rb:632:in `retrieve'"
> 0x29f032e0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type.rb:
> 695:in `retrieve_resource'"
> 0x29f03330:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type.rb:
> 1853:in `to_trans'"
> 0x29f03380:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type/
> file.rb:701:in `to_trans'"
> 0x29f033d0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type.rb:
> 1878:in `to_resource'"
> 0x29f03420:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/type.rb:
> 203:in `uniqueness_key'"
> 0x29f041c0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:83:in `add_resource'"
> 0x29f03470:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:72:in `each'"
> 0x29f04220:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:72:in `add_resource'"
> 0x29f04280:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:561:in `to_catalog'"
> 0x29f034c0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:531:in `each'"
> 0x29f042e0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:531:in `to_catalog'"
> 0x29f03510:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/
> catalog.rb:468:in `to_ral'"
> 0x29f03560:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> configurer.rb:117:in `convert_catalog'"
> 0x29f04340:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> configurer.rb:112:in `retrieve_catalog'"
> 0x29f035b0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> configurer.rb:143:in `run'"
> 0x29f02400:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 39"
> 0x29f03600:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent/
> locker.rb:21:in `lock'"
> 0x29f02440:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 39"
> 0x29f02480:      "/usr/local/lib/ruby/1.8/sync.rb:230:in
> `synchronize'"
> 0x29f024c0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 39"
> 0x29f03650:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 101:in `with_client'"
> 0x29f02500:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 37"
> 0x29f036a0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> application.rb:171:in `call'"
> 0x29f036f0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> application.rb:171:in `controlled_run'"
> 0x29f02540:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/agent.rb:
> 35:in `run'"
> 0x29f03740:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/application/
> agent.rb:114:in `onetime'"
> 0x29f043a0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/application/
> agent.rb:88:in `run_command'"
> 0x29f03790:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> application.rb:300:in `run'"
> 0x29f037e0:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> application.rb:397:in `exit_on_fail'"
> 0x29f03830:      "/usr/local/lib/ruby/site_ruby/1.8/puppet/
> application.rb:300:in `run'"
> 0x29f01040:      "/usr/local/sbin/puppetd:4"

It is definitely stuck waiting for something.
It looks like a deadlock in the kernel or libc.

> 
> (gdb) where
> #0  0x281b81a7 in __error () from /lib/libthr.so.3
> #1  0x281b7d88 in __error () from /lib/libthr.so.3
> #2  0x29f02060 in ?? ()
> #3  0x00000008 in ?? ()
> #4  0x00000001 in ?? ()
> #5  0x29f02040 in ?? ()
> #6  0xbc7fbf2c in ?? ()
> #7  0x80018718 in ?? ()
> #8  0x00000000 in ?? ()
> #9  0x281b66df in pthread_setcancelstate () from /lib/libthr.so.3
> #10 0x281b5f7d in pthread_cond_signal () from /lib/libthr.so.3
> #11 0x280c34d4 in thread_timer (dummy=0xbf9ad2ac) at eval.c:12318
> #12 0x281ad6ff in pthread_getprio () from /lib/libthr.so.3
> #13 0xbc6fafec in ?? ()

Hmm no good, we still don't have all the symbols, which means that those
unkown calls are certainly in the kernel (especially since the pc
address are a little bit strange).

Can you try in gdb to do:
thread apply all
bt full

I highly suspect a bug in the freebsd kernel or in ruby.
I google'd for "__error () from /lib/libthr.so.3" and found some other
program having the issues, some pointing to this bug:
http://security.freebsd.org/advisories/FreeBSD-EN-09:04.fork.asc

What freebsd version are you using?
-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Reply via email to