-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2013-10-31 11:11, John Arbash Meinel wrote: > So I managed to instrument a jujud with both CPU and Mem profiling > dumps. I then brought up 1000 units and did some poking around. > > The results were actually pretty enlightening. > > > 1) Guess what the #1 CPU time was. I know I was surprised: Total: > 25469 samples 14380 56.5% 56.5% 14404 56.6% > crypto/sha512.block 1261 5.0% 61.4% 1261 5.0% > crypto/hmac.(*hmac).tmpPad 1219 4.8% 66.2% 15737 61.8% > crypto/sha512.(*digest).Write 1208 4.7% 70.9% 9548 37.5% > crypto/sha512.(*digest).Sum 439 1.7% 72.7% 19046 74.8% > launchpad.net/juju-core/thirdparty/pbkdf2 >
One observation I didn't report. When testing this out, often machine-0's agent would work for a while, but eventually it would end up hitting 100% CPU and not getting any other work done. I didn't notice in Top, but it was actually spending all that time in sys. So I did some googling and found this: http://grokbase.com/t/gg/golang-dev/1388yzq7yb/code-review-12183044-syscall-disable-cpu-profiling-around-fork-issue-12183044 Given that I saw some of the hangups at the time we were trying to run "lxc-ls" it is possible that adding CPU profiling causes Fork to potentially hang. https://groups.google.com/forum/#!topic/golang-bugs/9Gyeef14Zaw http://code.google.com/p/go/issues/detail?id=5517 https://code.google.com/p/gperftools/issues/detail?id=278 So some of the hanging that I saw may not actually be a problem in practice, but not being able to profile the process seems pretty unfortunate. It looks like the fix landed on August 13: http://code.google.com/p/go/source/detail?r=9eb1dd061b1f Which is the day after Golang 1.1.2 was released. At least that alleviates my fear that when jujud restarts we have a high probability of hanging permanently. And I got enough profiling to see that pbkdf2 appears to be the primary cause of slow startup. 70ms per login * 10,000 agents = 700s ~ 12min or about 3min w/ 4-cpus. I'm still skeptical that we need pbkdf2 for Agent logins, though I do like it for user logins. (We are generating 18 character passwords because originally they were used by Mongo which "only" md5sum'd them. We could use sha512 and 64-byte/128-hex tokens if we cared.) John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJzmYwACgkQJdeBCYSNAANS2QCfb+iNU8CNuPKf8Cb94KQNoTjw ZgkAn23a5RYVhwDvKb2+tJ05aGuQxsQ+ =t1Ia -----END PGP SIGNATURE----- -- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
