On 11/17/2014 07:07 PM, Andrew Savchenko wrote: > Hello, > > On Sun, 09 Nov 2014 00:15:56 -0800 Zac Medico wrote: >> On 11/08/2014 10:53 PM, Andrew Savchenko wrote: > [...] >>> On old hardware it may take dozens of minutes of CPU time. I have >>> that *.pickle files, I have sqlite metadata cache, I have 100% CPU >>> usage. It's not an I/O problem. Just take into account that due to >>> instruction set, Lx cache, frequency and memory speed difference >>> old CPU-based system may be 20x times slower than recent one. >> >> It could be useful for us to collect profiling data generated on old >> hardware. If you'd like to help, you can use python's cProfile module to >> generate statistics for us to analyze. The statistics can be nicely >> visualized as a shaded call graph by using gprof2dot and graphviz [1]. > > Sorry for delay, I generated samples on two old hosts. > > Tarball contains per host directories. Each one contains: > - pstats file; > - generated pdf with call graphs and timing; > - host-related information: > * emerge --info > * /proc/cpuinfo > * /proc/memnifo > > *.pstats and *.pdf filename should describe command unambiguously, > e.g. emerge-pv_python:2.7_python:3.3-python-3.3.5-r1 means: > emerge -pv python:2.7 python:3.3 while using python-3.3.5-r1 as > interpreter.
Thank you for this data. I will see what I can to do optimize the problem areas that your data highlights. > hitomi system was not fully updated for a bit more than a year, > while another one for about half a year. So dependency calculations > may be of different intencities. Sets of packages installed are > similar but not the same: > 2502 on hitomi For hitomi, _slot_operator_update_probe/use_reduce is an obvious thing to optimize. It called use_reduce 53763 times there, so it seems to repeat use_reduce multiple times for the same packages. That means we should see a benefit from memoization. > 2953 on desktop For desktop, _dynamic_deps_preload is an obvious thing to optimize. You can avoid this function entirely if you use --dynamic-deps=n. You may need to run 'emerge @changed-deps' in order to ensure that emerge will be able to cope with --dynamic-deps=n. IIRC you need at least sys-apps/portage-2.2.14 for @changed-deps support. -- Thanks, Zac
