> -----Original Message----- > From: yocto-boun...@yoctoproject.org > [mailto:yocto-boun...@yoctoproject.org] On Behalf Of Xu, Dongxiao > Sent: Friday, June 17, 2011 10:19 AM > To: Richard Purdie > Cc: yocto@yoctoproject.org > Subject: Re: [yocto] configure optimization feature update > > Hi Richard, > > > -----Original Message----- > > From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] > > Sent: Thursday, June 16, 2011 11:01 PM > > To: Xu, Dongxiao > > Cc: yocto@yoctoproject.org > > Subject: Re: configure optimization feature update > > > > Hi Dongxiao, > > > > On Thu, 2011-06-16 at 08:57 +0800, Xu, Dongxiao wrote: > > > Recently I was doing the "configure optimization" feature and > > > collecting data for it. > > > > > > The main logic of this feature is straight forward: > > > > > > 1. Use the diff file as autoreconf cache. (I use command: "diff -ruN > > > SOURCE-ORIG SOURCE", here "SOURCE-ORIG" is the source directory > > > before running autoreconf, while "SOURCE" is the directory after > > > running autoreconf). > > > 2. Add SRC_URI checksum for all patches of the source code. > > > 3. Tag each autoreconf cache file with ${PN} and the SRC_URI > > > checksum of source code and all patches. > > > 4. If the currently SRC_URI checksum matches the cached checksum, > > > then we can patch the cache instead of running "autoreconf" stage. > > > > > > I did some testings for sato build, the result is not as good as we > > > expected: > > > > > > On a server build machine (Genuine Intel(R) CPU @ 2.40GHz, 2 sockets > > > with 6 > > core each and hyperthreading, thus 24 logical CPUs in all, 66G memory): > > > > > > w/o the optimization: > > > real 83m40.963s > > > user 496m58.550s > > > sys 329m1.590s > > > > > > w/ the optimization: > > > real 79m1.062s > > > user 460m58.600s > > > sys 347m42.120s > > > > > > It has about 5% performance gain. > > > > Whats interesting there is the relatively large sys times compared to > > user. Any idea why that's happening? Spinning locks? > > Yes, I also noticed the the in-consistent data of user and sys. > During the build, sometimes I found the build will suspend for some time and > system is doing "kjournald". > It happens relatively frequent on that 24 CPU's server with "48" and "-j48" > assigned for build parallel parameters. > I am not sure whether this caused the above phenomenon. > > > > > > I also tested the patch on a desktop core-i7 machine (Intel(R) > > > Core(TM) i7 > > CPU 870 @ 2.93GHz, 4 core 8 logical CPU, 4G memory): > > > > > > w/o the optimization: > > > real 105m25.436s > > > user 372m48.040s > > > sys 51m23.950s > > > > > > w/ the optimization: > > > real 103m38.314s > > > user 332m35.770s > > > sys 49m4.520s > > > > > > It only has about 2% performance gain. > > > > > > The result is not encouraging. > > > > Agreed, this isn't as good as we'd hoped for :(. > > > > > There are also some other things we need to take into consideration > > > for this feature: > > > > > > 1. If add this feature, the first build time should be longer than > > > current since it needs to build the autoreconf cache. > > > 2. Maintainers needs to maintain the SRC_URI checksums not only for > > > source code, but also all its patches. For some recipes, it has more > > > than 20 patches, which needs assignable maintenance effort. > > > 3. How to distribute the caches will be a problem. The total size of > > > such cache is about 900M (before compression) and 200M (after > > > compression). Since the size is not small, distributing it with Poky > > > source code doesn't make sense. On another aspect, we can use > > > something like "sstate". But since we already have caches of sstate, > > > I think it is not necessary for us to enable another similar cache > > > mechanism with little improvement. > > > > > > Therefore my opinion is we may give up this feature. What's your > > > comments and suggestions? > > > > I think we should put the patches together on a branch in contrib so > > we keep them somewhere in case we want them. Certainly tracking what > > changes the autoreconf process makes may be useful in other situations > > in future so its worth keeping the patches. I think you're right and > > we should shelve the idea for now though as it doesn't look to be worth the > pain it entails. > > OK, I will queue my patch into a contrib tree and keep it there.
The contrib tree to queue the configure optimization patch are located at: http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/configure-optimization it should have flaws and is not for merge. I will keep it there and see if we need it later. Thanks, Dongxiao > > > > > For reference, we probably do need to start tracking the file > > checksums for the benefit of sstate. > > Could you explain more here? Here the file checksums you mentioned is > SRC_URI checksum? > How can it help sstate? > > Thanks, > Dongxiao > > > > > The mediocre performance improvement is likely down to the size of the > > cache data but I can't immediately think of a way to improve that :(. > > > > Cheers, > > > > Richard > > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
