On 04/12/2012 07:08 AM, Björn Stenberg wrote: > Darren Hart wrote: >> /dev/md0 /build ext4 >> noauto,noatime,nodiratime,commit=6000 > > A minor detail: 'nodiratime' is a subset of 'noatime', so there is no > need to specify both.
Excellent, thanks for the tip. > >> I run on a beast with 12 cores, 48GB of RAM, OS and sources on a >> G2 Intel SSD, with two Seagate Barracudas in a RAID0 array for my >> /build partition. I run a headless Ubuntu 11.10 (x86_64) >> installation running the 3.0.0-16-server kernel. I can build >> core-image-minimal in < 30 minutes and core-image-sato in < 50 >> minutes from scratch. > > I'm guessing those are rather fast cores? They are: model name : Intel(R) Xeon(R) CPU X5680 @ 3.33GHz > I build on a different type > of beast: 64 cores at 2.1GHz and 128 GB ram. The OS is on a single > SSD and the build dir (and sources) is on a RAID0 array of Intel 520 > SSDs. Kernel is the same ubuntu 3.0.0-16-server as yours. Now that I think about it, my downloads are on the RAID0 array too. One thing that comes to mind is the parallel settings, BB_NUMBER_THREADS and PARALLEL_MAKE. I noticed a negative impact if I increased these beyond 12 and 14 respectively. I tested this with bb-matrix (scripts/contrib/bb-perf/bb-matrix.sh). The script is a bit fickle, but can provide useful results and killer 3D surface plots of build time with BB and PM on the axis. Can't seem to find a plot image at the moment for some reason... > > Yet for all the combined horsepower, I am unable to match your time > of 30 minutes for core-image-minimal. I clock in at around 37 minutes > for a qemux86-64 build with ipk output: > > ------ NOTE: Tasks Summary: Attempted 1363 tasks of which 290 didn't > need to be rerun and all succeeded. > > real 36m32.118s user 214m39.697s sys 108m49.152s ------ > > These numbers also show that my build is running less than 9x > realtime, indicating that 80% of my cores sit idle most of the time. Yup, that sounds about right. The build has a linear component to it, and anything above about 12 just doesn't help. In fact the added scheduling overhead seems to hurt. > This confirms what "ps xf" says during the builds: Only rarely is > bitbake running more than a handful tasks at once, even with > BB_NUMBER_THREADS at 64. And many of these tasks are in turn running > sequential loops on a single core. > > I'm hoping to find time soon to look deeper into this issue and > suggest remedies. It my distinct feeling that we should be able to > build significantly faster on powerful machines. > Reducing the dependency chains that result in the linear component of the build (forcing serialized execution) is one place we've focused, and could probably still use some attention. CC'ing RP as he's done a lot there. -- Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
