Patrick Lauer wrote: > On Fri, 2006-01-13 at 19:53 +0900, Kalin KOZHUHAROV wrote: > >>>Make this distributed tool for tar zip bzip2 and gzip and I'm in, I >>>don't think it would be useful with anything else than Gigabit Ethernet. > > One 2Ghz CPU can't even saturate a 100Mbit line with bzip2 as far as I > can tell. > Although the speedups won't be extreme it could just work. > > >>>We might want to have in the make.conf 2 separate variables, one of them >>>saying how many threads can be run on the machine, then How many >>>threads/process across a cluster. >>> >>>For example, my Dual Xeon EM64T file server can do make -j4 locally, >>>like in make install, make docs etc etc, But for compiling I can use >>>-j20, really not useful over -j8 anyway. But the point is, it would be >>>usefully to separate the load distribution on the local machine and >>>cluster nodes. >> >>As the discusison started... >> >>I would like to be able to limit the -jN when there is no distcc host >>available or when compiling c++ code, otherwise my poor laptop is dead with >>-j5 compiling pwlib when the network is down.... > > As far as I can tell distcc isn't smart enough for dynamic load balancing. > One could hack portage to "test" each server in the distcc host list and > remove missing servers for each run - doesn't look elegant to me.
Yes, might be a solution, even if not elegant. I am thinking also of automating distcc configuration (i.e. no need to run --set-hosts) and one idea is to use DNS with some TXT record, but that is just an idea - no patching is done yet. Not sure if distcc has local limiter, i.e. if it it set with "localhost/2" and portage user (or some other user != root) tries to start 3 processes, the 3rd just blocks (and not take memory). I think not, so this thing might be interesting to implement (for "old" laptops with less memory). I think I should resubscribe to the distcc list :-) >>It is particular example, but being able to limit portage in some way as >>total CPU, total MEM might be interesting (just nice-ing is not enough) > > Very difficult - usually gcc uses ~25M per process (small source files), but > I've seen >100M (most larger C++ files) and heard of ~600M per process for > MySQL > > Limiting that is beyond the scope of portage. Hmm, may be not limiting the total usage, but more like just adjusting MAKEOPTS='-j1' in some cases (NOTE: to /me, define "some cases"). Implementing the above "Local limiter" in distcc will solve that automagically. Kalin. -- |[ ~~~~~~~~~~~~~~~~~~~~~~ ]| +-> http://ThinRope.net/ <-+ |[ ______________________ ]| -- gentoo-dev@gentoo.org mailing list
