[gentoo-user] Basic questions about Distcc
I have a bunch of old laptops that large builds such as texlive and ghc fail on, I'm assuming because of insufficient memory and disk space. If I've understood correctly, with Distcc I could build everything on my main desktop PC and have the binaries transferred through network? How does this work, exactly, and is it a lot of work to set up? I currently have no networking devices besides a single modem/router, would something more be required? - Lasse
[gentoo-user] Re: FYI: Daily / weekly / monthly cron jobs run twice on DST - non-DST transition
On 2017-11-03 02:53, Kai Peter wrote: > 2. the shell script have to do some checks, e.g. the last run - I did > wrote a small 'include' script for that Isn't your 'small include script' just another implementation of run-crons? If not: how does it handle _missed_ jobs, if at all? If you want to run a monthly job on a host that is not always on, do you have to pretend it's an hourly job and check in the script itself? > Thus it's portable. The job have to be done once. If you want your _entire_ configuration to be portable, and my above point about monthly schedule sticks, then you have to ignore the monthly feature of all the crons on all your systems, because you cannot rely on it in any one system. -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. To reply privately _only_ on Usenet, fetch the TXT record for the domain.
Re: [gentoo-user] Basic questions about Distcc
On 03/11/17 16:54, Lasse Pouru wrote: I have a bunch of old laptops that large builds such as texlive and ghc fail on, I'm assuming because of insufficient memory and disk space. If I've understood correctly, with Distcc I could build everything on my main desktop PC and have the binaries transferred through network? How does this work, exactly, and is it a lot of work to set up? I currently have no networking devices besides a single modem/router, would something more be required? No. What distcc does is spread compilation across multiple machines to save time, so if it blows up on those machine currently, there's no guarantee it won't blow up with distcc. What I do is make sure my flags and stuff match across all machines, then compile using the -bk flags (I can't remember which says "create binary if it doesn't exist" and which is "use binary if it exists"). That way, it builds on the machine that works, and uses the binary on the machines that don't. Oh - and it's definitely possible you've got dodgy ram. I gather it's not uncommon for ram to work perfectly until gcc stresses it on a big build at which point everything comes crashing down ... Cheers, Wol
Re: [gentoo-user] Basic questions about Distcc
Wol's lists writes: > On 03/11/17 16:54, Lasse Pouru wrote: >> I have a bunch of old laptops that large builds such as texlive and >> ghc fail on, I'm assuming because of insufficient memory and disk >> space. If I've understood correctly, with Distcc I could build >> everything on my main desktop PC and have the binaries transferred >> through network? How does this work, exactly, and is it a lot of >> work to set up? I currently have no networking devices besides a >> single modem/router, would something more be required? >> > No. What distcc does is spread compilation across multiple machines to > save time, so if it blows up on those machine currently, there's no > guarantee it won't blow up with distcc. > > What I do is make sure my flags and stuff match across all machines, > then compile using the -bk flags (I can't remember which says "create > binary if it doesn't exist" and which is "use binary if it > exists"). That way, it builds on the machine that works, and uses the > binary on the machines that don't. > > Oh - and it's definitely possible you've got dodgy ram. I gather it's > not uncommon for ram to work perfectly until gcc stresses it on a big > build at which point everything comes crashing down ... > > Cheers, > Wol So the flags would have to be the same on all machines? What about the CPU architecture? Isn't there a way to cross-compile for a 32-bit machine (with minimal flags) on a 64-bit machine, and specify which machine to do the compiling on?
Re: [gentoo-user] Basic questions about Distcc
On Fri, Nov 3, 2017 at 10:51 AM, Lasse Pouru wrote: > > So the flags would have to be the same on all machines? What about the > CPU architecture? Isn't there a way to cross-compile for a 32-bit > machine (with minimal flags) on a 64-bit machine, and specify which > machine to do the compiling on? > I'm pretty sure that the only thing that matters is that the target is supported on all the workers. That said, I've never tried to use distcc with cross-compiling or even a 32/64-bit mix on the same major arch. My understanding is that the preprocessing is all done on the target machine, and the remote workers take all their marching orders from there. The contents of CFLAGS, libraries, and so on don't matter on the workers. You can build a program that requires qt using distcc workers that don't have qt installed, because all the includes are already resolved by the time the source code reaches them and linking is done later. I'm sure there are others around here with far more distcc experience. -- Rich
Re: [gentoo-user] FYI: Daily / weekly / monthly cron jobs run twice on DST - non-DST transition
On Thu, Nov 2, 2017 at 6:53 PM, Kai Peter wrote: > > *I* recommend fcron, it is a bit under estimated. Beside its progressive > design and w/o consulting the man page now again - AFAIR it can handle > DST issues like above through options in fcrontab. But with my concept I > don't need/use it. Be aware that some options could show an unexpected > behavior too - nothing is perfect. Anyhow, by using fcron it is possible > to eliminate the whole cronbase ebuild - it is part of the 20 percent > (round about) which are faulty in Gentoo. > If some people want to maintain Gentoo-specific solutions I certainly won't tell them that they can't, but 99% of the time when there is an upstream tool that does something out of the box, and a Gentoo-specific stack of shell scripts, the upstream tool is going to do a better job. That doesn't mean that the simpler tool isn't sometimes the better solution. I have hosts that use the systemd-timesyncd implementation, and I have hosts that run ntpd. There is a reason for each to exist. The nice thing about Gentoo is that you can use whatever you prefer, but if you care that much about what happens on the DST change or on the 31st day of the month, you're probably better off starting from a more full-featured cron implementation than trying to re-create one. -- Rich
Re: [gentoo-user] Basic questions about Distcc
On 03/11/17 18:02, Rich Freeman wrote: My understanding is that the preprocessing is all done on the target machine, and the remote workers take all their marching orders from there. The contents of CFLAGS, libraries, and so on don't matter on the workers. You can build a program that requires qt using distcc workers that don't have qt installed, because all the includes are already resolved by the time the source code reaches them and linking is done later. Yup. If you're cross-compiling (like I was - a mix of 32 and 64 bit), provided all machines are set up correctly it works fine. BUT. That's if the "master" is compiling for itself and just sharing out the work. But if all the machines are similar architecture (like mine are now all x86_64) you can do what I do - the fastest machine builds everything and makes binary install files, and the slower machines just install the binaries. (Or actually, the slower machine builds everything, because the faster machine has a habit of falling over during builds :-( Cheers, Wol
Re: [gentoo-user] Basic questions about Distcc
On 04/11/17 04:45, Wol's lists wrote: > On 03/11/17 18:02, Rich Freeman wrote: >> My understanding is that the preprocessing is all done on the target >> machine, and the remote workers take all their marching orders from >> there. The contents of CFLAGS, libraries, and so on don't matter on >> the workers. You can build a program that requires qt using distcc >> workers that don't have qt installed, because all the includes are >> already resolved by the time the source code reaches them and linking >> is done later. > > Yup. If you're cross-compiling (like I was - a mix of 32 and 64 bit), > provided all machines are set up correctly it works fine. BUT. That's if > the "master" is compiling for itself and just sharing out the work. > > But if all the machines are similar architecture (like mine are now all > x86_64) you can do what I do - the fastest machine builds everything and > makes binary install files, and the slower machines just install the > binaries. (Or actually, the slower machine builds everything, because > the faster machine has a habit of falling over during builds :-( > > Cheers, > Wol > I use ccache and distcc for raspberry pi (512M ram) to an 64bit intel vm set up to cross compile. Set MAKEOPTS to 1 in distcc use either 1 or zero for local jobs. Pi is 32 bit so use 2x 2G swapfiles for 4G swap - have not seen the second swap in use, but until I added it the last GCC build failed with out of memory. Most problematic package is GCC itself. Sllooo :) Suggestion - use an LCD (Lowest Common Denominator) host to build packages and then emerge them on other hosts as binary packages. BillK
Re: [gentoo-user] Re: FYI: Daily / weekly / monthly cron jobs run twice on DST - non-DST transition
On 2017-11-03 18:21, Ian Zimmerman wrote: On 2017-11-03 02:53, Kai Peter wrote: 2. the shell script have to do some checks, e.g. the last run - I did wrote a small 'include' script for that Isn't your 'small include script' just another implementation of run-crons? No. If not: how does it handle _missed_ jobs, if at all? Whatever _missed_ means - at least it's a question of error handling. If you want to run a monthly job on a host that is not always on, do you have to pretend it's an hourly job and check in the script itself? This is a special case to me. IMHO special cases have to be handled special or much better: avoid it. Thus it's portable. The job have to be done once. If you want your _entire_ configuration to be portable, and my above point about monthly schedule sticks, then you have to ignore the monthly feature of all the crons on all your systems, because you cannot rely on it in any one system. My _entire_ config isn't portable, but some (important) parts of it. -- Sent with eQmail-1.10
Re: [gentoo-user] Basic questions about Distcc
On Fri, Nov 3, 2017 at 11:54 AM, Lasse Pouru wrote: > I have a bunch of old laptops that large builds such as texlive and ghc fail > on, I'm assuming because of insufficient memory and disk space. If I've > understood correctly, with Distcc I could build everything on my main desktop > PC and have the binaries transferred through network? How does this work, > exactly, and is it a lot of work to set up? I currently have no networking > devices besides a single modem/router, would something more be required? > If you have a capable desktop PC you could cross compile for your other computers. Your main desktop would do all the build work and serve as a binhost. Distcc works brilliantly when it does work, but there are occasional failures. Of course, only failures get seen on this list. Cheers, R0b0t1
