Hi, Firstly I must warn you that I am an interested party since I do consulting related to this product but I did use it purely as a customer for several years.
Electric Accelerator is pretty much the pièce de résistance for running GNU Make and Ninja makefiles on clusters and there is a free evaluation version. It solves several of your problems: 1) Making sure the files you need (including tools and source) are available to every machine in your cluster. You never have to manage this or debug whether all your machines have been kept uptodate or provide NFS mounts everywhere or manage NIS authentication or any of that stuff. 2) It spots problems in your makefiles where tasks are run in the wrong order - this usually happens because there isn't enough dependency information in the makefiles. 3) There is a tool called Electric Insight which lets you visualise the build across your cluster and the analysis tools and reporting on this are incredible for helping to fix performance or correctness problems. http://electric-cloud.com/products/electricaccelerator/ Scroll down to the "Use it for free" link. There are a few limitations to the free version but they don't generally matter too much if you're not supporting 100s of users and if you find it very useful you can now get team licenses priced so that you don't have to be a huge company to afford it which is a welcome new change. Regards, Tim On 11 October 2017 at 03:15, Martin d'Anjou <martin.danjo...@gmail.com> wrote: > Hi David, > > I explored this problem a while ago. I captured my musings in a blog: > https://ninjaverification.wordpress.com/2008/10/21/ > overriding-gnu-make-shell-variable-for-massive-parallel-make/ > > In the end the idea is to have make call out to a job scheduler as part > of the target recipe. This is the method I use and it works really well. > > One problem I still have though is that if one of the parallel jobs > aborts and compromises the whole build, make will wait for all other > in-flight jobs to finish before itself returning. At the time I had > manually patched my local copy of GNU Make, but the patch no longer > applies and I have not gotten back to it. Here is a reference in case > you find it useful: > http://savannah.gnu.org/bugs/?41781 > > Best, > Martin > > On 17-10-10 09:53 AM, David Delahaye wrote: > > Hello, > > > > I wonder how the make command is compatible with clusters. I know the > > "-j" option, which allows us to run several jobs simultaneously and > > uses the several cores of a given processor. But if we have several > > nodes of computation of a cluster, is the command "make -j" able to > > run over these nodes (transparently for the user)? I know there are > > specific commands to run jobs on a cluster, I would like to know how > > "make -j" is compatible with these commands. Thanks in advance for > > your help. > > > > Best regards, > > > > David Delahaye. > > > > > _______________________________________________ > Help-make mailing list > Help-make@gnu.org > https://lists.gnu.org/mailman/listinfo/help-make > _______________________________________________ Help-make mailing list Help-make@gnu.org https://lists.gnu.org/mailman/listinfo/help-make
