haha OK. If I were you, I would have built lsmake functionality in GMAKE and not pay IBM lol.
Anyways, have a good day. :) On Mon, Sep 2, 2019 at 11:45 PM David Boyce <david.s.bo...@gmail.com> wrote: > I did not suggest using lsmake, I simply mentioned that we use it. > > On Mon, Sep 2, 2019 at 11:04 AM nikhil jain <jainnikhi...@gmail.com> > wrote: > >> Thanks for detailed information. >> >> I will see if I can use shell wrapper program as mentioned by you. >> >> I had used LSF a lot like for 5 years. I still use it. bsub, bjobs. >> bkill, lim, sbatchd, mbatchd etc. it is easy to understand and use >> >> lsmake - I do not want to use IBM's proprietary stuff. >> >> Thanks for your suggestions. >> >> Nikhil >> >> On Mon, Sep 2, 2019 at 11:10 PM David Boyce <david.s.bo...@gmail.com> >> wrote: >> >>> I'm not going to address the remote execution topic since it sounds like >>> you already have the solution and are not looking for help. However, I do >>> have fairly extensive experience with the NFS/retry area so will try to >>> contribute there. >>> >>> First, I don't think what Paul says: >>> >>> > As for your NFS issue, another option would be to enable the .ONESHELL >>> > feature available in newer versions of GNU make: that will ensure that >>> > all lines in a recipe are invoked in a single shell, which means that >>> > they should all be invoked on the same remote host. >>> >>> Is sufficient. Consider the typical case of compiling foo.c to foo.o and >>> linking it into foo.exe. Typically, and correctly, those actions would be >>> in two separate recipes which in a distributed-build scenario could run on >>> different hosts so the linker may not find the .o file from a previous >>> recipe. Here .ONSHELL cannot help since they're different recipes. >>> >>> In my day job we use a product from IBM called LSF (Load Sharing >>> F-something, >>> https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_welcome.html) >>> which exists to distribute jobs over a server farm (typically using NFS) >>> according to various metrics like load and free memory and so on. Part of >>> the LSF package is a program called lsmake ( >>> https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_command_ref/lsmake.1.html) >>> which under the covers is a version of GNU make with enhancement to enable >>> remote/distributed recipes and also adds retry-with-delay feature Nikhil >>> requested). Since GNU make is GPL, IBM is required to make its package of >>> enhancements available under GPL as well. Much of it is not of direct >>> interest to the open source community because it's all about communicating >>> with IBM's proprietary daemons but their retry logic could probably be >>> taken directly from the patch. At the very least, if retries were to be >>> added to GNU make per se it would be nice if the flags were compatible with >>> lsmake. >>> >>> However, my personal belief is that retries are God's way of telling us >>> to think harder and better. Retrying (and worse, delay-and-retry) is a form >>> of defeatism which I call "sleep and hope". Computers are deterministic, >>> there's always a root cause which can usually be found and addressed with >>> sufficient analysis, etc. Granted there are cases where you understand the >>> problem but can't address it for administrative/permissions/business >>> reasons but that can't be known until the problem is understood. >>> >>> NFS caching is the root cause of unreliable distributed builds, as >>> you've already described, but most or all of these issues can be addressed >>> with a less blunt instrument than sleep-and-retry. Even LSF engineers threw >>> up their hands and did retries but what we did here was take their patch, >>> which at last check was still targeted to 3.81, and while porting it to 4.1 >>> added some of the cache-flushing strategies detailed below. This has solved >>> most if not all of our NFS sync problems. Caveat: most of our people still >>> use the LSF retry logic in addition, because they're not as absolutist as I >>> am and just want to get their jobs done (go figure), which makes it harder >>> to determine what percentage of problems are solved by cache flushing vs >>> retries but I'm pretty sure flushing has resolved the great majority of >>> problems. >>> >>> One problem with sleep-and-hope is that there's no amount of time >>> guaranteed to be enough so you're just driving the incidence rate down, not >>> fixing it. >>> >>> Since we were already working with a hacked version of GNU make we found >>> it most convenient to implement flushing directly in the make program but >>> it can also be done within recipes. In fact we have 3 different >>> implementations of the same NFS cache flushing logic: >>> >>> 1. Directly within our enhanced version of lsmake. >>> 2. In a standalone binary called "nfsflush". >>> 3. In a Python script called nfsflush.py. >>> >>> The Python script is a lab for trying out new strategies but it's too >>> slow for production use. The binary is a faster version of the same >>> techniques for direct use in recipes, and that same C code is linked >>> directly into lsmake as well. Here's the usage message of our Python script: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *$ nfsflush.py --helpusage: nfsflush.py [-h] [-f] [-l] [-r] [-t] [-u] >>> [-V] path [path ...]positional arguments: path directory paths >>> to flushoptional arguments: -h, --help show this help message and >>> exit -f, --fsync Use fsync() on named files -l, --lock Lock >>> and unlock the named files -r, --recursive flush recursively -t, --touch >>> additional flush action - touch and remove a temp file -u, --upflush >>> flush parent directories back to the root -V, --verbose increment >>> verbosity levelFlush the NFS filehandle caches of NFS directories.Newly >>> created files are sometimes unavailable via NFS for a periodof time due to >>> filehandle caching, leading to apparent race problems.See >>> http://tss.iki.fi/nfs-coding-howto.html#fhcache >>> <http://tss.iki.fi/nfs-coding-howto.html#fhcache> for details.This script >>> forces a flush using techniques mentioned in the URL. Itcan optionally do >>> so recursively.This always does an opendir/closedir sequence on each >>> directoryvisited, as described in the URL, because that's cheap and safe >>> andoften sufficient. Other strategies, such as creating and removing atemp >>> file, are optional.EXAMPLES: nfsflush.py /nfs/path/...* >>> >>> The most important thing is to read the URL given above and/or to google >>> for similar resource of which there are many. While I'm not an NFS guru >>> myself, the summary of my understanding is that NFS caches all sorts of >>> things (metadata like atime/mtime, directory updates, etc) with various >>> degrees of aggression according to NFS vendor and internal configuration. >>> We've seen substantial variation between NAS providers such as NetApp, EMC, >>> etc, so much depends on whose NFS server you're using. However, the NFS >>> spec _requires_ that caches be flushed on a write operation so all >>> implementations will do this. >>> >>> Bottom line, the most common failure case is as mentioned above: foo.o >>> is compiled on host A and immediately linked on host B. The close() system >>> call following the final write() of foo.o on host A will cause its data to >>> be flushed. Similarly I *believe* the directory write (assuming foo.o is >>> newly created and not just updated) will cause the filehandle cache to be >>> flushed. Thus, after these two write ops (directory and file) the NFS >>> server will know about the new foo.o as soon as it's created. >>> >>> The problem typically arises on host B because no write operation has >>> taken place there after foo.o was created on A so no one has told it to >>> update its caches and as a result it doesn't know foo.o exists and the link >>> fails with ENOENT. All the flushing techniques in the script above are >>> attempts to address this. One takeaway from all this is that even if you do >>> retries, a "dumb" retry is immeasurably enhanced by adding a flush. In >>> other words the most efficient retry formula in a distributed build >>> scenario would be: >>> >>> <recipe> || flush || <recipe> >>> >>> This never flushes a cache unless the first attempt fails. It presumes >>> that NFS implementors and admins know what they're doing and thus caching >>> helps with performance so it's not done unless needed. This is what we >>> built into our variant of lsmake. However, the same can also be done in the >>> shell. >>> >>> Details about implemented cache flushing techniques: the filehandle >>> cache is the biggest source of problems in distributed builds and the >>> simplest solution for it seems to be opening and reading the directory >>> entry. Thus our script and its parallel C implementation always do that. >>> We've also seen cases where forcing a directory write operation is required >>> which the -t, --touch option does. Sometimes you can't easily enumerate all >>> directories involved (vpath etc) so the recurse-downward (-r) and recurse >>> upward (-u) flags may be helpful though they (especially -u) may also be >>> overkill. The -f and -l options were added based on advice found on the net >>> but have not been shown to be helpful in our environment. >>> >>> Some techniques may be of limited utility because they require write >>> and/or ownership privileges. For instance I've seen statements that >>> umounts, even failed umounts, will force flushes. Thus a command like "cd >>> <dir> && umount $(pwd)" would have to fail since the moount is busy but >>> would flush as a side effect. However I believe this requires root >>> privileges so is not helpful in the normal case. >>> >>> In summary: although I don't believe in retries, if they're going to be >>> used I think they should be implemented in a shell wrapper program which >>> could be passed to make as SHELL=<wrapper> and the wrapper should use >>> flushing in addition to, or instead of, retries. We didn't do it that way >>> but I think our nfsflush program could just as well have been implemented >>> as say "nfsshell" such that "nfsshell [other-options] -c <recipe>" would >>> run the recipe along with added flushing and retrying options. I agree with >>> Paul that I see no reason to implement any of these features, retry and/or >>> flush, directly in make. >>> >>> David >>> >>> On Mon, Sep 2, 2019 at 6:05 AM Paul Smith <psm...@gnu.org> wrote: >>> >>>> On Sun, 2019-09-01 at 23:23 -0700, Kaz Kylheku (gmake) wrote: >>>> > If your R&D team would allow you to add just one line to the >>>> > legacy GNU Makefile to assign the SHELL variable, you can assign that >>>> > to a shell wrapper program which performs command re-trying. >>>> >>>> You don't have to add any lines to the makefile. You can reset SHELL >>>> on the command line, just like any other make variable: >>>> >>>> make SHELL=/my/special/sh >>>> >>>> You can even override it only for specific targets using the --eval >>>> command line option: >>>> >>>> make --eval 'somerule: SHELL := /my/special/sh' >>>> >>>> Or, you can add '-f mymakefile.mk -f Makefile' options to the command >>>> line to force reading of a personal makefile before the standard >>>> makefile. >>>> >>>> Clearly you can modify the command line, otherwise adding new options >>>> to control a putative retry on error option would not be possible. >>>> >>>> As for your NFS issue, another option would be to enable the .ONESHELL >>>> feature available in newer versions of GNU make: that will ensure that >>>> all lines in a recipe are invoked in a single shell, which means that >>>> they should all be invoked on the same remote host. This can also be >>>> done from the command line, as above. If your recipes are written well >>>> it should Just Work. If they aren't, and you can't fix them, then >>>> obviously this solution won't work for you. >>>> >>>> Regarding changes to set re-invocation on failure, at this time I don't >>>> believe it's something I'd be willing to add to GNU make directly, >>>> especially not an option that simply retries every failed job. This is >>>> almost never useful (why would you want to retry a compile, or link, or >>>> similar? It will always just fail again, take longer, and generate >>>> confusing duplicate output--at best). >>>> >>>> The right answer for this problem is to modify the makefile to properly >>>> retry those specific rules which need it. >>>> >>>> I commiserate with you that your environment is static and you're not >>>> permitted to modify it, however adding new specialized capabilities to >>>> GNU make so that makefiles don't have to be modified isn't a design >>>> philosophy I want to adopt. >>>> >>>> >>>> _______________________________________________ >>>> Help-make mailing list >>>> Help-make@gnu.org >>>> https://lists.gnu.org/mailman/listinfo/help-make >>>> >>> _______________________________________________ Help-make mailing list Help-make@gnu.org https://lists.gnu.org/mailman/listinfo/help-make
