On Fri, Oct 04, 2019 at 11:15:46AM +0200, Rasmus Villemoes wrote:
> On 25/09/2019 01.29, Kees Cook wrote:
> > +# Extract and prepare jobserver file descriptors from envirnoment.
> > +try:
> > + # Fetch the make environment options.
> > + flags = os.environ['MAKEFLAGS']
> > +
> > + # Look for "--jobserver=R,W"
> > + opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
>
> OK, this handles the fact that Make changed from --jobserver-fds to
> --jobserver-auth at some point. Probably the comment could be more accurate.
I can update that, sure.
> > +# Read out as many jobserver slots as possible.
> > +jobs = b""
> > +while True:
> > + try:
> > + slot = os.read(reader, 1)
> > + jobs += slot
> > + except (OSError, IOError) as e:
> > + if e.errno == errno.EWOULDBLOCK:
> > + # Stop when reach the end of the jobserver queue.
> > + break
> > + raise e
>
> Ah, reading more carefully you set O_NONBLOCK explicitly. Well, older
> Makes are going to be very unhappy about that (remember that it's a
> property of the file description and not file descriptor). They don't
> expect EAGAIN when fetching a token, so fail hard. Probably not when
> htmldocs is the only target, because in that case the toplevel Make just
> reads back the exact number of tokens it put in as a sanity check, but
> if it builds other targets/spawns other submakes, I think this breaks.
Do you mean the processes sharing the file will suddenly gain
O_NONBLOCK? I didn't think that was true, but I can test. If it is true,
we could easily just restore the state before exit.
> > +# Return all the reserved slots.
> > +os.write(writer, jobs)
>
> Well, that probably works ok for the isolated case of a toplevel "make
> -j12 htmldocs", but if you're building other targets ("make -j12
> htmldocs vmlinux") this will effectively inject however many tokens the
> above loop grabbed (which might not be all if the top-level make has
> started things related to the vmlinux target), so for the duration of
> the docs build, there will be more processes running than asked for.
That is true, yes, though I still think it's an improvement over the
existing case of sphinx-build getting run with -jauto which lands us in
the same (or worse) position.
The best solution would be to teach sphinx-build about the Make
jobserver, though I expect that would be weird. Another idea would be to
hold the reservation until sphinx-build finishes and THEN return the
slots? That would likely need to change from a utility to a sphinx-build
wrapper...
> > +# If the jobserver was (impossibly) full or communication failed, use
> > default.
> > +if len(jobs) < 1:
> > + print(default)
> > +
> > +# Report available slots (with a bump for our caller's reserveration).
> > +print(len(jobs) + 1)
>
> There's a missing exit() or else: here; if len(jobs) < 1 you print both
> default (probably "1") and 0+1 aka "1".
Whoops! Yes, that needs fixing too. I'll send a patch...
--
Kees Cook
