On Sat, Jun 02, 2012 at 03:50:06PM -0700, Zac Medico wrote: > On 06/02/2012 02:31 PM, Micha?? G??rny wrote: > > On Sat, 2 Jun 2012 15:54:03 -0400 > > Mike Frysinger <vap...@gentoo.org> wrote: > > > >> # @FUNCTION: redirect_alloc_fd > >> # @USAGE: <var> <file> [redirection] > >> # @DESCRIPTION: > > > > (...and a lot of code) > > > > I may be wrong but wouldn't it be simpler to just stick with a named > > pipe here? Well, at first glance you wouldn't be able to read exactly > > one result at a time but is it actually useful? > > I'm pretty sure that the pipe has remain constantly open in read mode > (which can only be done by assigning it a file descriptor). Otherwise, > there's a race condition that can occur, where a write is lost because > it's written just before the reader closes the pipe.
There isn't a race; write side, it'll block once it exceeds pipe buf size; read side, bash's read functionality is explicitly byte by byte reads to avoid consuming data it doesn't need. That said, Mgorny's suggestion ignores that the the code already is pointed at a fifo. Presume he's suggesting "Just open it everytime you need to fuck with it"... which, sure, 'cept that complicates the read side (either having to find a free fd, open to it, then close it), or abuse cat or $(<) to pull the results and make the reclaim code handle multiple results in a single shot. Frankly, don't see the point in doing that. The code isn't that complex frankly, and we *need* the overhead of this to be minimal- the hand off/reclaim is effectively the bottleneck for scaling. If the jobs you've backgrounded are a second a piece, it matters less; if they're quick little bursts of activity, the scaling *will* be limited by how fast we can blast off/reclaim jobs. Keep in mind that the main process has to go find more work to queue up between the reclaims, thus this matters more than you'd think. Either way, that limit varies dependent on time required for each job vs # of cores; that said, you run code like this on a 48 core and you see it start becoming an actual bottleneck (which is why I came up with this hacky bash semaphore). ~harring
