I think I've just figured out why my workaround works. When ssh sets O_NONBLOCK on stdout, it's setting it on the write side of the pipe that git created. Nothing else writes to that pipe, so this does not cause a problem. Presumably, ssh itself is able to deal with writing to a non-blocking "file description". When ssh sets O_NONBLOCK on stderr, it's setting it on the pipe created by Jenkins. But that's the same pipe that my workaround program does the select() on, so my workaround can compensate. When I run "make | workaround", the write side of *that* pipe is never made non-blocking, and that's the one that make itself writes to.
Ok, I think all mysteries are solved, but I'm still not sure what the moral of the story is. This seems like it could lead to some really difficult to debug problems, but it's not clear to me what the right long-term fix should be. For now, we'll just leave our workaround in place and think about tweaking our makefile to redirect git's stderr.
