On 2/24/24 12:11 AM, Koichi Murase wrote:
I have a question. Maybe it's not technically a bug as fd > 9 is involved, but the resulting behavior appears to be strange. I received some reports from users in my project and tried to find out what is happening. This is a reduced case:#!/usr/bin/env bash enable -f /path/to/lib/bash/fdflags fdflags exec 50>1.txt fdflags -s +cloexec 50 exec 50>2.txt echo hello >&50 ls -l 1.txt 2.txt The result is $ ./reduced.sh -rw-r--r-- 1 murase murase 6 2024-02-24 13:53:01 1.txt -rw-r--r-- 1 murase murase 0 2024-02-24 13:53:01 2.txt I expect that the string `hello' is saved in 2.txt, but it actually goes into 1.txt. It turned out that the redirection "50>2.txt" is actually performed, but it's immediately undone at the end of `exec' command. The undo redirection is set up in the if statement starting on redir.c:1288 (devel 43ecbeb3). The code comment says: /* experimental: if we're saving a redirection to undo for a file descriptor above SHELL_FD_BASE, add a redirection to be undone if the exec builtin causes redirections to be discarded. There needs to be a difference between fds that are used to save other fds and then are the target of user redirections and fds that are just the target of user redirections. We use the close-on-exec flag to tell the difference; fds > SHELL_FD_BASE that have the close-on-exec flag set are assumed to be fds used internally to save others. */ This piece of the code seems to be introduced in commit cac4cdbf (ChangeLog says the change to redir.c was made on 2011-10-09): https://git.savannah.gnu.org/cgit/bash.git/commit/?h=devel&id=cac4cdbf5 I'm just curious about the background of this different treatment for "fds that are used to save other fds and then are the target of userredirections" (quoted from the code comment).
OK. Life used to be simple: users could only access file descriptors 0-9, so any other file descriptor >= 10 could be used for the shell's own purposes -- one being to save copies of file descriptors that were temporarily changed through redirections. The shell builds an undo list, duplicating the file descriptor (e.g., 2) to a private one (e.g., 10) and saying what you want to do with it (dup it back or close it) as part of performing a redirection like 2>/dev/null. You dup it back if the file descriptor was valid; you close it if not. If the redirections accompany the `exec' builtin, you just close all these file descriptors and throw away the undo list. File descriptors that are opened or duplicated by user redirections are always open on exec. File descriptors the shell uses to save others are always close on exec. The shell could always stay out of the user's way by not using fds 0-9 for private purposes. Now it's more complicated. User redirections can access arbitrary file descriptors -- POSIX makes the upper limit implementation-defined -- even though there are shells (dash and its siblings, mksh, ksh93, zsh) that don't allow it. There is no longer a safe range the shell can use. Consider echo abc 2>/dev/null The shell saves fd 2 to fd 10 (using fcntl), sets the close-on-exec flag on fd 10, and creates a pair of internal redirections to undo it: one to dup2 10 back to 2, clearing the close-on-exec flag, and then one to close 10. Simple. Now what happens with something like echo abc 2>/dev/null 10<>fd10 The shell saves fd 2 to fd 10 again, as above, but then finds fd 10 in use when it goes to perform the next redirection. What does it do then? It knows that fd 10 is greater than the lower limit the shell uses to save file descriptors, so fd 10 has either been the subject of a previous user redirection, in which case the close-on-exec flag is clear, or it's been used to save a file descriptor, in which case it's set. That determines what you do to undo it, and whether or not it persists if the `exec' builtin causes user redirections to be discarded. So you save fd 10 to fd 11, since that's eventually going to be used to restore fd 2 when the command completes, and add a new redirection to make sure it's undone or closed even if exec discards redirections. There's this chain of redirections that gets processed in order to undo everything.
Also, it says "experimental", but is there any alternative implementation, where the user-supplied O_CLOEXEC fds are not undone?
No. User-manipulated file descriptors are always open on exec. There is no standard way to modify this state. Using the example fdflags loadable builtin (which, of course, did not exist when I wrote this code and isn't part of the bash the vast majority of people use) can have unforeseen consequences.
Even if there is a technical background, the current behavior observed by the users is strange, and this behavior doesn't seem to be specified in the Bash documentation. The documentation just says Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally.
What else should it say? What additional guidance would you find useful? The man page isn't going to describe something it doesn't allow the user to do here.
but this behavior is non-trivial even if the user treats the fds with "care". Note that, in the actual code, the number 50 in the above reduced case is chosen to be a number that is not used (which can be tested by e.g. « ! : >&50 ». There can be false-negatives with this test, but that's fine).
Using the scheme I described above, you can't distinguish these two
cases if some external mechanism manipulates the close-on-exec flag.
You could, I suppose, go through the redirection undo list looking
for a redirection with (in this example), redirector.dest == 10 and
instruction == r_close_this or instruction == r_duplicating_output
and (flags & RX_INTERNAL), and use that to check that the open file
descriptor >= 10 with the close-on-exec flag set had been used by
this command to save a file descriptor. I've obviously not done that,
since the shell doesn't offer a standard way to manipulate the close-
on-exec flag.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
OpenPGP_signature.asc
Description: OpenPGP digital signature
