Configuration Information [Automatically generated, do not change]:
Machine: aarch64
OS: darwin24.2.0
Compiler: gcc
Compilation CFLAGS: -g -O2
uname output: Darwin MacBookPro.lan.internal 24.2.0 Darwin Kernel
Version 24.2.0: Fri Dec 6 19:01:59 PST 2024;
root:xnu-11215.61.5~2/RELEASE_ARM64_T6000 arm64
Machine Type: aarch64-apple-darwin24.2.0
Bash Version: 5.3
Patch Level: 15
Release Status: release
Description:
In the XNU kernel versions 6153.11.26 and later on macOS,
the size of pipe buffers are dynamic. Defined in the source
code under bsd/kern/sys_pipe.c as the `pipesize_blocks`
variable [1], there are 7 possible capacities:
512, 1024, 2048, 4096, 8192, 16384, 65536
Of these, only the 512-byte size appears to be guaranteed.
The kernel can expand a pipe to one of the larger capacities,
but it will only do so if the total capacity of all pipes on
the system is less than [2] the 16 MiB `maxpipekva` high-water
mark defined at build time [3].
A brief look at the code responsible for pipe sizing ([2])
suggests that it should be possible for a pipe's buffer to
initially be allocated at one of the larger capacities, but
this is only the case when writing into `pipefds[0]`. The
buffer for the "normal" direction is immediately allocated
when the pipe is created [4].
The kernel's behavior of no longer expanding pipes when the
high-water mark is reached causes bash to hang when writing
medium-sized (512 B < size < 64 KiB) heredocs. This is due to
an optimization in Bash 5.1+ where `here_document_to_fd` in
redir.c will write the heredoc into a pipe if its smaller
than the `HEREDOC_PIPESIZE` constant computed at build time.
When bash is built on a macOS system where the high-water
mark hasn't been reached, the max pipe size will be detected
as 64 KiB. If the high-water mark is later reached and bash
tries to write a medium-sized heredoc into a pipe, it will
block due to the pipe only having a 512-byte capacity.
Repeat-By:
1. Use a macOS system running XNU 6153 (macOS 10.15) or later.
2. Create and hold 16 MiB worth of pipes. The most consistent
way to do this is to create a small program that tries to
open 256 pipes and write 64K of data into each of them.
I included such a program in the 0002-*.patch attachment.
3. In a separate terminal window, run bash and try to pipe a
513-byte heredoc or herestring into `cat`. It should not
end up printing anything and will instead just hang until
interrupted with Ctrl+C.
bash -c 'cat <<< "$(printf %0513d 0)"'
Fix:
The simplest fix is to add `-DHEREDOC_PIPESIZE=512` to the
compiler flags when building bash for Darwin. A more
comprehensive fix would be to have `here_document_to_fd`
set the pipe as O_NONBLOCK and fall back to using a tempfile
if the write into the pipe was only partial.
I implemented both of these along with a regression test and
have attached them as patch files based on commit b4608166 from
the git repo hosted at https://savannah.gnu.org/git/?group=bash
When creating the patches, I tried to follow any existing
conventions that I noticed in the source code. This is the first
time I have written a patch for bash, though, so please let me
know if I missed anything.
Citations:
[1]:
https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/kern/sys_pipe.c#L307
[2]:
https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/kern/sys_pipe.c#L956-L960
[3]:
https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/kern/sys_pipe.c#L241
[4]:
https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/kern/sys_pipe.c#L633-L638
P.S. If the attached patch files end up lost somewhere along the way, I also
made a GitHub gist containing them:
https://gist.github.com/eth-p/40d763ce3eb13293c2ba4a6848af06de
0002-add-regression-test.patch
Description: Binary data
0003-nonblocking-write-fix.patch
Description: Binary data
0001-simple-fix.patch
Description: Binary data
