Hi all,While trying to use cp for a fairly simple script I'm writing, I noticed two sort of odd things going on. The first might not really be a bug, but I find it rather inconvenient. The second seems to involve a bug, but it doesn't really affect my use-case.
The command I can see the reproduced with is the following:strace cp --verbose --preserve=all --no-preserve=timestamps --parents --recursive --update=older --target-directory="/home/kye/test" "." where the "." directory contains three files (some random build files), two of which happen to be identical binaries. (I've attached the full trace to this message.)
More specifically, the odd behavior occurs when I run the same command multiple times in a row, without changing any of the data being copied. Due to the --update=older flag, I would more or less expect that no writes should be made, but a few writes continue to occur every time the command is rerun. From the strace output, we can see that there are two things going on:
1. The one that is more inconvenient for me is that at the end of the process, cp rewrites the permission bits on the target directory, even when they match the current permission bits (line 93 of the trace). Writes to the target directory are specifically undesirable because in my case the target directory will be an external btrfs subvolume which has very little IO, so the usual way to tell if there have been any changes to the subvolume is with its "generation number"; if cp is unnecessarily overwriting identical permission bits though, the generation number will be incremented even when no data has changed. Even outside of my use-case, an extra chmod that isn't actually changing anything is still a small opportunity for better optimization, especially since, when recursively copying (updating) a large directory tree, every directory in the tree will have it's permission bits overwritten. I should also note that this kind of behavior is not limited to btrfs, but I have also observed it on ZFS. On ZFS, a diff taken between two snapshots of a volume taken before and after a (ideally) null copy will also show that also the directories were modified. 2. The other odd thing in the trace is that there are some kind of odd shenanigans going on with the two identical binary files, and a third temporary file that cp makes and then removes. I don't really follow the logic of the series of "linkat"s, "renameat"s, and "unlinkat"s around lines 70-80 of the trace, but while I could accept that they're somehow useful, what definitely seems like a bug is the message printed (due to being in verbose mode) at line 98 of the trace. The message makes the claim that a certain file in the target directory was removed, but it's definitely still present, and the trace doesn't show any equivalent writes to the target directory, only the ones in the source directory that I mentioned above. This kind of behavior occurs with a lot of files in the directory tree I would like to be able to copy, but not all of them, nor even the majority of them. It's not really clear what causes it besides that in this case the files are identical binaries. It's also not really causing me any problems, it's just weird.While I think it's true that there isn't really any reason that cp should need to guarantee that --update=older creates exactly 0 writes in the target directory when there are no differences between the source and the target, in this case it seems like the current behavior is already close enough to getting it right that it would be worth making the change (avoiding "chmod"s that don't actually change any mode), if possible.
Let me know if you need any other information from me! Thanks, -Kye -- Kye E. Hunter PGP: 6859 E2DE D598 49EA 9319 10CD DEF2 BA03 A6BE 3062 --
execve("/usr/bin/cp", ["cp", "--verbose", "--preserve=all",
"--no-preserve=timestamps", "--parents", "--recursive", "--update=older",
"--target-directory=/home/kye/tes"..., "."], 0x7ffff4ac2b30 /* 38 vars */) = 0
brk(NULL) = 0x56522d8ab000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=174571, ...}) = 0
mmap(NULL, 174571, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb613780000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=34688, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fb61377e000
mmap(NULL, 32800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb613775000
mmap(0x7fb613777000, 16384, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb613777000
mmap(0x7fb61377b000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x6000) = 0x7fb61377b000
mmap(0x7fb61377c000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fb61377c000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=26496, ...}) = 0
mmap(NULL, 28696, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb61376d000
mmap(0x7fb61376f000, 12288, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb61376f000
mmap(0x7fb613772000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x5000) = 0x7fb613772000
mmap(0x7fb613773000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7fb613773000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000x\2\0\0\0\0\0"...,
832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"...,
896, 64) = 896
fstat(3, {st_mode=S_IFREG|0755, st_size=2149728, ...}) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"...,
896, 64) = 896
mmap(NULL, 2174000, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb613400000
mmap(0x7fb613424000, 1515520, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) = 0x7fb613424000
mmap(0x7fb613596000, 454656, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3,
0x196000) = 0x7fb613596000
mmap(0x7fb613605000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x204000) = 0x7fb613605000
mmap(0x7fb61360b000, 31792, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb61360b000
close(3) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fb61376a000
arch_prctl(ARCH_SET_FS, 0x7fb61376a740) = 0
set_tid_address(0x7fb61376aa10) = 56095
set_robust_list(0x7fb61376aa20, 24) = 0
rseq(0x7fb61376a680, 0x20, 0, 0x53053053) = 0
mprotect(0x7fb613605000, 16384, PROT_READ) = 0
mprotect(0x7fb613773000, 4096, PROT_READ) = 0
mprotect(0x7fb61377c000, 4096, PROT_READ) = 0
mprotect(0x56520054e000, 4096, PROT_READ) = 0
mprotect(0x7fb6137ec000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=RLIM64_INFINITY}) = 0
getrandom("\x73\xb1\xab\x58\x43\x79\x03\x41", 8, GRND_NONBLOCK) = 8
munmap(0x7fb613780000, 174571) = 0
brk(NULL) = 0x56522d8ab000
brk(0x56522d8cc000) = 0x56522d8cc000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3063024, ...}) = 0
mmap(NULL, 3063024, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb613000000
close(3) = 0
geteuid() = 1000
openat(AT_FDCWD, "/home/kye/test", O_RDONLY|O_PATH|O_DIRECTORY) = 3
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=4096, ...},
AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3, ".", {st_mode=S_IFDIR|0755, st_size=4096, ...},
AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents64(4, 0x56522d8acdb0 /* 5 entries */, 32768) = 208
getdents64(4, 0x56522d8acdb0 /* 0 entries */, 32768) = 0
close(4) = 0
newfstatat(AT_FDCWD, "./build_script_build-ed34004021de38b5.d",
{st_mode=S_IFREG|0644, st_size=468, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3, "./build_script_build-ed34004021de38b5.d", {st_mode=S_IFREG|0644,
st_size=468, ...}, 0) = 0
newfstatat(AT_FDCWD, "./build-script-build", {st_mode=S_IFREG|0755,
st_size=4164448, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3, "./build-script-build", {st_mode=S_IFREG|0755, st_size=4164448,
...}, 0) = 0
newfstatat(AT_FDCWD, "./build_script_build-ed34004021de38b5",
{st_mode=S_IFREG|0755, st_size=4164448, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3, "./build_script_build-ed34004021de38b5", {st_mode=S_IFREG|0755,
st_size=4164448, ...}, 0) = 0
linkat(3, "./build-script-build", 3, "./build_script_build-ed34004021de38b5",
0) = -1 EEXIST (File exists)
rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_DROPPABLE|MAP_ANONYMOUS, -1, 0) =
0x7fb6137aa000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7fb6137a9000
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
getrandom("\xb1\x9d\x7d\x12\xaf\x81\xa0\xd0\x95\xb9\x7e\x7f\x6a\xd3\x64\x57\x21\x82\x79\x4c\xa5\xfe\xa0\xbc\x4b\x74\x25\xa0\xc6\x5a\xd1\xa5",
32, 0) = 32
linkat(3, "./build-script-build", 3, "./CuYCWUCU", 0) = 0
renameat(3, "./CuYCWUCU", 3, "./build_script_build-ed34004021de38b5") = 0
unlinkat(3, "./CuYCWUCU", 0) = 0
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2998, ...}) = 0
read(4, "# Locale name alias data base.\n#"..., 4096) = 2998
read(4, "", 4096) = 0
close(4) = 0
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFREG|0644, st_size=7021, ...}) = 0
llistxattr(".", NULL, 0) = 0
llistxattr(".", 0x7fff3e1c89f0, 0) = 0
chmod("/home/kye/test/.", 040755) = 0
llistxattr(".", NULL, 0) = 0
llistxattr(".", 0x7fff3e1c89e0, 0) = 0
lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
close(0) = 0
write(1, "removed '/home/kye/test/./build_"..., 63removed
'/home/kye/test/./build_script_build-ed34004021de38b5'
) = 63
close(1) = 0
close(2) = 0
exit_group(0) = ?
+++ exited with 0 +++
OpenPGP_0xDEF2BA03A6BE3062.asc
Description: OpenPGP public key
OpenPGP_signature.asc
Description: OpenPGP digital signature
