This series adds a --reflink option to tar, which makes extraction clone the file data from the archive with the Linux FICLONERANGE ioctl instead of copying it.
First, the fine print: this only helps on Linux, and only when the archive is a seekable, uncompressed file on the same reflink-capable filesystem (Btrfs, XFS and a few others) as the extracted files. A compressed or piped archive has no plain data extents on disk to share, so the most common .tar.gz case gets no benefit: tar detects these cases and extracts with a regular copy as usual. Within its niche, though, the effect is large. Cloning means the member data is never read nor written, so the extraction time depends on the number of members rather than on the amount of data. Measured best of 3 runs; the page cache is dropped and the fs synced before each run, and the final sync of the extracted data is inside the timed window. The benchmark aborts unless it confirms shared extents with filefrag, so no reported result can silently include a copy fallback. The same archives, repacked with --reflink, extracted with and without it, on two different machines: Intel Xeon server, spinning HDD copy reflink speedup firefox sources 73.6s 56.2s 1.3x linux sources (~90k files) 28.9s 16.4s 1.8x linux-firmware (~400 KB files) 27.8s 6.8s 4.1x 0ad game data (few big files) 53.2s 0.048s ~1100x ARM notebook, NVMe SSD copy reflink speedup firefox sources 18.8s 7.3s 2.6x linux sources (~90k files) 11.4s 4.1s 2.8x linux-firmware (~400 KB files) 3.9s 1.0s 3.9x 0ad game data (few big files) 3.33s 0.129s 25.8x On both, the gain grows with member size, since the amount of copying avoided grows too: the data-bound 0ad archive, dominated by a few big files, wins by orders of magnitude. And since the extracted files share their blocks with the archive, extraction takes no extra disk space at all. Compression does not rule this out. An archive can be shipped aligned and compressed and, once decompressed to a plain file on a reflink-capable filesystem, extracted with --reflink. curl https://.../archive.tar.gz | gunzip > archive.tar tar -xf archive.tar --reflink When decompression keeps up with the link (the CPU out-runs the network most of the time) archive.tar is written at download speed and the extraction adds almost nothing on top, rather than the second full-data pass a plain extraction would do. It is also frugal with space: the pipe never stores the compressed copy on disk, and since the extracted tree shares its blocks with archive.tar, the archive can be kept for free -- the data exists on disk once, not as the separate archive and extracted copies a copying extraction leaves behind. Real-world workloads with large, uncompressed local tar archives are common: game asset bundles (like the 0ad data above), machine- learning dataset shards (e.g. WebDataset), and source trees unpacked repeatedly in CI. Package managers are also adopting copy-on-write installs (RPM CoW [1]; dpkg [2], which I am working on) -- they use their own archive formats, so they share the concept rather than this code, which this series brings to general-purpose tar. On the archive creation side, --reflink aligns the data of every member on a 4 KiB boundary, as required by FICLONERANGE, which needs the data aligned to the filesystem block size (4 KiB being the most common), by padding the pax extended header with a standard "comment" record, which is ignored on extraction (the option implies --format=posix). The resulting archives are ordinary posix archives, fully compatible with any tar implementation. Changes since v1 [3]: - fixed a buffer over-read in the pad record sizing with large extended headers, and integer overflows on 32-bit hosts - the clone is attempted only when it can work (seekable local uncompressed archive, plain destination file), and a failed clone is discarded before falling back to the regular copy - the last member is cloned up to the archive end, so the archive tail no longer needs padding - warn about option combinations that defeat the alignment - added texinfo documentation and a NEWS entry with the expected performance gains - added a second test asserting real extent sharing via filefrag, skipped on filesystems without reflink support - GNU ChangeLog-style commit logs I have completed the FSF copyright assignment for GNU tar. Benchmark script: [4]. A note on XFS: its current kernel implementation makes the post-clone ftruncate of each member expensive, so extracting many small files with --reflink there can be slower than a plain copy. This was analyzed with the XFS maintainers [5] (the cost is the copy-on-write zeroing of the shared tail block on truncate-down). Links: [1] RPM CoW: https://fedoraproject.org/wiki/Changes/RPMCoW [2] dpkg CoW: https://bugs.debian.org/1086976 [3] v1: https://lists.gnu.org/archive/html/bug-tar/2024-10/msg00028.html [4] benchmark script: https://gist.github.com/teknoraver/ef63104c0a6358a0a1ec9caf7c6ffd24 [5] XFS analysis: https://lore.kernel.org/linux-xfs/CAFnufp2NJj=eXCpfoU4ZzPwSX31oUbsUr=kstz6nedtz3ds...@mail.gmail.com/ Regards, -- per aspera ad upstream
0001-reflink-align-data-on-block-size-on-archive-create.patch
Description: Binary data
0002-reflink-add-support-for-reflink-when-extracting.patch
Description: Binary data
0003-reflink-add-documentation.patch
Description: Binary data
0004-reflink-add-unit-tests.patch
Description: Binary data
