Hi hackers,
Currently, pg_basebackup can fail when creating plain-format backups if
a file path or symbolic link target cannot be represented in a
traditional ustar header.
The ustar format has fixed-size fields for member names and symbolic link
targets. Although long member names can be split between the `name` and
`prefix` fields, some paths still cannot be represented. Symbolic link
targets, on the other hand, are limited to 100 bytes.
This can be problematic when backing up tablespaces located in
directories with long paths, resulting in errors such as:
ERROR: symbolic link target too long for tar format:
file name "pg_tblspc/16386", target ".../007_tablespace/data/tblspc_djiS"
This patch adds support for POSIX PAX extended headers to handle these cases.
The implementation has the following changes:
- Add `tarCreateHeaderExt()` and `tarCreateHeaderExtSize()` to generate
tar headers with optional PAX extended headers.
- Prefer the traditional ustar representation whenever possible,
including splitting member names between the `name` and `prefix`
fields, so existing archives remain unchanged when their paths fit.
- Use the PAX `path` and `linkpath` attributes when member names or
symbolic link targets cannot be represented in a ustar header.
- Teach the astreamer tar parser to consume local PAX extended headers
and apply their attributes to the following archive member.
- Reject global PAX extended headers and ignore unknown attributes that
do not affect the properties handled by the parser.
- Update the tar archiver to reconstruct headers when PAX attributes need
to be preserved during re-archiving.
- Update the file extractor and pg_verifybackup to handle longer member
paths.
TAP tests are added to cover:
- Backups containing long file paths.
- Tablespaces with long paths and symbolic link targets.
- Verification of PAX-format backups using pg_verifybackup.
- Extraction of generated archives using an external tar utility.
- Re-archiving when pg_basebackup writes tar output to stdout.
Comments and reviews are welcome.
--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.
>From 04d8874e40fcab53d990b3b05ad810db1ba1c961 Mon Sep 17 00:00:00 2001
From: Japin Li <[email protected]>
Date: Thu, 5 Nov 2026 11:16:34 +0800
Subject: [PATCH v1] Support long tar paths and link targets using PAX
Traditional ustar headers have fixed-size fields for member names and symbolic
link targets. Although the name can be split between the name and prefix
fields, some paths cannot be represented by these fields, and symbolic link
targets are limited to 100 bytes.
This can cause pg_basebackup to fail when backing up tablespaces with long
paths or when the data directory contains files whose paths cannot be
represented in a ustar header.
Add support for POSIX PAX extended headers to handle these cases. When a
member path cannot be represented using the traditional name and prefix
fields, write a PAX "path" attribute. Similarly, when a symbolic link target
exceeds the traditional linkname field, write a PAX "linkpath" attribute.
Both attributes can be included in the same extended header when necessary.
Prefer the traditional ustar representation whenever possible, including
splitting paths between the name and prefix fields. This keeps existing
archives unchanged for members that fit within the traditional format.
Teach the astreamer tar parser to consume local PAX extended headers and apply
their path and linkpath attributes to the following member. Reject global PAX
headers, and ignore unknown PAX attributes that do not affect the properties
handled by the parser.
Update the tar archiver to reconstruct headers when PAX attributes need to be
preserved, so that long paths and link targets are not lost when an archive is
processed and re-archived. Also update the file extractor and pg_verifybackup
to handle longer member paths.
Add TAP tests covering long member paths, long tablespace paths, PAX archive
verification, extraction by an external tar utility, and the re-archiving path
used when pg_basebackup writes tar output to stdout.
---
src/backend/backup/basebackup.c | 82 +++-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 149 +++++-
src/bin/pg_verifybackup/astreamer_verify.c | 19 +-
src/fe_utils/astreamer_file.c | 13 +-
src/fe_utils/astreamer_tar.c | 478 ++++++++++++++++---
src/include/fe_utils/astreamer.h | 5 +
src/include/pgtar.h | 28 ++
src/port/tar.c | 418 +++++++++++++++-
8 files changed, 1080 insertions(+), 112 deletions(-)
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index f8c3520e5b8..18c51886045 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -121,6 +121,7 @@ static void sendFileWithContent(bbsink *sink, const char *filename,
static int64 _tarWriteHeader(bbsink *sink, const char *filename,
const char *linktarget, struct stat *statbuf,
bool sizeonly);
+static int64 _tarGetHeaderSize(const char *filename, const char *linktarget);
static void _tarWritePadding(bbsink *sink, int len);
static void convert_link_to_directory(const char *pathbuf, struct stat *statbuf);
static void perform_base_backup(basebackup_options *opt, bbsink *sink,
@@ -1559,8 +1560,8 @@ sendDir(bbsink *sink, const char *path, int basepathlen, bool sizeonly,
/* Pad to a multiple of the tar block size. */
size += tarPaddingBytesRequired(statbuf.st_size);
- /* Size of the header for the file. */
- size += TAR_BLOCK_SIZE;
+ /* Size of the header for the file, including PAX if needed. */
+ size += _tarGetHeaderSize(tarfilename, NULL);
}
}
else
@@ -2124,9 +2125,14 @@ _tarWriteHeader(bbsink *sink, const char *filename, const char *linktarget,
struct stat *statbuf, bool sizeonly)
{
enum tarError rc;
+ size_t hdrsize = _tarGetHeaderSize(filename, linktarget);
if (!sizeonly)
{
+ char *header;
+ size_t written;
+ bool allocated = false;
+
/*
* As of this writing, the smallest supported block size is 1kB, which
* is twice TAR_BLOCK_SIZE. Since the buffer size is required to be a
@@ -2138,14 +2144,28 @@ _tarWriteHeader(bbsink *sink, const char *filename, const char *linktarget,
"BLCKSZ too small for tar block");
Assert(sink->bbs_buffer_length >= TAR_BLOCK_SIZE);
- rc = tarCreateHeader(sink->bbs_buffer, filename, linktarget,
- statbuf->st_size, statbuf->st_mode,
- statbuf->st_uid, statbuf->st_gid,
- statbuf->st_mtime);
+ /*
+ * A PAX extended header and its payload may be larger than the sink
+ * buffer, so allocate a temporary buffer in that case.
+ */
+ if (hdrsize <= sink->bbs_buffer_length)
+ header = sink->bbs_buffer;
+ else
+ {
+ header = palloc(hdrsize);
+ allocated = true;
+ }
+
+ rc = tarCreateHeaderExt(header, hdrsize, filename, linktarget,
+ statbuf->st_size, statbuf->st_mode,
+ statbuf->st_uid, statbuf->st_gid,
+ statbuf->st_mtime, &written);
+
switch (rc)
{
case TAR_OK:
+ Assert(written == hdrsize);
break;
case TAR_NAME_TOO_LONG:
ereport(ERROR,
@@ -2160,14 +2180,60 @@ _tarWriteHeader(bbsink *sink, const char *filename, const char *linktarget,
"file name \"%s\", target \"%s\"",
filename, linktarget)));
break;
+ case TAR_BUFFER_TOO_SMALL:
+ elog(ERROR, "tar header buffer is too small");
+ break;
default:
elog(ERROR, "unrecognized tar error: %d", rc);
}
- bbsink_archive_contents(sink, TAR_BLOCK_SIZE);
+ if (!allocated)
+ bbsink_archive_contents(sink, hdrsize);
+ else
+ {
+ size_t done = 0;
+
+ while (done < hdrsize)
+ {
+ size_t nbytes = Min(hdrsize - done, sink->bbs_buffer_length);
+
+ memcpy(sink->bbs_buffer, header + done, nbytes);
+ bbsink_archive_contents(sink, nbytes);
+ done += nbytes;
+ }
+
+ pfree(header);
+ }
+ }
+
+ return (int64) hdrsize;
+}
+
+/*
+ * Compute the size of the tar header for a member, including the PAX extended
+ * header and its padding when onne is required.
+ */
+static int64
+_tarGetHeaderSize(const char *filename, const char *linktarget)
+{
+ size_t hdrsize = tarCreateHeaderExtSize(filename, linktarget);
+
+ if (hdrsize == 0)
+ {
+ if (linktarget != NULL && strlen(linktarget) >= MAXPGPATH)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("symbolic link target too long for tar format: "
+ "file name \"%s\", target \"%s\"",
+ filename, linktarget)));
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("file name too long for tar format: \"%s\"",
+ filename)));
}
- return TAR_BLOCK_SIZE;
+ return (int64) hdrsize;
}
/*
diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
index 23a757b1c86..074f0d102af 100644
--- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl
+++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl
@@ -5,7 +5,7 @@ use strict;
use warnings FATAL => 'all';
use Config;
use File::Basename qw(basename dirname);
-use File::Path qw(rmtree);
+use File::Path qw(make_path rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
@@ -380,24 +380,64 @@ $node->command_fails_like(
'--tablespace-mapping with invalid format fails');
my $superlongname = "superlongname_" . ("x" x 100);
-# Tar format doesn't support filenames longer than 100 bytes.
+# Tar format supports long filenames using PAX extended headers.
SKIP:
{
my $superlongpath = "$pgdata/$superlongname";
- skip "File path too long", 1
+ skip "File path too long", 4
if $windows_os && length($superlongpath) > 255;
open my $file, '>', "$superlongpath"
or die "unable to create file $superlongpath";
close $file;
- $node->command_fails(
+ $node->command_ok(
[
@pg_basebackup_defs,
'--pgdata' => "$tempdir/tarbackup_l1",
'--format' => 'tar'
],
- 'pg_basebackup tar with long name fails');
+ 'pg_basebackup tar with long name succeeds');
+ rmtree("$tempdir/tarbackup_l1");
+
+ # Tar output to stdout forces pg_basebackup to parse and re-archive the
+ # stream (to inject the manifest). This exercises the PAX archiver path.
+ SKIP:
+ {
+ my $tar = $ENV{TAR};
+
+ skip "no tar program available", 3
+ if (!defined $tar || $tar eq '');
+
+ my $stdout_tar = "$tempdir/long_name_stdout.tar";
+ my $stdout_extract = "$tempdir/long_name_stdout";
+ my $stderr = '';
+ my $result;
+
+ $result = IPC::Run::run(
+ [
+ @pg_basebackup_defs,
+ '--pgdata' => '-',
+ '--format' => 'tar',
+ '--wal-method' => 'none',
+ '--dbname' => $node->connstr('postgres')
+ ],
+ '>', $stdout_tar,
+ '2>', \$stderr);
+ ok($result,
+ 'pg_basebackup tar to stdout with long name succeeds');
+ diag("pg_basebackup stderr: $stderr") unless $result;
+ mkdir($stdout_extract)
+ or BAIL_OUT("could not create $stdout_extract");
+ command_ok(
+ [ $tar, '-xf', $stdout_tar, '-C', $stdout_extract ],
+ 'external tar extracts re-archived PAX archive');
+ ok(-f "$stdout_extract/$superlongname",
+ 'long name preserved through re-archiving');
+ rmtree($stdout_extract);
+ unlink($stdout_tar);
+ }
+
unlink "$superlongpath";
}
@@ -429,7 +469,7 @@ $node->start;
# Test backup of a tablespace using tar format.
# Symlink the system located tempdir to our physical temp location.
# That way we can use shorter names for the tablespace directories,
-# which hopefully won't run afoul of the 99 character length limit.
+# which makes the resulting archives easier to inspect.
my $real_sys_tempdir = "$sys_tempdir/tempdir";
dir_symlink "$tempdir", $real_sys_tempdir;
@@ -1104,4 +1144,101 @@ $node2->command_fails_like(
qr/system identifier in backup manifest is .*, but database system identifier is/,
"pg_basebackup fails with different database system manifest");
+# Test support for long tablespace paths and long archive member paths. Both
+# require PAX extended headers, because they exceed the traditional ustar
+# name/prefix/linkname limits.
+my $long_ts_dir = "$tempdir/" . ('a' x 100);
+
+mkdir($long_ts_dir)
+ or BAIL_OUT("could not create $long_ts_dir");
+$node->safe_psql('postgres',
+ "CREATE TABLESPACE tblspc_long LOCATION '$long_ts_dir';");
+$node->safe_psql('postgres',
+ "CREATE TABLE test_long_ts (a int) TABLESPACE tblspc_long;"
+ . "INSERT INTO test_long_ts VALUES (1234);");
+my $long_ts_oid = $node->safe_psql('postgres',
+ "SELECT oid from pg_tablespace WHERE spcname = 'tblspc_long';");
+
+# Create a file whose relative path is much longer than the ustar limit.
+my $long_relpath = join('/', map { $_ x 80 } qw(d e f h));
+my $long_filename = 'g' x 100;
+my $long_dir = "$pgdata/$long_relpath";
+
+make_path($long_dir)
+ or BAIL_OUT("could not create $long_dir");
+append_to_file("$long_dir/$long_filename", "long path contents\n");
+
+# Plain format: the long tablespace symlink target and the long path must both
+# be restored intact.
+my $long_plain_backup = "$tempdir/long_plain_backup";
+my $long_map_dir = "$tempdir/long_tblspc";
+
+rmtree($long_map_dir);
+rmtree($long_plain_backup);
+$node->command_ok(
+ [
+ @pg_basebackup_defs,
+ '--pgdata' => $long_plain_backup,
+ '--format' => 'plain',
+ '--tablespace-mapping' => "$long_ts_dir=$long_map_dir",
+ ],
+ 'plain backup with long tablespace path and long file path');
+
+ok(-f "$long_plain_backup/$long_relpath/$long_filename",
+ 'long file path restored from plain backup');
+is(readlink("$long_plain_backup/pg_tblspc/$long_ts_oid"),
+ $long_map_dir,
+ 'long tablespace symlink target restored');
+
+# Tar format: the long path must be represented with a PAX extended header,
+# and must survive both our own parser and an external tar implementation.
+my $long_tar_backup = "$tempdir/long_tar_backup";
+
+rmtree($long_tar_backup);
+$node->command_ok(
+ [
+ @pg_basebackup_defs,
+ '--pgdata' => $long_tar_backup,
+ '--format' => 'tar',
+ ],
+ 'tar backup with long file path');
+ok(-f "$long_tar_backup/base.tar", 'long-path tar backup was created');
+
+command_ok(
+ [
+ 'pg_verifybackup', '-F', 't',
+ '-m', "$long_tar_backup/backup_manifest",
+ '-n', $long_tar_backup
+ ],
+ 'pg_verifybackup verifies a PAX tar backup');
+
+SKIP:
+{
+ my $tar = $ENV{TAR};
+
+ skip "no tar program available", 1
+ if (!defined $tar || $tar eq '');
+
+ my $long_tar_extract = "$tempdir/long_tar_extract";
+
+ rmtree($long_tar_extract);
+ mkdir($long_tar_extract)
+ or BAIL_OUT("could not create $long_tar_extract");
+ command_ok(
+ [ $tar, '-xf', "$long_tar_backup/base.tar", '-C', $long_tar_extract ],
+ 'external tar extracts PAX base archive');
+ ok(-f "$long_tar_extract/$long_relpath/$long_filename",
+ 'long file path restored from tar backup');
+ rmtree($long_tar_extract);
+}
+
+# Clean up.
+$node->safe_psql('postgres',
+ "DROP TABLE test_long_ts; DROP TABLESPACE tblspc_long;");
+rmtree($long_dir);
+rmtree($long_ts_dir);
+rmtree($long_map_dir);
+rmtree($long_plain_backup);
+rmtree($long_tar_backup);
+
done_testing();
diff --git a/src/bin/pg_verifybackup/astreamer_verify.c b/src/bin/pg_verifybackup/astreamer_verify.c
index 2578765b0ca..c5d7b579fff 100644
--- a/src/bin/pg_verifybackup/astreamer_verify.c
+++ b/src/bin/pg_verifybackup/astreamer_verify.c
@@ -162,7 +162,8 @@ member_verify_header(astreamer *streamer, astreamer_member *member)
{
astreamer_verify *mystreamer = (astreamer_verify *) streamer;
manifest_file *m;
- char pathname[MAXPGPATH];
+ char pathname[MAXPGPATH + 32];
+ int pathlen;
/* We are only interested in normal files. */
if (!member->is_regular)
@@ -180,10 +181,20 @@ member_verify_header(astreamer *streamer, astreamer_member *member)
* to make sure that canonicalize_path() does what we want.
*/
if (OidIsValid(mystreamer->tblspc_oid))
- snprintf(pathname, MAXPGPATH, "%s/%u/%s",
- "pg_tblspc", mystreamer->tblspc_oid, member->pathname);
+ pathlen = snprintf(pathname, sizeof(pathname), "%s/%u/%s", "pg_tblspc",
+ mystreamer->tblspc_oid, member->pathname);
else
- snprintf(pathname, MAXPGPATH, "./%s", member->pathname);
+ pathlen = snprintf(pathname, sizeof(pathname), "./%s",
+ member->pathname);
+
+ if (pathlen < 0 || pathlen >= (int) sizeof(pathname))
+ {
+ report_backup_error(mystreamer->context,
+ "file name \"%s\" in archive \"%s\" is too long",
+ member->pathname, mystreamer->archive_name);
+ return;
+ }
+
canonicalize_path(pathname);
/* Ignore any files that are listed in the ignore list. */
diff --git a/src/fe_utils/astreamer_file.c b/src/fe_utils/astreamer_file.c
index fb36cecc22a..5d0a85b4797 100644
--- a/src/fe_utils/astreamer_file.c
+++ b/src/fe_utils/astreamer_file.c
@@ -35,7 +35,7 @@ typedef struct astreamer_extractor
char *basepath;
const char *(*link_map) (const char *);
void (*report_output_file) (const char *);
- char filename[MAXPGPATH];
+ char *filename;
FILE *file;
} astreamer_extractor;
@@ -223,12 +223,15 @@ astreamer_extractor_content(astreamer *streamer, astreamer_member *member,
member->pathname);
/* Prepend basepath. */
- snprintf(mystreamer->filename, sizeof(mystreamer->filename),
- "%s/%s", mystreamer->basepath, member->pathname);
+ if (mystreamer->filename != NULL)
+ pfree(mystreamer->filename);
+ mystreamer->filename = psprintf("%s/%s",
+ mystreamer->basepath,
+ member->pathname);
/* Remove any trailing slash. */
fnamelen = strlen(mystreamer->filename);
- if (mystreamer->filename[fnamelen - 1] == '/')
+ if (fnamelen > 0 && mystreamer->filename[fnamelen - 1] == '/')
mystreamer->filename[fnamelen - 1] = '\0';
/* Dispatch based on file type. */
@@ -409,5 +412,7 @@ astreamer_extractor_free(astreamer *streamer)
astreamer_extractor *mystreamer = (astreamer_extractor *) streamer;
pfree(mystreamer->basepath);
+ if (mystreamer->filename)
+ pfree(mystreamer->filename);
pfree(mystreamer);
}
diff --git a/src/fe_utils/astreamer_tar.c b/src/fe_utils/astreamer_tar.c
index 8ab59e92238..0b85185faa6 100644
--- a/src/fe_utils/astreamer_tar.c
+++ b/src/fe_utils/astreamer_tar.c
@@ -27,13 +27,30 @@
#include "fe_utils/astreamer.h"
#include "pgtar.h"
+typedef enum
+{
+ TAR_PARSE_MEMBER_HEADER,
+ TAR_PARSE_PAX_PAYLOAD,
+ TAR_PARSE_PAX_PADDING,
+ TAR_PARSE_MEMBER_CONTENTS,
+ TAR_PARSE_MEMBER_TRAILER,
+ TAR_PARSE_ARCHIVE_TRAILER,
+} tar_parser_state;
+
typedef struct astreamer_tar_parser
{
astreamer base;
- astreamer_archive_context next_context;
+ tar_parser_state next_context;
astreamer_member member;
size_t file_bytes_sent;
size_t pad_bytes_expected;
+ size_t pax_bytes_expected;
+ size_t pax_pad_bytes_expected;
+ bool pax_pending;
+ bool pax_has_path;
+ bool pax_has_linkpath;
+ char pax_path[MAXPGPATH];
+ char pax_linkpath[MAXPGPATH];
} astreamer_tar_parser;
typedef struct astreamer_tar_archiver
@@ -42,13 +59,24 @@ typedef struct astreamer_tar_archiver
bool rearchive_member;
} astreamer_tar_archiver;
+typedef enum
+{
+ TAR_HEADER_MEMBER,
+ TAR_HEADER_PAX,
+ TAR_HEADER_ARCHIVE_END,
+} tar_header_result;
+
static void astreamer_tar_parser_content(astreamer *streamer,
astreamer_member *member,
const char *data, int len,
astreamer_archive_context context);
static void astreamer_tar_parser_finalize(astreamer *streamer);
static void astreamer_tar_parser_free(astreamer *streamer);
-static bool astreamer_tar_header(astreamer_tar_parser *mystreamer);
+static tar_header_result astreamer_tar_header(astreamer_tar_parser *mystreamer);
+static void astreamer_tar_parse_pax(astreamer_tar_parser *mystreamer,
+ const char *data, size_t len);
+static void tarReadStringField(char *dst, size_t dstsize,
+ const char *src, size_t srclen);
static const astreamer_ops astreamer_tar_parser_ops = {
.content = astreamer_tar_parser_content,
@@ -99,7 +127,7 @@ astreamer_tar_parser_new(astreamer *next)
&astreamer_tar_parser_ops;
streamer->base.bbs_next = next;
initStringInfo(&streamer->base.bbs_buffer);
- streamer->next_context = ASTREAMER_MEMBER_HEADER;
+ streamer->next_context = TAR_PARSE_MEMBER_HEADER;
return &streamer->base;
}
@@ -123,48 +151,105 @@ astreamer_tar_parser_content(astreamer *streamer, astreamer_member *member,
{
switch (mystreamer->next_context)
{
- case ASTREAMER_MEMBER_HEADER:
+ case TAR_PARSE_MEMBER_HEADER:
+ {
+ tar_header_result result;
+
+ /*
+ * If we're expecting an archive member header, accumulate
+ * a full block of data before doing anything futher.
+ */
+ if (!astreamer_buffer_until(streamer, &data, &len,
+ TAR_BLOCK_SIZE))
+ return;
+
+ /*
+ * Now we can process the header and get ready to process
+ * the file contents; however, we might find out that what
+ * we thought was the next file header is actually the
+ * start of the archive trailer. Alternatively, it might
+ * be a PAX extended header, in which case we need to read
+ * its payload before the actual member header.
+ */
+ result = astreamer_tar_header(mystreamer);
+ if (result == TAR_HEADER_ARCHIVE_END)
+ {
+ if (mystreamer->pax_pending)
+ pg_fatal("tar archive ended while a pax extended header was pending");
+ mystreamer->next_context = TAR_PARSE_ARCHIVE_TRAILER;
+ }
+ else if (result == TAR_HEADER_PAX)
+ {
+ mystreamer->base.bbs_buffer.len = 0;
+ mystreamer->file_bytes_sent = 0;
+ mystreamer->next_context = TAR_PARSE_PAX_PAYLOAD;
+ }
+ else
+ {
+ if (mystreamer->member.size == 0)
+ {
+ /* No content; trailer is zero-length. */
+ astreamer_content(mystreamer->base.bbs_next,
+ &mystreamer->member,
+ NULL, 0,
+ ASTREAMER_MEMBER_TRAILER);
+
+ /* Expect next header. */
+ mystreamer->next_context = TAR_PARSE_MEMBER_HEADER;
+ }
+ else
+ {
+ /* Expect contents. */
+ mystreamer->next_context = TAR_PARSE_MEMBER_CONTENTS;
+ }
+ mystreamer->base.bbs_buffer.len = 0;
+ mystreamer->file_bytes_sent = 0;
+ }
+ }
+ break;
- /*
- * If we're expecting an archive member header, accumulate a
- * full block of data before doing anything further.
- */
+ case TAR_PARSE_PAX_PAYLOAD:
+
+ /* Accumulate the complete PAX extended header payload. */
if (!astreamer_buffer_until(streamer, &data, &len,
- TAR_BLOCK_SIZE))
+ mystreamer->pax_bytes_expected))
return;
- /*
- * Now we can process the header and get ready to process the
- * file contents; however, we might find out that what we
- * thought was the next file header is actually the start of
- * the archive trailer. Switch modes accordingly.
- */
- if (astreamer_tar_header(mystreamer))
+ Assert(mystreamer->base.bbs_buffer.len ==
+ mystreamer->pax_bytes_expected);
+
+ /* Parse the records and remember path/linkpath overrides. */
+ astreamer_tar_parse_pax(mystreamer,
+ mystreamer->base.bbs_buffer.data,
+ mystreamer->base.bbs_buffer.len);
+
+ mystreamer->base.bbs_buffer.len = 0;
+
+ if (mystreamer->pax_pad_bytes_expected > 0)
+ mystreamer->next_context = TAR_PARSE_PAX_PADDING;
+ else
+ mystreamer->next_context = TAR_PARSE_MEMBER_HEADER;
+ break;
+
+ case TAR_PARSE_PAX_PADDING:
{
- if (mystreamer->member.size == 0)
- {
- /* No content; trailer is zero-length. */
- astreamer_content(mystreamer->base.bbs_next,
- &mystreamer->member,
- NULL, 0,
- ASTREAMER_MEMBER_TRAILER);
+ /* Consume the NUL padding that follows the PAX payload. */
+ if (!astreamer_buffer_until(streamer, &data, &len,
+ mystreamer->pax_pad_bytes_expected))
+ return;
- /* Expect next header. */
- mystreamer->next_context = ASTREAMER_MEMBER_HEADER;
- }
- else
+ for (size_t i = 0; i < mystreamer->pax_pad_bytes_expected; ++i)
{
- /* Expect contents. */
- mystreamer->next_context = ASTREAMER_MEMBER_CONTENTS;
+ if (mystreamer->base.bbs_buffer.data[i] != '\0')
+ pg_fatal("invalid padding after pax extended header");
}
+
mystreamer->base.bbs_buffer.len = 0;
- mystreamer->file_bytes_sent = 0;
+ mystreamer->next_context = TAR_PARSE_MEMBER_HEADER;
}
- else
- mystreamer->next_context = ASTREAMER_ARCHIVE_TRAILER;
break;
- case ASTREAMER_MEMBER_CONTENTS:
+ case TAR_PARSE_MEMBER_CONTENTS:
/*
* Send as much content as we have, but not more than the
@@ -199,18 +284,18 @@ astreamer_tar_parser_content(astreamer *streamer, astreamer_member *member,
ASTREAMER_MEMBER_TRAILER);
/* Expect next header. */
- mystreamer->next_context = ASTREAMER_MEMBER_HEADER;
+ mystreamer->next_context = TAR_PARSE_MEMBER_HEADER;
}
else
{
/* Trailer is not zero-length. */
- mystreamer->next_context = ASTREAMER_MEMBER_TRAILER;
+ mystreamer->next_context = TAR_PARSE_MEMBER_TRAILER;
}
mystreamer->base.bbs_buffer.len = 0;
}
break;
- case ASTREAMER_MEMBER_TRAILER:
+ case TAR_PARSE_MEMBER_TRAILER:
/*
* If we're expecting an archive member trailer, accumulate
@@ -229,11 +314,11 @@ astreamer_tar_parser_content(astreamer *streamer, astreamer_member *member,
ASTREAMER_MEMBER_TRAILER);
/* Expect next file header. */
- mystreamer->next_context = ASTREAMER_MEMBER_HEADER;
+ mystreamer->next_context = TAR_PARSE_MEMBER_HEADER;
mystreamer->base.bbs_buffer.len = 0;
break;
- case ASTREAMER_ARCHIVE_TRAILER:
+ case TAR_PARSE_ARCHIVE_TRAILER:
/*
* We've seen an end-of-archive indicator, so anything more is
@@ -256,20 +341,135 @@ astreamer_tar_parser_content(astreamer *streamer, astreamer_member *member,
}
}
+/*
+ * Parse the payload of a PAX extended header ('x') member.
+ *
+ * Records have the form "<length> <key>=<value>\n", where <length> counts the
+ * entire record including the length digits and the trailing newline. Unknown
+ * keys are ignored, as required by POSIX. If a key is repeated, the last
+ * occurrence wins.
+ */
+static void
+astreamer_tar_parse_pax(astreamer_tar_parser *mystreamer, const char *data,
+ size_t len)
+{
+ size_t pos = 0;
+
+ while (pos < len)
+ {
+ size_t start = pos;
+ size_t reclen = 0;
+ size_t keylen;
+ size_t valuelen;
+ const char *key;
+ const char *value;
+ const char *eq;
+ const char *rec_end;
+
+ /* parse the record length */
+ while (pos < len && data[pos] >= '0' && data[pos] <= '9')
+ {
+ size_t digit = data[pos] - '0';
+
+ if (reclen > (SIZE_MAX - digit) / 10)
+ pg_fatal("invalid pax extended header: record length overflow");
+ reclen = reclen * 10 + digit;
+ pos++;
+ }
+
+ if (pos == start || pos >= len || data[pos] != ' ')
+ pg_fatal("invalid pax extended header record");
+
+ if (reclen == 0 || reclen > len - start)
+ pg_fatal("invalid pax extended header: record length exceeds payload");
+
+ if (data[start + reclen - 1] != '\n')
+ pg_fatal("invalid pax extended header: record does not end with newline");
+
+ rec_end = data + start + reclen;
+
+ /* the key starts after the space following the length */
+ key = data + pos + 1;
+ if ((size_t) (rec_end - 1 - key) < 2)
+ pg_fatal("invalid pax extended header: malformed key/value record");
+ eq = memchr(key, '=', (rec_end - 1) - key);
+ if (eq == NULL || eq == key)
+ pg_fatal("invalid pax extended header: malformed key/value record");
+
+ keylen = eq - key;
+
+ /* neither the key or the value may contain an embedded NUL */
+ if (memchr(key, '\0', keylen) != NULL)
+ pg_fatal("invalid pax extended header: NUL in key");
+
+ value = eq + 1;
+ valuelen = (rec_end - 1) - value;
+
+ if (memchr(value, '\0', valuelen) != NULL)
+ pg_fatal("invalid pax extended header: NUL in value");
+
+ if (keylen == 4 && memcmp(key, "path", 4) == 0)
+ {
+ if (valuelen >= MAXPGPATH)
+ pg_fatal("pax extended header path is too long");
+
+ memcpy(mystreamer->pax_path, value, valuelen);
+ mystreamer->pax_path[valuelen] = '\0';
+ mystreamer->pax_has_path = true;
+ }
+ else if (keylen == 8 && memcmp(key, "linkpath", 8) == 0)
+ {
+ if (valuelen >= MAXPGPATH)
+ pg_fatal("pax extended header linkpath is too long");
+
+ memcpy(mystreamer->pax_linkpath, value, valuelen);
+ mystreamer->pax_linkpath[valuelen] = '\0';
+ mystreamer->pax_has_linkpath = true;
+ }
+ else
+ pg_log_debug("ignoring unknown pax extended header key \"%.*s\"",
+ (int) keylen, key);
+
+ pos = start + reclen;
+ }
+}
+
+/*
+ * Copy a fixed-size string field from a tar header. The field is not
+ * guaranteed to be NUL-terminated, and may be followed immediately by other
+ * header fields, so we must never scan past its declared length.
+ */
+static void
+tarReadStringField(char *dst, size_t dstsize, const char *src, size_t srclen)
+{
+ const char *nul = memchr(src, '\0', srclen);
+ size_t len = nul ? (size_t) (nul - src) : srclen;
+
+ if (len >= dstsize)
+ pg_fatal("tar member field is too long");
+
+ memcpy(dst, src, len);
+ dst[len] = '\0';
+}
+
/*
* Parse a file header within a tar stream.
*
- * The return value is true if we found a file header and passed it on to the
- * next astreamer; it is false if we have found the archive trailer.
+ * We pass member headers on to the next astreamer. A PAX extended header
+ * ('x') is consumed internally, and its path/linkpath attributes are applied
+ * to the member that follows it. Global PAX headers ('g') are rejected.
* We throw error if we see invalid data.
*/
-static bool
+static tar_header_result
astreamer_tar_header(astreamer_tar_parser *mystreamer)
{
bool has_nonzero_byte = false;
int i;
astreamer_member *member = &mystreamer->member;
char *buffer = mystreamer->base.bbs_buffer.data;
+ char prefix[TAR_PREFIX_FIELD_SIZE + 1];
+ char typeflag;
+ bool pax_applied;
Assert(mystreamer->base.bbs_buffer.len == TAR_BLOCK_SIZE);
@@ -291,7 +491,7 @@ astreamer_tar_header(astreamer_tar_parser *mystreamer)
* start of the next file.
*/
if (!has_nonzero_byte)
- return false;
+ return TAR_HEADER_ARCHIVE_END;
/*
* Verify that we have a reasonable-looking header.
@@ -299,10 +499,69 @@ astreamer_tar_header(astreamer_tar_parser *mystreamer)
if (!isValidTarHeader(buffer))
pg_fatal("input file does not appear to be a valid tar archive");
+ typeflag = buffer[TAR_OFFSET_TYPEFLAG];
+
+ /*
+ * A PAX extended header describes the member that follows it. Read and
+ * parse its payload in the caller's state machine.
+ */
+ if (typeflag == TAR_FILETYPE_PAX_EXTENDED)
+ {
+ uint64 paxsize;
+
+ if (mystreamer->pax_pending)
+ pg_fatal("consecutive pax extended headers are not supported");
+
+ paxsize = read_tar_number(&buffer[TAR_OFFSET_SIZE], 12);
+ if (paxsize == 0)
+ pg_fatal("pax extended header has zero size");
+ if (paxsize > TAR_MAX_PAX_PAYLOAD)
+ pg_fatal("pax extended header is too large");
+
+ mystreamer->pax_bytes_expected = (size_t) paxsize;
+ mystreamer->pax_pad_bytes_expected =
+ tarPaddingBytesRequired(mystreamer->pax_bytes_expected);
+ mystreamer->pax_pending = true;
+ mystreamer->pax_has_path = false;
+ mystreamer->pax_has_linkpath = false;
+ mystreamer->pax_path[0] = '\0';
+ mystreamer->pax_linkpath[0] = '\0';
+
+ return TAR_HEADER_PAX;
+ }
+ else if (typeflag == TAR_FILETYPE_PAX_EXTENDED_GLOBAL)
+ pg_fatal("global pax extended headers are not supported");
+
/*
- * Parse key fields out of the header.
+ * Parse key fields out of the header. The ustar prefix field, when
+ * present, is prepended to the name field. Both fields are fixed-width
+ * and need not be NUL-terminated.
*/
- strlcpy(member->pathname, &buffer[TAR_OFFSET_NAME], MAXPGPATH);
+ tarReadStringField(member->pathname, MAXPGPATH,
+ &buffer[TAR_OFFSET_NAME], TAR_NAME_FIELD_SIZE);
+ tarReadStringField(prefix, sizeof(prefix),
+ &buffer[TAR_OFFSET_PREFIX], TAR_PREFIX_FIELD_SIZE);
+
+ if (prefix[0] != '\0')
+ {
+ char fullpath[MAXPGPATH];
+ int pathlen;
+
+ /*
+ * Build the combined path in a separate buffer: snprintf() must not
+ * have the same buffer as both an input and the output.
+ */
+ pathlen = snprintf(fullpath, sizeof(fullpath), "%s/%s", prefix,
+ member->pathname);
+ if (pathlen < 0 || pathlen >= MAXPGPATH)
+ pg_fatal("tar member path is too long");
+ strlcpy(member->pathname, fullpath, MAXPGPATH);
+ }
+
+ /* PAX path, when present, overrides the traditional fields. */
+ if (mystreamer->pax_pending && mystreamer->pax_has_path)
+ strlcpy(member->pathname, mystreamer->pax_path, MAXPGPATH);
+
if (member->pathname[0] == '\0')
pg_fatal("tar member has empty name");
if (!path_is_safe_for_extraction(member->pathname))
@@ -314,7 +573,9 @@ astreamer_tar_header(astreamer_tar_parser *mystreamer)
member->uid = read_tar_number(&buffer[TAR_OFFSET_UID], 8);
member->gid = read_tar_number(&buffer[TAR_OFFSET_GID], 8);
- switch (buffer[TAR_OFFSET_TYPEFLAG])
+ pax_applied = mystreamer->pax_pending;
+
+ switch (typeflag)
{
case TAR_FILETYPE_PLAIN:
case TAR_FILETYPE_PLAIN_OLD:
@@ -325,26 +586,46 @@ astreamer_tar_header(astreamer_tar_parser *mystreamer)
break;
case TAR_FILETYPE_SYMLINK:
member->is_symlink = true;
- strlcpy(member->linktarget, &buffer[TAR_OFFSET_LINKNAME], 100);
- break;
- case TAR_FILETYPE_PAX_EXTENDED:
- case TAR_FILETYPE_PAX_EXTENDED_GLOBAL:
- pg_fatal("pax extensions to tar format are not supported");
+ tarReadStringField(member->linktarget, MAXPGPATH,
+ &buffer[TAR_OFFSET_LINKNAME],
+ TAR_LINKNAME_FIELD_SIZE);
+
+ /* PAX linkpath, when present, overrides the linkname field. */
+ if (mystreamer->pax_pending && mystreamer->pax_has_linkpath)
+ strlcpy(member->linktarget, mystreamer->pax_linkpath, MAXPGPATH);
break;
default:
/* For special filetypes, set none of the three is_xxx flags */
break;
}
+ /*
+ * Whether or not we used them, PAX attributes apply only to this member.
+ */
+ mystreamer->pax_pending = false;
+ mystreamer->pax_has_path = false;
+ mystreamer->pax_has_linkpath = false;
+
/* Compute number of padding bytes. */
mystreamer->pad_bytes_expected = tarPaddingBytesRequired(member->size);
- /* Forward the entire header to the next astreamer. */
- astreamer_content(mystreamer->base.bbs_next, member,
- buffer, TAR_BLOCK_SIZE,
- ASTREAMER_MEMBER_HEADER);
-
- return true;
+ /*
+ * If a PAX header was in effect, the traditional header block may contain
+ * onnly a placeholder for the real path or link target. Do not forward
+ * it; telling the next astreamer that no header block is available forces
+ * anyone who needs to emit a tar stream (e.g. astreamer_tar_archiver) to
+ * reconstruct it from member.
+ */
+ if (pax_applied)
+ astreamer_content(mystreamer->base.bbs_next, member,
+ NULL, 0,
+ ASTREAMER_MEMBER_HEADER);
+ else
+ astreamer_content(mystreamer->base.bbs_next, member,
+ buffer, TAR_BLOCK_SIZE,
+ ASTREAMER_MEMBER_HEADER);
+
+ return TAR_HEADER_MEMBER;
}
/*
@@ -355,8 +636,11 @@ astreamer_tar_parser_finalize(astreamer *streamer)
{
astreamer_tar_parser *mystreamer = (astreamer_tar_parser *) streamer;
- if (mystreamer->next_context != ASTREAMER_ARCHIVE_TRAILER &&
- (mystreamer->next_context != ASTREAMER_MEMBER_HEADER ||
+ if (mystreamer->pax_pending)
+ pg_fatal("COPY stream ended while a pax extended header was pending");
+
+ if (mystreamer->next_context != TAR_PARSE_ARCHIVE_TRAILER &&
+ (mystreamer->next_context != TAR_PARSE_MEMBER_HEADER ||
mystreamer->base.bbs_buffer.len > 0))
pg_fatal("COPY stream ended before last file was finished");
@@ -405,9 +689,11 @@ astreamer_tar_archiver_new(astreamer *next)
* Fix up the stream of input chunks to create a valid tar file.
*
* If a ASTREAMER_MEMBER_HEADER chunk is of size 0, it is replaced with a
- * newly-constructed tar header. If it is of size TAR_BLOCK_SIZE, it is
- * passed through without change. Any other size is a fatal error (and
- * indicates a bug).
+ * newly-constructed tar header. A header is also reconstructed when the
+ * member's path or link target requires a PAX extended header; this happens
+ * when the input stream used PAX and the parser supplied the real values via
+ * the astreamer_member. Otherwise, a TAR_BLOCK_SIZE header is passed through
+ * without change. Any other size is a fatal error (and indicates a bug).
*
* Whenever a new ASTREAMER_MEMBER_HEADER chunk is constructed, the
* corresponding ASTREAMER_MEMBER_TRAILER chunk is also constructed from
@@ -430,22 +716,65 @@ astreamer_tar_archiver_content(astreamer *streamer,
{
astreamer_tar_archiver *mystreamer = (astreamer_tar_archiver *) streamer;
char buffer[2 * TAR_BLOCK_SIZE];
+ char *allocated = NULL;
Assert(context != ASTREAMER_UNKNOWN);
- if (context == ASTREAMER_MEMBER_HEADER && len != TAR_BLOCK_SIZE)
+ if (context == ASTREAMER_MEMBER_HEADER)
{
- Assert(len == 0);
-
- /* Replace zero-length tar header with a newly constructed one. */
- tarCreateHeader(buffer, member->pathname, NULL,
- member->size, member->mode, member->uid, member->gid,
- time(NULL));
- data = buffer;
- len = TAR_BLOCK_SIZE;
-
- /* Also make a note to replace padding, in case size changed. */
- mystreamer->rearchive_member = true;
+ const char *linktarget;
+ size_t hdrsize;
+ bool constructed = (len != TAR_BLOCK_SIZE);
+
+ Assert(len == 0 || len == TAR_BLOCK_SIZE);
+
+ linktarget = member->is_symlink ? member->linktarget : NULL;
+ hdrsize = tarCreateHeaderExtSize(member->pathname, linktarget);
+ if (hdrsize == 0)
+ pg_fatal("tar member name or link target is too long: \"%s\"",
+ member->pathname);
+
+ /*
+ * We must replace the header not only when the input did not have one,
+ * but also when the member's full path or link target requires a PAX
+ * extended header. The latter happens when the input archive used PAX
+ * and the parser supplied the long values via 'member' while the
+ * original header block contains only a placeholder.
+ */
+ if (constructed || hdrsize != TAR_BLOCK_SIZE)
+ {
+ size_t written;
+ char *header;
+ enum tarError rc;
+
+ if (hdrsize <= sizeof(buffer))
+ header = buffer;
+ else
+ header = allocated = palloc(hdrsize);
+
+ rc = tarCreateHeaderExt(header, hdrsize, member->pathname,
+ linktarget, member->size, member->mode,
+ member->uid, member->gid, time(NULL),
+ &written);
+ if (rc == TAR_NAME_TOO_LONG)
+ pg_fatal("file name too long for tar format: \"%s\"",
+ member->pathname);
+ else if (rc == TAR_SYMLINK_TOO_LONG)
+ pg_fatal("symbolic link target too long for tar format: "
+ " file name \"%s\", target \"%s\"",
+ member->pathname, linktarget);
+ else if (rc != TAR_OK)
+ pg_fatal("could not generate tar header for \"%s\"",
+ member->pathname);
+
+ Assert(written == hdrsize);
+ data = header;
+ len = (int) hdrsize;
+
+ /* Also replace padding if we had to construct a header. */
+ if (constructed)
+ mystreamer->rearchive_member = true;
+ }
}
else if (context == ASTREAMER_MEMBER_TRAILER &&
mystreamer->rearchive_member)
@@ -469,6 +798,9 @@ astreamer_tar_archiver_content(astreamer *streamer,
}
astreamer_content(streamer->bbs_next, member, data, len, context);
+
+ if (allocated != NULL)
+ pfree(allocated);
}
/*
diff --git a/src/include/fe_utils/astreamer.h b/src/include/fe_utils/astreamer.h
index c5ebcefd3d1..1edc511e3e9 100644
--- a/src/include/fe_utils/astreamer.h
+++ b/src/include/fe_utils/astreamer.h
@@ -89,6 +89,11 @@ typedef enum
* to be present in the archive header and used to fill the struct, after
* which all subsequent calls for the same archive member are expected to
* pass the same details.
+ *
+ * For tar archives, a member may be preceded by a PAX extended header. In
+ * that case pathname and linktarget hold the values of the PAX "path" and
+ * "linkpath" attributes, which override any (possibly truncated) values in
+ * the traditional header fields.
*/
typedef struct
{
diff --git a/src/include/pgtar.h b/src/include/pgtar.h
index 323ba051f8b..71fcc03cc2d 100644
--- a/src/include/pgtar.h
+++ b/src/include/pgtar.h
@@ -16,11 +16,31 @@
#define TAR_BLOCK_SIZE 512
+/*
+ * Sizes of the fixed-width string fields in a traditional ustar header.
+ * These are part of the on-disk format and must not be changed.
+ */
+#define TAR_NAME_FIELD_SIZE 100
+#define TAR_PREFIX_FIELD_SIZE 155
+#define TAR_LINKNAME_FIELD_SIZE 100
+
+/*
+ * Maximum size of a PAX extended header payload that we are willing to accept.
+ * The only extensions we care about are "path" and "linkpath", both of which
+ * are bounded by MAXPGPATH, so there is no reason to accept arbitrarily large
+ * payloads.
+ */
+#define TAR_MAX_PAX_PAYLOAD (MAXPGPATH * 4)
+
+/* Name used for the PAX extended header member itself. */
+#define TAR_PAX_HEADER_NAME "PaxHeader"
+
enum tarError
{
TAR_OK = 0,
TAR_NAME_TOO_LONG,
TAR_SYMLINK_TOO_LONG,
+ TAR_BUFFER_TOO_SMALL,
};
/*
@@ -70,6 +90,14 @@ extern enum tarError tarCreateHeader(char *h, const char *filename,
const char *linktarget, pgoff_t size,
mode_t mode, uid_t uid, gid_t gid,
time_t mtime);
+extern enum tarError tarCreateHeaderExt(char *h, size_t hsize,
+ const char *filename,
+ const char *linktarget,
+ pgoff_t size, mode_t mode,
+ uid_t uid, gid_t gid, time_t mtime,
+ size_t *bytes_written);
+extern size_t tarCreateHeaderExtSize(const char *filename,
+ const char *linktarget);
extern uint64 read_tar_number(const char *s, int len);
extern void print_tar_number(char *s, int len, uint64 val);
extern int tarChecksum(const char *header);
diff --git a/src/port/tar.c b/src/port/tar.c
index fee9dbbf5be..cd4042b4c86 100644
--- a/src/port/tar.c
+++ b/src/port/tar.c
@@ -4,6 +4,22 @@
#include "pgtar.h"
+static bool tarSplitPathname(const char *pathname, char *name, char *prefix);
+static bool tarCanEncodePath(const char *pathname);
+static enum tarError tarCreateHeaderInternal(char *h,
+ const char *filename,
+ const char *linktarget,
+ pgoff_t size,
+ mode_t mode,
+ uid_t uid,
+ gid_t gid,
+ time_t mtime,
+ bool allow_prefix,
+ bool truncate);
+static size_t tarPaxRecordSize(const char *key, const char *value);
+static size_t tarPaxWriteRecord(char *p, const char *key, const char *value);
+static void tarCreatePaxHeaderBlock(char *h, size_t payload_size, time_t mtime);
+
/*
* Print a numeric field in a tar header. The field starts at *s and is of
* length len; val is the value to be written.
@@ -138,62 +154,329 @@ isValidTarHeader(const char *header)
* Fill in the buffer pointed to by h with a tar format header. This buffer
* must always have space for 512 characters, which is a requirement of
* the tar format.
+ *
+ * This function retains the traditional PostgreSQL behavior of rejecting names
+ * longer than the name field. Use tarCreateHeaderExt() if support for the
+ * ustar prefix field and PAX extended headers is desired.
*/
enum tarError
tarCreateHeader(char *h, const char *filename, const char *linktarget,
pgoff_t size, mode_t mode, uid_t uid, gid_t gid, time_t mtime)
{
- if (strlen(filename) > 99)
+ return tarCreateHeaderInternal(h, filename, linktarget, size, mode,
+ uid, gid, mtime, false, false);
+}
+
+/*
+ * Build the on-disk representation of a tar member at h, which must have room
+ * for tarCreateHeaderExtSize(filename, linktarget) bytes. If a PAX extended
+ * header is required, it is written before the traditional header, and the
+ * traditional header uses a truncated placeholder for the long field(s);
+ * consumers must give precedence to the PAX attributes.
+ *
+ * On success, *bytes_written is set to the total number of bytes emitted.
+ */
+enum tarError
+tarCreateHeaderExt(char *h, size_t hsize, const char *filename,
+ const char *linktarget, pgoff_t size, mode_t mode,
+ uid_t uid, gid_t gid, time_t mtime, size_t *bytes_written)
+{
+ bool need_path;
+ bool need_link;
+ size_t paxsize = 0;
+ size_t padlen;
+ size_t total;
+ char *p;
+ enum tarError rc;
+
+ *bytes_written = 0;
+
+ if (strlen(filename) >= MAXPGPATH)
return TAR_NAME_TOO_LONG;
+ if (linktarget != NULL && strlen(linktarget) >= MAXPGPATH)
+ return TAR_SYMLINK_TOO_LONG;
+
+ need_path = !tarCanEncodePath(filename);
+ need_link = (linktarget != NULL &&
+ strlen(linktarget) > TAR_LINKNAME_FIELD_SIZE - 1);
+
+ if (!need_path && !need_link)
+ {
+ rc = tarCreateHeaderInternal(h, filename, linktarget, size, mode,
+ uid, gid, mtime, true, false);
+ if (rc == TAR_OK)
+ *bytes_written = TAR_BLOCK_SIZE;
+ return rc;
+ }
+
+ if (need_path)
+ paxsize += tarPaxRecordSize("path", filename);
+ if (need_link)
+ paxsize += tarPaxRecordSize("linkpath", linktarget);
+
+ /*
+ * See tarCreateHeaderExtSize() for the layout; 'total' must match the
+ * value returned by that function.
+ */
+ padlen = tarPaddingBytesRequired(paxsize);
+ total = TAR_BLOCK_SIZE + paxsize + padlen + TAR_BLOCK_SIZE;
+
+ if (hsize < total)
+ return TAR_BUFFER_TOO_SMALL;
+
+ p = h;
+
+ /* PAX extended header member. */
+ tarCreatePaxHeaderBlock(p, paxsize, mtime);
+ p += TAR_BLOCK_SIZE;
+
+ /* PAX payload */
+ if (need_path)
+ p += tarPaxWriteRecord(p, "path", filename);
+ if (need_link)
+ p += tarPaxWriteRecord(p, "linkpath", linktarget);
+
+ /* Padding to 512-byte boundary. */
+ memset(p, 0, padlen);
+ p += padlen;
+
+ Assert(p == h + TAR_BLOCK_SIZE + paxsize + padlen);
+
+ /* The actual member header, with placeholders for the PAX fields. */
+ rc = tarCreateHeaderInternal(p, filename, linktarget, size, mode,
+ uid, gid, mtime, true, true);
+ if (rc == TAR_OK)
+ *bytes_written = total;
+
+ return rc;
+}
+
+/*
+ * Compute the number of bytes needed to represent a tar member, including a
+ * PAX extended header if the file name or symbolic link target cannot be
+ * represented using the traditional ustar fields. The result covers the PAX
+ * header block, the PAX payload and its padding, and the traditional member
+ * header block, but not the member contents or their padding.
+ *
+ * Returns 0 if the name or link target is longer than MAXPGPATH, in which case
+ * no archive representation is possible.
+ */
+size_t
+tarCreateHeaderExtSize(const char *filename, const char *linktarget)
+{
+ bool need_path;
+ bool need_link;
+ size_t paxsize = 0;
+ size_t padlen;
+
+ /*
+ * The PAX attributes are stored in MAXPGPATH-sized buffers by the
+ * consumers, so refuse anything that could not be represented there.
+ */
+ if (strlen(filename) >= MAXPGPATH)
+ return 0;
+ if (linktarget != NULL && strlen(linktarget) >= MAXPGPATH)
+ return 0;
- if (linktarget && strlen(linktarget) > 99)
+ need_path = !tarCanEncodePath(filename);
+ need_link = (linktarget != NULL &&
+ strlen(linktarget) > TAR_LINKNAME_FIELD_SIZE - 1);
+
+ if (!need_path && !need_link)
+ return TAR_BLOCK_SIZE;
+
+ if (need_path)
+ paxsize += tarPaxRecordSize("path", filename);
+ if (need_link)
+ paxsize += tarPaxRecordSize("linkpath", linktarget);
+
+ /*
+ * A member that needs PAX is laid out as:
+ *
+ * [ PAX extended header block ] TAR_BLOCK_SIZE, typeflag 'x'
+ * [ PAX payload ] paxsize bytes
+ * [ padding to 512 boundary ] tarPaddingBytesRequired(paxsize)
+ * [ traditional header block ] TAR_BLOCK_SIZE, typeflag '2' etc
+ *
+ * The second block is not redundant: POSIX requires a conventional ustar
+ * header to follow a PAX extended header, and the parser applies the PAX
+ * attributes to that header's member. The long path or link target lives
+ * in the PAX payload; the traditional header carries only a placeholder.
+ */
+ padlen = tarPaddingBytesRequired(paxsize);
+ return TAR_BLOCK_SIZE + paxsize + padlen + TAR_BLOCK_SIZE;
+}
+
+/*
+ * Try to split a pathname into the ustar prefix and name fields.
+ *
+ * Return true if the pathname can be represented that way, false otherwise.
+ * On success, name and prefix are NUL-terminated.
+ */
+static bool
+tarSplitPathname(const char *pathname, char *name, char *prefix)
+{
+ size_t len = strlen(pathname);
+ size_t lower;
+ size_t upper;
+
+ name[0] = prefix[0] = '\0';
+
+ /* A short pathname fits in the name field by itself. */
+ if (len < TAR_NAME_FIELD_SIZE)
+ {
+ memcpy(name, pathname, len + 1);
+ return true;
+ }
+
+ /*
+ * Otherwise, we need a slash that separates a prefix of at most 154
+ * characters from a name of at most 99 characters.
+ */
+ if (len > (TAR_PREFIX_FIELD_SIZE - 1) + 1 + (TAR_NAME_FIELD_SIZE - 1))
+ return false;
+
+ /*
+ * Search backwards for a slash that splits the pathname into a prefix that
+ * fits in the prefix field and a name that fits in the name field.
+ */
+ lower = Max((size_t) 1, len - TAR_NAME_FIELD_SIZE);
+ upper = Min(len - 1, (size_t) TAR_PREFIX_FIELD_SIZE - 1);
+ for (size_t i = upper + 1; i > lower; /* void */)
+ {
+ size_t prefixlen;
+ size_t namelen;
+
+ i--;
+ if (pathname[i] != '/')
+ continue;
+
+ prefixlen = i;
+ namelen = len - i - 1;
+ if (prefixlen == 0 || namelen == 0)
+ continue;
+
+ memcpy(prefix, pathname, prefixlen);
+ prefix[prefixlen] = '\0';
+ memcpy(name, pathname + i + 1, namelen);
+ name[namelen] = '\0';
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Can this pathname be represented using the traditional ustar name and
+ * prefix fields?
+ */
+static bool
+tarCanEncodePath(const char *pathname)
+{
+ char name[TAR_NAME_FIELD_SIZE + 1];
+ char prefix[TAR_PREFIX_FIELD_SIZE + 1];
+
+ return tarSplitPathname(pathname, name, prefix);
+}
+
+static enum tarError
+tarCreateHeaderInternal(char *h, const char *filename, const char *linktarget,
+ pgoff_t size, mode_t mode, uid_t uid, gid_t gid,
+ time_t mtime, bool allow_prefix, bool truncate)
+{
+ char name[TAR_NAME_FIELD_SIZE + 1];
+ char prefix[TAR_PREFIX_FIELD_SIZE + 1];
+
+ name[0] = prefix[0] = '\0';
+
+ if (allow_prefix)
+ {
+ if (!tarSplitPathname(filename, name, prefix))
+ {
+ const char *base;
+
+ if (!truncate)
+ return TAR_NAME_TOO_LONG;
+
+ /*
+ * The traditional header must still contain something reasonable,
+ * since consumers that do not understand PAX may look at it. Use
+ * the basename when it fits, otherwise a truncated form of the
+ * path.
+ */
+ base = last_dir_separator(filename);
+ if (base != NULL && base[1] != '\0')
+ base++;
+ else
+ base = filename;
+
+ strlcpy(name, base, TAR_NAME_FIELD_SIZE);
+ }
+ }
+ else
+ {
+ if (strlen(filename) > TAR_NAME_FIELD_SIZE - 1)
+ return TAR_NAME_TOO_LONG;
+ strlcpy(name, filename, TAR_NAME_FIELD_SIZE);
+ }
+
+ if (name[0] == '\0')
+ return TAR_NAME_TOO_LONG;
+
+ if (linktarget != NULL &&
+ strlen(linktarget) > TAR_LINKNAME_FIELD_SIZE - 1 &&
+ !truncate)
return TAR_SYMLINK_TOO_LONG;
memset(h, 0, TAR_BLOCK_SIZE);
/* Name 100 */
- strlcpy(&h[TAR_OFFSET_NAME], filename, 100);
+ strlcpy(&h[TAR_OFFSET_NAME], name, TAR_NAME_FIELD_SIZE);
if (linktarget != NULL || S_ISDIR(mode))
{
/*
- * We only support symbolic links to directories, and this is
- * indicated in the tar format by adding a slash at the end of the
- * name, the same as for regular directories.
+ * We only support symbolic links to directories, and this is indicated
+ * in the tar format by adding a slash at the end of the name, the same
+ * as for regular directories.
*/
- int flen = strlen(filename);
+ int flen = strlen(&h[TAR_OFFSET_NAME]);
- flen = Min(flen, 99);
- h[flen] = '/';
- h[flen + 1] = '\0';
+ if (flen < TAR_NAME_FIELD_SIZE - 1)
+ {
+ h[TAR_OFFSET_NAME + flen] = '/';
+ h[TAR_OFFSET_NAME + flen + 1] = '\0';
+ }
+ else
+ h[TAR_OFFSET_NAME + flen - 1] = '/';
}
- /* Mode 8 - this doesn't include the file type bits (S_IFMT) */
+ /* Mode 8 - this doesn't include the file type bits (S_IFMT) */
print_tar_number(&h[TAR_OFFSET_MODE], 8, (mode & 07777));
/* User ID 8 */
print_tar_number(&h[TAR_OFFSET_UID], 8, uid);
- /* Group 8 */
+ /* Group ID 8 */
print_tar_number(&h[TAR_OFFSET_GID], 8, gid);
/* File size 12 */
if (linktarget != NULL || S_ISDIR(mode))
- /* Symbolic link or directory has size zero */
+ /* Symbolic link or directory has size zero. */
print_tar_number(&h[TAR_OFFSET_SIZE], 12, 0);
else
print_tar_number(&h[TAR_OFFSET_SIZE], 12, size);
- /* Mod Time 12 */
+ /* Mode Time 12 */
print_tar_number(&h[TAR_OFFSET_MTIME], 12, mtime);
- /* Checksum 8 cannot be calculated until we've filled all other fields */
+ /* Checksum 8 cannot be calculated until we've filled all other fields. */
if (linktarget != NULL)
{
/* Type - Symbolic link */
h[TAR_OFFSET_TYPEFLAG] = TAR_FILETYPE_SYMLINK;
/* Link Name 100 */
- strlcpy(&h[TAR_OFFSET_LINKNAME], linktarget, 100);
+ strlcpy(&h[TAR_OFFSET_LINKNAME], linktarget, TAR_LINKNAME_FIELD_SIZE);
}
else if (S_ISDIR(mode))
{
@@ -226,10 +509,111 @@ tarCreateHeader(char *h, const char *filename, const char *linktarget,
/* Minor Dev 8 */
print_tar_number(&h[TAR_OFFSET_DEVMINOR], 8, 0);
- /* Prefix 155 - not used, leave as nulls */
+ /* Prefix 155 */
+ if (prefix[0] != '\0')
+ strlcpy(&h[TAR_OFFSET_PREFIX], prefix, TAR_PREFIX_FIELD_SIZE);
/* Finally, compute and insert the checksum */
print_tar_number(&h[TAR_OFFSET_CHECKSUM], 8, tarChecksum(h));
return TAR_OK;
}
+
+/*
+ * Number of bytes in a PAX extended header record of the form
+ *
+ * <length> <key>=<value>\n
+ *
+ * The length includes its own decimal digits, so we must iterate until the
+ * length and its digit count are consistent. This converges in at most two
+ * iterations.
+ */
+static size_t
+tarPaxRecordSize(const char *key, const char *value)
+{
+ size_t body = strlen(key) + strlen(value) + 3;
+ size_t len = body + 1;
+
+ for (;;)
+ {
+ size_t digits = 1;
+ size_t tmp = len;
+
+ while (tmp >= 10)
+ {
+ tmp /= 10;
+ digits++;
+ }
+ if (len == body + digits)
+ return len;
+ len = body + digits;
+ }
+}
+
+/*
+ * Write on PAX record. The buffer must have room for tarPaxRecordSize(), plus
+ * one byte for the terminating NUL that snprintf() writes.
+ */
+static size_t
+tarPaxWriteRecord(char *p, const char *key, const char *value)
+{
+ size_t len = tarPaxRecordSize(key, value);
+ int n;
+
+ n = snprintf(p, len + 1, "%zu %s=%s\n", len, key, value);
+ Assert(n == (int) len);
+
+ return len;
+}
+
+/*
+ * Build the header for the PAX extended header member itself. The payload
+ * follows immediately after this block.
+ */
+static void
+tarCreatePaxHeaderBlock(char *h, size_t payload_size, time_t mtime)
+{
+ memset(h, 0, TAR_BLOCK_SIZE);
+
+ /* Name 100 */
+ strlcpy(&h[TAR_OFFSET_NAME], TAR_PAX_HEADER_NAME, TAR_NAME_FIELD_SIZE);
+
+ /* Mode 8 */
+ print_tar_number(&h[TAR_OFFSET_MODE], 8, 0600);
+
+ /* User ID 8 */
+ print_tar_number(&h[TAR_OFFSET_UID], 8, 0);
+
+ /* Group ID 8 */
+ print_tar_number(&h[TAR_OFFSET_GID], 8, 0);
+
+ /* File size 12: the unpadded PAX payload size. */
+ print_tar_number(&h[TAR_OFFSET_SIZE], 12, payload_size);
+
+ /* Mod Time 12 */
+ print_tar_number(&h[TAR_OFFSET_MTIME], 12, mtime);
+
+ /* Type - PAX extended header */
+ h[TAR_OFFSET_TYPEFLAG] = TAR_FILETYPE_PAX_EXTENDED;
+
+ /* Magic 6 */
+ strcpy(&h[TAR_OFFSET_MAGIC], "ustar");
+
+ /* Version 2 */
+ memcpy(&h[TAR_OFFSET_VERSION], "00", 2);
+
+ /* User 32 */
+ strlcpy(&h[TAR_OFFSET_UNAME], "postgres", 32);
+
+ /* Group 32 */
+ strlcpy(&h[TAR_OFFSET_GNAME], "postgres", 32);
+
+ /* Major Dev 8 */
+ print_tar_number(&h[TAR_OFFSET_DEVMAJOR], 8, 0);
+
+ /* Minor Dev 8 */
+ print_tar_number(&h[TAR_OFFSET_DEVMINOR], 8, 0);
+
+ /* Finally, compute and insert the checksum */
+ print_tar_number(&h[TAR_OFFSET_CHECKSUM], 8, tarChecksum(h));
+}
--
2.53.0
