On Sun, Mar 15, 2020 at 06:15:02PM +0100, Fabien COELHO wrote: > Some feedback on v10:
Thanks for looking. I'm hoping to hear from Alvaro what he thinks of this approach (all functions to show isdir, rather than one function which lists recursively). > All patches apply cleanly, one on top of the previous. I really wish there > would be less than 9 patches… I kept them separate to allow the earlier patches to be applied. And intended to make easier to review, even if it's more work for me.. If you mean that it's a pain to apply 9 patches, I will suggest to use: |git am -3 ./mailbox where ./mailbox is either a copy of the mail you received, or retrieved from the web interface. To test that each one works (compiles, passes tests, etc), I use git rebase -i HEAD~11 and "e"edit the target (set of) patches. > * v10.1 doc change: ok > > * v10.2 doc change: ok, not sure why it is not merged with previous As I mentioned, separate since I'm proposing that they're backpatched to different releases. Those could be applied now (and Tom already applied a patch identical to 0001 in a prior patchset). > * v10.3 test add: could be merge with both previous > Tests seem a little contrived. I'm wondering whether something more > straightforward could be proposed. For instance, once the tablespace is just > created but not used yet, probably we do know that the tmp file exists and > is empty? The tmpdir *doesn't* exist until someone creates tmpfiles there. As it mentions: +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet > * v10.4 at least, some code! > I'm not sure of the "FLAG_" prefix which seems too generic, even if it is > local. I'd suggest "LS_DIR_*", maybe, as a more specific prefix. Done. > ISTM that Pg style requires spaces around operators. I'd suggest some > parenthesis would help as well, eg: "flags&FLAG_MISSING_OK" -> "(flags & > FLAG_MISSING_OK)" and other instances. Partially took your suggestion. > About: > > if (S_ISDIR(attrib.st_mode)) { > if (flags&FLAG_SKIP_DIRS) > continue; > } > > and similars, why not the simpler: > > if (S_ISDIR(attrib.st_mode) && (flags & FLAG_SKIP_DIRS)) > continue; That's not the same - if SKIP_DIRS isn't set, your way would fail that test for dirs, and then hit the !ISREG test, and skip them anyway. |else if (!S_ISREG(attrib.st_mode)) | continue > Maybe I'd create defines for long and common flag specs, eg: > > #define ..._LS_SIMPLE > (FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_SKIP_SPECIAL|FLAG_METADATA) Done > I'm not sure about these asserts: > > /* isdir depends on metadata */ > Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA)); > > Hmmm. Why? It's not supported to show isdir without showing metadata (because that case isn't needed to support the old and the new behaviors). + if (flags & FLAG_METADATA) + { + values[1] = Int64GetDatum((int64) attrib.st_size); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & FLAG_ISDIR) + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + } > /* Unreasonable to show isdir and skip dirs */ > Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS)); > > Hmmm. Why would I prevent that, even if it has little sense, it should work. > I do not see having false on the isdir column as an actual issue. It's unimportant, but testing for intended use of flags during development. > * v10.6 behavior change for existing functions, always show isdir column, > and removal of IS_DIR flag. > > I'm unsure why the features are removed, some use case may benefit from the > more complete function? > Maybe flags defs should not be changed anyway? Maybe. I put them back...but it means they're not being exercized by any *used* case. > I do not like much the "if (...) /* empty */;" code. Maybe it could be > caught more cleanly later in the conditional structure. This went away when I put back the SKIP_DIRS flag. > * v10.7 adds "pg_ls_dir_recurse" function > Doc looks incomplete and the example is very contrived and badly indented. Why you think it's contrived? Listing a tmpdir recursively is the initial motivation of this patch. Maybe you think I should list just the tmpdir for one tablespace ? Note that for temp_tablespaces parameter: |When there is more than one name in the list, PostgreSQL chooses a random member of the list each time a temporary object is to be created; except that within a transaction, successively created temporary objects are placed in successive tablespaces from the list. > The function definition does not follow the style around: uppercase whereas > all others are lowercase, "" instead of '', no "as"… I used "" because of this: | x.name||'/'||a.name I don't know if there's a better way to join paths in SQL, or if that suggests this is a bad way to do it. > I do not understand why oid 8511 is given to the new function. I used: ./src/include/catalog/unused_oids (maybe not correctly). > I do not understand why UNION ALL and not UNION. In general, union ALL can avoid a "distinct" plan node, but it doesn't seem to have any effect here. > I would have put the definition after "pg_ls_dir_metadata" definition. Done > pg_ls_dir_metadata seems defined as (text,bool,bool) but called as > (text,bool,bool,bool). fixed, thanks. > Maybe a better alias could be given instead of x? > > There are no tests for the new function. I'm not sure it would work. I added something which would've caught the issue with number of arguments. > * v10.8 change flags & add test on pg_ls_logdir(). > > I'm unsure why it is done at this stage. I think it makes sense to allow ls_logdir to succeed even if ./log doesn't exist, since it isn't created by initdb or during postmaster start, and since we already using MISSING_OK for tmpdir. But a separate patch since we didn't previous discuss changing logdir. > * v10.9 change some ls functions and fix patch 10.7 issue > I'm unsure why it is done at this stage. "make check" ok. This is the last patch in the series, since I think it's least likely to be agreed on. -- Justin
>From 1bb8e0efb4f14fa344cd5ee66c3138184a9fa9e2 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 16:50:07 -0600 Subject: [PATCH v11 1/9] Document historic behavior about hiding directories and special files Should backpatch to v10: tmpdir, waldir and archive_statusdir --- doc/src/sgml/func.sgml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 323366feb6..4c0ea5ab3f 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); (mtime) of each file in the log directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> @@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> @@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> -- 2.17.0
>From c8b89e104694645873b77f987d795269fc330bf7 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 17:12:04 -0600 Subject: [PATCH v11 2/9] Document historic behavior about hiding directories and special files Should backpatch to v12: tmpdir --- doc/src/sgml/func.sgml | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 4c0ea5ab3f..fc4d7f0f78 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> -- 2.17.0
>From 96de8a92294eec8c33ecb2ef0dbf48add281b763 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 22:40:24 -0500 Subject: [PATCH v11 3/9] Add tests exercizing pg_ls_tmpdir.. ..and its backing function pg_ls_dir_files --- src/test/regress/expected/misc_functions.out | 7 +++++++ src/test/regress/input/tablespace.source | 5 +++++ src/test/regress/output/tablespace.source | 8 ++++++++ src/test/regress/sql/misc_functions.sql | 4 ++++ 4 files changed, 24 insertions(+) diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index e217b678d7..9947d9ef9d 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -180,6 +180,13 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir(); t (1 row) +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +select * from pg_ls_tmpdir() where name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index a5f61a35dc..0b9cfe615e 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 162b591b31..a42714bf40 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith'; DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 1e11eb3554..656aad93e6 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -51,6 +51,10 @@ from (select pg_ls_waldir() w) ss where length((w).name) = 24 limit 1; select count(*) >= 0 as ok from pg_ls_archive_statusdir(); +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +select * from pg_ls_tmpdir() where name='Does not exist'; + -- -- Test adding a support function to a subject function -- -- 2.17.0
>From 3156f1c19e0389a0a23546971ca8715d378e0370 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 17:15:02 -0500 Subject: [PATCH v11 4/9] Add pg_ls_dir_metadata to list a dir with file metadata.. Generalize pg_ls_dir_files and retire pg_ls_dir Need catversion bumped? --- doc/src/sgml/func.sgml | 17 +++ src/backend/utils/adt/genfile.c | 190 ++++++++++++++++++-------------- src/include/catalog/pg_proc.dat | 6 + 3 files changed, 129 insertions(+), 84 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index fc4d7f0f78..dfea9f8061 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21342,6 +21342,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); List the contents of a directory. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. </entry> </row> + <row> + <entry> + <literal><function>pg_ls_dir_metadata(<parameter>dirname</parameter> <type>text</type> [, <parameter>missing_ok</parameter> <type>boolean</type>, <parameter>include_dot_dirs</parameter> <type>boolean</type>])</function></literal> + </entry> + <entry><type>setof text</type></entry> + <entry> + For each file in a directory, list the file and its metadata. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. + </entry> + </row> <row> <entry> <literal><function>pg_ls_logdir()</function></literal> @@ -21442,6 +21451,14 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); empty directory from an non-existent directory. </para> + <indexterm> + <primary>pg_ls_dir_metadata</primary> + </indexterm> + <para> + <function>pg_ls_dir_metadata</function> lists the files in the specified + directory along with the file's metadata. + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index bcf9bd1b97..b105363903 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -36,14 +36,23 @@ #include "utils/syscache.h" #include "utils/timestamp.h" -typedef struct -{ - char *location; - DIR *dirdesc; - bool include_dot_dirs; -} directory_fctx; +static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags); + +#define LS_DIR_ISDIR (1<<0) /* Show column: isdir */ +#define LS_DIR_METADATA (1<<1) /* Show columns: mtime, size */ +#define LS_DIR_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */ +#define LS_DIR_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */ +#define LS_DIR_SKIP_HIDDEN (1<<4) /* Do not show anything begining with . */ +#define LS_DIR_SKIP_DIRS (1<<5) /* Do not show directories */ +#define LS_DIR_SKIP_SPECIAL (1<<6) /* Do not show special file types */ +/* + * Shortcut for the historic behavior of the pg_ls_* functions (not including + * pg_ls_dir, which skips different files and doesn't show metadata. + */ +#define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) + /* * Convert a "text" filename argument to C string, and check it's allowable. * @@ -447,67 +456,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS) Datum pg_ls_dir(PG_FUNCTION_ARGS) { - FuncCallContext *funcctx; - struct dirent *de; - directory_fctx *fctx; - MemoryContext oldcontext; - - if (SRF_IS_FIRSTCALL()) - { - bool missing_ok = false; - bool include_dot_dirs = false; - - /* check the optional arguments */ - if (PG_NARGS() == 3) - { - if (!PG_ARGISNULL(1)) - missing_ok = PG_GETARG_BOOL(1); - if (!PG_ARGISNULL(2)) - include_dot_dirs = PG_GETARG_BOOL(2); - } - - funcctx = SRF_FIRSTCALL_INIT(); - oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); - - fctx = palloc(sizeof(directory_fctx)); - fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); - - fctx->include_dot_dirs = include_dot_dirs; - fctx->dirdesc = AllocateDir(fctx->location); - - if (!fctx->dirdesc) - { - if (missing_ok && errno == ENOENT) - { - MemoryContextSwitchTo(oldcontext); - SRF_RETURN_DONE(funcctx); - } - else - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not open directory \"%s\": %m", - fctx->location))); - } - funcctx->user_fctx = fctx; - MemoryContextSwitchTo(oldcontext); - } - - funcctx = SRF_PERCALL_SETUP(); - fctx = (directory_fctx *) funcctx->user_fctx; - - while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL) - { - if (!fctx->include_dot_dirs && - (strcmp(de->d_name, ".") == 0 || - strcmp(de->d_name, "..") == 0)) - continue; - - SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name)); - } - - FreeDir(fctx->dirdesc); - - SRF_RETURN_DONE(funcctx); + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS); } /* @@ -520,7 +471,9 @@ pg_ls_dir(PG_FUNCTION_ARGS) Datum pg_ls_dir_1arg(PG_FUNCTION_ARGS) { - return pg_ls_dir(fcinfo); + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, LS_DIR_SKIP_DOT_DIRS); } /* @@ -530,7 +483,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) * Other unreadable-directory cases throw an error. */ static Datum -pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) +pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) { ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; bool randomAccess; @@ -539,6 +492,32 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) DIR *dirdesc; struct dirent *de; MemoryContext oldcontext; + TypeFuncClass tuptype ; + + /* isdir depends on metadata */ + Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA)); + /* Unreasonable to show isdir and skip dirs */ + Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS)); + + /* check the optional arguments */ + if (PG_NARGS() == 3) + { + if (!PG_ARGISNULL(1)) + { + if (PG_GETARG_BOOL(1)) + flags |= LS_DIR_MISSING_OK; + else + flags &= ~LS_DIR_MISSING_OK; + } + + if (!PG_ARGISNULL(2)) + { + if (PG_GETARG_BOOL(2)) + flags &= ~LS_DIR_SKIP_DOT_DIRS; + else + flags |= LS_DIR_SKIP_DOT_DIRS; + } + } /* check to see if caller supports us returning a tuplestore */ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo)) @@ -554,8 +533,18 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) /* The tupdesc and tuplestore must be created in ecxt_per_query_memory */ oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory); - if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) - elog(ERROR, "return type must be a row type"); + tuptype = get_call_result_type(fcinfo, NULL, &tupdesc); + if (flags & LS_DIR_METADATA) + { + if (tuptype != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + } else { + /* pg_ls_dir returns a simple scalar */ + if (tuptype != TYPEFUNC_SCALAR) + elog(ERROR, "return type must be a scalar type"); + tupdesc = CreateTemplateTupleDesc(1); + TupleDescInitEntry(tupdesc, (AttrNumber) 1, "column", TEXTOID, -1, 0); + } randomAccess = (rsinfo->allowedModes & SFRM_Materialize_Random) != 0; tupstore = tuplestore_begin_heap(randomAccess, false, work_mem); @@ -574,7 +563,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) if (!dirdesc) { /* Return empty tuplestore if appropriate */ - if (missing_ok && errno == ENOENT) + if (flags&LS_DIR_MISSING_OK && errno == ENOENT) { tuplestore_donestoring(tupstore); return (Datum) 0; @@ -584,13 +573,19 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) while ((de = ReadDir(dirdesc, dir)) != NULL) { - Datum values[3]; - bool nulls[3]; + Datum values[4]; + bool nulls[4]; char path[MAXPGPATH * 2]; struct stat attrib; + if (flags & LS_DIR_SKIP_DOT_DIRS && + (strcmp(de->d_name, ".") == 0 || + strcmp(de->d_name, "..") == 0)) + continue; + /* Skip hidden files */ - if (de->d_name[0] == '.') + if (flags & LS_DIR_SKIP_HIDDEN && + de->d_name[0] == '.') continue; /* Get the file info */ @@ -600,13 +595,26 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", path))); - /* Ignore anything but regular files */ - if (!S_ISREG(attrib.st_mode)) - continue; + if (S_ISDIR(attrib.st_mode)) + { + if (flags & LS_DIR_SKIP_DIRS) + continue; + } + else if (!S_ISREG(attrib.st_mode)) + { + if (flags & LS_DIR_SKIP_SPECIAL) + continue; + } values[0] = CStringGetTextDatum(de->d_name); - values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & LS_DIR_METADATA) + { + values[1] = Int64GetDatum((int64) attrib.st_size); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & LS_DIR_ISDIR) + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + } + memset(nulls, 0, sizeof(nulls)); tuplestore_putvalues(tupstore, tupdesc, values, nulls); @@ -621,14 +629,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, false); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, false); + return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC); } /* @@ -646,7 +654,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) tblspc))); TempTablespacePath(path, tblspc); - return pg_ls_dir_files(fcinfo, path, true); + return pg_ls_dir_files(fcinfo, path, + LS_DIR_HISTORIC|LS_DIR_MISSING_OK); } /* @@ -675,5 +684,18 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS) Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true); + return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", + LS_DIR_HISTORIC|LS_DIR_MISSING_OK); +} + +/* + * Function to return the list of files and metadata in an arbitrary directory. + */ +Datum +pg_ls_dir_metadata(PG_FUNCTION_ARGS) +{ + char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); + + return pg_ls_dir_files(fcinfo, dirname, + LS_DIR_METADATA|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7fb574f9dc..0a1859f709 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10741,6 +10741,12 @@ proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', proargnames => '{tablespace,name,size,modification}', prosrc => 'pg_ls_tmpdir_1arg' }, +{ oid => '5032', descr => 'list directory with metadata', + proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool', + proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}', + prosrc => 'pg_ls_dir_metadata' }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', -- 2.17.0
>From ede11bc6af39a7234c31508bf981ee193dc0e0cb Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:57:54 -0500 Subject: [PATCH v11 5/9] pg_ls_tmpdir to show "isdir" argument.. similar to pg_stat_file(). It's worth breaking the function's return type, since core postgres creates "shared filesets" underneath the temp dirs, and it's unreasonable to not show them here, and the alternative query to show them is unreasaonbly complicated. See following commit which also adds these columns to the other pg_ls_* functions. Although I don't think it matters that they're easily UNIONed, it'd still make great sense if they returned the same columns. Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column. The other functions both skip dirs and only show 3 columns, and ls_tmpdir is changing to both show dirs and show 4 columns. It wouldn't make sense to skip dirs and then show a 4th column always false. Need catversion bump --- doc/src/sgml/func.sgml | 22 +++++++++++--------- src/backend/utils/adt/genfile.c | 4 ++-- src/include/catalog/pg_proc.dat | 8 +++---- src/test/regress/expected/misc_functions.out | 4 ++-- src/test/regress/output/tablespace.source | 4 ++-- 5 files changed, 22 insertions(+), 20 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index dfea9f8061..82891e2d64 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21391,12 +21391,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the - temporary directory for <parameter>tablespace</parameter>. If - <parameter>tablespace</parameter> is not provided, the - <literal>pg_default</literal> tablespace is used. Access is granted - to members of the <literal>pg_monitor</literal> role and may be - granted to other non-superuser roles. + For the temporary directory within <parameter>tablespace</parameter>, + list each file's name, size, last modification time, and a boolean + indicating if it is a directory. If <parameter>tablespace</parameter> + is not provided, the <literal>pg_default</literal> tablespace is used. + Access is granted to members of the <literal>pg_monitor</literal> role + and may be granted to other non-superuser roles. </entry> </row> <row> @@ -21499,14 +21499,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <primary>pg_ls_tmpdir</primary> </indexterm> <para> - <function>pg_ls_tmpdir</function> returns the name, size, and last modified - time (mtime) of each file in the temporary file directory for the specified - <parameter>tablespace</parameter>. If <parameter>tablespace</parameter> is + <function>pg_ls_tmpdir</function> lists each file in the temporary file + directory for the specified <parameter>tablespace</parameter>, along with + its size, last modified time (mtime) and a boolean indicating if the file is a + directory. Directories are used for temporary files shared by parallel + processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index b105363903..99a540baf2 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -477,7 +477,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) } /* - * Generic function to return a directory listing of files. + * Generic function to return a directory listing of files (and optionally dirs). * * If the directory isn't there, silently return an empty set if missing_ok. * Other unreadable-directory cases throw an error. @@ -655,7 +655,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) TempTablespacePath(path, tblspc); return pg_ls_dir_files(fcinfo, path, - LS_DIR_HISTORIC|LS_DIR_MISSING_OK); + LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK); } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 0a1859f709..7789d029ea 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10733,13 +10733,13 @@ { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', - proargnames => '{tablespace,name,size,modification}', + proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', + proargnames => '{tablespace,name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 9947d9ef9d..b0cf145dc5 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir(); -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) -- diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index a42714bf40..1c88e914e3 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0
>From 3398c7eaeb2c3ee2be8d80a726bea6d9a191e470 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 01:00:42 -0500 Subject: [PATCH v11 6/9] pg_ls_*dir to show directories and "isdir" column.. pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir XXX: ..FLAG_ISDIR is collapsed into FLAG_METADATA. Need catversion bump --- doc/src/sgml/func.sgml | 39 +++++++++++--------- src/backend/utils/adt/genfile.c | 11 ++++-- src/include/catalog/pg_proc.dat | 19 +++++----- src/test/regress/expected/misc_functions.out | 4 +- 4 files changed, 41 insertions(+), 32 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 82891e2d64..e4ba3fee40 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21357,8 +21357,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the log - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the log directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21368,8 +21369,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the WAL directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21379,8 +21381,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - archive status directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file's + name, size, last modification time, and a boolean indicating if it is a + directory. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21463,36 +21466,38 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <primary>pg_ls_logdir</primary> </indexterm> <para> - <function>pg_ls_logdir</function> returns the name, size, and last modified time - (mtime) of each file in the log directory. By default, only superusers + <function>pg_ls_logdir</function> lists each file in the log directory, + along with file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_waldir</primary> </indexterm> <para> - <function>pg_ls_waldir</function> returns the name, size, and last modified time - (mtime) of each file in the write ahead log (WAL) directory. By - default only superusers and members of the <literal>pg_monitor</literal> role + <function>pg_ls_waldir</function> lists each file in the WAL directory, + along with the file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers + and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_archive_statusdir</primary> </indexterm> <para> - <function>pg_ls_archive_statusdir</function> returns the name, size, and - last modified time (mtime) of each file in the WAL archive status - directory <filename>pg_wal/archive_status</filename>. By default only + <function>pg_ls_archive_statusdir</function> lists each file in the WAL + archive status directory, along with the file's size, last modification + time, and a boolean indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 99a540baf2..9c2feb0986 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -53,6 +53,9 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags */ #define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) +/* Shortcut for showing ... */ +#define LS_DIR_MODERN (LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA) + /* * Convert a "text" filename argument to C string, and check it's allowable. * @@ -629,14 +632,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_HISTORIC); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_HISTORIC); + return pg_ls_dir_files(fcinfo, XLOGDIR, LS_DIR_MODERN); } /* @@ -655,7 +658,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) TempTablespacePath(path, tblspc); return pg_ls_dir_files(fcinfo, path, - LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_ISDIR|LS_DIR_METADATA|LS_DIR_MISSING_OK); + LS_DIR_MODERN|LS_DIR_MISSING_OK); } /* @@ -685,7 +688,7 @@ Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", - LS_DIR_HISTORIC|LS_DIR_MISSING_OK); + LS_DIR_MODERN|LS_DIR_MISSING_OK); } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7789d029ea..b4c3d15495 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6143,6 +6143,7 @@ proname => 'pg_ls_dir', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool', prosrc => 'pg_ls_dir' }, + { oid => '2626', descr => 'sleep for the specified time in seconds', proname => 'pg_sleep', provolatile => 'v', prorettype => 'void', proargtypes => 'float8', prosrc => 'pg_sleep' }, @@ -10717,18 +10718,18 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz}', - proargmodes => '{o,o,o}', proargnames => '{name,size,modification}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', + proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', @@ -10743,9 +10744,9 @@ prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', - provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool', - proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}', + provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', prosrc => 'pg_ls_dir_metadata' }, # hash partitioning constraint function diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index b0cf145dc5..448901d841 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss; -- Test not-run-to-completion cases. select * from pg_ls_waldir() limit 0; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss; -- 2.17.0
>From ed17a9fb0975d35f36f6925e480029c461f03724 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:52:14 -0500 Subject: [PATCH v11 7/9] Add pg_ls_dir_recurse to show dir recursively.. ..possibly there's a better place to put this, like maybe a doc-only example ? TODO: src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public; Need catversion bumped ? --- doc/src/sgml/func.sgml | 24 ++++++++++++++++++++ src/include/catalog/pg_proc.dat | 6 +++++ src/test/regress/expected/misc_functions.out | 14 ++++++++++++ src/test/regress/sql/misc_functions.sql | 6 +++++ 4 files changed, 50 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index e4ba3fee40..f3422808d9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21351,6 +21351,15 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); For each file in a directory, list the file and its metadata. Restricted to superusers by default, but other users can be granted EXECUTE to run the function. </entry> </row> + <row> + <entry> + <literal><function>pg_ls_dir_recurse(<parameter>dirname</parameter> <type>text</type>)</function></literal> + </entry> + <entry><type>setof text</type></entry> + <entry> + Call pg_ls_dir_metadata to recursively list the files in the specified directory, along with each file's metadata. + </entry> + </row> <row> <entry> <literal><function>pg_ls_logdir()</function></literal> @@ -21462,6 +21471,21 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); directory along with the file's metadata. </para> + <indexterm> + <primary>pg_ls_dir_recurse</primary> + </indexterm> + <para> + <function>pg_ls_dir_recurse</function> recursively lists the files + in the specified directory. + To recursively list temporary directories in all tablespaces: +<programlisting> +SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp') AS dir +FROM pg_tablespace b, pg_control_system() pcs, +format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) AS suffix) AS dir, +pg_ls_dir_recurse(dir) AS a; +</programlisting> + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b4c3d15495..667ea5721e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10748,6 +10748,12 @@ proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', prosrc => 'pg_ls_dir_metadata' }, +{ oid => '8511', descr => 'list all files in a directory recursively', + proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text', + proallargtypes => '{text,text,int8,timestamptz,bool}', + proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 448901d841..82d967a62d 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -187,6 +187,20 @@ select * from pg_ls_tmpdir() where name='Does not exist'; ------+------+--------------+------- (0 rows) +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + name | isdir +-----------------------+------- + pg_wal | t + pg_wal/archive_status | t +(2 rows) + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- -- Test adding a support function to a subject function -- diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql index 656aad93e6..2cfdac386d 100644 --- a/src/test/regress/sql/misc_functions.sql +++ b/src/test/regress/sql/misc_functions.sql @@ -55,6 +55,12 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir(); -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; +-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix +SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; + +-- Check that expected columns are present +SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; + -- -- Test adding a support function to a subject function -- -- 2.17.0
>From e5b3bbf8fcf2a89ad5e273a939ef7257c0323fb2 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 17:23:51 -0600 Subject: [PATCH v11 8/9] pg_ls_logdir to ignore error if initial/top dir is missing --- src/backend/utils/adt/genfile.c | 2 +- src/test/regress/input/tablespace.source | 4 ++++ src/test/regress/output/tablespace.source | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 9c2feb0986..2d4f561ae0 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -632,7 +632,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN); + return pg_ls_dir_files(fcinfo, Log_directory, LS_DIR_MODERN|LS_DIR_MISSING_OK); } /* Function to return the list of files in the WAL directory */ diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index 0b9cfe615e..2a1268e17c 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 1c88e914e3..ba9a3fe29a 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa ------+------+--------------+------- (0 rows) +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + name | size | modification | isdir +------+------+--------------+------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail -- 2.17.0
>From e03b906cc3e00989a3ed8db93ea655b57b71b3ce Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 21:56:21 -0500 Subject: [PATCH v11 9/9] pg_ls_*dir to return all the metadata from pg_stat_file.. ..but it doesn't seem worth factoring out the common bits, since stat_file doesn't return a name, so all the field numbers are off by one. NOTE, the atime is now shown where the mtime used to be. Need catversion bump --- doc/src/sgml/func.sgml | 30 +++++++++----------- src/backend/utils/adt/genfile.c | 18 ++++++++---- src/include/catalog/pg_proc.dat | 30 ++++++++++---------- src/test/regress/expected/misc_functions.out | 12 ++++---- src/test/regress/output/tablespace.source | 8 +++--- 5 files changed, 51 insertions(+), 47 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index f3422808d9..7f7d81da6e 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21366,8 +21366,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the log directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the log directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21378,8 +21377,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the WAL directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21390,9 +21388,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL archive status directory, list the file's - name, size, last modification time, and a boolean indicating if it is a - directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file and its metadata. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21404,8 +21401,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <entry><type>setof record</type></entry> <entry> For the temporary directory within <parameter>tablespace</parameter>, - list each file's name, size, last modification time, and a boolean - indicating if it is a directory. If <parameter>tablespace</parameter> + list each file and its metadata. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. @@ -21491,8 +21487,8 @@ pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_logdir</function> lists each file in the log directory, - along with file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. Filenames beginning with a dot and special file types are not shown. @@ -21503,8 +21499,8 @@ pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_waldir</function> lists each file in the WAL directory, - along with the file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21516,8 +21512,8 @@ pg_ls_dir_recurse(dir) AS a; </indexterm> <para> <function>pg_ls_archive_statusdir</function> lists each file in the WAL - archive status directory, along with the file's size, last modification - time, and a boolean indicating if the file is a directory. By default, only + archive status directory, along with the metadata columns returned by + <function>pg_stat_file</function>. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21530,8 +21526,8 @@ pg_ls_dir_recurse(dir) AS a; <para> <function>pg_ls_tmpdir</function> lists each file in the temporary file directory for the specified <parameter>tablespace</parameter>, along with - its size, last modified time (mtime) and a boolean indicating if the file is a - directory. Directories are used for temporary files shared by parallel + the metadata columns returned by <function>pg_stat_file</function>. + Directories are used for temporary files shared by parallel processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 2d4f561ae0..7459371e48 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -576,8 +576,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) while ((de = ReadDir(dirdesc, dir)) != NULL) { - Datum values[4]; - bool nulls[4]; + Datum values[7]; + bool nulls[7]; char path[MAXPGPATH * 2]; struct stat attrib; @@ -613,9 +613,17 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) if (flags & LS_DIR_METADATA) { values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); - if (flags & LS_DIR_ISDIR) - values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime)); + values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + /* Unix has file status change time, while Win32 has creation time */ +#if !defined(WIN32) && !defined(__CYGWIN__) + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); + nulls[5] = true; +#else + nulls[4] = true; + values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); +#endif + values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode)); } memset(nulls, 0, sizeof(nulls)); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 667ea5721e..b74ed64b71 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10718,42 +10718,42 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', - proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', - proargnames => '{tablespace,name,size,modification,isdir}', + proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', - proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_dir_metadata' }, { oid => '8511', descr => 'list all files in a directory recursively', proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,bool}', - proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', - prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.modification, a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}', + prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification,a.change,a.creation,a.isdir from ls, pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 82d967a62d..3544977a51 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss; -- Test not-run-to-completion cases. select * from pg_ls_waldir() limit 0; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss; @@ -183,8 +183,8 @@ select count(*) >= 0 as ok from pg_ls_archive_statusdir(); -- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet -- The name='' condition is never true, so the function runs to completion but returns zero rows. select * from pg_ls_tmpdir() where name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix @@ -197,8 +197,8 @@ SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal'; -- Check that expected columns are present SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index ba9a3fe29a..1e1e02b589 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. -- The name='' condition is never true, so the function runs to completion but returns zero rows. SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0