I took a step back, and I wondered whether we should add a generic function for listing a dir with metadata, possibly instead of changing the existing functions. Then one could do pg_ls_dir_metadata('pg_wal',false,false);
Since pg8.1, we have pg_ls_dir() to show a list of files. Since pg10, we've had pg_ls_logdir and pg_ls_waldir, which show not only file names but also (some) metadata (size, mtime). And since pg12, we've had pg_ls_tmpfile and pg_ls_archive_statusdir, which also show metadata. ...but there's no a function which lists the metadata of an directory other than tmp, wal, log. One can do this: |SELECT b.*, c.* FROM (SELECT 'base' a)a, LATERAL (SELECT a||'/'||pg_ls_dir(a.a)b)b, pg_stat_file(b)c; ..but that's not as helpful as allowing: |SELECT * FROM pg_ls_dir_metadata('.',true,true); There's also no function which recurses into an arbitrary directory, so it seems shortsighted to provide a function to recursively list a tmpdir. Also, since pg_ls_dir_metadata indicates whether the path is a dir, one can write a SQL function to show the dir recursively. It'd be trivial to plug in wal/log/tmp (it seems like tmpdirs of other tablespace's are not entirely trivial). |SELECT * FROM pg_ls_dir_recurse('base/pgsql_tmp'); Also, on a neighboring thread[1], Tom indicated that the pg_ls_* functions should enumerate all files during the initial call, which sounds like a bad idea when recursively showing directories. If we add a function recursing into a directory, we'd need to discuss all the flags to expose to it, like recurse, ignore_errors, one_filesystem?, show_dotfiles (and eventually bikeshed all the rest of the flags in find(1)). My initial patch [2] changed ls_tmpdir to show metadata columns including is_dir, but not decend. It's pretty unfortunate if a function called pg_ls_tmpdir hides shared filesets, so maybe it really is best to change that (it's new in v12). I'm interested to in feedback on the alternative approach, as attached. The final patch to include all the rest of columns shown by pg_stat_file() is more of an idea/proposal and not sure if it'll be desirable. But pg_ls_tmpdir() is essentially the same as my v1 patch. This is intended to be mostly independent of any fix to the WARNING I reported [1]. Since my patch collapses pg_ls_dir into pg_ls_dir_files, we'd only need to fix one place. I'm planning to eventually look into Tom's suggestion of returning tuplestore to fix that, and maybe rebase this patchset on top of that. -- Justin [1] https://www.postgresql.org/message-id/flat/20200308173103.GC1357%40telsasoft.com [2] https://www.postgresql.org/message-id/20191214224735.GA28433%40telsasoft.com
>From 2c4b2c408490ecde3cfb4e336a78942f7a6f8197 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 27 Dec 2019 23:34:14 -0600 Subject: [PATCH v9 01/11] BUG: in errmsg Note there's two changes here. Should backpatch to v12, where pg_ls_tmpdir was added. --- src/backend/utils/adt/genfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 3741b87486..897b11a77d 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -590,7 +590,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) if (stat(path, &attrib) < 0) ereport(ERROR, (errcode_for_file_access(), - errmsg("could not stat directory \"%s\": %m", dir))); + errmsg("could not stat file \"%s\": %m", path))); /* Ignore anything but regular files */ if (!S_ISREG(attrib.st_mode)) -- 2.17.0
>From f3ef0c6ff664f2f26e95ce97e8b50a813bd1aab8 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 16:50:07 -0600 Subject: [PATCH v9 02/11] Document historic behavior about hiding directories and special files Should backpatch to v10: tmpdir, waldir and archive_statusdir --- doc/src/sgml/func.sgml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 323366feb6..4c0ea5ab3f 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21450,6 +21450,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); (mtime) of each file in the log directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> @@ -21461,6 +21462,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> @@ -21473,6 +21475,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> -- 2.17.0
>From b1fb8ef86178fff474f0205143b6f37e156d0d9f Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 17:12:04 -0600 Subject: [PATCH v9 03/11] Document historic behavior about hiding directories and special files Should backpatch to v12: tmpdir --- doc/src/sgml/func.sgml | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 4c0ea5ab3f..fc4d7f0f78 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21489,6 +21489,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. + Filenames beginning with a dot, directories, and other special files are not shown. </para> <indexterm> -- 2.17.0
>From f1350c7880c8ca94358c114808f90d9fc94aa08f Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 17:15:02 -0500 Subject: [PATCH v9 04/11] Add pg_ls_dir_metadata to list a dir with file metadata.. Need catversion bumped? --- src/backend/utils/adt/genfile.c | 51 ++++++++++++++++++++++++++------- src/include/catalog/pg_proc.dat | 6 ++++ 2 files changed, 47 insertions(+), 10 deletions(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 897b11a77d..4699aea568 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -524,7 +524,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) /* Generic function to return a directory listing of files */ static Datum -pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) +pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok) { FuncCallContext *funcctx; struct dirent *de; @@ -540,13 +540,17 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) fctx = palloc(sizeof(directory_fctx)); - tupdesc = CreateTemplateTupleDesc(3); + tupdesc = CreateTemplateTupleDesc(dir_ok ? 4:3); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name", TEXTOID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size", INT8OID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification", TIMESTAMPTZOID, -1, 0); + if (dir_ok) + TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", + BOOLOID, -1, 0); + funcctx->tuple_desc = BlessTupleDesc(tupdesc); fctx->location = pstrdup(dir); @@ -575,8 +579,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL) { - Datum values[3]; - bool nulls[3]; + Datum values[4]; + bool nulls[4]; char path[MAXPGPATH * 2]; struct stat attrib; HeapTuple tuple; @@ -592,13 +596,21 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", path))); - /* Ignore anything but regular files */ - if (!S_ISREG(attrib.st_mode)) + if (S_ISDIR(attrib.st_mode)) + { + if (!dir_ok) + continue; + } + else if (!S_ISREG(attrib.st_mode)) + /* Ignore anything but regular files */ continue; values[0] = CStringGetTextDatum(de->d_name); values[1] = Int64GetDatum((int64) attrib.st_size); values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (dir_ok) + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + memset(nulls, 0, sizeof(nulls)); tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); @@ -613,14 +625,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok) Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, false); + return pg_ls_dir_files(fcinfo, Log_directory, false, false); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, false); + return pg_ls_dir_files(fcinfo, XLOGDIR, false, false); } /* @@ -638,7 +650,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) tblspc))); TempTablespacePath(path, tblspc); - return pg_ls_dir_files(fcinfo, path, true); + return pg_ls_dir_files(fcinfo, path, true, false); } /* @@ -667,5 +679,24 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS) Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true); + return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false); +} + +/* + * Function to return the list of files and metadata in an arbitrary directory. + */ +Datum +pg_ls_dir_metadata(PG_FUNCTION_ARGS) +{ + char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); + bool missing_ok = false; + bool include_dot_dirs = false; + + if (!PG_ARGISNULL(1)) + missing_ok = PG_GETARG_BOOL(1); + if (!PG_ARGISNULL(2)) + /* XXX: Not implemented */ + include_dot_dirs = PG_GETARG_BOOL(2); + + return pg_ls_dir_files(fcinfo, dirname, missing_ok, true); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7fb574f9dc..0a1859f709 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10741,6 +10741,12 @@ proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', proargnames => '{tablespace,name,size,modification}', prosrc => 'pg_ls_tmpdir_1arg' }, +{ oid => '5032', descr => 'list directory with metadata', + proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool', + proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}', + prosrc => 'pg_ls_dir_metadata' }, # hash partitioning constraint function { oid => '5028', descr => 'hash partition CHECK constraint', -- 2.17.0
>From a90befd3d648bb591af4aba2f903e90b02640bb0 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 22:40:24 -0500 Subject: [PATCH v9 05/11] Add tests exercizing pg_ls_tmpdir.. ..and its backing function pg_ls_dir_files --- src/test/regress/input/tablespace.source | 5 +++++ src/test/regress/output/tablespace.source | 8 ++++++++ 2 files changed, 13 insertions(+) diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index a5f61a35dc..0b9cfe615e 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -11,6 +11,11 @@ DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 162b591b31..a42714bf40 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -13,6 +13,14 @@ SELECT spcoptions FROM pg_tablespace WHERE spcname = 'regress_tblspacewith'; DROP TABLESPACE regress_tblspacewith; -- create a tablespace we can use CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; +-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() +SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail -- 2.17.0
>From 018b74872089021ba1319aef01ebfb8154ee4b58 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:57:54 -0500 Subject: [PATCH v9 06/11] pg_ls_tmpdir to show "isdir" argument.. similar to pg_stat_file(). It's worth breaking the function's return type, since core postgres creates "shared filesets" underneath the temp dirs, and it's unreasonable to not show them here, and the alternative query to show them is unreasaonbly complicated. Note that "dir_ok" means both: do not skip dirs, *and* show "isdir" column. The other functions both skip dirs and only show 3 columns, and ls_tmpdir is changing to both show dirs and show 4 columns. It wouldn't make sense to skip dirs and then show a 4th column always false. Need catversion bump --- doc/src/sgml/func.sgml | 22 ++++++++++++---------- src/backend/utils/adt/genfile.c | 4 ++-- src/include/catalog/pg_proc.dat | 8 ++++---- src/test/regress/output/tablespace.source | 4 ++-- 4 files changed, 20 insertions(+), 18 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index fc4d7f0f78..7c73d3b82a 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21382,12 +21382,12 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the - temporary directory for <parameter>tablespace</parameter>. If - <parameter>tablespace</parameter> is not provided, the - <literal>pg_default</literal> tablespace is used. Access is granted - to members of the <literal>pg_monitor</literal> role and may be - granted to other non-superuser roles. + For the temporary directory within <parameter>tablespace</parameter>, + list each file's name, size, last modification time, and a boolean + indicating if it is a directory. If <parameter>tablespace</parameter> + is not provided, the <literal>pg_default</literal> tablespace is used. + Access is granted to members of the <literal>pg_monitor</literal> role + and may be granted to other non-superuser roles. </entry> </row> <row> @@ -21482,14 +21482,16 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <primary>pg_ls_tmpdir</primary> </indexterm> <para> - <function>pg_ls_tmpdir</function> returns the name, size, and last modified - time (mtime) of each file in the temporary file directory for the specified - <parameter>tablespace</parameter>. If <parameter>tablespace</parameter> is + <function>pg_ls_tmpdir</function> lists each file in the temporary file + directory for the specified <parameter>tablespace</parameter>, along with + its size, last modified time (mtime) and a boolean indicating if the file is a + directory. Directories are used for temporary files shared by parallel + processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 4699aea568..7168e249a4 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -522,7 +522,7 @@ pg_ls_dir_1arg(PG_FUNCTION_ARGS) return pg_ls_dir(fcinfo); } -/* Generic function to return a directory listing of files */ +/* Generic function to return a directory listing of files (and optionally dirs) */ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok) { @@ -650,7 +650,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) tblspc))); TempTablespacePath(path, tblspc); - return pg_ls_dir_files(fcinfo, path, true, false); + return pg_ls_dir_files(fcinfo, path, true, true); } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 0a1859f709..7789d029ea 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10733,13 +10733,13 @@ { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz}', proargmodes => '{i,o,o,o}', - proargnames => '{tablespace,name,size,modification}', + proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', + proargnames => '{tablespace,name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index a42714bf40..1c88e914e3 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,8 +17,8 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | modification | isdir +------+------+--------------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0
>From 907f88c171bd22f382b707da1cdbbdfd63fb177a Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 22:52:14 -0500 Subject: [PATCH v9 07/11] Add pg_ls_dir_recurse to show dir recursively.. ..possibly there's a better place to put this, like maybe a doc-only example ? TODO: src/backend/catalog/system_views.sql:REVOKE EXECUTE ON FUNCTION pg_ls_dir(text) FROM public; Need catversion bumped ? --- doc/src/sgml/func.sgml | 13 +++++++++++++ src/include/catalog/pg_proc.dat | 8 ++++++++ 2 files changed, 21 insertions(+) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 7c73d3b82a..672cbab7b9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21442,6 +21442,19 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); empty directory from an non-existent directory. </para> + <indexterm> + <primary>pg_ls_dir_recurse</primary> + </indexterm> + <para> + <function>pg_ls_dir_recurse</function> recursively lists the files + in the specified directory. + <parameter>missing_ok</parameter> ??? + To recusively list temporary directories in all tablespaces: +<programlisting> +SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'')||suffix, 'base/pgsql_tmp')dir FROM pg_tablespace b, pg_control_system() pcs, format('/PG_%s_%s', left(current_setting('server_version_num'), 2), catalog_version_no) suffix)dir, pg_ls_dir_recurse(dir)a; +</programlisting> + </para> + <indexterm> <primary>pg_ls_logdir</primary> </indexterm> diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 7789d029ea..cc2c6f6571 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6143,6 +6143,14 @@ proname => 'pg_ls_dir', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'text', proargtypes => 'text bool bool', prosrc => 'pg_ls_dir' }, + +{ oid => '8511', descr => 'list all files in a directory recursively', + proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', + provolatile => 'v', prorettype => 'record', proargtypes => 'text', + proallargtypes => '{text,text,int8,timestamptz,bool}', + proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', + prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" }, + { oid => '2626', descr => 'sleep for the specified time in seconds', proname => 'pg_sleep', provolatile => 'v', prorettype => 'void', proargtypes => 'float8', prosrc => 'pg_sleep' }, -- 2.17.0
>From 78e8824063de0170022595644e1be68e341dcb10 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Sun, 8 Mar 2020 23:51:22 -0500 Subject: [PATCH v9 08/11] generalize pg_ls_dir_files and retire pg_ls_dir --- src/backend/utils/adt/genfile.c | 172 +++++++++++++++----------------- 1 file changed, 79 insertions(+), 93 deletions(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 7168e249a4..888560f78e 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -42,6 +42,14 @@ typedef struct bool include_dot_dirs; } directory_fctx; +static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags); + +#define FLAG_ISDIR (1<<0) /* Show column: isdir */ +#define FLAG_METADATA (1<<1) /* Show columns: mtime, size */ +#define FLAG_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */ +#define FLAG_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */ +#define FLAG_SKIP_HIDDEN (1<<4) /* Do not show anything begining with . */ +#define FLAG_SKIP_DIRS (1<<5) /* Do not show directories */ /* * Convert a "text" filename argument to C string, and check it's allowable. @@ -446,67 +454,9 @@ pg_stat_file_1arg(PG_FUNCTION_ARGS) Datum pg_ls_dir(PG_FUNCTION_ARGS) { - FuncCallContext *funcctx; - struct dirent *de; - directory_fctx *fctx; - MemoryContext oldcontext; - - if (SRF_IS_FIRSTCALL()) - { - bool missing_ok = false; - bool include_dot_dirs = false; - - /* check the optional arguments */ - if (PG_NARGS() == 3) - { - if (!PG_ARGISNULL(1)) - missing_ok = PG_GETARG_BOOL(1); - if (!PG_ARGISNULL(2)) - include_dot_dirs = PG_GETARG_BOOL(2); - } - - funcctx = SRF_FIRSTCALL_INIT(); - oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); - - fctx = palloc(sizeof(directory_fctx)); - fctx->location = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); - - fctx->include_dot_dirs = include_dot_dirs; - fctx->dirdesc = AllocateDir(fctx->location); - - if (!fctx->dirdesc) - { - if (missing_ok && errno == ENOENT) - { - MemoryContextSwitchTo(oldcontext); - SRF_RETURN_DONE(funcctx); - } - else - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not open directory \"%s\": %m", - fctx->location))); - } - funcctx->user_fctx = fctx; - MemoryContextSwitchTo(oldcontext); - } - - funcctx = SRF_PERCALL_SETUP(); - fctx = (directory_fctx *) funcctx->user_fctx; - - while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL) - { - if (!fctx->include_dot_dirs && - (strcmp(de->d_name, ".") == 0 || - strcmp(de->d_name, "..") == 0)) - continue; - - SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name)); - } - - FreeDir(fctx->dirdesc); - - SRF_RETURN_DONE(funcctx); + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS); } /* @@ -519,17 +469,44 @@ pg_ls_dir(PG_FUNCTION_ARGS) Datum pg_ls_dir_1arg(PG_FUNCTION_ARGS) { - return pg_ls_dir(fcinfo); + text *filename_t = PG_GETARG_TEXT_PP(0); + char *filename = convert_and_check_filename(filename_t); + return pg_ls_dir_files(fcinfo, filename, FLAG_SKIP_DOT_DIRS); } /* Generic function to return a directory listing of files (and optionally dirs) */ static Datum -pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool dir_ok) +pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) { FuncCallContext *funcctx; struct dirent *de; directory_fctx *fctx; + /* isdir depends on metadata */ + Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA)); + /* Unreasonable to show isdir and skip dirs */ + Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS)); + + /* check the optional arguments */ + if (PG_NARGS() == 3) + { + if (!PG_ARGISNULL(1)) + { + if (PG_GETARG_BOOL(1)) + flags |= FLAG_MISSING_OK; + else + flags &= ~FLAG_MISSING_OK; + } + + if (!PG_ARGISNULL(2)) + { + if (PG_GETARG_BOOL(2)) + flags &= ~FLAG_SKIP_DOT_DIRS; + else + flags |= FLAG_SKIP_DOT_DIRS; + } + } + if (SRF_IS_FIRSTCALL()) { MemoryContext oldcontext; @@ -540,16 +517,20 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool fctx = palloc(sizeof(directory_fctx)); - tupdesc = CreateTemplateTupleDesc(dir_ok ? 4:3); + tupdesc = CreateTemplateTupleDesc((flags&FLAG_ISDIR) ? 4 : + (flags&FLAG_METADATA) ? 3 : 1); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name", TEXTOID, -1, 0); - TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size", - INT8OID, -1, 0); - TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification", - TIMESTAMPTZOID, -1, 0); - if (dir_ok) - TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", - BOOLOID, -1, 0); + if (flags&FLAG_METADATA) + { + TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size", + INT8OID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification", + TIMESTAMPTZOID, -1, 0); + if (flags&FLAG_ISDIR) + TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", + BOOLOID, -1, 0); + } funcctx->tuple_desc = BlessTupleDesc(tupdesc); @@ -558,7 +539,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool if (!fctx->dirdesc) { - if (missing_ok && errno == ENOENT) + if (flags&FLAG_MISSING_OK && errno == ENOENT) { MemoryContextSwitchTo(oldcontext); SRF_RETURN_DONE(funcctx); @@ -585,8 +566,14 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool struct stat attrib; HeapTuple tuple; + if (flags&FLAG_SKIP_DOT_DIRS && + (strcmp(de->d_name, ".") == 0 || + strcmp(de->d_name, "..") == 0)) + continue; + /* Skip hidden files */ - if (de->d_name[0] == '.') + if (flags&FLAG_SKIP_HIDDEN && + de->d_name[0] == '.') continue; /* Get the file info */ @@ -598,18 +585,21 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool if (S_ISDIR(attrib.st_mode)) { - if (!dir_ok) + if (flags&FLAG_SKIP_DIRS) continue; } else if (!S_ISREG(attrib.st_mode)) - /* Ignore anything but regular files */ continue; - values[0] = CStringGetTextDatum(de->d_name); - values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); - if (dir_ok) - values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + if (flags & FLAG_METADATA) + { + values[0] = CStringGetTextDatum(de->d_name); + values[1] = Int64GetDatum((int64) attrib.st_size); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + if (flags & FLAG_ISDIR) + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + } else + SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name)); memset(nulls, 0, sizeof(nulls)); @@ -625,14 +615,16 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, bool missing_ok, bool Datum pg_ls_logdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, Log_directory, false, false); + return pg_ls_dir_files(fcinfo, Log_directory, + FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* Function to return the list of files in the WAL directory */ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR, false, false); + return pg_ls_dir_files(fcinfo, XLOGDIR, + FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* @@ -650,7 +642,8 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) tblspc))); TempTablespacePath(path, tblspc); - return pg_ls_dir_files(fcinfo, path, true, true); + return pg_ls_dir_files(fcinfo, path, + FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA|FLAG_ISDIR); } /* @@ -679,7 +672,8 @@ pg_ls_tmpdir_1arg(PG_FUNCTION_ARGS) Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { - return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", true, false); + return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", + FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* @@ -689,14 +683,6 @@ Datum pg_ls_dir_metadata(PG_FUNCTION_ARGS) { char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); - bool missing_ok = false; - bool include_dot_dirs = false; - - if (!PG_ARGISNULL(1)) - missing_ok = PG_GETARG_BOOL(1); - if (!PG_ARGISNULL(2)) - /* XXX: Not implemented */ - include_dot_dirs = PG_GETARG_BOOL(2); - return pg_ls_dir_files(fcinfo, dirname, missing_ok, true); + return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_ISDIR); } -- 2.17.0
>From 43c8bc8611b53ff172e5c7caa052d80d9e4ac3e9 Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Fri, 6 Mar 2020 17:23:51 -0600 Subject: [PATCH v9 09/11] pg_ls_logdir to ignore error if initial/top dir is missing --- src/backend/utils/adt/genfile.c | 2 +- src/test/regress/input/tablespace.source | 4 ++++ src/test/regress/output/tablespace.source | 7 +++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 888560f78e..4ce39516d7 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -616,7 +616,7 @@ Datum pg_ls_logdir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, Log_directory, - FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); + FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* Function to return the list of files in the WAL directory */ diff --git a/src/test/regress/input/tablespace.source b/src/test/regress/input/tablespace.source index 0b9cfe615e..2a1268e17c 100644 --- a/src/test/regress/input/tablespace.source +++ b/src/test/regress/input/tablespace.source @@ -16,6 +16,10 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index 1c88e914e3..dacaafcae6 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -21,6 +21,13 @@ SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspa ------+------+--------------+------- (0 rows) +-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. +-- The name='' condition is never true, so the function runs to completion but returns zero rows. +SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; + name | size | modification +------+------+-------------- +(0 rows) + -- try setting and resetting some properties for the new tablespace ALTER TABLESPACE regress_tblspace SET (random_page_cost = 1.0, seq_page_cost = 1.1); ALTER TABLESPACE regress_tblspace SET (some_nonexistent_parameter = true); -- fail -- 2.17.0
>From adf2977cf2553a10a9c2cb6cb3a52590365783bf Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 01:00:42 -0500 Subject: [PATCH v9 10/11] pg_ls_*dir to show directories and "isdir" column.. pg_ls_logdir, pg_ls_waldir, pg_ls_archive_statusdir ..FLAG_ISDIR is collapsed into FLAG_METADATA. Need catversion bump --- doc/src/sgml/func.sgml | 39 +++++++++++++++++-------------- src/backend/utils/adt/genfile.c | 41 +++++++++++---------------------- src/include/catalog/pg_proc.dat | 20 ++++++++-------- 3 files changed, 46 insertions(+), 54 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 672cbab7b9..d0b782d803 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21348,8 +21348,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the log - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the log directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21359,8 +21360,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - directory. Access is granted to members of the <literal>pg_monitor</literal> + For each file in the WAL directory, list the file's name, size, last + modification time, and a boolean indicating if it is a directory. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> </row> @@ -21370,8 +21372,9 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - List the name, size, and last modification time of files in the WAL - archive status directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file's + name, size, last modification time, and a boolean indicating if it is a + directory. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21459,36 +21462,38 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'') <primary>pg_ls_logdir</primary> </indexterm> <para> - <function>pg_ls_logdir</function> returns the name, size, and last modified time - (mtime) of each file in the log directory. By default, only superusers + <function>pg_ls_logdir</function> lists each file in the log directory, + along with file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_waldir</primary> </indexterm> <para> - <function>pg_ls_waldir</function> returns the name, size, and last modified time - (mtime) of each file in the write ahead log (WAL) directory. By - default only superusers and members of the <literal>pg_monitor</literal> role + <function>pg_ls_waldir</function> lists each file in the WAL directory, + along with the file's size, last modification time, and a boolean + indicating if the file is a directory. By default, only superusers + and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> <primary>pg_ls_archive_statusdir</primary> </indexterm> <para> - <function>pg_ls_archive_statusdir</function> returns the name, size, and - last modified time (mtime) of each file in the WAL archive status - directory <filename>pg_wal/archive_status</filename>. By default only + <function>pg_ls_archive_statusdir</function> lists each file in the WAL + archive status directory, along with the file's size, last modification + time, and a boolean indicating if the file is a directory. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. - Filenames beginning with a dot, directories, and other special files are not shown. + Filenames beginning with a dot and special file types are not shown. </para> <indexterm> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 4ce39516d7..387114d4ee 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -44,12 +44,10 @@ typedef struct static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags); -#define FLAG_ISDIR (1<<0) /* Show column: isdir */ -#define FLAG_METADATA (1<<1) /* Show columns: mtime, size */ -#define FLAG_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */ -#define FLAG_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */ -#define FLAG_SKIP_HIDDEN (1<<4) /* Do not show anything begining with . */ -#define FLAG_SKIP_DIRS (1<<5) /* Do not show directories */ +#define FLAG_METADATA (1<<0) /* Show columns: mtime, size */ +#define FLAG_MISSING_OK (1<<1) /* Ignore ENOENT if the toplevel dir is missing */ +#define FLAG_SKIP_DOT_DIRS (1<<2) /* Do not show . or .. */ +#define FLAG_SKIP_HIDDEN (1<<3) /* Do not show anything begining with . */ /* * Convert a "text" filename argument to C string, and check it's allowable. @@ -482,11 +480,6 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) struct dirent *de; directory_fctx *fctx; - /* isdir depends on metadata */ - Assert(!(flags&FLAG_ISDIR) || (flags&FLAG_METADATA)); - /* Unreasonable to show isdir and skip dirs */ - Assert(!(flags&FLAG_ISDIR) || !(flags&FLAG_SKIP_DIRS)); - /* check the optional arguments */ if (PG_NARGS() == 3) { @@ -517,8 +510,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) fctx = palloc(sizeof(directory_fctx)); - tupdesc = CreateTemplateTupleDesc((flags&FLAG_ISDIR) ? 4 : - (flags&FLAG_METADATA) ? 3 : 1); + tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 4 : 1); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name", TEXTOID, -1, 0); if (flags&FLAG_METADATA) @@ -527,9 +519,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) INT8OID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification", TIMESTAMPTZOID, -1, 0); - if (flags&FLAG_ISDIR) - TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", - BOOLOID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", + BOOLOID, -1, 0); } funcctx->tuple_desc = BlessTupleDesc(tupdesc); @@ -584,10 +575,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) errmsg("could not stat file \"%s\": %m", path))); if (S_ISDIR(attrib.st_mode)) - { - if (flags&FLAG_SKIP_DIRS) - continue; - } + ; /* Do nothing, fall through */ else if (!S_ISREG(attrib.st_mode)) continue; @@ -596,8 +584,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) values[0] = CStringGetTextDatum(de->d_name); values[1] = Int64GetDatum((int64) attrib.st_size); values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); - if (flags & FLAG_ISDIR) - values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); } else SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name)); @@ -616,7 +603,7 @@ Datum pg_ls_logdir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, Log_directory, - FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); + FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* Function to return the list of files in the WAL directory */ @@ -624,7 +611,7 @@ Datum pg_ls_waldir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, XLOGDIR, - FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); + FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* @@ -643,7 +630,7 @@ pg_ls_tmpdir(FunctionCallInfo fcinfo, Oid tblspc) TempTablespacePath(path, tblspc); return pg_ls_dir_files(fcinfo, path, - FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA|FLAG_ISDIR); + FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* @@ -673,7 +660,7 @@ Datum pg_ls_archive_statusdir(PG_FUNCTION_ARGS) { return pg_ls_dir_files(fcinfo, XLOGDIR "/archive_status", - FLAG_MISSING_OK|FLAG_SKIP_DIRS|FLAG_SKIP_HIDDEN|FLAG_METADATA); + FLAG_MISSING_OK|FLAG_SKIP_HIDDEN|FLAG_METADATA); } /* @@ -684,5 +671,5 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS) { char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0)); - return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA|FLAG_ISDIR); + return pg_ls_dir_files(fcinfo, dirname, FLAG_METADATA); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index cc2c6f6571..0e5a570285 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6149,7 +6149,7 @@ provolatile => 'v', prorettype => 'record', proargtypes => 'text', proallargtypes => '{text,text,int8,timestamptz,bool}', proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', - prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false, true) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false, true)a WHERE x.isdir) SELECT * FROM x" }, + prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" }, { oid => '2626', descr => 'sleep for the specified time in seconds', proname => 'pg_sleep', provolatile => 'v', prorettype => 'void', @@ -10725,18 +10725,18 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz}', proargmodes => '{o,o,o}', - proargnames => '{name,size,modification}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', + proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz}', - proargmodes => '{o,o,o}', proargnames => '{name,size,modification}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', + proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', @@ -10751,9 +10751,9 @@ prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', - provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool bool', - proallargtypes => '{text,bool,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,i,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,dir_ok,name,size,modification,isdir}', + provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', prosrc => 'pg_ls_dir_metadata' }, # hash partitioning constraint function -- 2.17.0
>From e8960ff73967e5ca855ee8f392e6a7711e2ee64c Mon Sep 17 00:00:00 2001 From: Justin Pryzby <pryz...@telsasoft.com> Date: Mon, 9 Mar 2020 21:56:21 -0500 Subject: [PATCH v9 11/11] pg_ls_*dir to return all the metadata from pg_stat_file.. ..but it doesn't seem worth factoring out the common bits, since stat_file doesn't return a name, so all the field numbers are off by one. Need catversion bump --- doc/src/sgml/func.sgml | 30 +++++++++------------ src/backend/utils/adt/genfile.c | 32 ++++++++++++++++------- src/include/catalog/pg_proc.dat | 30 ++++++++++----------- src/test/regress/output/tablespace.source | 8 +++--- 4 files changed, 55 insertions(+), 45 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index d0b782d803..ae6c004a6d 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21348,8 +21348,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the log directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the log directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21360,8 +21359,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL directory, list the file's name, size, last - modification time, and a boolean indicating if it is a directory. + For each file in the WAL directory, list the file and its metadata. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21372,9 +21370,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </entry> <entry><type>setof record</type></entry> <entry> - For each file in the WAL archive status directory, list the file's - name, size, last modification time, and a boolean indicating if it is a - directory. Access is granted to members of the + For each file in the WAL archive status directory, list the file and its metadata. + Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. </entry> @@ -21386,8 +21383,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <entry><type>setof record</type></entry> <entry> For the temporary directory within <parameter>tablespace</parameter>, - list each file's name, size, last modification time, and a boolean - indicating if it is a directory. If <parameter>tablespace</parameter> + list each file and its metadata. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. Access is granted to members of the <literal>pg_monitor</literal> role and may be granted to other non-superuser roles. @@ -21463,8 +21459,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'') </indexterm> <para> <function>pg_ls_logdir</function> lists each file in the log directory, - along with file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. Filenames beginning with a dot and special file types are not shown. @@ -21475,8 +21471,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'') </indexterm> <para> <function>pg_ls_waldir</function> lists each file in the WAL directory, - along with the file's size, last modification time, and a boolean - indicating if the file is a directory. By default, only superusers + along with the metadata columns returned by <function>pg_stat_file</function>. + By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21488,8 +21484,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'') </indexterm> <para> <function>pg_ls_archive_statusdir</function> lists each file in the WAL - archive status directory, along with the file's size, last modification - time, and a boolean indicating if the file is a directory. By default, only + archive status directory, along with the metadata columns returned by + <function>pg_stat_file</function>. By default, only superusers and members of the <literal>pg_monitor</literal> role can use this function. Access may be granted to others using <command>GRANT</command>. @@ -21502,8 +21498,8 @@ SELECT * FROM (SELECT DISTINCT COALESCE(NULLIF(pg_tablespace_location(b.oid),'') <para> <function>pg_ls_tmpdir</function> lists each file in the temporary file directory for the specified <parameter>tablespace</parameter>, along with - its size, last modified time (mtime) and a boolean indicating if the file is a - directory. Directories are used for temporary files shared by parallel + the metadata columns returned by <function>pg_stat_file</function>. + Directories are used for temporary files shared by parallel processes. If <parameter>tablespace</parameter> is not provided, the <literal>pg_default</literal> tablespace is used. By default only superusers and members of the <literal>pg_monitor</literal> diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 387114d4ee..1fe7e0181d 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -510,16 +510,22 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) fctx = palloc(sizeof(directory_fctx)); - tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 4 : 1); + tupdesc = CreateTemplateTupleDesc((flags&FLAG_METADATA) ? 7 : 1); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "name", TEXTOID, -1, 0); if (flags&FLAG_METADATA) { TupleDescInitEntry(tupdesc, (AttrNumber) 2, "size", INT8OID, -1, 0); - TupleDescInitEntry(tupdesc, (AttrNumber) 3, "modification", + TupleDescInitEntry(tupdesc, (AttrNumber) 3, + "access", TIMESTAMPTZOID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 4, "modification", TIMESTAMPTZOID, -1, 0); - TupleDescInitEntry(tupdesc, (AttrNumber) 4, "isdir", + TupleDescInitEntry(tupdesc, (AttrNumber) 5, + "change", TIMESTAMPTZOID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 6, + "creation", TIMESTAMPTZOID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 7, "isdir", BOOLOID, -1, 0); } @@ -551,8 +557,8 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) while ((de = ReadDir(fctx->dirdesc, fctx->location)) != NULL) { - Datum values[4]; - bool nulls[4]; + Datum values[7]; + bool nulls[7]; char path[MAXPGPATH * 2]; struct stat attrib; HeapTuple tuple; @@ -579,17 +585,25 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags) else if (!S_ISREG(attrib.st_mode)) continue; + memset(nulls, 0, sizeof(nulls)); if (flags & FLAG_METADATA) { values[0] = CStringGetTextDatum(de->d_name); values[1] = Int64GetDatum((int64) attrib.st_size); - values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); - values[3] = BoolGetDatum(S_ISDIR(attrib.st_mode)); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_atime)); + values[3] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_mtime)); + /* Unix has file status change time, while Win32 has creation time */ +#if !defined(WIN32) && !defined(__CYGWIN__) + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); + nulls[5] = true; +#else + nulls[4] = true; + values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime)); +#endif + values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode)); } else SRF_RETURN_NEXT(funcctx, CStringGetTextDatum(de->d_name)); - memset(nulls, 0, sizeof(nulls)); - tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 0e5a570285..b571666a79 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6147,9 +6147,9 @@ { oid => '8511', descr => 'list all files in a directory recursively', proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text', - proallargtypes => '{text,text,int8,timestamptz,bool}', - proargnames => '{dirname,name,size,modification,isdir}', proargmodes => '{i,o,o,o,o}', - prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size, a.modification, a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" }, + proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}', + prolang => 'sql', prosrc => "WITH RECURSIVE x AS (SELECT * FROM pg_ls_dir_metadata(dirname, true, false) UNION ALL SELECT x.name||'/'||a.name, a.size,a.access,a.modification,a.change,a.creation,a.isdir FROM x, pg_ls_dir_metadata(dirname||'/'||x.name, true, false)a WHERE x.isdir) SELECT * FROM x" }, { oid => '2626', descr => 'sleep for the specified time in seconds', proname => 'pg_sleep', provolatile => 'v', prorettype => 'void', @@ -10725,35 +10725,35 @@ { oid => '3353', descr => 'list files in the log directory', proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_logdir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' }, { oid => '3354', descr => 'list of files in the WAL directory', proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_waldir' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' }, { oid => '5031', descr => 'list of files in the archive_status directory', proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{text,int8,timestamptz,bool}', - proargmodes => '{o,o,o,o}', proargnames => '{name,size,modification,isdir}', + proargtypes => '', proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', + proargmodes => '{o,o,o,o,o,o,o}', proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_archive_statusdir' }, { oid => '5029', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{text,int8,timestamptz,bool}', proargmodes => '{o,o,o,o}', - proargnames => '{name,size,modification,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, + proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' }, { oid => '5030', descr => 'list files in the pgsql_tmp directory', proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'oid', - proallargtypes => '{oid,text,int8,timestamptz,bool}', proargmodes => '{i,o,o,o,o}', - proargnames => '{tablespace,name,size,modification,isdir}', + proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}', + proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_1arg' }, { oid => '5032', descr => 'list directory with metadata', proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool', - proallargtypes => '{text,bool,bool,text,int8,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o}', - proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,modification,isdir}', + proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}', + proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_dir_metadata' }, # hash partitioning constraint function diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source index dacaafcae6..1e1e02b589 100644 --- a/src/test/regress/output/tablespace.source +++ b/src/test/regress/output/tablespace.source @@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@'; -- The name='' condition is never true, so the function runs to completion but returns zero rows. -- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir() SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist'; - name | size | modification | isdir -------+------+--------------+------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet. -- The name='' condition is never true, so the function runs to completion but returns zero rows. SELECT * FROM pg_ls_logdir() WHERE name='Does not exist'; - name | size | modification -------+------+-------------- + name | size | access | modification | change | creation | isdir +------+------+--------+--------------+--------+----------+------- (0 rows) -- try setting and resetting some properties for the new tablespace -- 2.17.0