On Wed, Jan 26, 2022 at 03:29:29PM +0900, Michael Paquier wrote:
> On Tue, Jan 25, 2022 at 09:44:26PM -0600, Justin Pryzby wrote:
> > It seems like an arbitrary and short-sighted policy to expose a handful of
> > flags in the view for the purpose of retiring ./check_guc, but not expose
> > other
> > flags, because we thought we knew that no user could ever want them.
> >
> > We should either expose all the flags, or should put them into an
> > undocumented
> > function. Otherwise, how would we document the flags argument ? "Shows
> > some
> > of the flags" ? An undocumented function avoids this issue.
>
> My vote would be to have a documented function, with a minimal set of
> the flags exposed and documented, with the option to expand that in
> the future. COMPUTED and EXPLAIN are useful, and allow some of the
> automated tests to happen. NOT_IN_SAMPLE and GUC_NO_SHOW_ALL are less
> useful for the user, and are more developer oriented, but are useful
> for the tests. So having these four seem like a good first cut.
I implemented that (But my own preference would still be for an *undocumented*
function which returns whatever flags we find to be useful to include. Or
alternately, a documented function which exposes every flag).
> +SELECT lower(name) FROM pg_settings_flags WHERE NOT not_in_sample EXCEPT
> +SELECT regexp_replace(ln, '^#?([_[:alpha:]]+) (= .*|[^ ]*$)', '\1') AS guc
> +FROM (SELECT regexp_split_to_table(pg_read_file('postgresql.conf'),
> '\n') AS ln) conf
>
> Tests reading postgresql.conf would break on instances started with a
> custom config_file provided by a command line, no?
Maybe you misunderstood - I'm not reading the file specified by
current_setting('config_file'). Rather, I'm reading
tmp_check/data/postgresql.conf, which is copied from the sample conf.
Do you see an issue with that ?
The regression tests are only intended run from a postgres source dir, and if
someone runs the from somewhere else, and they "fail", I think that's because
they violated their assumption, not because of a problem with the test.
I wondered if it should chomp off anything added by pg_regress --temp-regress.
However that's either going to be a valid guc (or else it would fail some other
test). Or an extention's guc (which this isn't testing), which has a dot, and
which this regex doesn't match, so doesn't cause false positives.
--
Justin
>From d09cc770a3e307f55843942a850f7859c63d4c76 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sun, 5 Dec 2021 23:22:04 -0600
Subject: [PATCH 1/2] check_guc: fix absurd number of false positives
Avoid false positives;
Avoid various assumptions;
Avoid list of exceptions;
Simplify shell/awk/sed/grep;
This requires GNU awk for RS as a regex.
---
src/backend/utils/misc/check_guc | 69 +++++---------------------------
1 file changed, 10 insertions(+), 59 deletions(-)
diff --git a/src/backend/utils/misc/check_guc b/src/backend/utils/misc/check_guc
index b171ef0e4ff..323ca13191b 100755
--- a/src/backend/utils/misc/check_guc
+++ b/src/backend/utils/misc/check_guc
@@ -1,78 +1,29 @@
-#!/bin/sh
+#! /bin/sh
+set -e
-## currently, this script makes a lot of assumptions:
+## this script makes some assumptions about the formatting of files it parses.
## in postgresql.conf.sample:
## 1) the valid config settings may be preceded by a '#', but NOT '# '
## (we use this to skip comments)
-## 2) the valid config settings will be followed immediately by ' ='
-## (at least one space preceding the '=')
-## in guc.c:
-## 3) the options have PGC_ on the same line as the option
-## 4) the options have '{' on the same line as the option
-
-## Problems
-## 1) Don't know what to do with TRANSACTION ISOLATION LEVEL
-
-## if an option is valid but shows up in only one file (guc.c but not
-## postgresql.conf.sample), it should be listed here so that it
-## can be ignored
-INTENTIONALLY_NOT_INCLUDED="debug_deadlocks in_hot_standby \
-is_superuser lc_collate lc_ctype lc_messages lc_monetary lc_numeric lc_time \
-pre_auth_delay role seed server_encoding server_version server_version_num \
-session_authorization trace_lock_oidmin trace_lock_table trace_locks trace_lwlocks \
-trace_notify trace_userlocks transaction_isolation transaction_read_only \
-zero_damaged_pages"
### What options are listed in postgresql.conf.sample, but don't appear
### in guc.c?
-# grab everything that looks like a setting and convert it to lower case
-SETTINGS=`grep ' =' postgresql.conf.sample |
-grep -v '^# ' | # strip comments
-sed -e 's/^#//' |
-awk '{print $1}'`
-
-SETTINGS=`echo "$SETTINGS" | tr 'A-Z' 'a-z'`
+# grab everything that looks like a setting
+SETTINGS=`sed '/^#[[:alnum:]]/!d; s/^#//; s/ =.*//; /^include/d' postgresql.conf.sample`
for i in $SETTINGS ; do
- hidden=0
## it sure would be nice to replace this with an sql "not in" statement
- ## it doesn't seem to make sense to have things in .sample and not in guc.c
-# for hidethis in $INTENTIONALLY_NOT_INCLUDED ; do
-# if [ "$hidethis" = "$i" ] ; then
-# hidden=1
-# fi
-# done
- if [ "$hidden" -eq 0 ] ; then
- grep -i '"'$i'"' guc.c > /dev/null
- if [ $? -ne 0 ] ; then
- echo "$i seems to be missing from guc.c";
- fi;
- fi
+ grep -i "\"$i\"" guc.c >/dev/null ||
+ echo "$i seems to be missing from guc.c";
done
### What options are listed in guc.c, but don't appear
### in postgresql.conf.sample?
# grab everything that looks like a setting and convert it to lower case
-
-SETTINGS=`grep '{.* PGC_' guc.c | awk '{print $1}' | \
- sed -e 's/{//g' -e 's/"//g' -e 's/,//'`
-
-SETTINGS=`echo "$SETTINGS" | tr 'A-Z' 'a-z'`
-
+SETTINGS=`gawk -F '[",]' 'BEGIN{RS="\n\t\\\\{\n"} /",[[:space:]]*PGC_.*.*gettext_noop/ && !/NOT_IN_SAMPLE/{print tolower($2)}' guc.c`
for i in $SETTINGS ; do
- hidden=0
- ## it sure would be nice to replace this with an sql "not in" statement
- for hidethis in $INTENTIONALLY_NOT_INCLUDED ; do
- if [ "$hidethis" = "$i" ] ; then
- hidden=1
- fi
- done
- if [ "$hidden" -eq 0 ] ; then
- grep -i '#'$i' ' postgresql.conf.sample > /dev/null
- if [ $? -ne 0 ] ; then
- echo "$i seems to be missing from postgresql.conf.sample";
- fi
- fi
+ grep "#$i " postgresql.conf.sample >/dev/null ||
+ echo "$i seems to be missing from postgresql.conf.sample";
done
--
2.17.1
>From ee2b323f329cc6ff1412bfbc5ad0b64c3584ff03 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 8 Dec 2021 12:06:35 -0600
Subject: [PATCH 2/2] Expose GUC flags in SQL function; retire ./check_guc
---
doc/src/sgml/func.sgml | 15 ++++
src/backend/utils/misc/check_guc | 29 -------
src/backend/utils/misc/guc.c | 37 +++++++++
src/include/catalog/pg_proc.dat | 6 ++
src/include/utils/guc.h | 3 +-
src/test/regress/expected/guc.out | 122 ++++++++++++++++++++++++++++++
src/test/regress/sql/guc.sql | 72 ++++++++++++++++++
7 files changed, 254 insertions(+), 30 deletions(-)
delete mode 100755 src/backend/utils/misc/check_guc
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 0ee6974f1c6..cbdbccb63d1 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -23596,6 +23596,21 @@ SELECT pg_type_is_visible('myschema.widget'::regtype);
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_get_flags</primary>
+ </indexterm>
+ <function>pg_get_get_flags</function> ( <parameter>guc</parameter> <type>text</type> )
+ <returnvalue>text[]</returnvalue>
+ </para>
+ <para>
+ Return an array of flags associated with the given GUC, or NULL if the
+ GUC does not exist. Not all flags are exposed; the set of flags which
+ are exposed is subject to change.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/utils/misc/check_guc b/src/backend/utils/misc/check_guc
deleted file mode 100755
index 323ca13191b..00000000000
--- a/src/backend/utils/misc/check_guc
+++ /dev/null
@@ -1,29 +0,0 @@
-#! /bin/sh
-set -e
-
-## this script makes some assumptions about the formatting of files it parses.
-## in postgresql.conf.sample:
-## 1) the valid config settings may be preceded by a '#', but NOT '# '
-## (we use this to skip comments)
-
-### What options are listed in postgresql.conf.sample, but don't appear
-### in guc.c?
-
-# grab everything that looks like a setting
-SETTINGS=`sed '/^#[[:alnum:]]/!d; s/^#//; s/ =.*//; /^include/d' postgresql.conf.sample`
-
-for i in $SETTINGS ; do
- ## it sure would be nice to replace this with an sql "not in" statement
- grep -i "\"$i\"" guc.c >/dev/null ||
- echo "$i seems to be missing from guc.c";
-done
-
-### What options are listed in guc.c, but don't appear
-### in postgresql.conf.sample?
-
-# grab everything that looks like a setting and convert it to lower case
-SETTINGS=`gawk -F '[",]' 'BEGIN{RS="\n\t\\\\{\n"} /",[[:space:]]*PGC_.*.*gettext_noop/ && !/NOT_IN_SAMPLE/{print tolower($2)}' guc.c`
-for i in $SETTINGS ; do
- grep "#$i " postgresql.conf.sample >/dev/null ||
- echo "$i seems to be missing from postgresql.conf.sample";
-done
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 4c94f09c645..d3b56b2007c 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -9634,6 +9634,43 @@ GetConfigOptionByName(const char *name, const char **varname, bool missing_ok)
return _ShowOption(record, true);
}
+/*
+ * Return some flags for the specified GUC, or NULL if it doesn't exist.
+ */
+Datum
+pg_get_guc_flags(PG_FUNCTION_ARGS)
+{
+ char *varname = TextDatumGetCString(PG_GETARG_DATUM(0));
+ struct config_generic *record;
+ int cnt = 0;
+#define MAX_NUM_FLAGS 5
+ Datum flags[MAX_NUM_FLAGS];
+ ArrayType *a;
+
+ record = find_option(varname, false, true, ERROR);
+
+ /* return NULL if no such variable */
+ if (record == NULL)
+ PG_RETURN_NULL();
+
+ if (record->flags & GUC_NO_SHOW_ALL)
+ flags[cnt++] = CStringGetTextDatum("NO_SHOW_ALL");
+ if (record->flags & GUC_NO_RESET_ALL)
+ flags[cnt++] = CStringGetTextDatum("NO_RESET_ALL");
+ if (record->flags & GUC_NOT_IN_SAMPLE)
+ flags[cnt++] = CStringGetTextDatum("NOT_IN_SAMPLE");
+ if (record->flags & GUC_EXPLAIN)
+ flags[cnt++] = CStringGetTextDatum("EXPLAIN");
+ if (record->flags & GUC_RUNTIME_COMPUTED)
+ flags[cnt++] = CStringGetTextDatum("RUNTIME_COMPUTED");
+
+ Assert(cnt <= MAX_NUM_FLAGS);
+
+ /* Returns the record as Datum */
+ a = construct_array(flags, cnt, TEXTOID, -1, false, TYPALIGN_INT);
+ PG_RETURN_ARRAYTYPE_P(a);
+}
+
/*
* Return GUC variable value by variable number; optionally return canonical
* form of name. Return value is palloc'd.
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0859dc81cac..562c1d779cb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6096,6 +6096,12 @@
proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
proargnames => '{name,setting,unit,category,short_desc,extra_desc,context,vartype,source,min_val,max_val,enumvals,boot_val,reset_val,sourcefile,sourceline,pending_restart}',
prosrc => 'show_all_settings' },
+
+{ oid => '8921', descr => 'return flags for specified GUC',
+ proname => 'pg_get_guc_flags', provolatile => 's',
+ prorettype => '_text', proargtypes => 'text',
+ prosrc => 'pg_get_guc_flags' },
+
{ oid => '3329', descr => 'show config file settings',
proname => 'pg_show_all_file_settings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 6bb81707b09..1ac20f85ab3 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -198,8 +198,9 @@ typedef enum
#define GUC_QUALIFIER_SEPARATOR '.'
-/*
+/* --
* bit values in "flags" of a GUC variable
+ * Consider if any new flags should be exposed in pg_get_guc_flags().
*/
#define GUC_LIST_INPUT 0x0001 /* input can be list format */
#define GUC_LIST_QUOTE 0x0002 /* double-quote list elements */
diff --git a/src/test/regress/expected/guc.out b/src/test/regress/expected/guc.out
index 59da91ff04d..29173f3ce8e 100644
--- a/src/test/regress/expected/guc.out
+++ b/src/test/regress/expected/guc.out
@@ -813,3 +813,125 @@ set default_with_oids to f;
-- Should not allow to set it to true.
set default_with_oids to t;
ERROR: tables declared WITH OIDS are not supported
+--
+-- Test GUC categories and flags.
+--
+CREATE TABLE pg_settings_flags AS SELECT name, category,
+ 'NO_SHOW_ALL' =ANY(flags) AS no_show_all,
+ 'NO_RESET_ALL' =ANY(flags) AS no_reset_all,
+ 'NOT_IN_SAMPLE' =ANY(flags) AS not_in_sample,
+ 'EXPLAIN' =ANY(flags) AS guc_explain,
+ 'COMPUTED' =ANY(flags) AS guc_computed
+ FROM pg_show_all_settings() AS psas, pg_get_guc_flags(psas.name) AS flags;
+-- test that GUCS are in postgresql.conf
+SELECT lower(name) FROM pg_settings_flags WHERE NOT not_in_sample EXCEPT
+SELECT regexp_replace(ln, '^#?([_[:alpha:]]+) (= .*|[^ ]*$)', '\1') AS guc
+FROM (SELECT regexp_split_to_table(pg_read_file('postgresql.conf'), '\n') AS ln) conf
+WHERE ln ~ '^#?[[:alpha:]]'
+ORDER BY 1;
+ lower
+-----------------------------
+ config_file
+ plpgsql.check_asserts
+ plpgsql.extra_errors
+ plpgsql.extra_warnings
+ plpgsql.print_strict_params
+ plpgsql.variable_conflict
+(6 rows)
+
+-- test that lines in postgresql.conf that look like GUCs are GUCs
+SELECT regexp_replace(ln, '^#?([_[:alpha:]]+) (= .*|[^ ]*$)', '\1') AS guc
+FROM (SELECT regexp_split_to_table(pg_read_file('postgresql.conf'), '\n') AS ln) conf
+WHERE ln ~ '^#?[[:alpha:]]'
+EXCEPT SELECT lower(name) FROM pg_settings_flags WHERE NOT not_in_sample
+ORDER BY 1;
+ guc
+-------------------
+ include
+ include_dir
+ include_if_exists
+(3 rows)
+
+-- test that section:DEVELOPER GUCs are flagged GUC_NOT_IN_SAMPLE:
+SELECT * FROM pg_settings_flags
+WHERE category='Developer Options' AND NOT not_in_sample
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------+----------+-------------+--------------+---------------+-------------+--------------
+(0 rows)
+
+-- Maybe the converse:
+SELECT * FROM pg_settings_flags
+WHERE category NOT IN ('Developer Options', 'Preset Options') AND not_in_sample
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------------------------+-------------------------------------------------+-------------+--------------+---------------+-------------+--------------
+ application_name | Reporting and Logging / What to Log | f | f | t | f | f
+ transaction_deferrable | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+ transaction_isolation | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+ transaction_read_only | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+(4 rows)
+
+-- Most Query Tuning GUCs are flagged as EXPLAIN:
+SELECT * FROM pg_settings_flags
+WHERE category ~ '^Query Tuning' AND NOT guc_explain
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+---------------------------+--------------------------------------+-------------+--------------+---------------+-------------+--------------
+ default_statistics_target | Query Tuning / Other Planner Options | f | f | f | f | f
+(1 row)
+
+-- Maybe the converse:
+SELECT * FROM pg_settings_flags
+WHERE guc_explain AND NOT category ~ '^Query Tuning|^Resource Usage'
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+---------------------+-------------------------------------------------+-------------+--------------+---------------+-------------+--------------
+ force_parallel_mode | Developer Options | f | f | t | t | f
+ search_path | Client Connection Defaults / Statement Behavior | f | f | f | t | f
+(2 rows)
+
+-- GUCs flagged RUNTIME are Preset
+SELECT * FROM pg_settings_flags
+WHERE guc_computed AND NOT category='Preset Options'
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------+----------+-------------+--------------+---------------+-------------+--------------
+(0 rows)
+
+-- PRESET GUCs are flagged NOT_IN_SAMPLE
+SELECT * FROM pg_settings_flags
+WHERE category='Preset Options' AND NOT not_in_sample
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------+----------+-------------+--------------+---------------+-------------+--------------
+(0 rows)
+
+-- NO_SHOW_ALL implies NO_RESET_ALL:
+SELECT * FROM pg_settings_flags
+WHERE no_show_all AND NOT no_reset_all
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------+----------+-------------+--------------+---------------+-------------+--------------
+(0 rows)
+
+-- Usually the converse:
+SELECT * FROM pg_settings_flags
+WHERE NOT no_show_all AND no_reset_all
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------------------------+-------------------------------------------------+-------------+--------------+---------------+-------------+--------------
+ transaction_deferrable | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+ transaction_isolation | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+ transaction_read_only | Client Connection Defaults / Statement Behavior | f | t | t | f | f
+(3 rows)
+
+-- NO_SHOW_ALL implies NOT_IN_SAMPLE:
+SELECT * FROM pg_settings_flags
+WHERE no_show_all AND NOT not_in_sample
+ORDER BY 1;
+ name | category | no_show_all | no_reset_all | not_in_sample | guc_explain | guc_computed
+------+----------+-------------+--------------+---------------+-------------+--------------
+(0 rows)
+
+DROP TABLE pg_settings_flags;
diff --git a/src/test/regress/sql/guc.sql b/src/test/regress/sql/guc.sql
index c39c11388d5..47b6a233d5c 100644
--- a/src/test/regress/sql/guc.sql
+++ b/src/test/regress/sql/guc.sql
@@ -311,3 +311,75 @@ reset check_function_bodies;
set default_with_oids to f;
-- Should not allow to set it to true.
set default_with_oids to t;
+
+--
+-- Test GUC categories and flags.
+--
+CREATE TABLE pg_settings_flags AS SELECT name, category,
+ 'NO_SHOW_ALL' =ANY(flags) AS no_show_all,
+ 'NO_RESET_ALL' =ANY(flags) AS no_reset_all,
+ 'NOT_IN_SAMPLE' =ANY(flags) AS not_in_sample,
+ 'EXPLAIN' =ANY(flags) AS guc_explain,
+ 'COMPUTED' =ANY(flags) AS guc_computed
+ FROM pg_show_all_settings() AS psas, pg_get_guc_flags(psas.name) AS flags;
+
+-- test that GUCS are in postgresql.conf
+SELECT lower(name) FROM pg_settings_flags WHERE NOT not_in_sample EXCEPT
+SELECT regexp_replace(ln, '^#?([_[:alpha:]]+) (= .*|[^ ]*$)', '\1') AS guc
+FROM (SELECT regexp_split_to_table(pg_read_file('postgresql.conf'), '\n') AS ln) conf
+WHERE ln ~ '^#?[[:alpha:]]'
+ORDER BY 1;
+
+-- test that lines in postgresql.conf that look like GUCs are GUCs
+SELECT regexp_replace(ln, '^#?([_[:alpha:]]+) (= .*|[^ ]*$)', '\1') AS guc
+FROM (SELECT regexp_split_to_table(pg_read_file('postgresql.conf'), '\n') AS ln) conf
+WHERE ln ~ '^#?[[:alpha:]]'
+EXCEPT SELECT lower(name) FROM pg_settings_flags WHERE NOT not_in_sample
+ORDER BY 1;
+
+-- test that section:DEVELOPER GUCs are flagged GUC_NOT_IN_SAMPLE:
+SELECT * FROM pg_settings_flags
+WHERE category='Developer Options' AND NOT not_in_sample
+ORDER BY 1;
+
+-- Maybe the converse:
+SELECT * FROM pg_settings_flags
+WHERE category NOT IN ('Developer Options', 'Preset Options') AND not_in_sample
+ORDER BY 1;
+
+-- Most Query Tuning GUCs are flagged as EXPLAIN:
+SELECT * FROM pg_settings_flags
+WHERE category ~ '^Query Tuning' AND NOT guc_explain
+ORDER BY 1;
+
+-- Maybe the converse:
+SELECT * FROM pg_settings_flags
+WHERE guc_explain AND NOT category ~ '^Query Tuning|^Resource Usage'
+ORDER BY 1;
+
+-- GUCs flagged RUNTIME are Preset
+SELECT * FROM pg_settings_flags
+WHERE guc_computed AND NOT category='Preset Options'
+ORDER BY 1;
+
+-- PRESET GUCs are flagged NOT_IN_SAMPLE
+SELECT * FROM pg_settings_flags
+WHERE category='Preset Options' AND NOT not_in_sample
+ORDER BY 1;
+
+-- NO_SHOW_ALL implies NO_RESET_ALL:
+SELECT * FROM pg_settings_flags
+WHERE no_show_all AND NOT no_reset_all
+ORDER BY 1;
+
+-- Usually the converse:
+SELECT * FROM pg_settings_flags
+WHERE NOT no_show_all AND no_reset_all
+ORDER BY 1;
+
+-- NO_SHOW_ALL implies NOT_IN_SAMPLE:
+SELECT * FROM pg_settings_flags
+WHERE no_show_all AND NOT not_in_sample
+ORDER BY 1;
+
+DROP TABLE pg_settings_flags;
--
2.17.1
