Le 02/12/2016 à 02:08, Karl O. Pinc a écrit : > On Sun, 27 Nov 2016 21:54:46 +0100 > Gilles Darold <gilles.dar...@dalibo.com> wrote: > >> I've attached the v15 of the patch >> I've not applied patch patch_pg_current_logfile-v14.diff.backoff to >> prevent constant call of logfile_writename() on a busy system (errno = >> ENFILE | EMFILE). > I don't think it should be applied and included in the basic > functionality patch in any case. I think it needs to be submitted as a > separate patch along with the basic functionality patch. Backing off > the retry of the current_logfiles write could be overly fancy and > simply not needed. > >> I think this can be done quite simply by testing if >> log rotate is still enabled. This is possible because function >> logfile_rotate() is already testing if errno = ENFILE | EMFILE and in >> this case rotation_disabled is set to true. So the following test >> should do the work: >> >> if (log_metainfo_stale && !rotation_disabled) >> logfile_writename(); >> >> This is included in v15 patch. > I don't see this helping much, if at all. > > First, it's not clear just when rotation_disabled can be false > when log_metainfo_stale is true. The typical execution > path is for logfile_writename() to be called after rotate_logfile() > has already set rotataion_disabled to true. logfile_writename() > is the only place setting log_metainfo_stale to true and > rotate_logfile() the only place settig rotation_disabled to false. > > While it's possible > that log_metainfo_stale might be set to true when logfile_writename() > is called from within open_csvlogfile(), this does not help with the > stderr case. IMO better to just test for log_metaifo_stale at the > code snippet above. > > Second, the whole point of retrying the logfile_writename() call is > to be sure that the current_logfiles file is updated before the logs > rotate. Waiting until logfile rotation is enabled defeats the purpose.
Ok, sorry I've misunderstood your previous post. Current v16 attached patch removed your change about log_meta_info stale and fix the use of sscanf to read the file. It seems that all fixes have been included in this patch. Regards -- Gilles Darold Consultant PostgreSQL http://dalibo.com - http://dalibo.org
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 0fc4e57..41144cb 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4169,6 +4169,12 @@ SELECT * FROM parent WHERE key = 2400; <primary>server log</primary> </indexterm> + <para>When logs are written to the file-system their paths, names, and + types are recorded in + the <xref linkend="storage-pgdata-current-logfiles"> file. This provides + a convenient way to find and access log content without establishing a + database connection.</para> + <sect2 id="runtime-config-logging-where"> <title>Where To Log</title> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index eca98df..47ca846 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -15444,6 +15444,19 @@ SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n); </row> <row> + <entry><literal><function>pg_current_logfile()</function></literal></entry> + <entry><type>text</type></entry> + <entry>primary log file name in use by the logging collector</entry> + </row> + + <row> + <entry><literal><function>pg_current_logfile(<type>text</>)</function></literal></entry> + <entry><type>text</type></entry> + <entry>log file name, of log in the requested format, in use by the + logging collector</entry> + </row> + + <row> <entry><literal><function>pg_my_temp_schema()</function></literal></entry> <entry><type>oid</type></entry> <entry>OID of session's temporary schema, or 0 if none</entry> @@ -15661,6 +15674,39 @@ SET search_path TO <replaceable>schema</> <optional>, <replaceable>schema</>, .. the time when the postmaster process re-read the configuration files.) </para> + <indexterm> + <primary>pg_current_logile</primary> + </indexterm> + + <indexterm> + <primary>Logging</primary> + <secondary>pg_current_logfile function</secondary> + </indexterm> + + <para> + <function>pg_current_logfile</function> returns, as <type>text</type>, + the path of either the csv or stderr log file currently in use by the + logging collector. This is a path including the + <xref linkend="guc-log-directory"> directory and the log file name. + Log collection must be active or the return value + is <literal>NULL</literal>. When multiple logfiles exist, each in a + different format, <function>pg_current_logfile</function> called + without arguments returns the path of the file having the first format + found in the ordered + list: <systemitem>stderr</>, <systemitem>csvlog</>. + <literal>NULL</literal> is returned when no log file has any of these + formats. To request a specific file format supply, + as <type>text</type>, either <systemitem>csvlog</> + or <systemitem>stderr</> as the value of the optional parameter. The + return value is <literal>NULL</literal> when the log format requested + is not a configured <xref linkend="guc-log-destination">. + + <function>pg_current_logfiles</function> reflects the content of the + <xref linkend="storage-pgdata-current-logfiles"> file. All caveats + regards <filename>current_logfiles</filename> content are applicable + to <function>pg_current_logfiles</function>' return value. + </para> + <indexterm> <primary>pg_my_temp_schema</primary> </indexterm> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 5c52824..928133c 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -60,6 +60,79 @@ Item <entry>Subdirectory containing per-database subdirectories</entry> </row> +<row id="storage-pgdata-current-logfiles" xreflabel="current_logfiles"> + <entry> + <indexterm> + <primary><filename>current_logfiles</filename></primary> + </indexterm> + <indexterm> + <primary>Logging</primary> + <secondary><filename>current_logfiles</filename> file</secondary> + </indexterm> + <filename>current_logfiles</> + </entry> + <entry> + <para>A file recording the log file(s) currently written to by the syslogger + and the file's log formats, <systemitem>stderr</> + or <systemitem>csvlog</>. Each line of the file is a space separated list of + two elements: the log format and the full path to the log file including the + value of <xref linkend="guc-log-directory">. The log format must be present + in <xref linkend="guc-log-destination"> to be listed in + <filename>current_logfiles</filename>.</para> + + <note> + <indexterm> + <primary><application>pg_ctl</application></primary> + <secondary>and <filename>current_logfiles</filename></secondary> + </indexterm> + <indexterm> + <primary><filename>stderr</filename></primary> + <secondary>and <filename>current_logfiles</filename></secondary> + </indexterm> + <indexterm> + <primary>log_destination configuration parameter</primary> + <secondary>and <filename>current_logfiles</filename></secondary> + </indexterm> + + <para>Although logs directed to <filename>stderr</filename> may be written + to the filesystem, when the writing of <filename>stderr</filename> is + managed outside of the <productname>PostgreSQL</productname> database + server the location of such files in the filesystem is not reflected in + the content of <filename>current_logfiles</filename>. One such case is + when the <application>pg_ctl</application> command is used to start + the <command>postgres</command> database server, capture + the <filename>stderr</filename> output of the server, and direct it to a + file.</para> + + <para>There are other notable situations related + to <filename>stderr</filename> logging. + Non-<productname>PostgreSQL</productname> log sources, such as 3rd party + libraries, which deliver error messages directly + to <filename>stderr</filename> are always logged + by <productname>PostgreSQL</productname> + to <filename>stderr</filename>. <Filename>Stderr</Filename> is also the + destination for any incomplete log messages produced by + <productname>PostgreSQL</productname>. When + <systemitem>stderr</systemitem> is not in + <xref linkend="guc-log-destination">, + <filename>current_logfiles</filename> does not not contain the name of the + file where these sorts of log messages are written.</para> + </note> + + <para>The <filename>current_logfiles</filename> file + is present only when <xref linkend="guc-logging-collector"> is + activated and when at least one of <systemitem>stderr</> or + <systemitem>csvlog</> value is present in + <xref linkend="guc-log-destination">.</para> + + <para>On a busy system <filename>current_logfiles</filename> might not be + updated immediately upon logfile rotation. On an extremely busy system it + is possible, although highly unlikely, for a logfile to be "skipped" and + never appear in <filename>current_logfiles</filename>.</para> + </entry> + +</row> + <row> <entry><filename>global</></entry> <entry>Subdirectory containing cluster-wide tables, such as diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c index fd62d66..ea812e0 100644 --- a/src/backend/postmaster/syslogger.c +++ b/src/backend/postmaster/syslogger.c @@ -146,6 +146,8 @@ static char *logfile_getname(pg_time_t timestamp, const char *suffix); static void set_next_rotation_time(void); static void sigHupHandler(SIGNAL_ARGS); static void sigUsr1Handler(SIGNAL_ARGS); +static bool rm_log_metainfo(void); +static void logfile_writename(void); /* @@ -348,6 +350,14 @@ SysLoggerMain(int argc, char *argv[]) rotation_disabled = false; rotation_requested = true; } + + /* + * Force rewriting last log filename when reloading configuration, + * even if rotation_requested is false, log_destination may have + * been changed and we don't want to wait the next file rotation. + */ + if (!rm_log_metainfo()) + logfile_writename(); } if (Log_RotationAge > 0 && !rotation_disabled) @@ -511,10 +521,14 @@ int SysLogger_Start(void) { pid_t sysloggerPid; - char *filename; - if (!Logging_collector) + if (!Logging_collector) { + /* Logging collector is not enabled. We don't know where messages are + * logged. Remove outdated file holding the current log filenames. + */ + unlink(LOG_METAINFO_DATAFILE); return 0; + } /* * If first time through, create the pipe which will receive stderr @@ -570,11 +584,13 @@ SysLogger_Start(void) * a time-based rotation. */ first_syslogger_file_time = time(NULL); - filename = logfile_getname(first_syslogger_file_time, NULL); + last_file_name = logfile_getname(first_syslogger_file_time, NULL); - syslogFile = logfile_open(filename, "a", false); + syslogFile = logfile_open(last_file_name, "a", false); - pfree(filename); + if (!rm_log_metainfo()) + logfile_writename(); + pfree(last_file_name); #ifdef EXEC_BACKEND switch ((sysloggerPid = syslogger_forkexec())) @@ -1098,6 +1114,8 @@ open_csvlogfile(void) pfree(last_csv_file_name); last_csv_file_name = filename; + + logfile_writename(); } /* @@ -1151,9 +1169,11 @@ static void logfile_rotate(bool time_based_rotation, int size_rotation_for) { char *filename; - char *csvfilename = NULL; + char *csvfilename; pg_time_t fntime; FILE *fh; + bool rotate_csvlog; + bool rotate_stderr; rotation_requested = false; @@ -1166,9 +1186,6 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) fntime = next_rotation_time; else fntime = time(NULL); - filename = logfile_getname(fntime, NULL); - if (csvlogFile != NULL) - csvfilename = logfile_getname(fntime, ".csv"); /* * Decide whether to overwrite or append. We can overwrite if (a) @@ -1178,8 +1195,12 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) * * Note: last_file_name should never be NULL here, but if it is, append. */ - if (time_based_rotation || (size_rotation_for & LOG_DESTINATION_STDERR)) + rotate_stderr = time_based_rotation || + (size_rotation_for & LOG_DESTINATION_STDERR); + if (rotate_stderr) { + filename = logfile_getname(fntime, NULL); + if (Log_truncate_on_rotation && time_based_rotation && last_file_name != NULL && strcmp(filename, last_file_name) != 0) @@ -1202,10 +1223,7 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) rotation_disabled = true; } - if (filename) - pfree(filename); - if (csvfilename) - pfree(csvfilename); + pfree(filename); return; } @@ -1216,14 +1234,16 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) if (last_file_name != NULL) pfree(last_file_name); last_file_name = filename; - filename = NULL; } /* Same as above, but for csv file. */ - - if (csvlogFile != NULL && - (time_based_rotation || (size_rotation_for & LOG_DESTINATION_CSVLOG))) + rotate_csvlog = csvlogFile != NULL && + (time_based_rotation || (size_rotation_for & LOG_DESTINATION_CSVLOG)); + if (rotate_csvlog) { + if (csvlogFile != NULL) + csvfilename = logfile_getname(fntime, ".csv"); + if (Log_truncate_on_rotation && time_based_rotation && last_csv_file_name != NULL && strcmp(csvfilename, last_csv_file_name) != 0) @@ -1246,10 +1266,7 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) rotation_disabled = true; } - if (filename) - pfree(filename); - if (csvfilename) - pfree(csvfilename); + pfree(csvfilename); return; } @@ -1260,13 +1277,10 @@ logfile_rotate(bool time_based_rotation, int size_rotation_for) if (last_csv_file_name != NULL) pfree(last_csv_file_name); last_csv_file_name = csvfilename; - csvfilename = NULL; } - if (filename) - pfree(filename); - if (csvfilename) - pfree(csvfilename); + if (rotate_stderr || rotate_csvlog) + logfile_writename(); set_next_rotation_time(); } @@ -1365,3 +1379,77 @@ sigUsr1Handler(SIGNAL_ARGS) errno = save_errno; } + +/* + * Store the name of the file(s) where the log collector, when enabled, writes + * log messages. Useful for finding the name(s) of the current log file(s) + * when there is time-based logfile rotation. Filenames are stored in a + * temporary file and renamed into the final destination for atomicity. + */ +static void +logfile_writename() +{ + FILE *fh; + char tempfn[MAXPGPATH]; + + snprintf(tempfn, sizeof(tempfn), "%s.tmp", LOG_METAINFO_DATAFILE); + + if ((fh = logfile_open(tempfn, "w", true) ) == NULL) + { + if (errno == ENFILE || errno == EMFILE) + ereport(LOG, + (errmsg("system is too busy to write logfile meta info, %s will be updated on next rotation (or use SIGHUP to retry)", LOG_METAINFO_DATAFILE))); + return; + } + if (last_file_name && (Log_destination & LOG_DESTINATION_STDERR)) + { + if (fprintf(fh, "stderr %s\n", last_file_name) < 0) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write stderr log file path \"%s\": %m", + tempfn))); + fclose(fh); + return; + } + } + + if (last_csv_file_name && (Log_destination & LOG_DESTINATION_CSVLOG)) + { + if (fprintf(fh, "csvlog %s\n", last_csv_file_name) < 0) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not write csvlog log file path \"%s\": %m", + tempfn))); + fclose(fh); + return; + } + } + fclose(fh); + + if (rename(tempfn, LOG_METAINFO_DATAFILE) != 0) + { + ereport(LOG, + (errcode_for_file_access(), + errmsg("could not rename file \"%s\": %m", + tempfn))); + return; + } + +} + +/* + * Delete the LOG_METAINFO_DATAFILE if it's not going to be used. + */ +static bool +rm_log_metainfo(void) +{ + if (!(Log_destination & LOG_DESTINATION_STDERR) + && !(Log_destination & LOG_DESTINATION_CSVLOG)) + { + unlink(LOG_METAINFO_DATAFILE); + return true; + } + return false; +} diff --git a/src/backend/utils/adt/misc.c b/src/backend/utils/adt/misc.c index 0da051a..2da40b2 100644 --- a/src/backend/utils/adt/misc.c +++ b/src/backend/utils/adt/misc.c @@ -19,6 +19,7 @@ #include <dirent.h> #include <math.h> #include <unistd.h> +#include <sys/stat.h> #include "access/sysattr.h" #include "catalog/pg_authid.h" @@ -892,3 +893,141 @@ parse_ident(PG_FUNCTION_ARGS) PG_RETURN_DATUM(makeArrayResult(astate, CurrentMemoryContext)); } + +/* + * Report current log file used by log collector + */ +Datum +pg_current_logfile(PG_FUNCTION_ARGS) +{ + FILE *fd; + char lbuffer[MAXPGPATH]; + char log_filepath[MAXPGPATH]; + text *fmt; + char *logfmt; + struct stat stat_buf; + + /* The log format parameter is optional */ + if (PG_NARGS() == 1) { + fmt = PG_ARGISNULL(0) ? NULL : PG_GETARG_TEXT_PP(0); + if (fmt != NULL) { + logfmt = text_to_cstring(fmt); + if ( (strcmp(logfmt, "stderr") != 0) && + (strcmp(logfmt, "csvlog") != 0) ) { + ereport(ERROR, + (errmsg("log format %s not supported, possible values are stderr or csvlog", logfmt))); + PG_RETURN_NULL(); + } + } + } else { + logfmt = NULL; + } + + if (!Logging_collector) + PG_RETURN_NULL(); + + /* Check if current log file is present */ + if (stat(LOG_METAINFO_DATAFILE, &stat_buf) != 0) + PG_RETURN_NULL(); + + fd = AllocateFile(LOG_METAINFO_DATAFILE, "r"); + if (fd == NULL) + { + if (errno != ENOENT) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", + LOG_METAINFO_DATAFILE))); + PG_RETURN_NULL(); + } + + /* + * Read the file to gather current log filename(s) registered + * by the syslogger. + */ + while (fgets(lbuffer, sizeof(lbuffer), fd) != NULL) { + char log_format[10]; + int i = 0, space_pos = 0; + + /* Check for a read error. */ + if (ferror(fd)) { + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", LOG_METAINFO_DATAFILE))); + FreeFile(fd); + break; + } + + /* remove trailing newline */ + if (strchr(lbuffer, '\n') != NULL) + *strchr(lbuffer, '\n') = '\0'; + + /* extract log format and log file path from the line */ + for (i = 0; lbuffer[i] != '\0'; i++) + { + if ((space_pos == 0) && (isspace(lbuffer[i]) != 0)) + { + log_format[i] = '\0'; + space_pos = i; + } + else if (space_pos > 0) + log_filepath[i-space_pos-1] = lbuffer[i]; + else + log_format[i] = lbuffer[i]; + } + log_filepath[i] = '\0'; + if ((space_pos != (int)strlen("stderr")) && + (space_pos != (int)strlen("csvlog"))) + { + ereport(ERROR, + (errmsg("unexpected line format in file %s", LOG_METAINFO_DATAFILE))); + break; + } + + /* + * When no log format is provided as argument always reports + * the first registered log file in LOG_METAINFO_DATAFILE. + */ + if (logfmt == NULL) + break; + + /* report the entry corresponding to the requested format */ + if (strcmp(logfmt, log_format) == 0) + break; + } + /* Close the current log filename file. */ + if (FreeFile(fd)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", LOG_METAINFO_DATAFILE))); + + + + if (lbuffer[0] == '\0') + PG_RETURN_NULL(); + + /* Return null when csvlog is requested but we have a stderr log */ + if ( (logfmt != NULL) && (strcmp(logfmt, "csvlog") == 0) + && !(Log_destination & LOG_DESTINATION_CSVLOG) ) + PG_RETURN_NULL(); + + /* Return null when stderr is requested but we have a csv log */ + if ( (logfmt != NULL) && (strcmp(logfmt, "stderr") == 0) + && !(Log_destination & LOG_DESTINATION_STDERR) ) + PG_RETURN_NULL(); + + PG_RETURN_TEXT_P(cstring_to_text(log_filepath)); +} + +/* + * Report current log file used by log collector (1 argument version) + * + * note: this wrapper is necessary to pass the sanity check in opr_sanity, + * which checks that all built-in functions that share the implementing C + * function take the same number of arguments + */ +Datum +pg_current_logfile_1arg(PG_FUNCTION_ARGS) +{ + return pg_current_logfile(fcinfo); +} diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index 047a1ce..26db448 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -3181,6 +3181,10 @@ DATA(insert OID = 2621 ( pg_reload_conf PGNSP PGUID 12 1 0 0 0 f f f f t f v s DESCR("reload configuration files"); DATA(insert OID = 2622 ( pg_rotate_logfile PGNSP PGUID 12 1 0 0 0 f f f f t f v s 0 0 16 "" _null_ _null_ _null_ _null_ _null_ pg_rotate_logfile _null_ _null_ _null_ )); DESCR("rotate log file"); +DATA(insert OID = 3800 ( pg_current_logfile PGNSP PGUID 12 1 0 0 0 f f f f f f v s 0 0 25 "" _null_ _null_ _null_ _null_ _null_ pg_current_logfile _null_ _null_ _null_ )); +DESCR("current logging collector file location"); +DATA(insert OID = 3801 ( pg_current_logfile PGNSP PGUID 12 1 0 0 0 f f f f f f v s 1 0 25 "25" _null_ _null_ _null_ _null_ _null_ pg_current_logfile_1arg _null_ _null_ _null_ )); +DESCR("current logging collector file location"); DATA(insert OID = 2623 ( pg_stat_file PGNSP PGUID 12 1 0 0 0 f f f f t f v s 1 0 2249 "25" "{25,20,1184,1184,1184,1184,16}" "{i,o,o,o,o,o,o}" "{filename,size,access,modification,change,creation,isdir}" _null_ _null_ pg_stat_file_1arg _null_ _null_ _null_ )); DESCR("get information about file"); diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 999440f..afc636f 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -467,4 +467,13 @@ extern bool has_rolreplication(Oid roleid); extern bool BackupInProgress(void); extern void CancelBackup(void); +/* in backend/utils/adt/misc.c and backend/postmaster/syslogger.c */ +/* + * Name of file holding the paths, names, and types of the files where current + * log messages are written, when the log collector is enabled. Useful + * outside of Postgres when finding the name of the current log file in the + * case of time-based log rotation. + */ +#define LOG_METAINFO_DATAFILE "current_logfiles" + #endif /* MISCADMIN_H */ diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h index 90f5132..e532022 100644 --- a/src/include/utils/builtins.h +++ b/src/include/utils/builtins.h @@ -522,6 +522,8 @@ extern Datum pg_collation_for(PG_FUNCTION_ARGS); extern Datum pg_relation_is_updatable(PG_FUNCTION_ARGS); extern Datum pg_column_is_updatable(PG_FUNCTION_ARGS); extern Datum parse_ident(PG_FUNCTION_ARGS); +extern Datum pg_current_logfile(PG_FUNCTION_ARGS); +extern Datum pg_current_logfile_1arg(PG_FUNCTION_ARGS); /* oid.c */ extern Datum oidin(PG_FUNCTION_ARGS);
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers