Re: POC: track vacuum/analyze cumulative time per relation

2025-01-17 Thread Bertrand Drouvot
ts of pgstatfuncs.c Yup. > v4 attached. One comment: + PG_RETURN_FLOAT8(result);\ The "\" indentation looks wrong for this line (was ok in v3). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend I/O statistics

2025-01-17 Thread Bertrand Drouvot
Hi, On Fri, Jan 17, 2025 at 03:12:36PM +0900, Michael Paquier wrote: > On Fri, Jan 17, 2025 at 06:06:35AM +0000, Bertrand Drouvot wrote: > > On Fri, Jan 17, 2025 at 09:08:02AM +0900, Michael Paquier wrote: > >> I could tweak that around the beginning of next week with a prop

Re: per backend I/O statistics

2025-01-16 Thread Bertrand Drouvot
f1...@ip-10-97-1-34.eu-west-3.compute.internal > > I could tweak that around the beginning of next week with a proposal > of patch. Bertrand, perhaps you'd prefer hack on this one? Yeah, I had in mind to work on it (in the exact same line as you are describing above). I'll work o

Re: per backend WAL statistics

2025-01-16 Thread Bertrand Drouvot
Hi, On Thu, Jan 16, 2025 at 11:38:47AM -0500, Andres Freund wrote: > Hi, > > On 2025-01-16 15:59:31 +, Bertrand Drouvot wrote: > > On Wed, Jan 15, 2025 at 03:11:32PM +0900, Michael Paquier wrote: > > > On Fri, Jan 10, 2025 at 09:40:38AM +, Bertrand Drouvot wrot

Re: per backend I/O statistics

2025-01-16 Thread Bertrand Drouvot
Hi, On Thu, Jan 16, 2025 at 06:48:58AM +, Bertrand Drouvot wrote: > Hi, > > On Thu, Jan 16, 2025 at 09:55:10AM +0900, Michael Paquier wrote: > > On Wed, Jan 15, 2025 at 05:20:57PM +0300, Nazir Bilal Yavuz wrote: > > > I think allowing only pgSt

Re: per backend WAL statistics

2025-01-16 Thread Bertrand Drouvot
Hi, On Wed, Jan 15, 2025 at 03:11:32PM +0900, Michael Paquier wrote: > On Fri, Jan 10, 2025 at 09:40:38AM +0000, Bertrand Drouvot wrote: > > Please find attached v4 taking into account 2c14037bb5. > > +} PgStat_WalCounters; > + > +typedef struct PgStat_WalStats > +{ &g

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-15 Thread Bertrand Drouvot
ed "fix". >From what I can see, the above proposal does (at least) silent the warning here (clang 5.0.1 and same as demoiselle): https://godbolt.org/z/cGosfzGne (we can see the warning by using the current define and that the warning is gone with the new define). Let's see on

Re: per backend I/O statistics

2025-01-15 Thread Bertrand Drouvot
how much I could > break, and this was one pattern once I've forced the pending part. I > didn't get through the whole exercise. I'll look at it and come back with a proposal as part of [1]. [1]: https://www.postgresql.org/message-id/z4drlnuhsq3hp...@paquier.xyz Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend I/O statistics

2025-01-15 Thread Bertrand Drouvot
hands). So it seems to me that the "migrating WAL stats thread" is a better place to discuss about it. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: POC: track vacuum/analyze cumulative time per relation

2025-01-15 Thread Bertrand Drouvot
\ +} I did propose "double" up-thread to be consistent with the code around. That would mean to also cast to "double". That's just for consistency purpose. What do you think? Appart from the above that LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors

Re: per backend I/O statistics

2025-01-15 Thread Bertrand Drouvot
Hi, On Wed, Jan 15, 2025 at 06:27:22PM +0900, Michael Paquier wrote: > On Wed, Jan 15, 2025 at 08:30:20AM +0000, Bertrand Drouvot wrote: > > On Wed, Jan 15, 2025 at 11:03:54AM +0300, Nazir Bilal Yavuz wrote: > >> With this commit it may not be possible to count IOs in the crit

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-15 Thread Bertrand Drouvot
Hi, On Tue, Jan 14, 2025 at 06:55:03PM -0500, Andres Freund wrote: > Hi, > > On 2025-01-14 13:06:31 +, Bertrand Drouvot wrote: > > On Tue, Jan 14, 2025 at 12:58:44AM -0500, Andres Freund wrote: > > > The current code has the weird behaviour of going through &

Re: per backend I/O statistics

2025-01-15 Thread Bertrand Drouvot
g() usage. Michael mentioned in [2] that is not really consistent with the rest (what I agree with) and that "we should rethink a bit the way pending entries are retrieved". I did not think about it yet but that might be the way to go, thoughts? [1]: https://www.postgresql.org/message-i

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-14 Thread Bertrand Drouvot
Hi, On Tue, Jan 14, 2025 at 12:58:44AM -0500, Andres Freund wrote: > Hi, > > On 2025-01-13 12:20:39 +, Bertrand Drouvot wrote: > > > We have > > > multiple copies of code to go to FatalError, that doesn't seem great. > > > > + * FIXME: This sh

Re: POC: track vacuum/analyze cumulative time per relation

2025-01-13 Thread Bertrand Drouvot
RelationGetRelid(rel)); + starttime = GetCurrentTimestamp(); I wonder if it wouldn't make more sense to put the GetCurrentTimestamp() call before the pgstat_progress_start_command() call. That would be aligned with the "endtime" being after the pgstat_progress_end_command(

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Bertrand Drouvot
ing, it's treated as as a reason to crash-restart: > /* >* Any unexpected exit of the checkpointer > (including FATAL >* exit) is treated as a crash. >*/ > HandleChildCrash(pid, exitstatus, > > _("checkpointer process")); > I do agree. I was trying to reach this code path while replying to one of the above comments (about PM_WAIT_DEAD_END ordering) but had to replace with another if condition to reach it during testing. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: POC: track vacuum/analyze cumulative time per relation

2025-01-10 Thread Bertrand Drouvot
total_autoanalyze_time; /* autovacuum initiated */ Those comments look weird to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend I/O statistics

2025-01-10 Thread Bertrand Drouvot
Hi, On Thu, Dec 19, 2024 at 06:12:04AM +, Bertrand Drouvot wrote: > I think I'll start a dedicated thread to discuss the > stats_fetch_consistency/'snapshot' > point (will be easier to follow than resuming the discussion in this thread). I gave more thoughts on it

Re: per backend WAL statistics

2025-01-10 Thread Bertrand Drouvot
the pending stats > data. Yeah, makes sense, thanks! Please find attached v4 taking into account 2c14037bb5. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From c424cd02baba3f441768b1f049a5ec6ad11bc3ce Mon Se

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-10 Thread Bertrand Drouvot
Hi, On Fri, Jan 10, 2025 at 10:15:52AM +0300, Nazir Bilal Yavuz wrote: > Hi, > > On Fri, 10 Jan 2025 at 04:47, Michael Paquier wrote: > > > > On Thu, Jan 09, 2025 at 03:30:37PM +0000, Bertrand Drouvot wrote: > > > We first use an Assert in is_ioop_tracked_in_byte

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-09 Thread Bertrand Drouvot
mething like? #define is_ioop_tracked_in_bytes(io_op) \ ((io_op) < IOOP_NUM_TYPES && (io_op) >= IOOP_EXTEND) v7-0002: I wonder if it wouldn't make more sense to remove pgstat_count_io_op() first and then implement what currently is in v7-0001. What v7-0002 is removing is not produced by v7-0001. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-09 Thread Bertrand Drouvot
SIGINT, ReqShutdownXLOG); /* trigger shutdown checkpoint */ === 6 + * Wait until we're asked to shut down. By seperating the writing of the Typo: s/seperating/separating/ I don't really anything else in 0006 and 0007 but as you said it's probably worth a few more eyes as the code

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-09 Thread Bertrand Drouvot
Hi, On Wed, Jan 08, 2025 at 02:32:24PM -0500, Andres Freund wrote: > On 2025-01-08 14:48:21 +0000, Bertrand Drouvot wrote: > > === 2 > > > > + default: > > + /* all signals sent by postmaster should be listed > > here */ &

Re: per backend WAL statistics

2025-01-08 Thread Bertrand Drouvot
Hi, On Thu, Jan 09, 2025 at 01:03:15PM +0900, Michael Paquier wrote: > On Wed, Jan 08, 2025 at 11:11:59AM +0000, Bertrand Drouvot wrote: > > Yeah, that's more elegant as it also means that the main callback will not > > change > > (should we add even more stats in the

Re: improve DEBUG1 logging of parallel workers for CREATE INDEX?

2025-01-08 Thread Bertrand Drouvot
ially" above. That would make it more clear in case one just look for "building index" in the log. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-08 Thread Bertrand Drouvot
n. s/postmaster.c/PostmasterStateMachine()/? === 4 + * too. That allows checkpointer to perform some last bits of of Typo "of of" I'll give 0004 a closer look once it leaves the WIP state but want to share that it looks like a good way to fix the issue. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend WAL statistics

2025-01-08 Thread Bertrand Drouvot
Hi, On Wed, Jan 08, 2025 at 03:21:26PM +0900, Michael Paquier wrote: > On Tue, Jan 07, 2025 at 08:48:51AM +0000, Bertrand Drouvot wrote: > > Now that commit 9aea73fc61 added backend-level statistics to pgstats (and > > per backend IO statistics), we can more easily add per bac

Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)

2025-01-07 Thread Bertrand Drouvot
Hi, On Tue, Jan 07, 2025 at 10:19:36PM +0100, Tomas Vondra wrote: > On 1/7/25 21:42, Robert Treat wrote: > > On Tue, Jan 7, 2025 at 10:44 AM Bertrand Drouvot > > wrote: > >> > >> ... > >> > >> Another idea regarding the storage of those metric

Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)

2025-01-07 Thread Bertrand Drouvot
ry often either. Thanks for working on this! Another idea regarding the storage of those metrics: I think that one would want to see "precise" data for recent metrics but would probably be fine with some level of aggregation for historical ones. Something like being able to retrieve &qu

Re: per backend I/O statistics

2025-01-07 Thread Bertrand Drouvot
Hi, On Fri, Jan 03, 2025 at 10:48:41AM +, Bertrand Drouvot wrote: > I started to look at it and should be able to share a patch next week. A dedicated thread and a patch have been created, see [1]. [1]: https://www.postgresql.org/message-id/flat/Z3zqc4o09dM/Ezyz%40ip-10-97-1-34.eu-wes

per backend WAL statistics

2025-01-07 Thread Bertrand Drouvot
keep this change as we may want to add more per backend stats in the future. Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 5c2241fcee52f7fb3cfea9890d495f51b981a9c0 Mon

Re: relfilenode statistics

2025-01-03 Thread Bertrand Drouvot
Hi, On Tue, Dec 03, 2024 at 10:31:15AM +, Bertrand Drouvot wrote: > Hi, > > On Fri, Nov 29, 2024 at 08:52:13PM +0500, Kirill Reshke wrote: > > On Fri, 29 Nov 2024 at 20:20, Bertrand Drouvot > > wrote: > > > On Fri, Nov 29, 2024 at 11:23:12AM +0500, Kirill Res

Re: Autovacuum giving up on tables after crash because of lack of stats

2025-01-03 Thread Bertrand Drouvot
On Fri, Jan 03, 2025 at 09:47:58AM -0500, Andres Freund wrote: > On 2024-12-30 19:12:54 +0900, Michael Paquier wrote: > > On Mon, Dec 30, 2024 at 09:44:45AM +, Bertrand Drouvot wrote: > > > I think that replicating stats that are used by autovacuum would be an > > &g

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-03 Thread Bertrand Drouvot
to "activate logical decoding" and zero logical replication slot created would be synonym to "deactivate logical decoding". That way: 1. an end user don't need to manipulate new functions and would just rely on replication slots existence 2. we ensure that no extra WAL data is generated if not absolutely "needed" Thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend I/O statistics

2025-01-03 Thread Bertrand Drouvot
Hi, On Tue, Dec 24, 2024 at 02:35:16PM +0900, Michael Paquier wrote: > On Fri, Dec 20, 2024 at 09:57:19AM +0000, Bertrand Drouvot wrote: > > BTW, now that the per backend I/O statistics is done, I'll start working on > > per > > backend wal statistics. > >

Re: Avoid orphaned objects dependencies, take 3

2025-01-02 Thread Bertrand Drouvot
Hi, On Mon, Oct 28, 2024 at 09:30:19AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Aug 19, 2024 at 03:35:14PM +, Bertrand Drouvot wrote: > > Hi, > > > > On Wed, Jul 10, 2024 at 07:31:06AM +, Bertrand Drouvot wrote: > > > So, to sum up:

Re: WAL-logging facility for pgstats kinds

2024-12-31 Thread Bertrand Drouvot
So it makes === 7 useless as here we are setting all the fields. But I think it's good to keep === 7 as it acts as a template. === 9 + if (!RecoveryInProgress() && inj_stats_wal_enabled) s/!RecoveryInProgress()/XLogInsertAllowed()/ maybe? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Autovacuum giving up on tables after crash because of lack of stats

2024-12-30 Thread Bertrand Drouvot
Hi, On Mon, Dec 30, 2024 at 07:12:54PM +0900, Michael Paquier wrote: > On Mon, Dec 30, 2024 at 09:44:45AM +0000, Bertrand Drouvot wrote: > > I think that replicating stats that are used by autovacuum would be an > > additional > > benefit, so +1 for the idea number 2). Th

Re: Autovacuum giving up on tables after crash because of lack of stats

2024-12-30 Thread Bertrand Drouvot
n additional benefit, so +1 for the idea number 2). This is an "issue" that has been raised multiple times (like in [1]). [1]: https://www.postgresql.org/message-id/20240607033806.6gwgolihss72cj6r%40awork3.anarazel.de Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Fix handling of injection_points regarding pending stats

2024-12-30 Thread Bertrand Drouvot
should" be done). > While doing so, I have also noticed that injection_stats_flush_cb() > missed a pgstat_unlock_entry() when entries get locked. Right. > All that is addressed in the patch attached. Feel free if you have > any comments. The patch is pretty straightforward and LGTM. Regards, --

Re: per backend I/O statistics

2024-12-20 Thread Bertrand Drouvot
Hi, On Thu, Dec 19, 2024 at 06:12:04AM +, Bertrand Drouvot wrote: > On Thu, Dec 19, 2024 at 01:21:54PM +0900, Michael Paquier wrote: > > bumped the two version counters, and done. > >The existing structure could be expanded in the > >future to add more in

Re: Parametrization minimum password lenght

2024-12-20 Thread Bertrand Drouvot
On Thu, Dec 19, 2024 at 09:57:31AM -0600, Nathan Bossart wrote: > On Thu, Dec 19, 2024 at 09:36:17AM -0600, Nathan Bossart wrote: > > On Thu, Dec 19, 2024 at 07:25:30AM +, Bertrand Drouvot wrote: > >> - errmsg(&quo

Re: per backend I/O statistics

2024-12-20 Thread Bertrand Drouvot
16%3A28%3A58 Thanks for the report! I was not able able to reproduce (even with asan-enabled) but I think the test is wrong. Indeed the backend could fsync too during the test (see register_dirty_segment() and the case where the request queue is full). I think the test should look like the atta

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-19 Thread Bertrand Drouvot
Hi, On Thu, Dec 19, 2024 at 05:50:43PM +0300, Nazir Bilal Yavuz wrote: > Hi, > > On Thu, 19 Dec 2024 at 08:50, Bertrand Drouvot > wrote: > > > > Hi, > > > > On Wed, Dec 18, 2024 at 11:55:11AM -0500, Andres Freund wrote: > > > Yea, i think it'

Re: Parametrization minimum password lenght

2024-12-18 Thread Bertrand Drouvot
_0001.txt attached). Also, it seems to me that INT_MAX is too large (as mentioned in [1]), but that's probably a nit. [1]: https://www.postgresql.org/message-id/Zzzmw4IAvrypmFO4%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open

Re: per backend I/O statistics

2024-12-18 Thread Bertrand Drouvot
and done. Thanks! I think I'll start a dedicated thread to discuss the stats_fetch_consistency/'snapshot' point (will be easier to follow than resuming the discussion in this thread). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-18 Thread Bertrand Drouvot
instead: I'll try to have a look at adding IO stats for walwriter. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-18 Thread Bertrand Drouvot
Hi, On Wed, Dec 18, 2024 at 10:35:45AM -0500, Andres Freund wrote: > Hi, > > On 2024-12-18 15:14:07 +, Bertrand Drouvot wrote: > > While working on [1], it has been noticed that pgstat_flush_io() is called > > for > > the walwriter. Indeed, it's coming from

Removing the pgstat_flush_io() call from the walwriter

2024-12-18 Thread Bertrand Drouvot
https://www.postgresql.org/message-id/Z1rs/j5JMdTbUIJJ%40ip-10-97-1-34.eu-west-3.compute.internal [2]: https://www.postgresql.org/message-id/flat/20200124195226.lth52iydq2n2uilq%40alap3.anarazel.de Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Data

Re: per backend I/O statistics

2024-12-18 Thread Bertrand Drouvot
On Wed, Dec 18, 2024 at 08:11:55AM +, Bertrand Drouvot wrote: > Yeah, I also had something like this in mind (see R2 in [1]). So, +1 for it. [1] being: https://www.postgresql.org/message-id/Zy4bmvgHqGjcK1pI%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot Postgre

Re: per backend I/O statistics

2024-12-18 Thread Bertrand Drouvot
Hi, On Wed, Dec 18, 2024 at 01:57:53PM +0900, Michael Paquier wrote: > On Tue, Dec 17, 2024 at 09:35:37AM +0000, Bertrand Drouvot wrote: > > Agree, we may need to add parameters to it but we'll see when the time > > comes. > > Another thing that'

Re: per backend I/O statistics

2024-12-17 Thread Bertrand Drouvot
Hi, On Tue, Dec 17, 2024 at 06:13:44PM +0900, Michael Paquier wrote: > On Tue, Dec 17, 2024 at 08:13:59AM +0000, Bertrand Drouvot wrote: > > Having said that, v9-0002 looks good to me (except the > > pgstat_fetch_proc_stat_io > > renaming). > > This routine retu

Re: per backend I/O statistics

2024-12-17 Thread Bertrand Drouvot
Hi, On Tue, Dec 17, 2024 at 03:26:02PM +0900, Michael Paquier wrote: > On Mon, Dec 16, 2024 at 03:42:04PM +0000, Bertrand Drouvot wrote: > >> Perhaps there's an argument for an entirely separate > >> callback that would run before pgstat is plugged off, like a

Re: Track the amount of time waiting due to cost_delay

2024-12-16 Thread Bertrand Drouvot
Hi, On Mon, Dec 16, 2024 at 04:02:56PM -0600, Nathan Bossart wrote: > On Mon, Dec 16, 2024 at 10:11:23AM +0000, Bertrand Drouvot wrote: > > +#define PARALLEL_VACUUM_WORKER_DELAY_REPORT_INTERVAL_NS (NS_PER_S) > > > > Did not changed in v14, but "PARALLEL_VACUUM_REPORT_IN

Re: per backend I/O statistics

2024-12-16 Thread Bertrand Drouvot
Hi, On Mon, Dec 16, 2024 at 05:07:52PM +0900, Michael Paquier wrote: > On Fri, Dec 13, 2024 at 09:20:13AM +0000, Bertrand Drouvot wrote: > > Not feeling so sure about the value brought by the backend_type > returned in pg_stat_get_backend_io(), but well.. It's not necessar

Re: Track the amount of time waiting due to cost_delay

2024-12-16 Thread Bertrand Drouvot
gly. === 5 + if (INSTR_TIME_GET_NANOSEC(time_since_last_report) >= + PARALLEL_VACUUM_WORKER_DELAY_REPORT_INTERVAL_NS) + { + pgstat_progress_parallel_incr_param(PROGRESS_VACUUM_DELAY_TIME, Added a comment to mention that PROGRESS_ANALYZE_DELAY_TIME is out of interest here. v14 also fixes the typo mentio

Re: per backend I/O statistics

2024-12-13 Thread Bertrand Drouvot
Hi, On Fri, Dec 13, 2024 at 11:02:53AM +0900, Michael Paquier wrote: > On Thu, Dec 12, 2024 at 02:02:38PM +0000, Bertrand Drouvot wrote: > > Anyway, isn't it possible that this lookup loop finishes by finding > nothing depending on concurrent updates of other beentries? It so

Re: Track the amount of time waiting due to cost_delay

2024-12-12 Thread Bertrand Drouvot
Hi, On Thu, Dec 12, 2024 at 10:15:18AM -0600, Nathan Bossart wrote: > On Thu, Dec 12, 2024 at 04:36:21AM +0000, Bertrand Drouvot wrote: > > --- a/src/backend/catalog/system_views.sql > > +++ b/src/backend/catalog/system_views.sql > > @@ -1222,7 +1222,8 @@ CREATE VIEW pg_st

Re: per backend I/O statistics

2024-12-12 Thread Bertrand Drouvot
Hi, On Thu, Dec 12, 2024 at 01:52:03PM +0900, Michael Paquier wrote: > On Mon, Nov 25, 2024 at 03:47:59PM +0000, Bertrand Drouvot wrote: > +view. The function does not return I/O statistics for the > checkpointer, > +the background writer, the startup process and th

Re: Fix comments related to pending statistics

2024-12-11 Thread Bertrand Drouvot
Hi, On Thu, Dec 12, 2024 at 09:20:15AM +0900, Michael Paquier wrote: > On Wed, Dec 11, 2024 at 07:32:38AM +0000, Bertrand Drouvot wrote: > > On Wed, Dec 11, 2024 at 02:56:08PM +0900, Michael Paquier wrote: > >> Your suggestion does not look completely right to me. The

Re: Track the amount of time waiting due to cost_delay

2024-12-11 Thread Bertrand Drouvot
Yeah, fair enough. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 35e6075791000498ed05a7eb62fd34616957c4ce Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Mon, 24 Jun 2024 08:43:26 + Subject:

Re: Make use of pg_memory_is_all_zeros() in more places

2024-12-10 Thread Bertrand Drouvot
Hi, On Wed, Dec 11, 2024 at 03:03:34PM +0900, Michael Paquier wrote: > On Tue, Dec 10, 2024 at 02:18:33PM +0000, Bertrand Drouvot wrote: > > While searching for memcmp() calls in "*stat*.c" files (due to [1]), it > > appeared > > that we could $SUBJECT. Please

Re: Fix comments related to pending statistics

2024-12-10 Thread Bertrand Drouvot
Hi, On Wed, Dec 11, 2024 at 02:56:08PM +0900, Michael Paquier wrote: > On Tue, Dec 10, 2024 at 02:16:14PM +0000, Bertrand Drouvot wrote: > > > > 2. One linked to PgStat_FunctionCounts pending stats, mentioning the use of > > memcmp() against zeroes to detect whether there

Re: fix deprecation mention for age() and mxid_age()

2024-12-10 Thread Bertrand Drouvot
t; now. Thanks for the poke. D'oh I missed it too! > Thanks for the poke. +1 Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-12-10 Thread Bertrand Drouvot
Hi, On Tue, Dec 10, 2024 at 11:55:41AM -0600, Nathan Bossart wrote: > On Mon, Dec 09, 2024 at 04:41:03PM +0000, Bertrand Drouvot wrote: > > + time_delayed bigint > > I think it's also worth considering names like total_delay and > cumulative_delay. That's f

Make use of pg_memory_is_all_zeros() in more places

2024-12-10 Thread Bertrand Drouvot
t there is no others that could be replaced with pg_memory_is_all_zeros(). [1]: https://www.postgresql.org/message-id/flat/Z1hNLvcPgVLPxCoc%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://

Fix comments related to pending statistics

2024-12-10 Thread Bertrand Drouvot
/ZlGYokUIlERemvpB%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From a7a026fa0f183bf4d66d85ea05463b69422d20a8 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Tue, 10 Dec 2

Re: shared-memory based stats collector - v70

2024-12-09 Thread Bertrand Drouvot
Hi, On Tue, Dec 10, 2024 at 09:54:36AM +0900, Michael Paquier wrote: > On Mon, Dec 09, 2024 at 08:03:54AM +0000, Bertrand Drouvot wrote: > > Right. OTOH I think that could help the tap test added in da99fedf8c to not > > rely on assert enabled build (the tap test could "s

Re: Track the amount of time waiting due to cost_delay

2024-12-09 Thread Bertrand Drouvot
Hi, On Mon, Dec 09, 2024 at 08:34:13PM +0530, Dilip Kumar wrote: > On Mon, Dec 9, 2024 at 6:55 PM Bertrand Drouvot > > Yeah. I think we could change the wording that way: > > > > s/waiting due to/waiting during/ > > > > Does that make sense? I don't th

Re: Track the amount of time waiting due to cost_delay

2024-12-09 Thread Bertrand Drouvot
Hi, On Mon, Dec 09, 2024 at 05:18:30PM +0530, Dilip Kumar wrote: > On Mon, Dec 9, 2024 at 2:51 PM Masahiro Ikeda > wrote: > > > > On 2024-12-06 18:31, Bertrand Drouvot wrote: > > > Hi, > > > > > > On Thu, Dec 05, 2024 at 10:43:51AM +, Bertrand Dr

Re: shared-memory based stats collector - v70

2024-12-09 Thread Bertrand Drouvot
at could help the tap test added in da99fedf8c to not rely on assert enabled build (the tap test could "simply" check for the WARNING in the logfile instead). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-12-06 Thread Bertrand Drouvot
Hi, On Thu, Dec 05, 2024 at 10:43:51AM +, Bertrand Drouvot wrote: > Yeah, people would likely use this new field to monitor long running vacuum. > Long enough that this error should be acceptable. Do you agree? OTOH, adding the 100% accuracy looks as simple as v9-0002 attached (0001 i

Re: shared-memory based stats collector - v70

2024-12-05 Thread Bertrand Drouvot
Hi, On Thu, Dec 05, 2024 at 05:13:13PM +0900, Michael Paquier wrote: > On Thu, Dec 05, 2024 at 07:37:27AM +0000, Bertrand Drouvot wrote: > > That said, I think that's worth to update the comment a bit (like in the > > attached?) as I think that answers a legitimate questio

Re: Track the amount of time waiting due to cost_delay

2024-12-05 Thread Bertrand Drouvot
ld be acceptable. Do you agree? > I didn't see any comments regarding this, so I wanted to confirm. Added a comment to make it clear, thanks! [1]: https://www.postgresql.org/message-id/CAD21AoDOu%3DDZcC%2BPemYmCNGSwbgL1s-5OZkZ1Spd5pSxofWNCw%40mail.gmail.com Regards, -- Bertrand Drouvot PostgreSQL

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2024-12-05 Thread Bertrand Drouvot
Hi, On Wed, Dec 04, 2024 at 02:49:11PM +0300, Nazir Bilal Yavuz wrote: > On Thu, 28 Nov 2024 at 16:39, Bertrand Drouvot > wrote: > > > You are right, no need to have this check; it can not be less than 0. > I completely removed the function now. Thanks! Yeah I think that makes

Re: shared-memory based stats collector - v70

2024-12-04 Thread Bertrand Drouvot
Hi, On Thu, Dec 05, 2024 at 02:43:43PM +0900, Michael Paquier wrote: > On Wed, Dec 04, 2024 at 03:24:55PM +0000, Bertrand Drouvot wrote: > > Thanks! I've the feeling that something has to be fixed, see my comments in > > [1]. It might be that the failed assertion does

Re: Missing LWLock protection in pgstat_reset_replslot()

2024-12-04 Thread Bertrand Drouvot
Hi, On Thu, Dec 05, 2024 at 01:26:38PM +0900, Michael Paquier wrote: > On Wed, Dec 04, 2024 at 03:20:03PM +0000, Bertrand Drouvot wrote: > > I need to think more about it but it seems to me that those values make > > sense, > > so maybe we should drop the entry for

Re: shared-memory based stats collector - v70

2024-12-04 Thread Bertrand Drouvot
ething has to be fixed, see my comments in [1]. It might be that the failed assertion does not handle a "valid" scenario. [1]: https://www.postgresql.org/message-id/Z1BzI/eMTCOKA%2Bj6%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Missing LWLock protection in pgstat_reset_replslot()

2024-12-04 Thread Bertrand Drouvot
" is false (entry_ref->shared_entry->generation is 1 while entry_ref->generation is 0). I need to think more about it but it seems to me that those values make sense, so maybe we should drop the entry for this particular case (shmem_exit()). Thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: relfilenode statistics

2024-12-03 Thread Bertrand Drouvot
Hi, On Fri, Nov 29, 2024 at 08:52:13PM +0500, Kirill Reshke wrote: > On Fri, 29 Nov 2024 at 20:20, Bertrand Drouvot > wrote: > > On Fri, Nov 29, 2024 at 11:23:12AM +0500, Kirill Reshke wrote: > > > If we don’t have the relation OID when writing buffers out, can we &g

Drop back the redundant "Lock" suffix from LWLock wait event names

2024-12-02 Thread Bertrand Drouvot
https://www.postgresql.org/message-id/flat/18728-450924477056a339%40postgresql.org [2]: https://www.postgresql.org/message-id/flat/202401231025.gbv4nnte5fmm%40alvherre.pgsql Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases

Re: relfilenode statistics

2024-11-29 Thread Bertrand Drouvot
Hi, On Fri, Nov 29, 2024 at 11:23:12AM +0500, Kirill Reshke wrote: > On Tue, 5 Nov 2024 at 11:06, Bertrand Drouvot > wrote: > > > > > > Does it sound ok to you to move with the above principal? (I'm +1 on it). > > > > Hi! I looked through this thread. T

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2024-11-28 Thread Bertrand Drouvot
ontext are passed only for the assertions. What about removing them from there and put the asserts in other places? === 4 + /* Only IOOP_READ, IOOP_WRITE and IOOP_EXTEND can do IO in bytes. */ Not sure about "can do IO in bytes" (same wording is used in multiple places). === 5

Re: per backend I/O statistics

2024-11-27 Thread Bertrand Drouvot
Hi, On Wed, Nov 27, 2024 at 03:33:38PM +0900, Michael Paquier wrote: > On Mon, Nov 25, 2024 at 03:47:59PM +0000, Bertrand Drouvot wrote: > > It also takes care of most of the comments that you have made in [1], > > meaning > > that it: > > > > - removes the bac

Re: Potential ABI breakage in upcoming minor releases

2024-11-25 Thread Bertrand Drouvot
ostgresql.org/message-id/flat/ZzcR%2BoQmUOIm6RVF%40ip-10-97-1-34.eu-west-3.compute.internal#08182ae6a6719632acf52fe4d90e9778 Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: per backend I/O statistics

2024-11-25 Thread Bertrand Drouvot
Hi, On Mon, Nov 25, 2024 at 04:18:54PM +0900, Michael Paquier wrote: > On Mon, Nov 25, 2024 at 07:12:56AM +0000, Bertrand Drouvot wrote: > > Not sure here, could custom stats start incrementing before the database > > system > > is ready to accept connections? > &g

Re: per backend I/O statistics

2024-11-24 Thread Bertrand Drouvot
Hi, On Mon, Nov 25, 2024 at 10:06:44AM +0900, Michael Paquier wrote: > On Fri, Nov 22, 2024 at 07:49:58AM +0000, Bertrand Drouvot wrote: > > On Fri, Nov 22, 2024 at 10:36:29AM +0900, Michael Paquier wrote: > >> Hmm. created_entry only matters for pgstat_init_function_usage().

Re: per backend I/O statistics

2024-11-21 Thread Bertrand Drouvot
Hi, On Fri, Nov 22, 2024 at 10:36:29AM +0900, Michael Paquier wrote: > On Thu, Nov 21, 2024 at 05:23:42PM +0000, Bertrand Drouvot wrote: > > So, given that: > > > > - the end result would be the same > > - the code changes would be non negligible (unless we h

Re: per backend I/O statistics

2024-11-21 Thread Bertrand Drouvot
Hi, On Thu, Nov 21, 2024 at 07:18:24AM +0900, Michael Paquier wrote: > On Wed, Nov 20, 2024 at 02:20:18PM +0000, Bertrand Drouvot wrote: > > Right. I did not had in mind to go that far here (for the per backend stats > > needs). My idea was "just" to move the new p

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-21 Thread Bertrand Drouvot
Hi, On Thu, Nov 21, 2024 at 10:01:07AM +0900, Michael Paquier wrote: > On Wed, Nov 20, 2024 at 05:13:18PM +0000, Bertrand Drouvot wrote: > > I don't have a strong opinion for this particular case here (I think the > > code > > is harder to read but yeah there is some co

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-21 Thread Bertrand Drouvot
Hi, On Thu, Nov 21, 2024 at 04:26:47PM +0900, Michael Paquier wrote: > On Thu, Nov 21, 2024 at 06:32:03AM +0000, Bertrand Drouvot wrote: > > That was in fact the main reason why I added this test. But well, just > > adding the "write_to_file" in the injection test

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-21 Thread Bertrand Drouvot
Hi, On Thu, Nov 21, 2024 at 10:38:28AM +0300, Nazir Bilal Yavuz wrote: > Hi, > > On Thu, 21 Nov 2024 at 09:32, Bertrand Drouvot > wrote: > > That was in fact the main reason why I added this test. But well, just > > adding the "write_to_file" in the injec

Re: per backend I/O statistics

2024-11-20 Thread Bertrand Drouvot
Hi, On Tue, Nov 19, 2024 at 10:47:53AM +0900, Michael Paquier wrote: > On Thu, Nov 14, 2024 at 01:30:11PM +0000, Bertrand Drouvot wrote: > > - change the arguments to false in the pgstat_drop_entry_internal() call in > > pgstat_drop_all_entries() > > - start the engine &

Re: per backend I/O statistics

2024-11-20 Thread Bertrand Drouvot
Hi, On Wed, Nov 20, 2024 at 09:01:26AM +0900, Michael Paquier wrote: > On Tue, Nov 19, 2024 at 04:28:55PM +0000, Bertrand Drouvot wrote: > > So, for the startup process only, what about? > > > > - don't call pgstat_create_backend_stat() in pgstat_beinit()... > &

Re: Add a write_to_file member to PgStat_KindInfo

2024-11-20 Thread Bertrand Drouvot
211 insertions(+), 2 deletions(-) v2: 8 files changed, 152 insertions(+), 22 deletions(-) I don't have a strong opinion for this particular case here (I think the code is harder to read but yeah there is some code reduction): so I'm fine with v2 too. Regards, -- Bertrand Drouvot Post

Re: per backend I/O statistics

2024-11-20 Thread Bertrand Drouvot
eah, also this could useful for custom statistics. So I created a dedicated thread and a patch proposal (see [1]). [1]: https://www.postgresql.org/message-id/Zz3skBqzBncSFIuY%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Add a write_to_file member to PgStat_KindInfo

2024-11-20 Thread Bertrand Drouvot
(the code changes are simple enough). [1]: https://www.postgresql.org/message-id/Zy4bmvgHqGjcK1pI%40ip-10-97-1-34.eu-west-3.compute.internal Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https

Re: Parametrization minimum password lenght

2024-11-19 Thread Bertrand Drouvot
uto_explain means "In ordinary usage, these parameters are set in postgresql.conf," [1]: https://github.com/postgres/postgres/blob/master/src/tools/ci/README [2]: https://www.postgresql.org/message-id/ZzsZZY3YrO6hinnT%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Ber

Re: fix deprecation mention for age() and mxid_age()

2024-11-18 Thread Bertrand Drouvot
Hi, On Tue, Nov 19, 2024 at 10:09:57AM +0900, Michael Paquier wrote: > On Mon, Nov 18, 2024 at 08:11:57AM +0000, Bertrand Drouvot wrote: > > Also it looks like that for PG 16, 7471230cc6, did put the functions > > description > > in the "Control Data Functions". T

Re: Parametrization minimum password lenght

2024-11-18 Thread Bertrand Drouvot
g (see auto_explain.c for example). === 10 + + In postgresql.conf you may set the minimum password length + by setting passwordcheck.min_password_length = INT. I think that would make sense to add a "Configuration Parameters" section and follow the format done in auto-explain or pg_p

  1   2   3   4   5   6   7   >