Re: pg_stat_io for the startup process

2023-06-12 Thread Kyotaro Horiguchi
At Mon, 12 Jun 2023 21:13:35 -0700, Andres Freund wrote in > Thanks for looking - I already had pushed the commit by the time I read your > email, otherwise I'd have mentioned you reviewing it... Oops! But I anticipated that and that's no problem (I should have confirmed commits.). Thanks for th

Re: pg_stat_io for the startup process

2023-06-12 Thread Andres Freund
Hi, On 2023-06-13 12:54:19 +0900, Kyotaro Horiguchi wrote: > I think that the reason is that we only pass true only when we're in a > sort of idle time. Addition to that XLOG_RUNNING_XACTS comes so > infrequently to cause noticeable degradation. If it causes > sufficiently frequent updates, we wil

Re: pg_stat_io for the startup process

2023-06-12 Thread Kyotaro Horiguchi
At Fri, 9 Jun 2023 21:28:23 -0700, Andres Freund wrote in > Hi, > > On 2023-05-22 13:36:42 +0900, Kyotaro Horiguchi wrote: > > At Sun, 21 May 2023 15:14:23 -0700, Andres Freund > > wrote in > > > Hi, > > > > > > I don't really feel confident we're going to get the timeout approach > > > sol

Re: pg_stat_io for the startup process

2023-06-09 Thread Andres Freund
Hi, On 2023-05-22 13:36:42 +0900, Kyotaro Horiguchi wrote: > At Sun, 21 May 2023 15:14:23 -0700, Andres Freund wrote > in > > Hi, > > > > I don't really feel confident we're going to get the timeout approach solid > > enough for something going into the tree around beta 1. > > > > How about t

Re: pg_stat_io for the startup process

2023-05-21 Thread Kyotaro Horiguchi
At Sun, 21 May 2023 15:14:23 -0700, Andres Freund wrote in > Hi, > > I don't really feel confident we're going to get the timeout approach solid > enough for something going into the tree around beta 1. > > How about this, imo a lot simpler, approach: We flush stats whenever replaying > a XLOG_

Re: pg_stat_io for the startup process

2023-05-21 Thread Andres Freund
Hi, I don't really feel confident we're going to get the timeout approach solid enough for something going into the tree around beta 1. How about this, imo a lot simpler, approach: We flush stats whenever replaying a XLOG_RUNNING_XACTS record. Unless the primary is idle, it will log those at a re

Re: pg_stat_io for the startup process

2023-05-11 Thread Kyotaro Horiguchi
Thanks for clarifying! At Mon, 8 May 2023 14:46:43 -0700, Andres Freund wrote in > Hi, > > On 2023-04-26 18:47:14 +0900, Kyotaro Horiguchi wrote: > > I see four issues here. > > > > 1. The current database stats omits buffer fetches that don't > > originate from a relation. > > > > In this

Re: pg_stat_io for the startup process

2023-05-08 Thread Andres Freund
Hi, On 2023-04-26 18:47:14 +0900, Kyotaro Horiguchi wrote: > I see four issues here. > > 1. The current database stats omits buffer fetches that don't > originate from a relation. > > In this case pgstat_relation can't work since recovery isn't conscious > of relids. We might be able to resol

Re: pg_stat_io for the startup process

2023-05-08 Thread Melanie Plageman
On Wed, May 03, 2023 at 04:11:33PM +0300, Melih Mutlu wrote: > Andres Freund , 27 Nis 2023 Per, 19:27 tarihinde şunu > yazdı: > > #ifdef WAL_DEBUG > > > if (XLOG_DEBUG || > > > (record->xl_rmid == RM_XACT_ID && > > trace_recovery_messages <= DEB

Re: pg_stat_io for the startup process

2023-05-08 Thread Melanie Plageman
I've reviewed both your patch versions and responded to the ideas in both of them and the associated emails below. On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi wrote: > > At Tue, 25 Apr 2023 16:04:23 -0700, Andres Freund wrote > in > > key those stats by oid. However, it *does* report the

Re: pg_stat_io for the startup process

2023-05-03 Thread Melih Mutlu
Hi, Andres Freund , 27 Nis 2023 Per, 19:27 tarihinde şunu yazdı: > Huh - do you have a link to the failure? That's how I would expect it to be > done. I think I should have registered it in the beginning of PerformWalRecovery() and not just before the main redo loop since HandleStartupProcInter

Re: pg_stat_io for the startup process

2023-04-27 Thread Kyotaro Horiguchi
At Fri, 28 Apr 2023 11:15:51 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 27 Apr 2023 17:30:40 -0400, Melanie Plageman > wrote in > > After a quick example implementation of this, I found that it seemed to > > try and flush the stats less often on an idle standby (good) than using > > en

Re: pg_stat_io for the startup process

2023-04-27 Thread Kyotaro Horiguchi
At Thu, 27 Apr 2023 17:30:40 -0400, Melanie Plageman wrote in > On Wed, Apr 26, 2023 at 2:53 PM Melih Mutlu wrote: > > > > Hi all, > > > > Robert Haas , 26 Nis 2023 Çar, 15:34 tarihinde şunu > > yazdı: > >> > >> On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi > >> wrote: > >> > 3. When shou

Re: pg_stat_io for the startup process

2023-04-27 Thread Melanie Plageman
On Wed, Apr 26, 2023 at 2:53 PM Melih Mutlu wrote: > > Hi all, > > Robert Haas , 26 Nis 2023 Çar, 15:34 tarihinde şunu > yazdı: >> >> On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi >> wrote: >> > 3. When should we call pgstat_report_stats on the startup process? >> > >> > During recovery, I t

Re: pg_stat_io for the startup process

2023-04-27 Thread Andres Freund
Hi, On 2023-04-26 21:53:16 +0300, Melih Mutlu wrote: > Robert Haas , 26 Nis 2023 Çar, 15:34 tarihinde şunu > yazdı: > > > On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi > > wrote: > > > 3. When should we call pgstat_report_stats on the startup process? > > > > > > During recovery, I think we

Re: pg_stat_io for the startup process

2023-04-26 Thread Melih Mutlu
Hi all, Robert Haas , 26 Nis 2023 Çar, 15:34 tarihinde şunu yazdı: > On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi > wrote: > > 3. When should we call pgstat_report_stats on the startup process? > > > > During recovery, I think we can call pgstat_report_stats() (or a > > subset of it) right

Re: pg_stat_io for the startup process

2023-04-26 Thread Robert Haas
On Wed, Apr 26, 2023 at 5:47 AM Kyotaro Horiguchi wrote: > 3. When should we call pgstat_report_stats on the startup process? > > During recovery, I think we can call pgstat_report_stats() (or a > subset of it) right before invoking WaitLatch and at segment > boundaries. I think this kind of idea

Re: pg_stat_io for the startup process

2023-04-26 Thread Kyotaro Horiguchi
At Tue, 25 Apr 2023 16:04:23 -0700, Andres Freund wrote in > I refreshed my memory: The startup process has indeed behaved that way for > much longer than pg_stat_io existed - but it's harder to spot, because the > stats are more coarsely aggregated :/. And it's very oddly inconsistent: > > The

Re: pg_stat_io for the startup process

2023-04-25 Thread Andres Freund
Hi, On 2023-04-25 16:00:24 -0400, Robert Haas wrote: > On Tue, Apr 25, 2023 at 2:39 PM Andres Freund wrote: > > I'm mildly inclined to not consider it a bug, given that this looks to have > > been true for other stats for quite a while? But it does still seem worth > > improving upon - I'd make t

Re: pg_stat_io for the startup process

2023-04-25 Thread Michael Paquier
On Tue, Apr 25, 2023 at 04:00:24PM -0400, Robert Haas wrote: > ISTM that it's pretty desirable to do something about this. If the > process isn't going to report statistics properly, at least remove it > from the view. If it can be made to report properly, that would be > even better. But shipping

Re: pg_stat_io for the startup process

2023-04-25 Thread Robert Haas
On Tue, Apr 25, 2023 at 2:39 PM Andres Freund wrote: > I'm mildly inclined to not consider it a bug, given that this looks to have > been true for other stats for quite a while? But it does still seem worth > improving upon - I'd make the consideration when to apply the relevant patches > depend o

Re: pg_stat_io for the startup process

2023-04-25 Thread Andres Freund
Hi, On 2023-04-25 13:54:43 -0400, Melanie Plageman wrote: > On Tue, Apr 25, 2023 at 10:51:14PM +0900, Fujii Masao wrote: > > Regarding pg_stat_io for the startup process, I noticed that the counters > > are only incremented after the startup process exits, not during WAL replay

Re: pg_stat_io for the startup process

2023-04-25 Thread Melanie Plageman
On Tue, Apr 25, 2023 at 10:51:14PM +0900, Fujii Masao wrote: > Hi, > > Regarding pg_stat_io for the startup process, I noticed that the counters > are only incremented after the startup process exits, not during WAL replay > in standby mode. This is because pgstat_flush_io() is o

pg_stat_io for the startup process

2023-04-25 Thread Fujii Masao
Hi, Regarding pg_stat_io for the startup process, I noticed that the counters are only incremented after the startup process exits, not during WAL replay in standby mode. This is because pgstat_flush_io() is only called when the startup process exits. Shouldn't it be called during WAL repl