Re: Fix race condition in InvalidatePossiblyObsoleteSlot()

2024-03-06 Thread Bertrand Drouvot
Hi, On Wed, Mar 06, 2024 at 05:45:56PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 6, 2024 at 4:51 PM Michael Paquier wrote: > > > > On Wed, Mar 06, 2024 at 09:17:58AM +, Bertrand Drouvot wrote: > > > Right, somehow out of context here. > > > > We&#x

Spurious pgstat_drop_replslot() call

2024-03-08 Thread Bertrand Drouvot
. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 56ade7f561c180ee0120cb8c33d1c39e32ab7863 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Fri, 8 Mar 2024 09:29:29 + Subject: [PATCH v1] Re

Re: Missing LWLock protection in pgstat_reset_replslot()

2024-03-08 Thread Bertrand Drouvot
Hi, On Thu, Mar 07, 2024 at 02:17:53PM +0900, Michael Paquier wrote: > On Wed, Mar 06, 2024 at 09:05:59AM +0000, Bertrand Drouvot wrote: > > Yeah, all of the above done in v3 attached. > > In passing.. pgstat_create_replslot() and pgstat_drop_replslot() rely > on the assumpti

Re: Spurious pgstat_drop_replslot() call

2024-03-08 Thread Bertrand Drouvot
Hi, On Fri, Mar 08, 2024 at 07:55:39PM +0900, Michael Paquier wrote: > On Fri, Mar 08, 2024 at 10:19:11AM +0000, Bertrand Drouvot wrote: > > Indeed, it does not seem appropriate to remove stats during slot > > invalidation as > > one could still be interested to look at the

Re: Spurious pgstat_drop_replslot() call

2024-03-11 Thread Bertrand Drouvot
makes sense to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Statistics Import and Export

2024-03-11 Thread Bertrand Drouvot
nd not throwing an error? (if ok, then I think we can get rid of returning a bool). 7 === +* If this relation is an index and that index has expressions in +* it, and the attnum specified s/is an index and that index has/is an index that has/? Regards, -- Bertrand Drouvot Postg

Re: Spurious pgstat_drop_replslot() call

2024-03-11 Thread Bertrand Drouvot
we hold > ReplicationSlotAllocationLock? Yeah, makes fully sense and looks good to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-12 Thread Bertrand Drouvot
validations for "synced" slot on standby? I mean, do they make sense given the fact that those slots are not usable until the standby is promoted? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-12 Thread Bertrand Drouvot
Hi, On Tue, Mar 12, 2024 at 05:51:43PM +0530, Amit Kapila wrote: > On Tue, Mar 12, 2024 at 1:24 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 08, 2024 at 10:42:20PM +0530, Bharath Rupireddy wrote: > > > On Wed, Mar 6, 2024 at 4:49 PM Amit Kapila > > > w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-13 Thread Bertrand Drouvot
Hi, On Tue, Mar 12, 2024 at 09:19:35PM +0530, Bharath Rupireddy wrote: > On Tue, Mar 12, 2024 at 9:11 PM Bertrand Drouvot > wrote: > > > > > AFAIR, we don't prevent similar invalidations due to > > > 'max_slot_wal_keep_size' for sync slots, >

Re: Minimal logical decoding on standbys

2023-04-07 Thread Bertrand Drouvot
Hi, New wording works for me, thanks! Bertrand Le sam. 8 avr. 2023, 08:26, Andres Freund a écrit : > Hi, > > On 2023-04-07 11:12:26 -0700, Andres Freund wrote: > > + > > + > > + confl_active_logicalslot > bigint > > + > > + > > + Number of active logical slot

Re: Add isCatalogRel in rmgrdesc

2023-12-20 Thread Bertrand Drouvot
show the flag when !isCatalogRel, but I cannot get > > excited about that either because that would require one to do more > > cross-checks with the core code when looking at WAL dumps. > > Thank you for the comments. Agreed. > > I've just pushed, bf6260b39. > Thank

Re: Synchronizing slots from primary to standby

2023-12-21 Thread Bertrand Drouvot
veral places that the replication can be resumed after a failover. Should we add a few words about possible lag? (see [1]) [1]: https://www.postgresql.org/message-id/CAA4eK1KihniOK21mEVYtSOHRQiGNyToUmENWp7hPbH_PMsqzkA%40mail.gmail.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track in pg_replication_slots the reason why slots conflict?

2023-12-21 Thread Bertrand Drouvot
gt; > > > We clearly can't just expose the numerical value for a C enum. So it has to > > be > > converted to something SQL representable. > > > > We can return int2 value from the function pg_get_replication_slots() > and then use that to display a strin

Re: Synchronizing slots from primary to standby

2023-12-22 Thread Bertrand Drouvot
restarted Then when the standby starts, the "synced" slots will be invalidated and later removed but not re-created on the next sync-cycle (because they don't exist anymore on the primary). Worth to reword a bit that part? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track in pg_replication_slots the reason why slots conflict?

2023-12-26 Thread Bertrand Drouvot
for replacing the existing boolean > with a text. Same here, I'd vote to avoid 2 columns having the same "meaning". Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track in pg_replication_slots the reason why slots conflict?

2024-01-02 Thread Bertrand Drouvot
ut the need to refer to the documentation) and that the fact that it is "insufficient" is more or less implicit. Basically I think that with "primary_wal_level" one would need to refer to the doc less frequently than with "wal_level_insufficient". But again, that's

Re: Track in pg_replication_slots the reason why slots conflict?

2024-01-02 Thread Bertrand Drouvot
Hi, On Wed, Jan 03, 2024 at 08:53:44AM +0530, Amit Kapila wrote: > On Wed, Jan 3, 2024 at 7:10 AM Michael Paquier wrote: > > > > On Tue, Jan 02, 2024 at 02:07:58PM +, Bertrand Drouvot wrote: > > > + wal_level_insufficient means that the > > > +

Re: verify predefined LWLocks have entries in wait_event_names.txt

2024-01-03 Thread Bertrand Drouvot
time instead of during testing, which might be kind of > > nice, too. > > Yeah, I think that would be better. +1 to add a test and put in a place that would produce failures at build time. I think that having the test in the script that generates the header file is more appr

Re: Synchronizing slots from primary to standby

2024-01-03 Thread Bertrand Drouvot
ons before we decide to add a new GUC (for > dbname) for slotsync worker. > I think that as long as enable_syncslot is off then there is no need to add the dbname in primary_conninfo (means there is no need to change an existing primary_conninfo for the ones that don't use the sync

Re: Synchronizing slots from primary to standby

2024-01-04 Thread Bertrand Drouvot
ReplSlotSyncWorkerMain(). That does make sense, but what do you think about creating dedicated libpqslotsyncwrkr_connect and slotsyncwrkr_connect (instead of using the libpqrcv_connect / walrcv_connect ones)? That way we could make use of slotsyncwrkr_connect() in ReplSlotSyncWorkerMain() as I think it's confusing to use "rcv" functions while the process using them is not of backend type walreceiver. I'm not sure that worth the extra complexity though, what do you think? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: verify predefined LWLocks have entries in wait_event_names.txt

2024-01-04 Thread Bertrand Drouvot
Hi, On Fri, Jan 05, 2024 at 12:11:44AM -0600, Nathan Bossart wrote: > On Wed, Jan 03, 2024 at 07:59:45AM +0000, Bertrand Drouvot wrote: > > +1 to add a test and put in a place that would produce failures at build > > time. > > I think that having the test in the script that

Re: Synchronizing slots from primary to standby

2024-01-05 Thread Bertrand Drouvot
Hi, On Fri, Jan 05, 2024 at 10:00:53AM +0530, Amit Kapila wrote: > On Fri, Jan 5, 2024 at 8:59 AM shveta malik wrote: > > > > On Thu, Jan 4, 2024 at 7:24 PM Bertrand Drouvot > > wrote: > > > > > > 4 === > > > > > > Looking clo

Re: verify predefined LWLocks have entries in wait_event_names.txt

2024-01-06 Thread Bertrand Drouvot
Hi, On Fri, Jan 05, 2024 at 11:46:20AM -0600, Nathan Bossart wrote: > On Fri, Jan 05, 2024 at 10:42:03AM -0600, Nathan Bossart wrote: > > On Fri, Jan 05, 2024 at 07:39:39AM +, Bertrand Drouvot wrote: > >> + die "lists of predefined LWLocks

Re: Add a perl function in Cluster.pm to generate WAL

2024-01-07 Thread Bertrand Drouvot
s somehow due to [1] (means something holding global xmin). BTW, I think we should resume working on [1] and having it fixed in all the cases. [1]: https://www.postgresql.org/message-id/d40d015f-03a4-1cf2-6c1f-2b9aca860762%40gmail.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team

Re: verify predefined LWLocks have entries in wait_event_names.txt

2024-01-07 Thread Bertrand Drouvot
Hi, On Sat, Jan 06, 2024 at 10:18:52AM -0600, Nathan Bossart wrote: > On Sat, Jan 06, 2024 at 09:03:52AM +0000, Bertrand Drouvot wrote: > > Sorry, I missed this in my first review, but instead of: > > > > - input: files('../../backend/storage/lmgr/lwlocknames.t

Re: verify predefined LWLocks have entries in wait_event_names.txt

2024-01-08 Thread Bertrand Drouvot
Hi, On Mon, Jan 08, 2024 at 04:02:12PM -0600, Nathan Bossart wrote: > Sorry for the noise. I spent some more time tidying this up for commit, > which I am hoping to do in the next day or two. Thanks! v6 looks good to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RD

Re: Add a perl function in Cluster.pm to generate WAL

2024-01-08 Thread Bertrand Drouvot
in stability once you begin using the > patch posted on [2], mentioned by Bertrand? Alexander, pleae find attached v3 which is more or less a rebased version of it. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-01-10 Thread Bertrand Drouvot
tests numbering are not the same (23 and 24). That is even more weird as those tests should be the 24 and 25 ones. Would it be possible to also send the standby logs? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-06-05 Thread Bertrand Drouvot
Hi, On Thu, May 23, 2024 at 02:10:54PM -0400, Robert Haas wrote: > On Thu, May 23, 2024 at 12:19 AM Bertrand Drouvot > wrote: > > The reason why we are using a dirty snapshot here is for the cases where we > > are > > recording a dependency on a referenced object tha

Re: relfilenode statistics

2024-06-06 Thread Bertrand Drouvot
Hi, On Mon, Jun 03, 2024 at 11:11:46AM +, Bertrand Drouvot wrote: > Yeah, I’ll update the commit message in V2 with better explanations once I get > feedback on V1 (should we decide to move on with the relfilenode stats idea). > Please find attached v2, mandatory rebase due to c

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Bertrand Drouvot
ossible orphaned dependencies cases. There is an ongoing thread (see [1]) to fix the orphaned dependencies issue. v9 attached in [1] fixes the case you describe here. [1]: https://www.postgresql.org/message-id/flat/ZiYjn0eVc7pxVY45%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 04:00:23PM -0400, Robert Haas wrote: > On Thu, Jun 6, 2024 at 1:56 AM Bertrand Drouvot > wrote: > > v9 is more invasive (as it changes code in much more places) than v8 but it > > is > > easier to follow (as it is now clear where the new loc

Re: relfilenode statistics

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 08:38:06PM -0700, Andres Freund wrote: > Hi, > > On 2024-06-03 11:11:46 +, Bertrand Drouvot wrote: > > The main argument is that we currently don’t have writes counters for > > relations. > > The reason is that we don’t have the relati

Re: relfilenode statistics

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 08:17:36PM -0700, Andres Freund wrote: > Hi, > > On 2024-06-06 12:27:49 -0400, Robert Haas wrote: > > On Wed, Jun 5, 2024 at 1:52 AM Bertrand Drouvot > > wrote: > > > I think we should keep the stats in the relation during relfilenode

Track the amount of time waiting due to cost_delay

2024-06-09 Thread Bertrand Drouvot
doing-that/ Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 750dfc26cd6fcf5a5618c3fe35fc42d5b5c66f00 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Thu, 6 Jun 2024 1

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-06-09 Thread Bertrand Drouvot
Hi Alexander, On Sat, Jun 08, 2024 at 07:00:00AM +0300, Alexander Lakhin wrote: > Hello Bertrand and Michael, > > 23.01.2024 11:07, Bertrand Drouvot wrote: > > On Tue, Jan 23, 2024 at 02:50:06PM +0900, Michael Paquier wrote: > > > > > Anyway, that's not th

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-06-10 Thread Bertrand Drouvot
Hi, On Mon, Jun 10, 2024 at 03:39:34PM +0900, Michael Paquier wrote: > On Mon, Jun 10, 2024 at 06:29:17AM +0000, Bertrand Drouvot wrote: > > Thanks for the report! I think it makes sense to add it to the list of known > > failures. > > > > One way to deal with those co

Re: Proposal to add page headers to SLRU pages

2024-06-10 Thread Bertrand Drouvot
of pg_upgrade. I think there is something in t/002_pg_upgrade.pl (see src/bin/pg_upgrade/TESTING), that could be used to run automated tests using an old and a current version. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: relfilenode statistics

2024-06-10 Thread Bertrand Drouvot
tables (like heap_blks_written), 2) ensure the user can still retrieve the stats from pg_statio_all_tables in such a way that it survives a rewrite, 3) provide dedicated APIs to retrieve relfilenode stats but only for the current relfilenode, 4) document this behavior. This is what the POC patc

Re: Track the amount of time waiting due to cost_delay

2024-06-10 Thread Bertrand Drouvot
Hi, On Mon, Jun 10, 2024 at 10:36:42AM -0500, Nathan Bossart wrote: > On Mon, Jun 10, 2024 at 06:05:13AM +0000, Bertrand Drouvot wrote: > > During the last pgconf.dev I attended Robert´s presentation about > > autovacuum and > > it made me remember of an idea I had so

Re: Track the amount of time waiting due to cost_delay

2024-06-10 Thread Bertrand Drouvot
On Mon, Jun 10, 2024 at 02:20:16PM -0500, Nathan Bossart wrote: > On Mon, Jun 10, 2024 at 05:48:22PM +0000, Bertrand Drouvot wrote: > > On Mon, Jun 10, 2024 at 10:36:42AM -0500, Nathan Bossart wrote: > >> I wonder if we should also > >> surface the effective cost limit

Re: Track the amount of time waiting due to cost_delay

2024-06-10 Thread Bertrand Drouvot
JOIN pg_stat_activity a using (pid) > > Maybe all progress views should just expose the > "beentry->st_activity_start_timestamp " > to let the user know when the current operation began. Yeah maybe, I think this is likely a separate discussion too, thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
hread) (I think that 3) could be misleading but I'll yield to majority opinion if people think it's not). Thoughts? [1]: https://www.postgresql.org/message-id/A0935130-7C4B-4094-B6E4-C7D5086D50EF%40amazon.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
ve in another dedicated view? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
d/Zmf712A5xcOM9Hlg%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 1345e99dcb.

Re: Allow logical failover slots to wait on synchronous replication

2024-06-11 Thread Bertrand Drouvot
plication to be held back unnecessarily if one of the > replicas falls behind for whatever reason. A way to tie standby_slot_names > to synchronous replication instead does seem like it would be useful in > this case. FWIW, I have the same understanding and also think your proposal would be useful in thi

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 11:40:36AM -0500, Nathan Bossart wrote: > On Tue, Jun 11, 2024 at 07:25:11AM +0000, Bertrand Drouvot wrote: > > So I think that in v2 we could: 1) measure the actual wait time instead, 2) > > count the number of times the vacuum slept. We could also

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
addition to the time waited). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 01:13:48PM -0400, Robert Haas wrote: > On Tue, Jun 11, 2024 at 5:49 AM Bertrand Drouvot > wrote: > > As we can see the actual wait time is 30ms less than the intended wait time > > with > > this simple test. So I still think we should go

Re: relfilenode statistics

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 03:35:23PM +0900, Kyotaro Horiguchi wrote: > At Mon, 10 Jun 2024 08:09:56 +0000, Bertrand Drouvot > wrote in > > > > My idea was to move all that is in pg_statio_all_tables to relfilenode stats > > and 1) add new stats to pg

Re: Track the amount of time waiting due to cost_delay

2024-06-13 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 08:26:23AM +, Bertrand Drouvot wrote: > Hi, > > On Tue, Jun 11, 2024 at 04:07:05PM +0900, Masahiko Sawada wrote: > > > Thank you for the proposal and the patch. I understand the motivation > > of this patch. > > Thanks for looking

Re: Avoid orphaned objects dependencies, take 3

2024-06-13 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 10:49:34AM -0400, Robert Haas wrote: > On Fri, Jun 7, 2024 at 4:41 AM Bertrand Drouvot > wrote: > > Do you still find the code hard to maintain with v9? > > I don't think it substantially changes my concerns as compared with > the earlie

Re: Avoid orphaned objects dependencies, take 3

2024-06-14 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 02:27:45PM -0400, Robert Haas wrote: > On Thu, Jun 13, 2024 at 12:52 PM Bertrand Drouvot > wrote: > > > > table_open(childRelId, ...) would lock any "ALTER TABLE > > > > DROP CONSTRAINT" > > > > already. Not sure I

Re: Avoid orphaned objects dependencies, take 3

2024-06-17 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 04:52:09PM +, Bertrand Drouvot wrote: > Hi, > > On Thu, Jun 13, 2024 at 10:49:34AM -0400, Robert Haas wrote: > > On Fri, Jun 7, 2024 at 4:41 AM Bertrand Drouvot > > wrote: > > > Do you still find the code hard to maintain with

Re: Allow logical failover slots to wait on synchronous replication

2024-06-17 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 10:00:46AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Jun 10, 2024 at 09:25:10PM -0500, Nathan Bossart wrote: > > On Mon, Jun 10, 2024 at 03:51:05PM -0700, John H wrote: > > > The existing 'standby_slot_names' isn't great f

Re: Avoid orphaned objects dependencies, take 3

2024-06-17 Thread Bertrand Drouvot
Hi, On Mon, Jun 17, 2024 at 12:24:46PM -0400, Robert Haas wrote: > On Fri, Jun 14, 2024 at 3:54 AM Bertrand Drouvot > wrote: > > > Ah, right. So, I was assuming that, with either this version of your > > > patch or the earlier version, we'd end up locking the constra

Re: Avoid orphaned objects dependencies, take 3

2024-06-19 Thread Bertrand Drouvot
Hi, On Mon, Jun 17, 2024 at 05:57:05PM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Jun 17, 2024 at 12:24:46PM -0400, Robert Haas wrote: > > So to try to sum up here: I'm not sure I agree with this design. But I > > also feel like the design is not as clear an

Re: Pluggable cumulative statistics

2024-06-20 Thread Bertrand Drouvot
om stats (so what you've done minus the in-core stats). This one would not be configurable and the extensions will not need to know about it. Would that remove the benefit from c) that you mentioned up-thread? [1]: https://www.postgresql.org/message-id/20220818195124.c7ipzf6c5v7vxymc%40aw

Re: Pluggable cumulative statistics

2024-06-20 Thread Bertrand Drouvot
es documentation. Did not look yet. > - 0005 is an example of how to use them, with a TAP test providing > coverage. Did not look yet. As I said, I've in mind to do a more in depth review. I've noted the above while doing a quick read of the patches so thought it makes sense to share them now while at it. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-06-21 Thread Bertrand Drouvot
Hi, On Wed, Jun 19, 2024 at 02:11:50PM +, Bertrand Drouvot wrote: > To sum up, I did not see any cases that did not lead to 1. or 2., so I think > it's safe to not add an extra lock for the RelationRelationId case. If, for > any > reason, there is still cases that are outs

Re: Track the amount of time waiting due to cost_delay

2024-06-22 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 11:56:26AM +, Bertrand Drouvot wrote: > == Observations > === > > As compare to v2: > > 1. scanning heap time is about the same > 2. vacuuming indexes time is about 3 minutes

Re: Track the amount of time waiting due to cost_delay

2024-06-24 Thread Bertrand Drouvot
Hi, On Sat, Jun 22, 2024 at 12:48:33PM +, Bertrand Drouvot wrote: > 1. vacuuming indexes time has been longer on master because with v2, the > leader > has been interrupted 342605 times while waiting, then making v2 "faster". > > 2. the leader being interrupted whil

Re: New standby_slot_names GUC in PG 17

2024-06-24 Thread Bertrand Drouvot
and that we want them to be in sync. So I'd vote for (in that order); synchronized_primary_slot_names, synchronized_standby_slots [1]: https://www.postgresql.org/message-id/bb437218-73bc-34c3-b8fb-8c1be4ddaec9%40enterprisedb.com [2]: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=93db6cbda037f1be9544932bd9a785dabf3ff712 Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-25 Thread Bertrand Drouvot
_stat_progress_vacuum at 1 second interval and see by how much the vaccum has been paused during that time could help too (specially if it is made of a lot of parallel workers that could lead to a lot of I/O). But it would miss data if we are reporting at a larger rate. > It just occurred to me

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Bertrand Drouvot
ot sure that wording is correct, if we feel the need to explain the GUC, maybe repeat some wording from bf279ddd1c? 2 Should we rename StandbySlotNamesConfigData too? 3 ==== Should we rename SlotExistsInStandbySlotNames too? 4 Should we rename validate_standby_slots() too? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Bertrand Drouvot
Hi, On Wed, Jun 26, 2024 at 11:39:45AM +0530, Amit Kapila wrote: > On Wed, Jun 26, 2024 at 10:19 AM Bertrand Drouvot > wrote: > > > > > > 2 > > > > Should we rename StandbySlotNamesConfigData too? > > > > How about SyncStandbySlotsCo

Re: Avoid orphaned objects dependencies, take 3

2024-06-26 Thread Bertrand Drouvot
Hi, On Fri, Jun 21, 2024 at 01:22:43PM +, Bertrand Drouvot wrote: > Another thought for the RelationRelationId class case: we could check if there > is a lock first and if there is no lock then acquire one. That way that would > ensure the relation is always locked (so no "risk&

Re: New standby_slot_names GUC in PG 17

2024-06-26 Thread Bertrand Drouvot
Hi, On Wed, Jun 26, 2024 at 09:15:48AM +, Zhijie Hou (Fujitsu) wrote: > Renamed these to the names suggested by Amit. > > Attach the v2 patch set which addressed above and removed > the changes in release-17.sgml according to the comment from Amit. > Thanks! LGTM. Regards

Re: walsender.c comment with no context is hard to understand

2024-06-28 Thread Bertrand Drouvot
lines became separated due to a 2023 patch [1], and you > > were one of the reviewers of that patch, so I assumed the code > > separation was deemed OK at that time. Unless it was some mistake that > > slipped past multiple reviewers? > > > > I don't know whether your assum

Re: Track the amount of time waiting due to cost_delay

2024-06-30 Thread Bertrand Drouvot
; [1] > https://www.postgresql.org/message-id/01000190606e3d2a-116ead16-84d2-4449-8d18-5053da66b1f4-00%40email.amazonses.com > Yeah, I think it would make sense to put this thread on hold until we know more about [1] (you mentioned above) outcome. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Surround CheckRelation[Oid]LockedByMe() with USE_ASSERT_CHECKING

2024-06-30 Thread Bertrand Drouvot
patch doing so. Thoughts? [1]: https://www.postgresql.org/message-id/flat/ZiYjn0eVc7pxVY45%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >F

Re: Surround CheckRelation[Oid]LockedByMe() with USE_ASSERT_CHECKING

2024-07-01 Thread Bertrand Drouvot
Hi, On Mon, Jul 01, 2024 at 12:35:34PM +0530, Bharath Rupireddy wrote: > Hi, > > On Mon, Jul 1, 2024 at 12:12 PM Bertrand Drouvot > wrote: > > > > Hi hackers, > > > > While working on a rebase for [1] due to 0cecc908e97, I noticed th

Re: Avoid orphaned objects dependencies, take 3

2024-07-01 Thread Bertrand Drouvot
Hi, On Wed, Jun 26, 2024 at 10:24:41AM +, Bertrand Drouvot wrote: > Hi, > > On Fri, Jun 21, 2024 at 01:22:43PM +, Bertrand Drouvot wrote: > > Another thought for the RelationRelationId class case: we could check if > > there > > is a lock first and if there

Re: Surround CheckRelation[Oid]LockedByMe() with USE_ASSERT_CHECKING

2024-07-01 Thread Bertrand Drouvot
Hi, On Mon, Jul 01, 2024 at 05:01:46PM +0900, Michael Paquier wrote: > On Mon, Jul 01, 2024 at 06:42:46AM +0000, Bertrand Drouvot wrote: > > While working on a rebase for [1] due to 0cecc908e97, I noticed that > > CheckRelationLockedByMe() and CheckRelationOidLockedByMe() a

Re: Surround CheckRelation[Oid]LockedByMe() with USE_ASSERT_CHECKING

2024-07-01 Thread Bertrand Drouvot
Hi, On Mon, Jul 01, 2024 at 10:21:35AM -0400, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Jul 01, 2024 at 06:42:46AM +, Bertrand Drouvot wrote: > >> I think it would make sense to declare / define those functions only for > >> assert enabled build: please

Re: Avoid orphaned objects dependencies, take 3

2024-07-01 Thread Bertrand Drouvot
Hi, On Mon, Jul 01, 2024 at 09:39:17AM +, Bertrand Drouvot wrote: > Hi, > > On Wed, Jun 26, 2024 at 10:24:41AM +, Bertrand Drouvot wrote: > > Hi, > > > > On Fri, Jun 21, 2024 at 01:22:43PM +, Bertrand Drouvot wrote: > > > Another thought for t

Re: Restart pg_usleep when interrupted

2024-07-02 Thread Bertrand Drouvot
might be another idea to explore. [1]: https://www.postgresql.org/message-id/flat/ZmaXmWDL829fzAVX%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Pluggable cumulative statistics

2024-07-04 Thread Bertrand Drouvot
struct). That would mean having things like: " typedef struct PgStatShared_Archiver { PgStat_ArchiverStats stats; /* lock protects ->reset_offset as well as stats->stat_reset_timestamp */ LWLock lock; uint32 changecount; PgStat_ArchiverStats reset_

Re: walsender.c comment with no context is hard to understand

2024-07-06 Thread Bertrand Drouvot
Hi, On Fri, Jul 05, 2024 at 11:10:00AM +0530, Amit Kapila wrote: > On Fri, Jun 28, 2024 at 6:30 PM Bertrand Drouvot > wrote: > > > > On Fri, Jun 28, 2024 at 03:15:22PM +0530, Amit Kapila wrote: > > > On Fri, Jun 28, 2024 at 12:55 PM Peter Smith > > > wro

Re: walsender.c comment with no context is hard to understand

2024-07-07 Thread Bertrand Drouvot
if that happens we can correctly determine the required > timeline because all the required WAL is already available, is that > correct Yeah that's correct. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: walsender.c comment with no context is hard to understand

2024-07-07 Thread Bertrand Drouvot
Hi, On Mon, Jul 08, 2024 at 11:20:45AM +0530, Amit Kapila wrote: > On Mon, Jul 8, 2024 at 11:08 AM Bertrand Drouvot > wrote: > > > > On Mon, Jul 08, 2024 at 08:46:19AM +0530, Amit Kapila wrote: > > > This sounds better but it is better to add just before we determine &

Re: Pluggable cumulative statistics

2024-07-07 Thread Bertrand Drouvot
Hi, On Mon, Jul 08, 2024 at 02:30:23PM +0900, Michael Paquier wrote: > On Fri, Jul 05, 2024 at 09:35:19AM +0900, Michael Paquier wrote: > > On Thu, Jul 04, 2024 at 11:30:17AM +, Bertrand Drouvot wrote: > >> On Wed, Jul 03, 2024 at 06:47:15PM +0900, Mic

Re: Pluggable cumulative statistics

2024-07-08 Thread Bertrand Drouvot
Hi, On Mon, Jul 08, 2024 at 03:49:34PM +0900, Michael Paquier wrote: > On Mon, Jul 08, 2024 at 06:39:56AM +0000, Bertrand Drouvot wrote: > > + for (int kind = PGSTAT_KIND_FIRST_VALID; kind <= PGSTAT_KIND_LAST; > > kind++) > > + { > > +

Re: Pluggable cumulative statistics

2024-07-08 Thread Bertrand Drouvot
Hi, On Mon, Jul 08, 2024 at 07:22:32AM +, Bertrand Drouvot wrote: > Except the above (which is just a Nit), 0001 LGTM. > Looking at 0002: It looks pretty straightforward, just one comment: + ptr = ((char *) ctl) + kind_info->share

Re: Pluggable cumulative statistics

2024-07-08 Thread Bertrand Drouvot
Hi, On Tue, Jul 09, 2024 at 10:45:05AM +0900, Michael Paquier wrote: > On Mon, Jul 08, 2024 at 07:22:32AM +0000, Bertrand Drouvot wrote: > > Yeah, what I meant to say is that one could think for example that's the > > PgStatShared_Archiver size while in fact it's the

Re: Restart pg_usleep when interrupted

2024-07-09 Thread Bertrand Drouvot
it would be enough to compare their ticks. > Since sub-millisecond sleep times are not guaranteed as suggested by > the vacuum_cost_delay docs ( see below ), an alternative idea > is to use clock_nanosleep for vacuum delay when it’s available, else > fallback to WaitLatch. Wouldn't that increase even more the cases where sub-millisecond won't be guaranteed? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-07-10 Thread Bertrand Drouvot
Hi, On Tue, Jul 02, 2024 at 05:56:23AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Jul 01, 2024 at 09:39:17AM +, Bertrand Drouvot wrote: > > Hi, > > > > On Wed, Jun 26, 2024 at 10:24:41AM +, Bertrand Drouvot wrote: > > > Hi, > > > &

Re: Pluggable cumulative statistics

2024-07-10 Thread Bertrand Drouvot
Hi, On Tue, Jul 09, 2024 at 03:54:37PM +0900, Michael Paquier wrote: > On Mon, Jul 08, 2024 at 02:07:58PM +0000, Bertrand Drouvot wrote: > > It looks pretty straightforward, just one comment: > > > > + ptr = ((char *) ctl) + kind

Re: Pluggable cumulative statistics

2024-07-10 Thread Bertrand Drouvot
Hi, On Wed, Jul 10, 2024 at 08:28:56AM +, Bertrand Drouvot wrote: > Hi, > > On Tue, Jul 09, 2024 at 03:54:37PM +0900, Michael Paquier wrote: > > On Mon, Jul 08, 2024 at 02:07:58PM +, Bertrand Drouvot wrote: > > > It looks pretty straightf

Re: relfilenode statistics

2024-07-10 Thread Bertrand Drouvot
Hi, On Wed, Jul 10, 2024 at 03:02:34PM +0900, Michael Paquier wrote: > On Sat, May 25, 2024 at 07:52:02AM +0000, Bertrand Drouvot wrote: > > But I think that it is in a state that can be used to discuss the approach > > it > > is implementing (so that we can agree or not

Re: Allow logical failover slots to wait on synchronous replication

2024-07-10 Thread Bertrand Drouvot
tion on lock */ + for (i = 0; i < NUM_SYNC_REP_WAIT_MODE; i++) + { + lsn[i] = walsndctl->lsn[i]; + } NUM_SYNC_REP_WAIT_MODE is small but as the goal is the keep the lock time as short as possible I wonder if it wouldn't be better to use memcpy() here instead of this for loop. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: relfilenode statistics

2024-07-10 Thread Bertrand Drouvot
Hi, On Thu, Jul 11, 2024 at 01:58:19PM +0900, Michael Paquier wrote: > On Wed, Jul 10, 2024 at 01:38:06PM +0000, Bertrand Drouvot wrote: > > So, I think it makes sense to link the hashkey to all the RelFileLocator > > fields, means: > > > > dboid (linked to RelFile

Re: Pluggable cumulative statistics

2024-07-11 Thread Bertrand Drouvot
Hi, On Fri, Jul 05, 2024 at 09:35:19AM +0900, Michael Paquier wrote: > On Thu, Jul 04, 2024 at 11:30:17AM +0000, Bertrand Drouvot wrote: > > > > /* > > * Reads in existing statistics file into the shared stats hash. > > > > This comment above pgstat_read_statsf

Re: Restart pg_usleep when interrupted

2024-07-12 Thread Bertrand Drouvot
+static void vacuum_sleep(double msec); What about a more generic name that could be used outside of vacuum? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Flush pgstats file during checkpoints

2024-07-12 Thread Bertrand Drouvot
ome decision over giving up entirely based on some previous state of > the stats. That sounds like a win for me with steady workloads, less > for bulby workloads.. I agree and it is not worst (though not ideal) that currently on HEAD. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Flush pgstats file during checkpoints

2024-07-12 Thread Bertrand Drouvot
e conflicts. Thanks! Looking at 0001: + /* error logged already */ Maybe mention it's already logged by durable_rename() (like it's done in InstallXLogFileSegment(), BaseBackup() for example). Except this nit, 0001 LGTM. Need to spend more time and thoughts on 0002

Re: Flush pgstats file during checkpoints

2024-07-12 Thread Bertrand Drouvot
Hi, On Fri, Jul 12, 2024 at 12:10:26PM +, Bertrand Drouvot wrote: > Need to spend more time and thoughts on 0002+. I think there is a corner case, say: 1. shutdown checkpoint at LSN1 2. startup->reads the stat file (contains LSN1)->all good->read stat file and remove it

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
s for the patch! > Now both worker and SQL > function set 'syncing' when they start and reset it when they exit. It means that it's not possible anymore to trigger a manual sync if sync_replication_slots is on. Indeed that would trigger: postgres=# select pg_sync_re

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
Hi, On Mon, Apr 15, 2024 at 06:29:49PM +0530, Amit Kapila wrote: > On Mon, Apr 15, 2024 at 6:21 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 15, 2024 at 03:38:53PM +0530, shveta malik wrote: > > > Now both worker and SQL > > > function set 'syncin

<    1   2   3   4   5   6   7   8   9   >