Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-21 Thread Nathan Bossart
we must keep it around so that * recovery restarts from the right place. */ if (ReachedNormalRunning) CancelBackup(); The attached patch removes this code. -- Nathan Bossart Amazon Web Services: https:/

Re: Add index scan progress to pg_stat_progress_vacuum

2022-02-21 Thread Nathan Bossart
more naturally in a hash table. +if (callback) +callback(values, 3); Why does this need to be set up as a callback function? Could we just call the function if cmdtype == PROGRESS_COMMAND_VACUUM? ISTM that is pretty much all this is doing. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

remove more archiving overhead

2022-02-21 Thread Nathan Bossart
I've attached a patch that makes the changes discussed above. Thoughts? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From a6d033aff90a6218345cba41847ccfdfbe6447d7 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Mon, 21 Feb 2022 16:29:14 -0800 Subject: [PATCH v1

Re: remove more archiving overhead

2022-02-22 Thread Nathan Bossart
On Mon, Feb 21, 2022 at 05:19:48PM -0800, Nathan Bossart wrote: > I also spent some time investigating whether durably renaming the archive > status files was even necessary. In theory, it shouldn't be. If a crash > happens before the rename is persisted, we might try to re-archiv

Re: remove more archiving overhead

2022-02-22 Thread Nathan Bossart
On Tue, Feb 22, 2022 at 09:37:11AM -0800, Nathan Bossart wrote: > In my testing, I found that when I killed the server just before unlink() > during WAL recyling, I ended up with links to the same file in pg_wal after > restarting. My latest test produced links to the same file for th

Re: C++ Trigger Framework

2022-02-22 Thread Nathan Bossart
iler incompatibilities)? This might answer your questions: https://www.postgresql.org/docs/devel/xfunc-c.html#EXTEND-CPP -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow file inclusion in pg_hba and pg_ident files

2022-02-23 Thread Nathan Bossart
optional ssl flag and returns the file, line and raw line > matching if any, or null. For instance: I think another use-case for this is testing updates to your configuration files. For example, I could ensure that hba_forbid_non_ssl.conf wasn't accidentally reverted as part of an unrelated change. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: remove more archiving overhead

2022-02-24 Thread Nathan Bossart
27;t it a big > difference? Yeah, it probably increases the chances quite a bit. > I think we don't want make checkpointer slower in exchange of making > archiver faster. Perhaps we need to work using a condensed > information rather than repeatedly scanning over the distributed >

Re: remove more archiving overhead

2022-02-24 Thread Nathan Bossart
. With the patch, it took around 62 seconds, which amounts to a ~43% reduction in overhead. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Fix overflow in justify_interval related functions

2022-02-25 Thread Nathan Bossart
On Fri, Feb 25, 2022 at 10:30:57AM -0500, Joseph Koshakow wrote: > Just checking because I'm not very familiar with the process, > are there any outstanding items that I need to do for this patch? Unless someone has additional feedback, I don't think so. -- Nathan Bossart Ama

Re: Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-25 Thread Nathan Bossart
run looks like when the synchronous replicas are in the same region. That is probably a more realistic use-case. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Add index scan progress to pg_stat_progress_vacuum

2022-02-25 Thread Nathan Bossart
cal context about the index work done before failsafe kicked in, but IMO it is misleading to still include it in the progress view. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-02-25 Thread Nathan Bossart
r a while. Should we mark it as returned-with-feedback? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-26 Thread Nathan Bossart
onsiders an LSN synchronously replicated, it would be okay to send it to the async standbys. I personally don't think it is worth taking on the extra complexity for that level of configuration just yet. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Nathan Bossart
licitly supplied, if it is practical to determine that from fn_expr. (I > haven't > looked yet to see how practical that is.) Agreed. I will look into updating this one, too. I think the 'exclusive' parameter should remain documented for now for both pg_start_backup() and pg_stop_backup(), but this documentation will just note that it is there for backward compatibility and must be set to false. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Nathan Bossart
patch > removes the comment describing it: > > > - * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs > - * describing the cluster's tablespaces. > > > which seems like collateral damage. Thanks. I will fix this and the proofre

Re: parse/analyze API refactoring

2022-02-28 Thread Nathan Bossart
I believe my only feedback was around deduplicating the pg_analyze_and_rewrite_*() functions. Would you rather handle that in a separate patch? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-28 Thread Nathan Bossart
On Mon, Feb 28, 2022 at 06:45:51PM +0530, Bharath Rupireddy wrote: > On Sat, Feb 26, 2022 at 9:37 PM Nathan Bossart > wrote: >> For >> this feature, I think we always need to consider what the primary considers >> synchronously replicated. My suggested approach d

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-28 Thread Nathan Bossart
On Sat, Feb 26, 2022 at 02:06:14PM -0800, Nathan Bossart wrote: > On Sat, Feb 26, 2022 at 04:48:52PM +, Chapman Flack wrote: >> Assuming the value is false, so no error is thrown, is it practical to >> determine >> from flinfo->fn_expr whether the value was default

Re: Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-28 Thread Nathan Bossart
On Tue, Mar 01, 2022 at 11:10:09AM +0530, Bharath Rupireddy wrote: > On Tue, Mar 1, 2022 at 12:27 AM Nathan Bossart > wrote: >> My feedback is specifically about this behavior. I don't think we should >> spin in XLogSend*() waiting for an LSN to be synchronously rep

Re: Pre-allocating WAL files

2022-03-01 Thread Nathan Bossart
; DROP TABLESPACE regress_tblspace_renamed; > +ERROR: tablespace "regress_tblspace_renamed" is not empty I believe this is due to an existing bug. This patch set seems to influence the timing to make it more likely. I'm tracking the fix here: https://commitfest.postgresql.org/37/

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
art/stop. I'm fine with this approach. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow async standbys wait for sync replication

2022-03-01 Thread Nathan Bossart
On Tue, Mar 01, 2022 at 04:34:31PM +0900, Kyotaro Horiguchi wrote: > At Mon, 28 Feb 2022 22:05:28 -0800, Nathan Bossart > wrote in >> replicated LSN. TBH there are probably other things that need to be >> considered (e.g., how do we ensure that the WAL sender sends the

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
at pg_start/stop_backup() are going to be removed soon (and always emit a WARNING when they are used). Would that address your concerns? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Proposal: Support custom authentication methods using hooks

2022-03-01 Thread Nathan Bossart
On Mon, Feb 28, 2022 at 03:46:34PM -0500, Stephen Frost wrote: > md5 should be removed. +1 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow async standbys wait for sync replication

2022-03-01 Thread Nathan Bossart
On Tue, Mar 01, 2022 at 11:09:57PM +0530, Bharath Rupireddy wrote: > On Tue, Mar 1, 2022 at 10:35 PM Nathan Bossart > wrote: >> Yes, perhaps the synchronous replication framework will need to alert WAL >> senders when the syncrep LSN advances so that the WAL is sent to the

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
athan Bossart Amazon Web Services: https://aws.amazon.com >From 7119f9063f22652fca1e2a44fdf6b4b6b3fbf679 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 1 Dec 2021 23:50:49 + Subject: [PATCH v4 1/1] remove exclusive backup mode --- doc/src/sgml/backup.sgml | 188 +---

Re: Allow async standbys wait for sync replication

2022-03-04 Thread Nathan Bossart
On Wed, Mar 02, 2022 at 09:47:09AM +0530, Bharath Rupireddy wrote: > On Wed, Mar 2, 2022 at 2:57 AM Nathan Bossart > wrote: >> I think there are a couple of advantages. For one, spinning is probably >> not the best from a resource perspective. > > Just to be on the sa

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread Nathan Bossart
eded..." AFAICT the more complicated strategy was around before pg_basebackup, and the pg_basebackup recommendation was added in 2012 as part of 920febd. Thoughts? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread Nathan Bossart
On Tue, Mar 08, 2022 at 03:09:50PM -0600, David Steele wrote: > On 3/8/22 14:01, Nathan Bossart wrote: >> On Wed, Mar 02, 2022 at 02:23:51PM -0500, Chapman Flack wrote: >> > I did not notice this earlier (sorry), but there seems to remain in >> > backup.sgml a programl

Re: parse/analyze API refactoring

2022-03-09 Thread Nathan Bossart
On Wed, Mar 09, 2022 at 11:35:32AM +0100, Peter Eisentraut wrote: > I have committed my original patches. I'll leave the above-mentioned topic > as ideas for the future. Sounds good. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Nathan Bossart
cumentation additions like this for Chap's new thread? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Nathan Bossart
alent about label => 'label'. It would be consistent, > but should we just agree for conciseness that there will always be > a label and it will always be first? > > You can pretty much tell in a call what's a label; it's those anonymous > trues and falses that a

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-09 Thread Nathan Bossart
e backends progress array. I think this can be removed. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-10 Thread Nathan Bossart
t_vacuum_worker_progress(). IMO the patches are in decent shape, and this should likely be marked as ready-for-committer in the near future. Before doing so, I think we should check that Sawada-san is okay with moving the deeper infrastructure changes to a separate threaḋ. -- Nathan Bo

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-10 Thread Nathan Bossart
ile_name_offset(b.lsn) w; but I think your suggestion is simpler. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 3283d2b85f38f46d1e2ada0e6c5ea59d8c8e9f9d Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 1 Dec 2021 23:50:49 + Subject: [PATCH v7 1/1] remove excl

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-11 Thread Nathan Bossart
On Fri, Mar 11, 2022 at 02:00:37PM -0500, Chapman Flack wrote: > v7 looks good to me. I have repeated the installcheck-world and given > the changed documentation sections one last read-through, and will > change the status to RfC. Thanks for reviewing! -- Nathan Bossart Amazon Web

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-11 Thread Nathan Bossart
y. +CheckpointStats.repl_snap_cutoff_lsn = cutoff; I have the same note for this one as I have for repl_map_cutoff_lsn. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-12 Thread Nathan Bossart
good to me. Barring additional feedback, I intend to mark this as ready-for-committer early next week. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Optimize external TOAST storage

2022-03-12 Thread Nathan Bossart
g(MemoryContextIsValid(context)); I didn't see an existing commitfest entry for this patch. I'd encourage you to add one in the July commitfest: https://commitfest.postgresql.org/38/ -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Allow async standbys wait for sync replication

2022-03-12 Thread Nathan Bossart
ck WAL replay on async standbys based on synchronous replication might be relatively easy. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-13 Thread Nathan Bossart
s. Do you think reporting the sizes is worth the extra system call? [0] https://postgr.es/m/20220215231123.GA2584239%40nathanxps13 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-14 Thread Nathan Bossart
more closely related to the number of files anyway. I'm -1 on splitting these new statistics to separate LOGs. In addition to making it more difficult to discover statistics for a given checkpoint, I think it actually adds even more bloat to the server log. If we are concerned about the amount of

Re: Estimating HugePages Requirements?

2022-03-14 Thread Nathan Bossart
27;s alright to leave it for the v16 development cycle? [0] https://postgr.es/m/C45224E1-29C8-414C-A8E6-B718C07ACB94%40amazon.com -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 7ee7b176c5280349631426ff047a9df394e26d59 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Mo

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread Nathan Bossart
My compiler is worried that syncslotname may be used uninitialized in start_table_sync(). The attached patch seems to silence this warning. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 05e4e03af5afa1658ede8d78b31c1c999b5c7deb Mon Sep 17 00:00:00 2001 From: Nathan Boss

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-15 Thread Nathan Bossart
AL segment. I don't have specific numbers to share, but as noted elsewhere [0], I routinely see lengthy checkpoints that spend a lot of time in these cleanup tasks. [0] https://postgr.es/m/18ED8B1F-7F5B-4ABF-848D-45916C938BC7%40amazon.com -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Optimize external TOAST storage

2022-03-15 Thread Nathan Bossart
we can choose one threshold that will work for all PostgreSQL installations in the known universe. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Estimating HugePages Requirements?

2022-03-15 Thread Nathan Bossart
a DEBUG* value). For comparison, 'postgres -C' for a non-runtime-computed GUC does not emit extra output as long as the user sets log_min_messages to DEBUG2 or higher. > The "able to run on a live cluster" seems a lot bigger and more scary > and not 15 material.

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-03-16 Thread Nathan Bossart
On Wed, Mar 16, 2022 at 06:40:23PM +1300, Thomas Munro wrote: > Pushed and back-patched (it's slightly different before 12). Thanks! Thank you! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Optimize external TOAST storage

2022-03-16 Thread Nathan Bossart
choose the extra disk space. These are admittedly extreme (and maybe even impossible) examples, but my point is that the amount of disk space you are willing to give up may be related to the size of the attribute. And maybe one way to extract additional read performance with this optimization is to use a variable threshold so that we are more likely to use it for large attributes. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-16 Thread Nathan Bossart
On Wed, Mar 16, 2022 at 03:02:41PM +0530, Bharath Rupireddy wrote: > On Mon, Mar 14, 2022 at 10:34 PM Nathan Bossart > wrote: >> I'm -1 on splitting these new statistics to separate LOGs. In addition to >> making it more difficult to discover statistics for a given ch

Re: Optimize external TOAST storage

2022-03-16 Thread Nathan Bossart
On Wed, Mar 16, 2022 at 11:36:56AM -0700, Nathan Bossart wrote: > Thinking further, is simply reducing the number of TOAST chunks the right > thing to look at? If I want to add a TOAST attribute that requires 100,000 > chunks, and you told me that I could save 10% in the read path for

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-16 Thread Nathan Bossart
whether Sawada-san has any comments on the revised approach. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Optimize external TOAST storage

2022-03-17 Thread Nathan Bossart
ed about the proposed configurability changes (assuming my proposal makes some amount of sense to you and others). -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-17 Thread Nathan Bossart
versions and maintainability problems. Having said that, I will leave > it to the committer to decide on that. I don't think v4-0003/option 1 needs to be unreadable. Perhaps we can use a StringInfo to build the message. That might be a net improvement in readability anyway. -- Nathan Bossart

Re: O(n) tasks cause lengthy startups and checkpoints

2022-03-17 Thread Nathan Bossart
It seems unlikely that anything discussed in this thread will be committed for v15, so I've adjusted the commitfest entry to v16 and moved it to the next commitfest. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Pre-allocating WAL files

2022-03-17 Thread Nathan Bossart
It seems unlikely that this will be committed for v15, so I've adjusted the commitfest entry to v16 and moved it to the next commitfest. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: support for CREATE MODULE

2022-03-17 Thread Nathan Bossart
It seems unlikely that this will be committed for v15. Swaha, should the commitfest entry be adjusted to v16 and moved to the next commitfest? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Patch to avoid orphaned dependencies

2022-03-17 Thread Nathan Bossart
Bertand, do you think this has any chance of making it into v15? If not, are you alright with adjusting the commitfest entry to v16 and moving it to the next commitfest? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: support for CREATE MODULE

2022-03-17 Thread Nathan Bossart
On Thu, Mar 17, 2022 at 04:26:31PM -0700, Swaha Miller wrote: > On Thu, Mar 17, 2022 at 4:16 PM Nathan Bossart > wrote: > >> It seems unlikely that this will be committed for v15. Swaha, should the >> commitfest entry be adjusted to v16 and moved to the next commitfes

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2022-03-17 Thread Nathan Bossart
I think this one requires some more work, and it needn't be a priority for v15, so I've adjusted the commitfest entry to v16 and moved it to the next commitfest. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-18 Thread Nathan Bossart
ogmsg, "%d WAL file(s) added, %d removed, %d recycled; ", CheckpointStats.ckpt_segs_added, CheckpointStats.ckpt_segs_removed, CheckpointStats.ckpt_segs

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-21 Thread Nathan Bossart
On Mon, Mar 21, 2022 at 11:27:15AM +0530, Bharath Rupireddy wrote: > On Sat, Mar 19, 2022 at 3:16 AM Nathan Bossart > wrote: >> /* buffer stats */ >> appendStringInfo(&logmsg, &

Re: Estimating HugePages Requirements?

2022-03-21 Thread Nathan Bossart
On Tue, Mar 15, 2022 at 03:44:39PM -0700, Nathan Bossart wrote: > A simple approach could be to just set log_min_messages to PANIC before > exiting. I've attached a patch for this. With this patch, we'll still see > a FATAL if we try to use 'postgres -C' for a runtim

Re: pgcrypto: Remove internal padding implementation

2022-03-21 Thread Nathan Bossart
ch and found nothing of concern. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Optimize external TOAST storage

2022-03-22 Thread Nathan Bossart
is needed before > changing anything in this area You've certainly got me thinking more about this. If the scope of this work is going to expand, a few months before the first v16 commitfest is probably the right time for it. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2022-03-22 Thread Nathan Bossart
On Thu, Mar 17, 2022 at 04:45:28PM -0700, Nathan Bossart wrote: > I think this one requires some more work, and it needn't be a priority for > v15, so I've adjusted the commitfest entry to v16 and moved it to the next > commitfest. Here is a new patch. The main differe

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2022-03-22 Thread Nathan Bossart
basic idea. Ah, my bad. I should've made sure the context was carried over better. I updated the commit message with some basic information about the intent. Please let me know if there is anything else that needs to be cleared up. -- Nathan Bossart Amazon Web Services:

Re: Use durable_unlink for .ready and .done files for WAL segment removal

2018-11-02 Thread Nathan Bossart
One argument for instead checking WAL file existence before calling archive_command might be to avoid the increased startup time. Granted, any added delay from this patch is unlikely to be noticeable unless your archiver is way behind and archive_status has a huge number of files. However, I have

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-10 Thread Nathan Bossart
KxHAC6srOfBH3M8iXUM%3DewqHV6n%3Dw1u8Q%40mail.gmail.com [1] https://postgr.es/m/VI1PR05MB62031A41186ACC3FC91ACFC70%40VI1PR05MB6206.eurprd05.prod.outlook.com [2] https://postgr.es/m/dd0e070809430a31f7ddd8483fbcce59%40mail.gmail.com -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: introduce dynamic shared memory registry

2024-01-10 Thread Nathan Bossart
On Thu, Jan 11, 2024 at 09:50:19AM +0700, Andrei Lepikhov wrote: > On 9/1/2024 00:16, Nathan Bossart wrote: >> On Mon, Jan 08, 2024 at 10:53:17AM +0530, Bharath Rupireddy wrote: >> > 2. FWIW, I'd like to call this whole feature "Support for named DSM >> > segme

Re: recovery modules

2024-01-11 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 6cee7d220b886e9be309929da5274c5770e59845 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v17 1/5] introduce routine for checking mutually exclusive string G

reorganize "Shared Memory and LWLocks" section of docs

2024-01-11 Thread Nathan Bossart
[0] https://postgr.es/m/20231205034647.GA2705267%40nathanxps13 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From dc5c1b37ced883021f4a92a17aa50b9d80d73fc6 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Thu, 11 Jan 2024 21:55:25 -0600 Subject: [PATCH v1 1/1] reorganize

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-12 Thread Nathan Bossart
one process allocates a new tranche_id > (LWLockNewTrancheId) and initializes each new LWLock > (LWLockInitialize). > """ > > Personally I think that reminding the corresponding function name here > is redundant and complicates reading just a bit. But maybe it's just > me. Yeah, I waffled on this one. I don't mind removing it. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: introduce dynamic shared memory registry

2024-01-12 Thread Nathan Bossart
it here. [0] https://postgr.es/m/20240112041430.GA3557928%40nathanxps13 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 7cf22727a96757bf212ec106bd471bf55a6981b9 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Thu, 11 Jan 2024 21:55:25 -0600 Subject: [PATCH v6 1/3] reorg

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 09:46:50AM -0600, Nathan Bossart wrote: > On Fri, Jan 12, 2024 at 05:12:28PM +0300, Aleksander Alekseev wrote: >> """ >> Any registered shmem_startup_hook will be executed shortly after each >> backend attaches to shared memory. >&g

Re: introduce dynamic shared memory registry

2024-01-12 Thread Nathan Bossart
reserve LWLocks on server startup. Like with shared >> memory, > > (Would "As with shared memory" read better? Maybe, but then again maybe > it should be left alone because you also write "Unlike with" elsewhere.) I think "As with shared memory...&q

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
that much benefit. However, we > might need to adjust the logic to make the size be 1000 divided > by the number of parallel restore jobs allowed. I wonder if it'd be worth making this configurable for pg_upgrade as an escape hatch in case the default setting is problematic. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
of setting both? I think it'd be unfortunate if pg_upgrade completely missed out on this improvement. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: pg_upgrade failing for 200+ million Large Objects

2024-01-12 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 04:42:27PM -0600, Nathan Bossart wrote: > On Wed, Dec 20, 2023 at 06:47:44PM -0500, Tom Lane wrote: >> +char *cmdEnd = psprintf(" OWNER TO %s", >> fmtId(te->owner)); >> + >> +IssueCo

Re: Fix minor memory leak in connection string validation

2024-01-12 Thread Nathan Bossart
errdetail("Non-superusers must provide > a password in the connection string."))); > + } > } > > PQconninfoFree(opts); Another option could be to surround this with PG_TRY/PG_FINALLY, but your patch seems sufficient, too. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-13 Thread Nathan Bossart
at. I've attached a v3 with a couple of fixes suggested in the other thread [0]. I'll wait a little while longer in case anyone else wants to take a look. [0] https://postgr.es/m/ZaF6UpYImGqVIhVp%40toroid.org -- Nathan Bossart Amazon Web Services

Re: introduce dynamic shared memory registry

2024-01-13 Thread Nathan Bossart
On Fri, Jan 12, 2024 at 01:45:55PM -0600, Nathan Bossart wrote: > On Fri, Jan 12, 2024 at 11:13:46PM +0530, Abhijit Menon-Sen wrote: >> At 2024-01-12 11:21:52 -0600, nathandboss...@gmail.com wrote: >>> + Each backend sould obtain a pointer to the reserved shared memo

Re: Postgres and --config-file option

2024-01-13 Thread Nathan Bossart
returned by getopt(), and I'm wondering why this doesn't use getopt_long() instead. AFAICT this dates back to the introduction of GUCs in 6a68f426 (May 2000). [0] https://www.postgresql.org/docs/devel/app-postgres.html -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: ALTER ROLE documentation improvement

2024-01-14 Thread Nathan Bossart
On Sun, Jan 14, 2024 at 04:17:41PM +0530, vignesh C wrote: > The attached v3 version patch has the changes for the same. LGTM. I'll wait a little while longer for additional feedback, but if none materializes, I'll commit this soon. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: archive modules loose ends

2024-01-15 Thread Nathan Bossart
for-committer, and I'm hoping to commit it in the near future. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: make pg_ctl more friendly

2024-01-15 Thread Nathan Bossart
the intent here, it might be worth fleshing out the messages a bit more. For example, instead of "PITR shutdown," perhaps we could say "shut down while in recovery." And maybe we should point to the specific settings in the latter message. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-16 Thread Nathan Bossart
s mistake is nearly old enough to vote (e0dece1, 5f78aa5). Good catch. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-16 Thread Nathan Bossart
On Tue, Jan 16, 2024 at 08:20:19AM -0600, Nathan Bossart wrote: > On Tue, Jan 16, 2024 at 10:02:15AM +0530, Bharath Rupireddy wrote: >> The v3 patch looks good to me except for a nitpick: the input >> parameter for RequestAddinShmemSpace is 'Size' not 'int' >&

Re: introduce dynamic shared memory registry

2024-01-16 Thread Nathan Bossart
I considered making it optional (i.e., you could pass in NULL if you didn't want it), but I didn't feel like the extra code in GetNamedDSMSegment() to allow this was worth it so that callers could avoid creating a single bool. -- Nathan Bossart Amazon Web Services: https://aws.amazon.c

Re: ALTER ROLE documentation improvement

2024-01-18 Thread Nathan Bossart
On Thu, Jan 18, 2024 at 02:44:35PM -0700, David G. Johnston wrote: > LGTM too. I didn't go looking for anything else related to this, but the > proposed changes all look needed. Committed. Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: reorganize "Shared Memory and LWLocks" section of docs

2024-01-19 Thread Nathan Bossart
On Wed, Jan 17, 2024 at 06:48:37AM +0530, Bharath Rupireddy wrote: > LGTM. Committed. Thanks for reviewing! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: introduce dynamic shared memory registry

2024-01-19 Thread Nathan Bossart
On Wed, Jan 17, 2024 at 08:00:00AM +0530, Bharath Rupireddy wrote: > The v8 patches look good to me. Committed. Thanks everyone for reviewing! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

cleanup patches for dshash

2024-01-19 Thread Nathan Bossart
ash(), and it uses them for the DSM registry's dshash table. This allows us to remove the hacky key padding code for lookups on this table. Thoughts? -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 5fe6d05f2cfa1401c2fb967fe4bb52cae3706228 Mon Sep 17 00:00:00

Re: introduce dynamic shared memory registry

2024-01-21 Thread Nathan Bossart
some way other than SIGSEGV. Oops. I've attached an attempt at fixing this. I took the opportunity to clean up the surrounding code a bit. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From f4c1c7a7ce8bccf7251e384f895f34beb33f839e Mon Sep 17 00:00:00 2001 From: Na

Re: cleanup patches for dshash

2024-01-21 Thread Nathan Bossart
[0] https://api.cirrus-ci.com/v1/artifact/task/5124848070950912/log/src/test/modules/test_dsm_registry/log/postmaster.log -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: cleanup patches for dshash

2024-01-21 Thread Nathan Bossart
On Sun, Jan 21, 2024 at 09:51:18PM -0600, Nathan Bossart wrote: > I did notice a cfbot failure [0]. After a quick glance, I'm assuming this > is caused by the memcpy() in insert_into_bucket(). Even if the string is > short, it will copy the maximum key size, which is bad. So, 000

Re: core dumps in auto_prewarm, tests succeed

2024-01-22 Thread Nathan Bossart
On Mon, Jan 22, 2024 at 12:41:17PM -0800, Andres Freund wrote: > I noticed that I was getting core dumps while executing the tests, without the > tests failing. Backtraces are vriations of this: Looking, thanks for the heads-up. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: core dumps in auto_prewarm, tests succeed

2024-01-22 Thread Nathan Bossart
On Mon, Jan 22, 2024 at 02:44:57PM -0600, Nathan Bossart wrote: > On Mon, Jan 22, 2024 at 12:41:17PM -0800, Andres Freund wrote: >> I noticed that I was getting core dumps while executing the tests, without >> the >> tests failing. Backtraces are vriations of this: > &

Re: core dumps in auto_prewarm, tests succeed

2024-01-22 Thread Nathan Bossart
On Mon, Jan 22, 2024 at 01:24:54PM -0800, Andres Freund wrote: > On 2024-01-22 15:19:36 -0600, Nathan Bossart wrote: >> I think this is because the autoprewarm state was moved to a DSM segment, >> and DSM segments are detached before the on_shmem_exit callbacks are called >>

  1   2   3   4   5   6   7   8   9   10   >