Re: please update ps display for recovery checkpoint

2020-12-03 Thread Bossart, Nathan
On 12/3/20, 1:58 PM, "Justin Pryzby" wrote: > On Thu, Dec 03, 2020 at 09:18:07PM +0000, Bossart, Nathan wrote: >> I considered also checking that update_process_title was enabled, but >> I figured that these ps display updates should happen sparsely enough >> t

Re: A few new options for CHECKPOINT

2020-12-05 Thread Bossart, Nathan
On 12/5/20, 6:41 AM, "Stephen Frost" wrote: > Assuming we actually want to do this, which I still generally don't > agree with since it isn't really clear if it'll actually end up doing > something, or not, wouldn't it make more sense to have a command that > just sits and waits for the currently

Re: Change default of checkpoint_completion_target

2020-12-08 Thread Bossart, Nathan
On 12/7/20, 9:53 AM, "Stephen Frost" wrote: > Concretely, attached is a patch which changes the default and updates > the documentation accordingly. +1 to setting checkpoint_completion_target to 0.9 by default. Nathan

pg_waldump error message fix

2020-12-10 Thread Bossart, Nathan
Hi, I noticed that when pg_waldump finds an invalid record, the corresponding error message seems to point to the last valid record read. rmgr: ... lsn: 0/090E5AF8, prev 0/090E59D0, ... pg_waldump: fatal: error in WAL record at 0/90E5AF8: invalid record length at 0/90E5B30: wanted 24, go

Re: pg_waldump error message fix

2020-12-11 Thread Bossart, Nathan
On 12/10/20, 9:23 PM, "Michael Paquier" wrote: > Please note that this is documented in xlogreader.h. Hmm. I can see > your point here, still I think that what both of you are suggesting > is not completely correct. For example, switching to EndRecPtr would > cause DecodeXLogRecord() to report

Re: please update ps display for recovery checkpoint

2020-12-11 Thread Bossart, Nathan
On 12/11/20, 4:00 PM, "Michael Paquier" wrote: > On Fri, Dec 11, 2020 at 06:54:42PM +0000, Bossart, Nathan wrote: >> This approach seems reasonable to me. I've attached my take on it. > > + /* Reset the process title */ > + set_ps_display("

Re: pg_waldump error message fix

2020-12-14 Thread Bossart, Nathan
On 12/13/20, 7:01 PM, "Michael Paquier" wrote: > On Mon, Dec 14, 2020 at 10:26:01AM +0900, Kyotaro Horiguchi wrote: >> Yeah, I had the same feeling. At least, the two LSNs in the message >> under discussion are simply redundant. So +1 to just remove the LSN at >> the caller site. > > That would me

Re: A few new options for CHECKPOINT

2020-12-14 Thread Bossart, Nathan
On 12/5/20, 9:11 AM, "Stephen Frost" wrote: > * Bossart, Nathan (bossa...@amazon.com) wrote: >> On 12/5/20, 6:41 AM, "Stephen Frost" wrote: >> > Assuming we actually want to do this, which I still generally don't >> > agree with since

Re: archive status ".ready" files may be created too early

2020-12-17 Thread Bossart, Nathan
On 12/15/20, 2:33 AM, "Kyotaro Horiguchi" wrote: > You're right in that regard. There's a window where partial record is > written when write location passes F0 after insertion location passes > F1. However, remembering all spanning records seems overkilling to me. I'm curious why you feel that r

Re: Invalidate acl.c caches for pg_authid.rolinherit changes

2020-12-21 Thread Bossart, Nathan
On 12/21/20, 1:51 AM, "Noah Misch" wrote: > Backends reflect "GRANT role_name" changes rather quickly, due to a syscache > invalidation callback. Let's register an additional callback to reflect > "ALTER ROLE ... [NO]INHERIT" with equal speed. I propose to back-patch this. > While pg_authid chan

Re: partial heap only tuples

2021-02-15 Thread Bossart, Nathan
On 2/13/21, 8:26 AM, "Andres Freund" wrote: > On 2021-02-09 18:48:21 +0000, Bossart, Nathan wrote: >> In order to be eligible for cleanup, the final tuple in the >> corresponding PHOT/HOT chain must also be eligible for cleanup, or all >> indexes must have been up

documentation fix for SET ROLE

2021-02-17 Thread Bossart, Nathan
Hi hackers, I noticed some interesting role behavior that seems to be either a bug or a miss in the documentation. The documentation for SET ROLE claims that RESET ROLE resets "the current user identifier to be the current session user identifier" [0], but this doesn't seem to hold true when "rol

Re: documentation fix for SET ROLE

2021-02-17 Thread Bossart, Nathan
On 2/17/21, 12:15 PM, "Joe Conway" wrote: > On 2/17/21 2:12 PM, David G. Johnston wrote: >> On Wednesday, February 17, 2021, Bossart, Nathan > <mailto:bossa...@amazon.com>> wrote: >> >> >> postgres=# ALTER ROLE test1 SET ROLE test2;

Re: partial heap only tuples

2021-02-23 Thread Bossart, Nathan
On 2/10/21, 2:43 PM, "Bruce Momjian" wrote: > I wonder if you should create a Postgres wiki page to document all of > this. I agree PG 15 makes sense. I would like to help with this if I > can. I will need to study this email more later. I've started the wiki page for this: https://wiki.p

Re: partial heap only tuples

2021-03-08 Thread Bossart, Nathan
On 3/8/21, 10:16 AM, "Ibrar Ahmed" wrote: > On Wed, Feb 24, 2021 at 3:22 AM Bossart, Nathan wrote: >> On 2/10/21, 2:43 PM, "Bruce Momjian" wrote: >>> I wonder if you should create a Postgres wiki page to document all of >>> this. I agree PG 15 ma

Re: partial heap only tuples

2021-03-09 Thread Bossart, Nathan
On 3/9/21, 8:24 AM, "Bruce Momjian" wrote: > On Mon, Feb 15, 2021 at 08:19:40PM +0000, Bossart, Nathan wrote: >> Yeah, this is something I'm concerned about. I think adding a bitmap >> of modified columns to the header of PHOT-updated tuples improves >> matte

Re: documentation fix for SET ROLE

2021-03-12 Thread Bossart, Nathan
On 3/12/21, 6:35 AM, "Laurenz Albe" wrote: > On Fri, 2021-03-12 at 10:16 +0100, I wrote: >> After sleeping on it, I have come to think that it is excessive to write >> so much documentation for a feature that is that unimportant. >> >> It takes some effort to come up with a good use case for it. >

Re: documentation fix for SET ROLE

2021-03-12 Thread Bossart, Nathan
On 3/12/21, 11:14 AM, "Joe Conway" wrote: > On 3/12/21 1:16 PM, Bossart, Nathan wrote: >> My main goal of this thread is to get the RESET ROLE documentation >> fixed. I don't have a terribly strong opinion on documenting these >> special uses of "ro

Re: documentation fix for SET ROLE

2021-03-15 Thread Bossart, Nathan
On 3/15/21, 7:06 AM, "Laurenz Albe" wrote: > On Fri, 2021-03-12 at 21:41 +0000, Bossart, Nathan wrote: >> On 3/12/21, 11:14 AM, "Joe Conway" wrote: >> > Looking back at the commit history it seems to me that this only works >> > accidentally. Pe

Re: A few new options for CHECKPOINT

2020-11-24 Thread Bossart, Nathan
On 11/24/20, 4:03 PM, "tsunakawa.ta...@fujitsu.com" wrote: > From: Bossart, Nathan >> The main purpose of this patch is to give users more control over their >> manually >> requested checkpoints or restartpoints. I suspect the most useful option is >

Re: A few new options for CHECKPOINT

2020-11-27 Thread Bossart, Nathan
On 11/27/20, 8:29 AM, "Stephen Frost" wrote: > Also note that, in all other cases (that is, when there *is* outstanding > WAL since the last checkpoint), pg_start_backup actually just waits for > the existing checkpoint to complete- and while it's waiting for that to > happen, there'll be addition

Re: A few new options for CHECKPOINT

2020-11-27 Thread Bossart, Nathan
On 11/27/20, 10:58 AM, "Stephen Frost" wrote: > If you'd like to show that I'm wrong, and it's entirely possible that I > am, then retry the above with actual load on the system, and also > actually look at how much outstanding WAL you end up with given the > different scenarios which has to be re

Re: A few new options for CHECKPOINT

2020-11-28 Thread Bossart, Nathan
On 11/28/20, 9:50 AM, "Stephen Frost" wrote: >> Granted, I used a rather aggressive checkpoint_timeout, but I think >> this demonstrates that waiting for a non-immediate checkpoint to >> complete can lower the amount of WAL needed for recovery, even though >> it might not lower it as much as waiti

Re: A few new options for CHECKPOINT

2020-11-30 Thread Bossart, Nathan
On 11/29/20, 7:21 PM, "Stephen Frost" wrote: > Checkpoints are always happening though, that's kind of my point..? > Sure, you get lucky sometimes that the time you snapshot might have less > outstanding WAL than at some other time, but I'm not convinced that this > patch is really going to give a

Re: archive status ".ready" files may be created too early

2021-01-26 Thread Bossart, Nathan
On 12/17/20, 9:15 PM, "Kyotaro Horiguchi" wrote: > At Thu, 17 Dec 2020 22:20:35 +0000, "Bossart, Nathan" > wrote in >> On 12/15/20, 2:33 AM, "Kyotaro Horiguchi" wrote: >> > You're right in that regard. There's a window where parti

Re: archive status ".ready" files may be created too early

2021-01-26 Thread Bossart, Nathan
On 1/2/21, 8:55 AM, "Andrey Borodin" wrote: > Recently we had somewhat related incident. Do I understand correctly that > this incident is related to the bug discussed in this thread? I'm not sure that we can rule it out, but the log pattern I've typically seen for this is "invalid contrecord le

Re: archive status ".ready" files may be created too early

2021-01-27 Thread Bossart, Nathan
On 1/26/21, 6:36 PM, "Kyotaro Horiguchi" wrote: > At Tue, 26 Jan 2021 19:13:57 +0000, "Bossart, Nathan" > wrote in >> On 12/17/20, 9:15 PM, "Kyotaro Horiguchi" wrote: >> > At Thu, 17 Dec 2020 22:20:35 +, "Bossart, Nathan"

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2021-01-28 Thread Bossart, Nathan
On 1/27/21, 5:08 PM, "Justin Pryzby" wrote: > Thanks, I wrote my message after running into the issue and remembered this > thread. I didn't necessarily mean to send another patch :) No worries. I lost track of this thread, but I don't mind picking it up again. > My only comment is on the name

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2021-02-09 Thread Bossart, Nathan
On 2/8/21, 9:19 PM, "Michael Paquier" wrote: > On Mon, Feb 08, 2021 at 06:59:45PM +0000, Bossart, Nathan wrote: >> These suggestions seem reasonable to me. I've applied them in v9. > > Sounds good to me, so applied. Thanks! Nathan

Re: partial heap only tuples

2021-02-10 Thread Bossart, Nathan
On 2/10/21, 2:43 PM, "Bruce Momjian" wrote: > On Tue, Feb 9, 2021 at 06:48:21PM +0000, Bossart, Nathan wrote: >> HOT works wonders when no indexed columns are updated. However, as >> soon as you touch one indexed column, you lose that optimization >> entirely, as

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 6:06 PM, "Euler Taveira" wrote: > Saying that a certain task is O(n) doesn't mean it needs a separate process to > handle it. Did you have a use case or even better numbers (% of checkpoint / > startup time) that makes your proposal worthwhile? I don't have specific numbers on hand, b

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 6:48 PM, "Bharath Rupireddy" wrote: > +1 for the overall idea of making the checkpoint faster. In fact, we > here at our team have been thinking about this problem for a while. If > there are a lot of files that checkpoint has to loop over and remove, > IMO, that task can be delegated

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread Bossart, Nathan
On 12/2/21, 9:50 AM, "David Steele" wrote: > On 12/2/21 12:38, David Steele wrote: >> On 12/2/21 11:00, Andrew Dunstan wrote: >>> >>> Should we really be getting rid of >>> PostgreSQL::Test::Cluster::backup_fs_hot() ? >> >> Agreed, it would be better to update backup_fs_hot() to use exclusive >> m

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-12-02 Thread Bossart, Nathan
On 10/29/21, 10:49 AM, "Bossart, Nathan" wrote: > On 10/29/21, 3:54 AM, "Greg Nancarrow" wrote: >> When recently debugging a vacuum-related issue, I noticed that there >> is some log-report processing/formatting code at the end of >> lazy_scan_heap()

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-02 Thread Bossart, Nathan
On 12/1/21, 5:59 AM, "Bharath Rupireddy" wrote: > On Wed, Dec 1, 2021 at 6:45 PM Tom Lane wrote: >> Considering the vanishingly small number of actual complaints we've >> seen about this, that sounds ridiculously over-engineered. >> A documentation example should be sufficient. > > Thanks. Here'

Re: Temporary tables versus wraparound... again

2021-12-02 Thread Bossart, Nathan
On 10/12/21, 3:07 PM, "Greg Stark" wrote: > Here's an updated patch. I added some warning messages to autovacuum. I think this is useful optimization, and I intend to review the patch more closely soon. It looks reasonable to me after a quick glance. > One thing I learned trying to debug this s

Re: Alter all tables in schema owner fix

2021-12-02 Thread Bossart, Nathan
On 12/2/21, 7:07 PM, "vignesh C" wrote: > Currently while changing the owner of ALL TABLES IN SCHEMA > publication, it is not checked if the new owner has superuser > permission or not. Added a check to throw an error if the new owner > does not have superuser permission. > Attached patch has the

Re: Alter all tables in schema owner fix

2021-12-03 Thread Bossart, Nathan
On 12/2/21, 11:57 PM, "tanghy.f...@fujitsu.com" wrote: > Thanks for your patch. > I tested it and it fixed this problem as expected. It also passed "make > check-world". +1, the patch looks good to me, too. My only other suggestion would be to move IsSchemaPublication() to pg_publication.c Nat

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 7:40 AM, "Peter Geoghegan" wrote: > If my patch to unite vacuum verbose and the autovacuum logging gets > in, then this issue also goes away. Perhaps this patch should be marked Rejected in favor of that one, then. Nathan

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 6:21 AM, "Bharath Rupireddy" wrote: > +1 to add here in the "Parameter Names and Values section", but do we > want to backlink every string parameter to this section? I think it > needs more effort. IMO, we can just backlink for > shared_preload_libraries alone. Thoughts? IMO this is

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-03 Thread Bossart, Nathan
On 12/3/21, 5:57 AM, "Bharath Rupireddy" wrote: > On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan wrote: >> >> On 12/1/21, 6:48 PM, "Bharath Rupireddy" >> wrote: >> > +1 for the overall idea of making the checkpoint faster. In fact, we &

Re: pg_replslotdata - a tool for displaying replication slot information

2021-12-06 Thread Bossart, Nathan
On 12/5/21, 11:10 PM, "Michael Paquier" wrote: > On Thu, Dec 02, 2021 at 08:32:08AM +0530, Bharath Rupireddy wrote: >> On Thu, Dec 2, 2021 at 4:22 AM Andres Freund wrote: I don't have any other compelling use- cases at the moment, but I will say that it is typically nice from an adminis

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 3:44 AM, "Bharath Rupireddy" wrote: > On Fri, Dec 3, 2021 at 11:50 PM Bossart, Nathan wrote: >> I might hack something together for the separate worker approach, if >> for no other reason than to make sure I really understand how these >> functio

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 4:34 AM, "Bharath Rupireddy" wrote: > While the database is performing end-of-recovery checkpoint, the > control file gets updated with db state as "shutting down" in > CreateCheckPoint (see the code snippet at [1]) and at the end it sets > it back to "shut down" for a brief moment an

Re: Do we need pre-allocate WAL files during end-of-recovery checkpoint?

2021-12-06 Thread Bossart, Nathan
On 12/6/21, 4:54 AM, "Bharath Rupireddy" wrote: > The function PreallocXlogFiles doesn't get called during > end-of-recovery checkpoint in CreateCheckPoint, see [1]. The server > becomes operational after the end-of-recovery checkpoint and may need > WAL files. However, I'm not sure how beneficia

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-07 Thread Bossart, Nathan
On 12/6/21, 8:19 PM, "Dilip Kumar" wrote: > If the subtransaction cache is overflowed in some of the transactions > then it will affect all the concurrent queries as they need to access > the SLRU for checking the visibility of each tuple. But currently > there is no way to identify whether in an

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:29 AM, "Bharath Rupireddy" wrote: > Why can't the walwriter pre-allocate some of the WAL segments instead > of a new background process? Of course, it might delay the main > functionality of the walwriter i.e. flush and sync the WAL files, but > having checkpointer do the pre-alloc

Re: Alter all tables in schema owner fix

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 2:47 AM, "Greg Nancarrow" wrote: > On Tue, Dec 7, 2021 at 9:01 PM Amit Kapila wrote: >> >> Thanks, the patch looks mostly good to me. I have slightly modified it >> to incorporate one of Michael's suggestions, ran pgindent, and >> modified the commit message. >> > > LGTM, except in th

Re: Pre-allocating WAL files

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 9:35 AM, "Bossart, Nathan" wrote: > On 12/7/21, 12:29 AM, "Bharath Rupireddy" > wrote: >> Why can't the walwriter pre-allocate some of the WAL segments instead >> of a new background process? Of course, it might delay the main >> f

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 12:10 AM, "Bharath Rupireddy" wrote: > Here's a patch that I've come up with. Please see if this looks okay > and let me know if we want to take it forward so that I can add a CF > entry. Overall, the patch seems reasonable to me. + case DB_IN_END_OF_RECOVERY_CHECKPOIN

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 4:17 AM Bossart, Nathan wrote: >> I agree with Tom. I would just s/server/backend/ (as per the >> attached) and call it a day. > > Thanks. v5 patch looks good to me. I've mark

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 5:21 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 2:50 AM Bossart, Nathan wrote: >> I noticed that some (but not all) of the surrounding messages say >> "last known up at" the control file time. I'm curious why you chose >>

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-07 Thread Bossart, Nathan
On 12/7/21, 8:42 PM, "Bharath Rupireddy" wrote: > On Wed, Dec 8, 2021 at 9:49 AM Bossart, Nathan wrote: >> I think that's alright. The only other small suggestion I have would >> be to say "during end-of-recovery checkpoint" instead of "while in

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2021-12-08 Thread Bossart, Nathan
On 12/8/21, 3:29 AM, "Bharath Rupireddy" wrote: > Thanks for your thoughts. I'm fine either way, hence attaching two > patches here with and I will leave it for the committer 's choice. > 1) v1-0001-Add-DB_IN_END_OF_RECOVERY_CHECKPOINT-state-for-co.patch -- > adds new db state DB_IN_END_OF_RECOVE

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 11:34 AM, "John Naylor" wrote: > When a user must shut down and restart in single-user mode to run > vacuum on an entire database, that does a lot of work that's > unnecessary for getting the system online again, even without > index_cleanup. We had a recent case where a single-user va

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 12:33 PM, "Bossart, Nathan" wrote: > On 12/9/21, 11:34 AM, "John Naylor" wrote: >> Now that we have a concept of a fail-safe vacuum, maybe it would be >> beneficial to skip a vacuum in single-user mode if the fail-safe >> criteria were not met

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 4:36 PM, "Peter Geoghegan" wrote: > We could then apply this criteria in new code that implements this > "big red button" (maybe this is a new option for the postgres > executable, a little like --single?). Something that's reasonably > targeted, and dead simple to use. +1 Nathan

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 5:06 PM, "Bossart, Nathan" wrote: > On 12/9/21, 4:36 PM, "Peter Geoghegan" wrote: >> We could then apply this criteria in new code that implements this >> "big red button" (maybe this is a new option for the postgres >> exe

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Bossart, Nathan
On 12/9/21, 5:27 PM, "Peter Geoghegan" wrote: > I imagine that this new function (to handle maintenance tasks in the > event of a wraparound emergency) would output information about its > progress. For example, it would make an up-front decision about which > tables needed to be vacuumed in order

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 5:54 AM, "Robert Haas" wrote: > I don't know whether this kind of idea is good or not. Thanks for chiming in. I have an almost-complete patch set that I'm hoping to post to the lists in the next couple of days. > One thing we've seen a number of times now is that entrusting the sam

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 9:20 AM, "Justin Pryzby" wrote: > On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote: >> Another issue is that we don't want to increase the number of >> processes without bound. Processes use memory and CPU resources and if >> we run too many of them it becomes a burden on t

Re: Add sub-transaction overflow status in pg_stat_activity

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 6:30 AM, "Dilip Kumar" wrote: > On Tue, Dec 7, 2021 at 11:11 AM Justin Pryzby wrote: >> Since I think this field is usually not interesting to most users of >> pg_stat_activity, maybe this should instead be implemented as a function like >> pg_backend_get_subxact_status(pid). >> >> P

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-13 Thread Bossart, Nathan
On 12/13/21, 12:37 PM, "Robert Haas" wrote: > On Mon, Dec 13, 2021 at 1:21 PM Bossart, Nathan wrote: >> > But against all that, if these tasks are slowing down checkpoints and >> > that's avoidable, that seems pretty important too. Interestingly, I >> &g

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-14 Thread Bossart, Nathan
On 12/14/21, 9:00 AM, "Bruce Momjian" wrote: > Have we changed temporary file handling in any recent major releases, > meaning is this a current problem or one already improved in PG 14. I haven't noticed any recent improvements while working in this area. Nathan

Re: O(n) tasks cause lengthy startups and checkpoints

2021-12-14 Thread Bossart, Nathan
On 12/14/21, 12:09 PM, "Bossart, Nathan" wrote: > On 12/14/21, 9:00 AM, "Bruce Momjian" wrote: >> Have we changed temporary file handling in any recent major releases, >> meaning is this a current problem or one already improved in PG 14. > > I have

Re: archive modules

2021-12-15 Thread Bossart, Nathan
On 11/2/21, 8:07 AM, "Bossart, Nathan" wrote: > The main motivation is provide a way to archive without shelling out. > This reduces the amount of overhead, which can improve archival rate > significantly. It should also make it easier to archive more safely. > For examp

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-15 Thread Bossart, Nathan
On 12/1/21, 3:02 PM, "Imseih (AWS), Sami" wrote: > The current implementation of pg_stat_progress_vacuum does not > provide progress on which index is being vacuumed making it > difficult for a user to determine if the "vacuuming indexes" phase > is making progress. By exposing which index is bein

Re: Strange path from pgarch_readyXlog()

2021-12-29 Thread Bossart, Nathan
On 12/29/21, 12:22 PM, "Thomas Munro" wrote: > Isn't this a corrupted pathname? > > 2021-12-29 03:39:55.708 CST [79851:1] WARNING: removal of orphan > archive status file > "pg_wal/archive_status/00010003.0028.backup00010004.ready" > failed too many times, will

Re: Strange path from pgarch_readyXlog()

2021-12-30 Thread Bossart, Nathan
On 12/29/21, 3:11 PM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> This crossed my mind, too. I also think one of the arrays can be >> eliminated in favor of just using the heap (after rebuilding with a >> reversed comparator). Here is a minimally-t

Re: Pre-allocating WAL files

2022-01-05 Thread Bossart, Nathan
On 12/30/21, 3:52 AM, "Maxim Orlov" wrote: > I did check the patch too and found it to be ok. Check and check-world are > passed. > Overall idea seems to be good in my opinion, but I'm not sure where is the > optimal place to put the pre-allocation. > > On Thu, Dec 30, 2021 at 2:46 PM Pavel Bor

Re: skip replication slot snapshot/map file removal during end-of-recovery checkpoint

2022-01-05 Thread Bossart, Nathan
On 12/23/21, 3:17 AM, "Bharath Rupireddy" wrote: > Currently the end-of-recovery checkpoint can be much slower, impacting > the server availability, if there are many replication slot files > .snap or map- to be enumerated and deleted. How about skipping > the .snap and map- file handling

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-06 Thread Bossart, Nathan
On 12/29/21, 8:44 AM, "Imseih (AWS), Sami" wrote: > In "pg_stat_progress_vacuum", introduce 2 columns: > > * total_index_vacuum : This is the # of indexes that will be vacuumed. Keep > in mind that if failsafe mode kicks in mid-flight to the vacuum, Postgres may > choose to forgo index scans. Th

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2022-01-06 Thread Bossart, Nathan
On 12/21/21, 11:42 AM, "Mark Dilger" wrote: > + /* pre-v9.3 lock-only bit pattern */ > + ereport(ERROR, > + (errcode(ERRCODE_DATA_CORRUPTED), > +errmsg_internal("found tuple with HEAP_XMAX_COMMITTED > and" > +

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-06 Thread Bossart, Nathan
On 12/2/21, 1:34 PM, "Bossart, Nathan" wrote: > On 12/2/21, 9:50 AM, "David Steele" wrote: >> On 12/2/21 12:38, David Steele wrote: >>> On 12/2/21 11:00, Andrew Dunstan wrote: >>>> >>>> Should we really be getting rid of >>>

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-07 Thread Bossart, Nathan
On 1/7/22, 5:52 AM, "David Steele" wrote: > On 1/6/22 20:20, Euler Taveira wrote: >> On Thu, Jan 6, 2022, at 9:48 PM, Bossart, Nathan wrote: >>> After a quick glance, I didn't see an easy way to hold a session open >>> while the test does ot

Re: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers

2022-01-07 Thread Bossart, Nathan
On 1/6/22, 11:25 PM, "Jeff Davis" wrote: > On Wed, 2022-01-05 at 23:59 -0800, SATYANARAYANA NARLAPURAM wrote: >> I would like to propose a GUC send_Wal_after_quorum_committed which >> when set to ON, walsenders corresponds to async standbys and logical >> replication workers wait until the LSN is

Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

2022-01-07 Thread Bossart, Nathan
On 11/23/21, 11:29 AM, "Thomas Munro" wrote: > Here's a patch for Linux and also FreeBSD. The latter OS decided to > turn on ASLR by default recently, causing my workstation to fail like > this quite reliably, which reminded me to follow up with this. It > disables ASLR in pg_ctl and pg_regress,

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-10 Thread Bossart, Nathan
On 1/6/22, 6:14 PM, "Imseih (AWS), Sami" wrote: > I am hesitant to make column name changes for obvious reasons, as it breaks > existing tooling. However, I think there is a really good case to change > "index_vacuum_count" as the name is confusing. > "index_vacuum_cycles_completed" is the name

Re: make MaxBackends available in _PG_init

2022-01-10 Thread Bossart, Nathan
On 1/7/22, 12:27 PM, "Robert Haas" wrote: > On Fri, Jan 7, 2022 at 1:09 PM Bossart, Nathan wrote: >> While that approach would provide a way to safely retrieve the value, >> I think it would do little to prevent the issue in practice. If the >> size of the pa

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2022-01-10 Thread Bossart, Nathan
I noticed this thread and thought I'd share my experiences building something similar for Multi-AZ DB clusters [0]. It's not a strict RPO mechanism, but it does throttle backends in an effort to keep the replay lag below a configured maximum. I can share the code if there is interest. I wrote it

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 10:06 AM, "John Naylor" wrote: > I pushed this with one small change -- I felt the comment didn't need > to explain the warning message, since it now simply matches the coding > more exactly. Also, v5 was a big enough change from v4 that I put > Nathan as the first author. Thanks! Na

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 5:01 PM, "Imseih (AWS), Sami" wrote: > I have attached the 3rd revision of the patch which also includes the > documentation changes. Also attached is a rendered html of the docs for > review. > > "max_index_vacuum_cycle_time" has been removed. > "index_rows_vacuumed" renamed to "ind

Re: improve CREATE EXTENSION error message

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 11:23 AM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> Okay, the message looks like this in v5: > >> postgres=# CREATE EXTENSION does_not_exist; >> ERROR: extension "does_not_exist" is not available >

Re: Add jsonlog log_destination for JSON server logs

2022-01-11 Thread Bossart, Nathan
On 1/10/22, 4:51 AM, "Michael Paquier" wrote: > The issue comes from an incorrect change in syslogger_parseArgs() > where I missed that the incrementation of argv by 3 has no need to be > changed. A build with -DEXEC_BACKEND is enough to show the failure, > which caused a crash when starting up t

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-11 Thread Bossart, Nathan
On 1/11/22, 12:33 PM, "Imseih (AWS), Sami" wrote: > What about something like "The number of indexes that are eligible for > vacuuming". > This covers the cases where either an individual index is skipped or the > entire "index vacuuming" phase is skipped. Hm. I don't know if "eligible" is th

Re: do only critical work during single-user vacuum?

2022-01-12 Thread Bossart, Nathan
On 1/12/22, 7:43 AM, "John Naylor" wrote: > On Wed, Jan 12, 2022 at 1:49 AM Masahiko Sawada wrote: >> As another idea, we might be able to add a new option that takes an >> optional integer value, like VACUUM (MIN_XID), VACUUM (MIN_MXID), and >> VACUUM (MIN_XID 50). We vacuum only tables whos

Re: Add index scan progress to pg_stat_progress_vacuum

2022-01-12 Thread Bossart, Nathan
On 1/11/22, 11:46 PM, "Masahiko Sawada" wrote: > Regarding the new pg_stat_progress_vacuum_index view, why do we need > to have a separate view? Users will have to check two views. If this > view is expected to be used together with and joined to > pg_stat_progress_vacuum, why don't we provide one

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-12 Thread Bossart, Nathan
On 12/31/21, 4:44 AM, "Bharath Rupireddy" wrote: > Currently the server is erroring out when unable to remove/parse a > logical rewrite file in CheckPointLogicalRewriteHeap wasting the > amount of work the checkpoint has done and preventing the checkpoint > from finishing. This is unlike CheckPoi

Re: parse/analyze API refactoring

2022-01-12 Thread Bossart, Nathan
On 12/28/21, 8:25 AM, "Peter Eisentraut" wrote: > (The "withcb" naming maybe isn't great; better ideas welcome.) FWIW I immediately understood that this meant "with callback," so it might be okay. > Not included in this patch set, but food for further thought: The > pg_analyze_and_rewrite_*()

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-01-13 Thread Bossart, Nathan
On 1/12/22, 10:03 PM, "Bharath Rupireddy" wrote: > On Thu, Jan 13, 2022 at 3:47 AM Bossart, Nathan wrote: >> The only feedback I have for the patch is that I don't think the new >> comments are necessary. > > I borrowed the comments as-is from the CheckPoint

Re: do only critical work during single-user vacuum?

2022-01-13 Thread Bossart, Nathan
On 1/13/22, 4:58 AM, "John Naylor" wrote: > On Wed, Jan 12, 2022 at 12:26 PM Bossart, Nathan wrote: >> As I've stated upthread, Sawada-san's suggested approach was my >> initial reaction to this thread. I'm not wedded to the idea of adding >>

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-13 Thread Bossart, Nathan
Thanks for the new patch! + +Returns a record of information about the backend's subtransactions. +The fields returned are subxact_count identifies +number of active subtransaction and subxact_overflow + shows whether the backend's subtransaction cache is +

Re: O(n) tasks cause lengthy startups and checkpoints

2022-01-14 Thread Bossart, Nathan
On 1/14/22, 3:43 AM, "Maxim Orlov" wrote: > The code seems to be in good condition. All the tests are running ok with no > errors. Thanks for your review. > I like the whole idea of shifting additional checkpointer jobs as much as > possible to another worker. In my view, it is more appropriat

Re: Add sub-transaction overflow status in pg_stat_activity

2022-01-14 Thread Bossart, Nathan
On 1/14/22, 8:26 AM, "Tom Lane" wrote: > Julien Rouhaud writes: >> Like many I previously had to investigate a slowdown due to sub-transaction >> overflow, and even with the information available in a monitoring view (I had >> to rely on a quick hackish extension as I couldn't patch postgres) it'

Re: Change default of checkpoint_completion_target

2021-03-23 Thread Bossart, Nathan
LGTM. I just have a few small wording suggestions. +completion overhead. Reducing this parameter is not recommended as that +causes the I/O from the checkpoint to have to complete faster, resulting +in a higher I/O rate, while then having a period of less I/O between t

Re: Change default of checkpoint_completion_target

2021-03-23 Thread Bossart, Nathan
On 3/23/21, 12:19 PM, "Stephen Frost" wrote: > * Bossart, Nathan (bossa...@amazon.com) wrote: > > LGTM. I just have a few small wording suggestions. > > Agreed, those looked like good suggestions and so I've incorporated > them. > > Updated patch attached. Looks good! Nathan

Re: documentation fix for SET ROLE

2021-04-02 Thread Bossart, Nathan
On 4/2/21, 10:54 AM, "Joe Conway" wrote: > On 4/2/21 10:21 AM, Laurenz Albe wrote: >> So I think that documenting this is the way to go. I'll mark it as >> "ready for committer". > > pushed Thanks! Nathan

Re: psql - add SHOW_ALL_RESULTS option

2021-04-12 Thread Bossart, Nathan
On 4/12/21, 9:25 AM, "Tom Lane" wrote: > Fabien COELHO writes: >>> Between this and the known breakage of control-C, it seems clear >>> to me that this patch was nowhere near ready for prime time. >>> I think shoving it in on the last day before feature freeze was >>> ill-advised, and it ought to

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2020-08-04 Thread Bossart, Nathan
On 8/2/20, 11:47 PM, "Michael Paquier" wrote: > + VACOPT_TOAST_CLEANUP = 1 << 6, /* process TOAST table, if any */ > + VACOPT_DISABLE_PAGE_SKIPPING = 1 << 7, /* don't skip any pages */ > + VACOPT_MAIN_REL_CLEANUP = 1 << 8/* process main relation */ > } VacuumOption; > > Do we actually

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2020-08-26 Thread Bossart, Nathan
On 8/26/20, 12:16 PM, "Alvaro Herrera" wrote: > On 2020-Aug-20, Jeremy Schneider wrote: >> Alternatively, if we don't want to take this approach, then I'd argue >> that we should update README.tuplock to explicitly state that >> XMAX_LOCK_ONLY and XMAX_COMMITTED are incompatible (just as it alread

  1   2   3   4   5   >