[Proposal] Add foreign-server health checks infrastructure

2021-10-29 Thread kuroda.hay...@fujitsu.com
Dear Hackers, I want to propose the feature that checks the health of foreign servers. As a first step I want to add an infrastructure for periodical checking to PostgreSQL core. Currently this is the WIP, it does not contain docs. ## Background Currently there is no way to check the status o

Re: Skipping logical replication transactions on subscriber side

2021-10-29 Thread Amit Kapila
On Fri, Oct 29, 2021 at 4:49 PM Amit Kapila wrote: > > On Fri, Oct 29, 2021 at 10:54 AM Masahiko Sawada > wrote: > > > > I'm not sure it's better to drop apply worker stats after resetting > > skip xid (i.g., after skipping the transaction). Since the view is a > > cumulative view and has last_e

Add additional information to src/test/ssl/README

2021-10-29 Thread Kevin Burke
Hi, I've been trying to run the SSL tests against my branch and that was tougher than expected because I didn't realize that additional output was being saved that I couldn't see - it wasn't even getting to the part where it could run the tests. This patch adds a note to the README explaining where

Re: Add support for ALTER INDEX .. ALTER [COLUMN] col_num {SET, RESET}

2021-10-29 Thread Michael Paquier
On Fri, Oct 29, 2021 at 06:23:34PM +, Bossart, Nathan wrote: > This is my concern as well. I remember seeing some weird errors in > the thread that led to fdd8857, and I was able to reproduce it with > this new patch. If you apply the following patch for the hstore test, > it will begin faili

Re: Add support for ALTER INDEX .. ALTER [COLUMN] col_num {SET,RESET}

2021-10-29 Thread Michael Paquier
On Fri, Oct 29, 2021 at 11:15:08PM +0300, Alexander Korotkov wrote: > All these cases require index rebuild to change the opclass option. I > can imagine there could be options, which don't require index rebuild > to change. For instance, a split algorithm for R-tree-like GiST index > could be ch

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-10-29 Thread Robert Haas
On Fri, Oct 29, 2021 at 6:56 PM Jeff Davis wrote: > On Wed, 2021-10-27 at 16:10 -0400, Robert Haas wrote: > > But the thing that defines a tenant need not be a role. It can be > > some > > other kind of object. Suppose we invent a CREATE TENANT command. > > Would it be a recursive concept? Could a

Re: Improving psql's \password command

2021-10-29 Thread Tom Lane
"Bossart, Nathan" writes: > On 10/29/21, 12:47 PM, "Tom Lane" wrote: >> While testing that, I noticed another bit of user-unfriendliness: >> there's no obvious way to get out of it if you realize you are >> setting the wrong user's password. simple_prompt() ignores >> control-C, and when you giv

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-10-29 Thread Jeff Davis
On Thu, 2021-10-28 at 11:02 -0700, Mark Dilger wrote: > It feels to me that the traditional concept of users and groups could > map, one-to-one, onto users and roles, but we've mapped both users > and groups, many-to-one, onto roles, leaving no distinct concept of > groups, and now we're proposing

Re: Feature request for adoptive indexes

2021-10-29 Thread Tomas Vondra
On 10/29/21 15:32, Hayk Manukyan wrote: Hi all First of all thank you all for fast and rich responses, that is really nice. I don't have that deep knowledge of how postgres  works under the hood so I will try to explain more user side. I want to refer for some points mentioned above.  - Firs

Re: Delegating superuser tasks to new security roles (Was: Granting control of SUSET gucs to non-superusers)

2021-10-29 Thread Jeff Davis
On Wed, 2021-10-27 at 16:10 -0400, Robert Haas wrote: > But the thing that defines a tenant need not be a role. It can be > some > other kind of object. Suppose we invent a CREATE TENANT command. Would it be a recursive concept? Could a tenant create a sub-tenant? Regards, Jeff Davis

Re: Improving psql's \password command

2021-10-29 Thread Bossart, Nathan
On 10/29/21, 12:47 PM, "Tom Lane" wrote: > While testing that, I noticed another bit of user-unfriendliness: > there's no obvious way to get out of it if you realize you are > setting the wrong user's password. simple_prompt() ignores > control-C, and when you give up and press return, you'll jus

Re: Add support for ALTER INDEX .. ALTER [COLUMN] col_num {SET,RESET}

2021-10-29 Thread Alexander Korotkov
Hi, Michael! On Fri, Oct 29, 2021 at 12:10 PM Michael Paquier wrote: > On Fri, Oct 29, 2021 at 02:09:34PM +0900, Michael Paquier wrote: > > The attached does the work, with tests and documentation added to all > > the places I could think about while reviewing the existing opclass > > code for in

Improving psql's \password command

2021-10-29 Thread Tom Lane
The attached patch responds to the discussion at [1]. That thread pointed out that if you issue "\password" with no argument, psql tries to set the password for the role identified by PQuser(conn), that is, the originally requested login name. That's problematic because if you've done SET SESSION

add checkpoint stats of snapshot and mapping files of pg_logical dir

2021-10-29 Thread Bharath Rupireddy
Hi, At times, there can be many snapshot and mapping files under pg_logical dir that the checkpoint might have to delete/fsync based on the cutoff LSN which can increase the checkpoint time. The user/admin/developer might want to get an answer for "what's the total time spent in deleting these fil

Re: when the startup process doesn't (logging startup delays)

2021-10-29 Thread Robert Haas
On Fri, Oct 29, 2021 at 9:10 AM Nitin Jadhav wrote: > > I think you're wrong. If we did that, the previous timer could fire > > right after we set startup_progress_timer_expired = false, and before > > we reschedule the timeout. It seems annoying to have to disable the > > timeout and immediately

Re: Add support for ALTER INDEX .. ALTER [COLUMN] col_num {SET, RESET}

2021-10-29 Thread Bossart, Nathan
On 10/29/21, 2:14 AM, "Michael Paquier" wrote: > By the way, while looking at this area of the code (particularly > tsvector and gist), I was under the impression that it is safe to > assume that we don't particularly need to rebuild the index when > changing its attoptions this way, but I have al

Re: ThisTimeLineID is used uninitialized in basebackup.c, too

2021-10-29 Thread Robert Haas
On Fri, Oct 29, 2021 at 7:28 AM Michael Paquier wrote: > Yes, I agree that it is a good idea to cut the dependency of those > code paths with ThisTimeLineID, expecting IDENTIFY_SYSTEM to have done > the job beforehand. One argument in favor of your change, though I'd > like to think that nobody d

Re: Better localization of errors in plpgsql variable initialization

2021-10-29 Thread Pavel Stehule
pá 29. 10. 2021 v 20:00 odesílatel Tom Lane napsal: > While preparing commit a2a731d6c, I chanced to notice what seemed > like an off-by-one error in the line number reported for an error > occurring while we initialize a plpgsql variable. For instance > > regression=# do $$ declare x int := 1/0

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2021-10-29 Thread Daniel Gustafsson
> On 29 Oct 2021, at 18:04, Tom Lane wrote: > Daniel Gustafsson writes: >> This patch still seems relevant for back-branches, but starting at 14 this >> time. > > I think the appropriate thing to do is stick your patch into all branches > for the moment. We can remove it again whenever we inv

Better localization of errors in plpgsql variable initialization

2021-10-29 Thread Tom Lane
While preparing commit a2a731d6c, I chanced to notice what seemed like an off-by-one error in the line number reported for an error occurring while we initialize a plpgsql variable. For instance regression=# do $$ declare x int := 1/0; regression$# begin end $$; ERROR: division by zero CONTEXT:

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-10-29 Thread Bossart, Nathan
On 10/29/21, 3:54 AM, "Greg Nancarrow" wrote: > When recently debugging a vacuum-related issue, I noticed that there > is some log-report processing/formatting code at the end of > lazy_scan_heap() that, unless the vacuum VERBOSE option is specified, > typically results in no log output (as the lo

Re: [PATCH] Support pg_ident mapping for LDAP

2021-10-29 Thread Jacob Champion
On Tue, 2021-09-28 at 18:15 +, Jacob Champion wrote: > | authn authz > -+--- > envvar | PGAUTHUSERPGUSER > conninfo | authuser user > frontend | conn->pgauthuser conn->pguser > backend | port->auth_user port->us

Re: inefficient loop in StandbyReleaseLockList()

2021-10-29 Thread Bossart, Nathan
On 10/28/21, 11:53 PM, "Michael Paquier" wrote: > On Thu, Oct 28, 2021 at 04:52:48PM -0700, Andres Freund wrote: >> I suspect the reverse lock order release could be tad faster. But I probably >> wouldn't change it either - I was more thinking of some of the other cases >> that deleted the first e

Re: Modifying TOAST policy will not affect the way existing data is stored ?

2021-10-29 Thread Marcos Pegoraro
Em qua., 27 de out. de 2021 às 16:48, Marcos Pegoraro escreveu: > Postgres weekly came today with this interesting article: > https://hakibenita.com/sql-medium-text-performance > > The problem it explains is when you have no one records being stored on > toast, even if they are toastable. That is

Re: enhance pg_log_backend_memory_contexts() to log memory contexts of auxiliary processes

2021-10-29 Thread Bharath Rupireddy
On Mon, Oct 11, 2021 at 9:55 AM Bharath Rupireddy wrote: > > On Mon, Oct 11, 2021 at 8:21 AM torikoshia wrote: > > > > Thanks for working on this! > > > > On 2021-10-09 22:23, Bharath Rupireddy wrote: > > > Hi, > > > > > > Currently pg_log_backend_memory_contexts() doesn't log the memory > > > co

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2021-10-29 Thread Tom Lane
Daniel Gustafsson writes: > On 13 Feb 2020, at 23:55, Tom Lane wrote: >>> Given the current behavior of SET ROLE and SET SESSION AUTHORIZATION, >>> I don't actually see any way that we could get these features to >>> play together. > The recent work on pg_dump reminded me about this thread, AFAI

Re: Vulnerability identified with Postgres 13.4 for Windows

2021-10-29 Thread David G. Johnston
On Friday, October 29, 2021, Joel Mariadasan (jomariad) wrote: > Detected by Automated Scanning tool: > > *libxml 2.9.10* > > > > Can you confirm if this is the same version of libxml used in Postgres? > > We want to confirm if the detection is a false positive or a vulnerability. > > > IIUC (t

Re: plpgsql: can I use a variable in a DECLARE later whithin the DECLARE?

2021-10-29 Thread tomas
On Fri, Oct 29, 2021 at 10:04:31AM -0400, Tom Lane wrote: > writes: > > In plpgsql, i'm trying to use a variable initialised in a DECLARE in > > a later initialisation expression whithin the same DECLARE: > > > DECLARE > > x int := 23; > > y int := x + 4 > > BEGIN > >... > > > Ex

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2021-10-29 Thread Daniel Gustafsson
> On 19 May 2020, at 23:07, Daniel Gustafsson wrote: > >> On 19 May 2020, at 17:34, Tom Lane wrote: >> >> Daniel Gustafsson writes: On 13 Feb 2020, at 23:55, Tom Lane wrote: Given the current behavior of SET ROLE and SET SESSION AUTHORIZATION, I don't actually see any way that

Re: Feature request for adoptive indexes

2021-10-29 Thread Hayk Manukyan
Hi all First of all thank you all for fast and rich responses, that is really nice. I don't have that deep knowledge of how postgres works under the hood so I will try to explain more user side. I want to refer for some points mentioned above. - First INCLUDE statement mostly eliminates the neces

Re: plpgsql: can I use a variable in a DECLARE later whithin the DECLARE?

2021-10-29 Thread Tom Lane
writes: > In plpgsql, i'm trying to use a variable initialised in a DECLARE in > a later initialisation expression whithin the same DECLARE: > DECLARE > x int := 23; > y int := x + 4 > BEGIN >... > Experiments suggest that it works, but the docs are silent about that. > Is this a

Re: refactoring basebackup.c

2021-10-29 Thread Robert Haas
On Fri, Oct 29, 2021 at 8:59 AM Jeevan Ladhe wrote:> > bbsink_gzip_ops have the cleanup() callback set to NULL, and when the > bbsink_cleanup() callback is triggered, it tries to invoke a function that > is NULL. I think either bbsink_gzip_ops should set the cleanup callback > to bbsink_forward_cl

Re: ThisTimeLineID is used uninitialized in basebackup.c, too

2021-10-29 Thread Robert Haas
On Thu, Oct 28, 2021 at 9:27 PM Kyotaro Horiguchi wrote: > The first hunk is useless but seems good for sanity. The second hunk > looks good. (I think we can call CheckXLogRemoved before collecting > file names but it would be another thing.) The third and fifth hunks > look fine. Thanks for your

Re: when the startup process doesn't (logging startup delays)

2021-10-29 Thread Nitin Jadhav
> I think you're wrong. If we did that, the previous timer could fire > right after we set startup_progress_timer_expired = false, and before > we reschedule the timeout. It seems annoying to have to disable the > timeout and immediately turn around and re-enable it, but I don't see > how to avoid

Re: refactoring basebackup.c

2021-10-29 Thread Jeevan Ladhe
Thanks, Robert for the patches. I tried to take a backup using gzip compression and got a core. $ pg_basebackup -t server:/tmp/data_gzip -Xnone --server-compression=gzip NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complet

Re: when the startup process doesn't (logging startup delays)

2021-10-29 Thread Robert Haas
On Fri, Oct 29, 2021 at 7:37 AM Nitin Jadhav wrote: > ereport_startup_progress() logs a message. So I feel just setting > 'startup_progress_timer_expired' to false in > begin_startup_progress_phase() would work. Please correct me if I am > wrong. I think you're wrong. If we did that, the previous

plpgsql: can I use a variable in a DECLARE later whithin the DECLARE?

2021-10-29 Thread tomas
Hi, In plpgsql, i'm trying to use a variable initialised in a DECLARE in a later initialisation expression whithin the same DECLARE: DECLARE x int := 23; y int := x + 4 BEGIN ... Experiments suggest that it works, but the docs are silent about that. Is this a good idea, or should

Re: Teach pg_receivewal to use lz4 compression

2021-10-29 Thread Michael Paquier
On Fri, Oct 29, 2021 at 09:45:41AM +, gkokola...@pm.me wrote: > On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier > wrote: >> We don't really care about contentSize as long as a segment is not >> completed. Rather than filling contentSize all the time we write >> something, we'd

Re: when the startup process doesn't (logging startup delays)

2021-10-29 Thread Nitin Jadhav
> I was fooling around with a test setup today, working on an unrelated > problem, and this happened: > > 2021-10-28 14:21:23.145 EDT [92010] LOG: resetting unlogged relations > (init), elapsed time: 0.00 s, current path: base/13020 Nice catch and interesting case. > That's not supposed to happe

Re: ThisTimeLineID is used uninitialized in basebackup.c, too

2021-10-29 Thread Michael Paquier
On Thu, Oct 28, 2021 at 11:39:52AM -0400, Robert Haas wrote: > I previously reported[1] that CreateReplicationSlot() was accessing > the global variable ThisTimeLineID when it might not be initialized. > Today I realized that the code in basebackup.c has the same problem. > perform_base_backup() bl

Re: Skipping logical replication transactions on subscriber side

2021-10-29 Thread Amit Kapila
On Fri, Oct 29, 2021 at 10:54 AM Masahiko Sawada wrote: > > On Thu, Oct 28, 2021 at 7:40 PM Amit Kapila wrote: > > > > On Thu, Oct 28, 2021 at 10:36 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 27, 2021 at 7:02 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Oct 21, 2021 at 10:29

Vulnerability identified with Postgres 13.4 for Windows

2021-10-29 Thread Joel Mariadasan (jomariad)
Hi, The scanning tool used by our organization has detected the presence of vulnerable libxml version in the latest Postgres 13.4 release for windows (Zip version). Detected by Automated Scanning tool: libxml 2.9.10 Can you confirm if this is the same version of libxml used in Postgres? We w

Re: Support for NSS as a libpq TLS backend

2021-10-29 Thread Kevin Burke
For anyone else trying to test out this branch I'm able to get the patches to apply cleanly if I check out e.g. commit 92e6a98c3636948e7ece9a3260f9d89dd60da278. Kevin -- Kevin Burke phone: 925-271-7005 | kevin.burke.dev On Thu, Oct 28, 2021 at 9:31 PM Kevin Burke wrote: > Hi all, apologies bu

Re: Support for NSS as a libpq TLS backend

2021-10-29 Thread Kevin Burke
Hi all, apologies but I'm having trouble applying the latest patch (v45) to the latest commit on master (6b0f6f79eef2168ce38a8ee99c3ed76e3df5d7ad) I downloaded all of the patches to my local filesystem, and then ran: for patch in ../../kevinburke/rustls-postgres/patchsets/2021-10-05-gustafsson-ma

Skip vacuum log report code in lazy_scan_heap() if possible

2021-10-29 Thread Greg Nancarrow
Hi, When recently debugging a vacuum-related issue, I noticed that there is some log-report processing/formatting code at the end of lazy_scan_heap() that, unless the vacuum VERBOSE option is specified, typically results in no log output (as the log-level is then DEBUG2). I think it is worth skipp

Re: Teach pg_receivewal to use lz4 compression

2021-10-29 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier wrote: > On Fri, Sep 17, 2021 at 08:12:42AM +, gkokola...@pm.me wrote: > > I have been digging into the issue I saw in the TAP tests when closing > a segment, and found the problem. The way you

Re: Add support for ALTER INDEX .. ALTER [COLUMN] col_num {SET,RESET}

2021-10-29 Thread Michael Paquier
On Fri, Oct 29, 2021 at 02:09:34PM +0900, Michael Paquier wrote: > The attached does the work, with tests and documentation added to all > the places I could think about while reviewing the existing opclass > code for indexes. There is no need to worry about pg_dump, as opclass > parameters are lo

Re: Skipping logical replication transactions on subscriber side

2021-10-29 Thread Amit Kapila
On Fri, Oct 29, 2021 at 6:18 AM Masahiko Sawada wrote: > > On Thu, Oct 28, 2021 at 6:34 PM Amit Kapila wrote: > > > > On Thu, Oct 28, 2021 at 10:56 AM Masahiko Sawada > > wrote: > > > > > > > > > Another thing I'm concerned is that the syntax "SKIP ( > > > subscription_parameter [=value] [, ...

Re: pg_receivewal starting position

2021-10-29 Thread Ronan Dunklau
Le vendredi 29 octobre 2021, 04:27:51 CEST Michael Paquier a écrit : > On Thu, Oct 28, 2021 at 03:55:12PM +0200, Ronan Dunklau wrote: > > Interesting ideas, thanks. For the record, the time drops from ~4.5s to 3s > > on average on my machine. > > I think if you reduce the size of the generate_serie

Re: Multi-Column List Partitioning

2021-10-29 Thread Amit Langote
Hi Nitin, On Fri, Oct 22, 2021 at 6:48 PM Nitin Jadhav wrote: > Thanks for sharing. I have fixed the issue in the attached patch. I noticed that there's no commitfest entry for this. Will you please add this to the next one? -- Amit Langote EDB: http://www.enterprisedb.com

Re: Improve logging when using Huge Pages

2021-10-29 Thread Masahiko Sawada
On Wed, Oct 27, 2021 at 3:40 PM Shinoda, Noriyoshi (PN Japan FSIP) wrote: > > Hi, > Thank you for your comment. > The attached patch stops message splitting. > This patch also limits the timing of message output when huge_pages = try and > HugePages is not used. > I've looked at the patch. Here