Re: Unwanted file mode modification?

2023-01-09 Thread Amit Kapila
On Mon, Jan 9, 2023 at 1:21 PM Japin Li wrote: > > Commit 216a784829 change the src/backend/replication/logical/worker.c file > mode > from 0644 to 0755, which is unwanted, right? > Right, it is by mistake. I'll fix it. -- With Regards, Amit Kapila.

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread Ankit Kumar Pandey
On 09/01/23 03:48, David Rowley wrote: On Mon, 9 Jan 2023 at 06:17, Ankit Kumar Pandey wrote: I have addressed all points 1-6 in the attached patch. A few more things: 1. You're still using the 'i' variable in the foreach loop. foreach_current_index() will work. 2. I think the "in

Re: Common function for percent placeholder replacement

2023-01-09 Thread Peter Eisentraut
On 04.01.23 01:37, Nathan Bossart wrote: In general, +1. On Tue, Dec 20, 2022 at 06:30:40AM +0100, Peter Eisentraut wrote: (Still need to think about Robert's comment about lack of error context.) Would adding the name of the GUC be sufficient? ereport(ERROR,

Re: RFC: logical publication via inheritance root?

2023-01-09 Thread Aleksander Alekseev
Hi Jacob, > we have to deal with multiple inheritance somehow I would like to point out that we shouldn't necessarily support multiple inheritance in all the possible cases, at least not in the first implementation. Supporting simple cases of inheritance would be already a valuable feature even i

Re: Progress report of CREATE INDEX for nested partitioned tables

2023-01-09 Thread Ilya Gladyshev
On Sun, 2023-01-08 at 10:48 -0600, Justin Pryzby wrote: > On Sat, Dec 17, 2022 at 08:30:02AM -0600, Justin Pryzby wrote: > > We have the common problem of too many patches. > > > > https://www.postgresql.org/message-id/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com > > This changes the

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Amit Kapila
On Sun, Jan 8, 2023 at 11:32 AM houzj.f...@fujitsu.com wrote: > > On Sunday, January 8, 2023 11:59 AM houzj.f...@fujitsu.com > wrote: > > Attach the updated patch set. > > Sorry, the commit message of 0001 was accidentally deleted, just attach > the same patch set again with commit message. > P

Re: An oversight in ExecInitAgg for grouping sets

2023-01-09 Thread David Rowley
On Thu, 5 Jan 2023 at 20:06, Richard Guo wrote: > I reviewed this patch and have some comments. Thanks for looking at this. I think I've fixed all the issues you mentioned. One extra thing I noticed was that I had to add a new VALGRIND_MAKE_MEM_DEFINED in AllocSetAlloc when grabbing an item off

Re: Privileges on PUBLICATION

2023-01-09 Thread Peter Eisentraut
On 16.12.22 17:37, Antonin Houska wrote: This is v4. The patch had to be rebased due to the commit 369f09e420. I think what this patch set needs first of all is a comprehensive description of what it is trying to do, exactly what commands and behaviors it adds, what are some of the subtleties

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Shinoda, Noriyoshi (PN Japan FSIP)
Hi, Thanks for the great new feature. Applied patches include adding wait events LogicalParallelApplyMain, LogicalParallelApplyStateChange. However, it seems that monitoring.sgml only contains descriptions for pg_locks. The attached patch adds relevant wait event information. Please update if y

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Aleksander Alekseev
Hi Alexander, > I'm going to push this if no objections. I took a fresh look at the patch and it LGTM. I only did a few cosmetic changes, PFA v7. Changes since v6 are: ``` @@ -318,12 +318,12 @@ heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, result = heap_delete(rela

Re: [PATCH] Simple code cleanup in tuplesort.c.

2023-01-09 Thread John Naylor
On Thu, Jan 5, 2023 at 8:18 AM John Naylor wrote: > > The label TSS_BUILDRUNS has a similar style and also the following comment, so I will push this patch with a similar comment added unless someone wants to make a case for doing otherwise. > > * Note that mergeruns sets the correct state->status

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Alexander Korotkov
Hi Aleksander, On Mon, Jan 9, 2023 at 12:56 PM Aleksander Alekseev wrote: > > I'm going to push this if no objections. > > I took a fresh look at the patch and it LGTM. I only did a few > cosmetic changes, PFA v7. > > Changes since v6 are: Thank you for looking into this. It appears that I've a

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread houzj.f...@fujitsu.com
On Monday, January 9, 2023 5:32 PM Shinoda, Noriyoshi (PN Japan FSIP) wrote: > > Hi, Thanks for the great new feature. > > Applied patches include adding wait events LogicalParallelApplyMain, > LogicalParallelApplyStateChange. > However, it seems that monitoring.sgml only contains descriptions

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Alexander Korotkov
On Mon, Jan 9, 2023 at 1:10 PM Alexander Korotkov wrote: > On Mon, Jan 9, 2023 at 12:56 PM Aleksander Alekseev > wrote: > > > I'm going to push this if no objections. > > > > I took a fresh look at the patch and it LGTM. I only did a few > > cosmetic changes, PFA v7. > > > > Changes since v6 are:

RE: Logical replication timeout problem

2023-01-09 Thread wangw.f...@fujitsu.com
On Fri, Jan 6, 2023 at 15:06 PM Ashutosh Bapat wrote: > Hi Wang, > Thanks for working on this. One of our customer faced a similar > situation when running BDR with PostgreSQL. > > I tested your patch and it solves the problem. > > Please find some review comments below Thanks for your testing

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Pavel Borisov
Hi, Alexander! On Mon, 9 Jan 2023 at 13:29, Alexander Korotkov wrote: > > On Mon, Jan 9, 2023 at 1:10 PM Alexander Korotkov > wrote: > > On Mon, Jan 9, 2023 at 12:56 PM Aleksander Alekseev > > wrote: > > > > I'm going to push this if no objections. > > > > > > I took a fresh look at the patch

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Aleksander Alekseev
Alexander, Pavel, > Sorry for being so detailed in small details. In my opinion the patch > now is ready to be committed. Agree. Personally I liked the version with (lockUpdated, lockedSlot) pair a bit more since it is a bit more readable, however the version without lockUpdated is less error pr

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Alexander Korotkov
Hi, Pavel! On Mon, Jan 9, 2023 at 1:41 PM Pavel Borisov wrote: > On Mon, 9 Jan 2023 at 13:29, Alexander Korotkov wrote: > > On Mon, Jan 9, 2023 at 1:10 PM Alexander Korotkov > > wrote: > > > On Mon, Jan 9, 2023 at 12:56 PM Aleksander Alekseev > > > wrote: > > > > > I'm going to push this if n

Re: Split index and table statistics into different types of stats

2023-01-09 Thread Nitin Jadhav
>> +/* - >> + * >> + * pgstat_index.c >> + *Implementation of index statistics. > > This is a fair bit of duplicated code. Perhaps it'd be worth keeping > pgstat_relation with code common to table/index stats? +1 to keep c

Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread shiy.f...@fujitsu.com
Hi hackers, I noticed that there is a problem about system view pg_publication_tables when looking into [1]. The column "attnames" contains generated columns when no column list is specified, but generated columns shouldn't be included because they are not replicated (see send_relation_and_attrs()

Re: [PATCH] random_normal function

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 00:20, Tom Lane wrote: > > This version seems committable to me, barring objections. > Whilst I have no objection to adding random_normal(), ISTM that we're at risk of adding an arbitrary set of random functions without a clear idea of where we'll end up, and how they'll aff

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread David Rowley
On Mon, 9 Jan 2023 at 21:34, Ankit Kumar Pandey wrote: > > > > On 09/01/23 03:48, David Rowley wrote: > > 3. I don't think you need to set "index" on every loop. Why not just > set it to foreach_current_index(l) - 1; break; > > Consider this query > > EXPLAIN (COSTS OFF) > SELECT empno, >

Re: MERGE ... RETURNING

2023-01-09 Thread Dean Rasheed
On Sun, 8 Jan 2023 at 20:09, Isaac Morland wrote: > > Would it be useful to have just the action? Perhaps "WITH ACTION"? My idea is > that this would return an enum of INSERT, UPDATE, DELETE (so is "action" the > right word?). It seems to me in many situations I would be more likely to > care a

Re: [PATCH] Simple code cleanup in tuplesort.c.

2023-01-09 Thread Xing Guo
On 1/9/23, John Naylor wrote: > On Thu, Jan 5, 2023 at 8:18 AM John Naylor > wrote: >> >> The label TSS_BUILDRUNS has a similar style and also the following > comment, so I will push this patch with a similar comment added unless > someone wants to make a case for doing otherwise. >> >> * Note th

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-09 Thread Shinoda, Noriyoshi (PN Japan FSIP)
Thanks for the reply. > Thanks for reporting. I think for LogicalParallelApplyStateChange we'd better > document it in a consistent style with LogicalSyncStateChange, > so I have slightly adjusted the patch for the same. I think the description in the patch you attached is better. Regards, Nor

Re: [EXTERNAL] Re: Support load balancing in libpq

2023-01-09 Thread Jelte Fennema
Attached an updated patch which should address your feedback and I updated the commit message. > I wonder whether making the parameter a boolean will paint us into a > corner I made it a string option, just like target_session_attrs. I'm pretty sure a round-robin load balancing policy could be i

Allow +group in pg_ident.conf

2023-01-09 Thread Andrew Dunstan
Over at [1] I speculated that it might be a good idea to allow +grouprole type user names in pg_ident.conf. The use case I have in mind is where the user authenticates to pgbouncer and then pgbouncer connects as the user using a client certificate. Without this mechanism that means that you need a

Re: Allow DISTINCT to use Incremental Sort

2023-01-09 Thread Richard Guo
On Sat, Jan 7, 2023 at 5:47 PM David Rowley wrote: > While working on the regression tests added in a14a58329, I noticed > that DISTINCT does not make use of Incremental Sort. It'll only ever > do full sorts on the cheapest input path or make use of a path that's > already got the required pathk

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
> > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type of pageno representation? Is SLRU wraparound still > exactly there after 0x byte? > After applying the whole patch set, SLRU will become 64–bit without a wraparound. Thus, no wraparound should be

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Aleksander Alekseev
Hi Andrey, > Hi! I think that 64-bit xids are a very important feature and I want > to help advance it. That's why I want to try to understand a patch > better. Thanks for your interest to the patchset! > Do I get it right that the proposed v51 patchset only changes the SLRU > filenames and type

Re: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread Amit Kapila
On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com wrote: > > I noticed that there is a problem about system view pg_publication_tables when > looking into [1]. The column "attnames" contains generated columns when no > column list is specified, but generated columns shouldn't be included becaus

Re: Add support for DEFAULT specification in COPY FROM

2023-01-09 Thread Andrew Dunstan
On 2022-12-02 Fr 09:11, Israel Barth Rubio wrote: > Hello all, > > I'm submitting a new version of the patch. Instead of changing signature > of several functions in order to use the defaults parameter, it is now > storing > that in the cstate structure, which is already passed to all functions >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
On Fri, 6 Jan 2023 at 09:51, Japin Li wrote: > > For v51-0003. We can use GetClogDirName instead of GET_MAJOR_VERSION in > copy_subdir_files(). > Of course! Tanks! I'll address this in the next iteration, v52. -- Best regards, Maxim Orlov.

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2023-01-09 Thread Maxim Orlov
Hi! Here is a new patch set. I've added comments and make use GetClogDirName call in copy_subdir_files. -- Best regards, Maxim Orlov. v52-0003-Make-pg_upgrade-from-32-bit-to-64-bit-SLRU.patch Description: Binary data v52-0002-Use-64-bit-pages-representation-in-SLRU-callers.patch Description:

Lazy allocation of pages required for verifying FPI consistency

2023-01-09 Thread Bharath Rupireddy
Hi, Postgres verifies consistency of FPI from WAL record with the replayed page during recovery in verifyBackupPageConsistency() when either wal_consistency_checking for the resource manager is enabled or a WAL record with XLR_CHECK_CONSISTENCY flag is inserted. While doing so, it uses two interme

Re: Timeout when changes are filtered out by the core during logical replication

2023-01-09 Thread Ashutosh Bapat
On Fri, Dec 23, 2022 at 2:45 PM Amit Kapila wrote: > On Thu, Dec 22, 2022 at 6:58 PM Ashutosh Bapat > wrote: > > > > Hi All, > > A customer ran a script dropping a few dozens of users in a transaction. > Before dropping a user they change the ownership of the tables owned by > that user to anoth

Re: Add SHELL_EXIT_CODE to psql

2023-01-09 Thread Maxim Orlov
Hi! In overall, I think we move in the right direction. But we could make code better, should we? + /* Capture exit code for SHELL_EXIT_CODE */ + close_exit_code = pclose(fd); + if (close_exit_code == -1) + { + pg_log_e

Re: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread Tom Lane
Amit Kapila writes: > On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com > wrote: >> I think one way to fix it is to modify pg_publication_tables query to exclude >> generated columns. But in this way, we need to bump catalog version when >> fixing >> it in back-branch. Another way is to modif

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-01-09 Thread Jelte Fennema
Thanks for clarifying your reasoning. I now agree that ssrootcert=system is now the best option. > > 2. Should we allow the same approach with ssl_ca_file on the server side, > > for client cert validation? > > I don't know enough about this use case to implement it safely. We'd > have to make su

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
Dean Rasheed writes: > So IMO all pseudorandom functions should share the same PRNG state and > seed-setting functions. That would mean they should all be in the same > (new) C file, so that the PRNG state can be kept private to that file. I agree with this in principle, but I feel no need to act

Re: [PATCH] psql: Add tab-complete for optional view parameters

2023-01-09 Thread Jim Jones
Hi Christoph, Thanks for the patch! I just tested it and I could reproduce the expected behaviour in these cases: postgres=# CREATE VIEW w AS  WITH ( postgres=# CREATE OR REPLACE VIEW w AS  WITH ( postgres=# CREATE VIEW w WITH ( CHECK_OPTION  SECURITY_BARRIER  SECURITY_INVOKER p

Re: [PATCH] Add `verify-system` sslmode to use system CA pool for server cert

2023-01-09 Thread Andrew Dunstan
On 2023-01-09 Mo 10:07, Jelte Fennema wrote: > Thanks for clarifying your reasoning. I now agree that ssrootcert=system > is now the best option. Cool, that looks like a consensus. > >>> 2. Should we allow the same approach with ssl_ca_file on the server side, >>> for client cert validation?

Re: PL/pgSQL cursors should get generated portal names by default

2023-01-09 Thread Pavel Stehule
Hi I wrote a new check in plpgsql_check, that tries to identify explicit work with the name of the referenced portal. create or replace function foo01() returns refcursor as $$#option dump declare c cursor for select 1; r refcursor; begin open c; r := 'c'; return r; end; $$ language plp

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 9 Jan 2023 08:09:02 +0100 Brar Piening wrote: > On 09.01.2023 at 03:31, vignesh C wrote: > > The patch does not apply on top of HEAD as in [1], please post a > > rebased patch: > This one applies on top of 3c569049b7b502bb4952483d19ce622ff0af5fd6 > and the documentation build succeeds.

psql: Add role's membership options to the \du+ command

2023-01-09 Thread Pavel Luzanov
When you include one role in another, you can specify three options: ADMIN, INHERIT (added in e3ce2de0) and SET (3d14e171). For example. CREATE ROLE alice LOGIN; GRANT pg_read_all_settings TO alice WITH ADMIN TRUE, INHERIT TRUE, SET TRUE; GRANT pg_stat_scan_tables TO alice WITH ADMIN FALSE, INH

Re: [PATCH] random_normal function

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 15:26, Tom Lane wrote: > > Dean Rasheed writes: > > So IMO all pseudorandom functions should share the same PRNG state and > > seed-setting functions. That would mean they should all be in the same > > (new) C file, so that the PRNG state can be kept private to that file. >

Re: MERGE ... RETURNING

2023-01-09 Thread Vik Fearing
On 1/9/23 13:29, Dean Rasheed wrote: On Sun, 8 Jan 2023 at 20:09, Isaac Morland wrote: Would it be useful to have just the action? Perhaps "WITH ACTION"? My idea is that this would return an enum of INSERT, UPDATE, DELETE (so is "action" the right word?). It seems to me in many situations I

Re: Make EXPLAIN generate a generic plan for a parameterized query

2023-01-09 Thread Laurenz Albe
On Tue, 2022-12-27 at 14:37 -0800, Michel Pelletier wrote: > I built and tested this patch for review and it works well, although I got > the following warning when building: > > analyze.c: In function 'transformStmt': > analyze.c:2919:35: warning: 'generic_plan' may be used uninitialized in this

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Matthias van de Meent
On Sun, 8 Jan 2023 at 04:09, Andres Freund wrote: > > Hi, > > On 2023-01-07 16:29:23 -0800, Andres Freund wrote: > > It's probably not too hard to fix specifically in this one place - we could > > just clamp vacuum_defer_cleanup_age to be smaller than latest_completed, but > > it strikes me as as

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-09 Thread Ankit Kumar Pandey
On 09/01/23 17:53, David Rowley wrote: We need to keep looping backwards until we find the first WindowClause which does not contain the pathkeys of the ORDER BY. I found the cause of confusion, *first* WindowClause means first from forward direction. Since we are looping from backward, I

Re: wake up logical workers after ALTER SUBSCRIPTION

2023-01-09 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index cf220c3bcb..7661c0c86e 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1996,6 +1996,16 @@ postgres 2709

Re: pub/sub - specifying optional parameters without values.

2023-01-09 Thread Zheng Li
Hi, This documentation change looks good to me. I verified in testing and in code that the value for boolean parameters in PUB/SUB commands can be omitted. which is equivalent to specifying TRUE. For example, CREATE PUBLICATIOIN mypub for ALL TABLES with (publish_via_partition_root); is equival

Re: suppressing useless wakeups in logical/worker.c

2023-01-09 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 2466001a3ae6f94aac8eff45b488765e619bea1b Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Thu, 1 Dec 2022 20:50:21 -0800 Subject: [PATCH v2 1/1] suppress unnecessary wakeups in logical/worker.c --- src/bac

Re: MERGE ... RETURNING

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 16:23, Vik Fearing wrote: > > Bikeshedding here. Instead of Yet Another WITH Clause, could we perhaps > make a MERGING() function analogous to the GROUPING() function that goes > with grouping sets? > > MERGE ... > RETURNING *, MERGING('clause'), MERGING('action'); > Hmm, p

Re: add \dpS to psql

2023-01-09 Thread Nathan Bossart
On Sat, Jan 07, 2023 at 11:18:59AM +, Dean Rasheed wrote: > It might be true that temp tables aren't usually interesting from a > permissions point of view, but it's not hard to imagine situations > where interesting things do happen. It's also probably the case that > most users won't have man

Re: Common function for percent placeholder replacement

2023-01-09 Thread Nathan Bossart
On Mon, Jan 09, 2023 at 09:36:12AM +0100, Peter Eisentraut wrote: > On 04.01.23 01:37, Nathan Bossart wrote: >> On Tue, Dec 20, 2022 at 06:30:40AM +0100, Peter Eisentraut wrote: >> > + * A value may be NULL. If the corresponding placeholder is found in the >> > + * input string, the whole function

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Sun, Jan 8, 2023 at 6:43 PM Peter Geoghegan wrote: > On further reflection even v2 won't repair the page-level > PD_ALL_VISIBLE flag in passing in this scenario. ISTM that on HEAD we > might actually leave the all-frozen bit set in the VM, while both the > all-visible bit and the page-level PD_

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Önder Kalacı
Hi, Thank you for the useful comments! > I took a very brief look through this. I'm not too pleased with > this whole line of development TBH. It seems to me that the core > design of execReplication.c and related code is "let's build our > own half-baked executor and much-less-than-half-baked

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Nathan Bossart
On Mon, Jan 09, 2023 at 08:00:26AM -0500, Andrew Dunstan wrote: > + If the database-username begins with a > + + character, then the operating system user can login > as > + any user belonging to that role, similarly to how user names beginning > with > + + are treated in pg_hba.conf. I

Re: Add SHELL_EXIT_CODE to psql

2023-01-09 Thread Corey Huinker
On Mon, Jan 9, 2023 at 10:01 AM Maxim Orlov wrote: > Hi! > > In overall, I think we move in the right direction. But we could make code > better, should we? > > + /* Capture exit code for SHELL_EXIT_CODE */ > + close_exit_code = pclose(fd); > + if (close_

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
I wrote: > Hmm ... it occurred to me to try the same check on the existing > random() tests (attached), and darn if they don't fail even more > often, usually within 50K iterations. So maybe we should rethink > that whole thing. I pushed Paul's patch with the previously-discussed tests, but the m

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Andres Freund
Hi, Robert, Mark, CCing you because of the amcheck aspect (see below). On 2023-01-09 17:50:10 +0100, Matthias van de Meent wrote: > On Sun, 8 Jan 2023 at 04:09, Andres Freund wrote: > > > For a bit I was thinking that we should introduce the notion that a > > > FullTransactionId can be from the

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Andres Freund
Hi, On 2023-01-08 16:27:59 -0800, Peter Geoghegan wrote: > On Sun, Jan 8, 2023 at 3:53 PM Andres Freund wrote: > > How? > > See the commit message for the scenario I have in mind, which involves > a concurrent HOT update that aborts. I looked at it. I specifically was wondering about this part

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 10:16:03 -0800, Peter Geoghegan wrote: > Attached is v3, which explicitly checks the need to set the PD_ALL_VISIBLE > flag at the relevant visibilitymap_set() call site. It also has improved > comments. Afaict we'll need to backpatch this all the way? > From e7788ebdb589fb7c6f

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Andres Freund
Hi, On 2023-01-07 13:50:04 -0500, Tom Lane wrote: > I think we should either live within the constraints set by this > overarching design, or else nuke execReplication.c from orbit and > start using the real planner and executor. Perhaps the foreign > key enforcement mechanisms could be a model -

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 11:44 AM Andres Freund wrote: > I think that's just an imprecise formulation though - the problem is that we > can call visibilitymap_set() with just VISIBILITYMAP_ALL_FROZEN, even though > VISIBILITYMAP_ALL_VISIBLE was concurrently unset. That's correct. You're right that

Re: Split index and table statistics into different types of stats

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 17:08:42 +0530, Nitin Jadhav wrote: > +1 to keep common functions/code between table and index stats. Only > the data structure should be different as the goal is to shrink the > current memory usage. I don't think the goal is solely to shrink memory usage - it's also to make it

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Tom Lane
Brar Piening writes: > On 09.01.2023 at 03:38, vignesh C wrote: >> There are couple of commitfest entries for this: >> https://commitfest.postgresql.org/41/4041/ >> https://commitfest.postgresql.org/41/4042/ Can one of them be closed? > I've split the initial patch into two parts upon Álvaro's re

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Thomas Munro
On Tue, Jan 10, 2023 at 8:34 AM Andres Freund wrote: > A different approach would be to represent fxids as *signed* 64bit > integers. That'd of course loose more range, but could represent everything > accurately, and would have a compatible on-disk representation on two's > complement platforms (

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-09 Thread Tom Lane
Andres Freund writes: > On 2023-01-07 13:50:04 -0500, Tom Lane wrote: >> Also ... maybe I am missing something, but is REPLICA IDENTITY FULL >> sanely defined in the first place? It looks to me that >> RelationFindReplTupleSeq assumes without proof that there is a unique >> full-tuple match, but

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 09 Jan 2023 15:18:18 -0500 Tom Lane wrote: > I pushed the ID-addition patch, with a few fixes: > > * AFAIK our practice is to use "-" never "_" in XML ID attributes. > You weren't very consistent about that even within this patch, and > the overall effect would have been to have no stand

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Robert Haas
On Mon, Jan 2, 2023 at 1:32 PM Peter Geoghegan wrote: > On Sat, Dec 31, 2022 at 4:53 PM Peter Geoghegan wrote: > > The first patch makes sure that the snapshotConflictHorizon cutoff > > (XID cutoff for recovery conflicts) is never a special XID, unless > > that XID is InvalidTransactionId, which

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 11:57 AM Andres Freund wrote: > Afaict we'll need to backpatch this all the way? I thought that we probably wouldn't need to, at first. But I now think that we really have to. I didn't realize that affected visibilitymap_set() calls could generate useless set-VM WAL record

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-09 Thread Melanie Plageman
Attached is v45 of the patchset. I've done some additional code cleanup and changes. The most significant change, however, is the docs. I've separated the docs into its own patch for ease of review. The docs patch here was edited and co-authored by Samay Sharma. I'm not sure if the order of pg_sta

Re: ATTACH PARTITION seems to ignore column generation status

2023-01-09 Thread Tom Lane
I wrote: > After thinking about this awhile, I feel that we ought to disallow > it in the traditional-inheritance case as well. The reason is that > there are semantic prohibitions on inserting or updating a generated > column, eg > regression=# create table t (f1 int, f2 int generated always as

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Andres Freund
Hi, On 2022-11-15 10:26:05 -0800, Peter Geoghegan wrote: > Pushed something like this earlier today, though without any changes > to VISIBLE records. While updating a patch to log various offsets in pg_waldump, I noticed a few minor issues in this patch: ISTM that some of the page level freezing

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Mark Dilger
> On Jan 9, 2023, at 11:34 AM, Andres Freund wrote: > > 1) Because ctx->next_xid is set after the XidFromFullTransactionId() call in > update_cached_xid_range(), we end up using the xid 0 (or an outdated value in > subsequent calls) to determine whether epoch needs to be reduced. Can you say

Show various offset arrays for heap WAL records

2023-01-09 Thread Andres Freund
Hi, A couple times when investigating data corruption issues, the last time just yesterday in [1], I needed to see the offsets affected by PRUNE and VACUUM records. As that's probably not just me, I think we should make that change in-tree. The attached patch adds details to XLOG_HEAP2_PRUNE, XLO

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 13:55:02 -0800, Mark Dilger wrote: > > On Jan 9, 2023, at 11:34 AM, Andres Freund wrote: > > > > 1) Because ctx->next_xid is set after the XidFromFullTransactionId() call in > > update_cached_xid_range(), we end up using the xid 0 (or an outdated value > > in > > subsequent ca

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 1:43 PM Andres Freund wrote: > ISTM that some of the page level freezing functions are misnamed. In heapam.c > the heap_xlog* routines are for replay, afaict. However > heap_xlog_freeze_plan() is used to WAL log the freeze > plan. heap_xlog_freeze_page() is used to replay th

Re: WAL Insertion Lock Improvements (was: Re: Avoid LWLockWaitForVar() for currently held WAL insertion lock in WaitXLogInsertionsToFinish())

2023-01-09 Thread Andres Freund
Hi, On 2022-12-08 12:29:54 +0530, Bharath Rupireddy wrote: > On Tue, Dec 6, 2022 at 12:00 AM Andres Freund wrote: > > I think it'd be safe to optimize LWLockConflictsWithVar(), due to some > > pre-existing, quite crufty, code. LWLockConflictsWithVar() says: > > > > * Test first to see if

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Karl O. Pinc
On Mon, 09 Jan 2023 15:18:18 -0500 Tom Lane wrote: > Brar Piening writes: > > On 09.01.2023 at 03:38, vignesh C wrote: > >> There are couple of commitfest entries for this: > >> https://commitfest.postgresql.org/41/4041/ > >> https://commitfest.postgresql.org/41/4042/ Can one of them be > >> c

Re: doc: add missing "id" attributes to extension packaging page

2023-01-09 Thread Tom Lane
"Karl O. Pinc" writes: > I think there's more to comment on regards the xslt in the > other patch, but I'll wait for the new thread for that patch. > That might be where there should be warnings raised, etc. We can continue using this thread, now that the other commit is in.

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Andrew Dunstan
On 2023-01-09 Mo 13:24, Nathan Bossart wrote: > On Mon, Jan 09, 2023 at 08:00:26AM -0500, Andrew Dunstan wrote: >> + If the database-username begins with a >> + + character, then the operating system user can login >> as >> + any user belonging to that role, similarly to how user names begi

Re: [PATCH] random_normal function

2023-01-09 Thread Dean Rasheed
On Mon, 9 Jan 2023 at 18:52, Tom Lane wrote: > > I pushed Paul's patch with the previously-discussed tests, but > the more I look at random.sql the less I like it. I propose > that we nuke the existing tests from orbit and replace with > something more or less as attached. Looks like a definite

Re: [PATCH] Support % wildcard in extension upgrade filenames

2023-01-09 Thread Tom Lane
I continue to think that this is a fundamentally bad idea. It creates all sorts of uncertainties about what is a valid update path and what is not. Restrictions like + Such wildcard update + scripts will only be used when no explicit path is found from + old to target version. are j

Re: allow granting CLUSTER, REFRESH MATERIALIZED VIEW, and REINDEX

2023-01-09 Thread Nathan Bossart
On Tue, Jan 03, 2023 at 03:45:49PM -0800, Nathan Bossart wrote: > I'd like to get this fixed, but I have yet to hear thoughts on the > suggested approach. I'll proceed with adjusting the tests and > documentation shortly unless someone objects. As promised, here is a new version of the patch with

Re: Fixing a couple of buglets in how VACUUM sets visibility map bits

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 12:51 PM Robert Haas wrote: > I feel that you should at least have a reproducer for these problems > posted to the thread, and ideally a regression test, before committing > things. I think it's very hard to understand what the problems are > right now. Hard to understand r

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Jelte Fennema
This seems very much related to my patch here: https://commitfest.postgresql.org/41/4081/ (yes, somehow the thread got split. I blame outlook) I'll try to review this one soonish.

Re: [PATCH] random_normal function

2023-01-09 Thread Tom Lane
Dean Rasheed writes: > On Mon, 9 Jan 2023 at 18:52, Tom Lane wrote: >> (We could probably go further >> than this, like trying to verify distribution properties. But >> it's been too long since college statistics for me to want to >> write such tests myself, and I'm not real sure we need them.)

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Andres Freund
Hi, On 2023-01-09 13:46:50 +0300, Alexander Korotkov wrote: > I'm going to push v9. Could you hold off for a bit? I'd like to look at this, I'm not sure I like the direction. Greetings, Andres Freund

Re: POC: Lock updated tuples in tuple_update() and tuple_delete()

2023-01-09 Thread Andres Freund
Hi, I'm a bit worried that this is optimizing the rare case while hurting the common case. See e.g. my point below about creating additional slots in the happy path. It's also not clear that change is right directionally. If we want to avoid re-fetching the "original" row version, why don't we pr

Re: Allow +group in pg_ident.conf

2023-01-09 Thread Michael Paquier
On Mon, Jan 09, 2023 at 05:33:14PM -0500, Andrew Dunstan wrote: > I've adapted a sentence from the pg_hba.conf documentation so we stay > consistent. + + If the database-username begins with a + + character, then the operating system user can login as + any user belonging to that role, sim

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Andres Freund
Hi, On 2023-01-08 17:49:20 -0800, Peter Geoghegan wrote: > Teach autovacuum.c to launch "table age" autovacuums at the same point > that it previously triggered antiwraparound autovacuums. Antiwraparound > autovacuums are retained, but are only used as a true option of last > resort, when regular

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-09 Thread Michael Paquier
On Mon, Jan 09, 2023 at 08:30:00AM +0530, Bharath Rupireddy wrote: > A recent commit [1] added --save-fullpage option to pg_waldump to > extract full page images (FPI) from WAL records and save them into > files (one file per FPI) under a specified directory. While it added > tests to check the LSN

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 5:22 PM Andres Freund wrote: > I've also seen the inverse, with recent versions of postgres: Autovacuum can > only ever make progress if it's an anti-wraparound vacuum, because it'll > always get cancelled otherwise. I'm worried that substantially increasing the > time unti

Re: Add index scan progress to pg_stat_progress_vacuum

2023-01-09 Thread Andres Freund
Hi, On 2023-01-07 01:59:40 +, Imseih (AWS), Sami wrote: > --- a/src/backend/access/nbtree/nbtree.c > +++ b/src/backend/access/nbtree/nbtree.c > @@ -998,6 +998,8 @@ btvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult > *stats, > if (info->report_progress) >

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-09 Thread Andres Freund
Hi, On 2023-01-03 12:05:20 -0800, Andres Freund wrote: > The attached patch adds libpq-be-fe-helpers.h and uses it in libpqwalreceiver, > dblink, postgres_fdw. > As I made libpq-be-fe-helpers.h handle reserving external fds, > libpqwalreceiver now does so. I briefly looked through its users witho

Re: refactoring relation extension and BufferAlloc(), faster COPY

2023-01-09 Thread Andres Freund
Hi, On 2023-01-06 11:52:04 +0530, vignesh C wrote: > On Sat, 29 Oct 2022 at 08:24, Andres Freund wrote: > > > > The patches here aren't fully polished (as will be evident). But they should > > be more than good enough to discuss whether this is a sane direction. > > The patch does not apply on t

Re: Allow DISTINCT to use Incremental Sort

2023-01-09 Thread David Rowley
Thanks for having a look at this. On Tue, 10 Jan 2023 at 02:28, Richard Guo wrote: > +1 for the changes. A minor comment is that previously on HEAD for > SELECT DISTINCT case, if we have to do an explicit full sort atop the > cheapest path, we try to make sure to always use the more rigorous > o

  1   2   >