Re: Vacuum statistics

2025-01-13 Thread Alena Rybakina
On 10.01.2025 17:51, Alena Rybakina wrote: Hi! I thought about this problem again and I think I have a solution. On 02.12.2024 23:12, Alena Rybakina wrote: * I've read the previous discussion on how important to keep all these fields regarding vacuum statistics including points by Andrei and J

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Bertrand Drouvot
Hi, On Fri, Jan 10, 2025 at 02:07:25PM -0500, Andres Freund wrote: > However, I think 0007 needs a bit more work. > > One thing that worries me a bit is that using SIGINT for triggering the > shutdown checkpoint could somehow be unintentionally triggered? Previously > checkpointer would effective

Re: Conflict detection for update_deleted in logical replication

2025-01-13 Thread Nisha Moond
Here are the performance test results and analysis with the recent patches. Test Setup: - Created Pub and Sub nodes with logical replication and below configurations. autovacuum_naptime = '30s' shared_buffers = '30GB' max_wal_size = 20GB min_wal_size = 10GB track_commit_timestamp =

Re: proposal: plpgsql, new check for extra_errors - strict_expr_check

2025-01-13 Thread Pavel Stehule
Hi fresh rebase Regards Pavel From 4e582d825af9d3463be332a90b9e959980480293 Mon Sep 17 00:00:00 2001 From: "ok...@github.com" Date: Wed, 12 Jun 2024 21:34:05 +0200 Subject: [PATCH 1/3] use strict rules for parsing PL/pgSQL expressions Originally the rule PLpgSQL_Expr allows almost all SQL clau

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-01-13 Thread Jacob Champion
On Mon, Jan 13, 2025 at 3:21 PM Jacob Champion wrote: > Next email will discuss the architectural bug that Kashif found. Okay, here goes. A standard OAuth connection attempt looks like this (oh, I hope Gmail doesn't mangle it): IssuerUserlibpq Backend |||

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2025-01-13 Thread Peter Smith
On Tue, Jan 14, 2025 at 8:22 AM Robert Treat wrote: > > On Mon, Jan 13, 2025 at 3:55 AM Amit Kapila wrote: > > On Mon, Jan 13, 2025 at 10:22 AM Robert Treat wrote: > > > On Sun, Jan 12, 2025 at 11:00 PM Amit Kapila > > > wrote: > > > > On Sat, Jan 11, 2025 at 7:28 PM Robert Treat wrote: > > >

Re: Several buildfarm animals fail tests because of shared memory error

2025-01-13 Thread Tom Lane
Robins Tharakan writes: > Unrelated, parula has been failing the libperl test (only v15 and older), > for the past > 3 weeks - to clarify, this test started to fail (~18 days ago) before I > fixed the > 'RemoveIPC' configuration (~5 days ago), so this is unrelated to that > change. > https://buil

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Peter Smith
Hi Sawada-San. Some review comments for patch v13-0002. == I think the v12 ambiguity of RBTXN_PREPARE versus RBTXN_SENT_PREPARE was mostly addressed already by the improved comments for the macros in patch 0001. Meanwhile, patch v13-0002 says it is renaming constants for better consistency,

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2025-01-13 Thread Robert Treat
On Mon, Jan 13, 2025 at 3:55 AM Amit Kapila wrote: > On Mon, Jan 13, 2025 at 10:22 AM Robert Treat wrote: > > On Sun, Jan 12, 2025 at 11:00 PM Amit Kapila > > wrote: > > > On Sat, Jan 11, 2025 at 7:28 PM Robert Treat wrote: > > > +If a table with replica identity set to NOTHING > > > +

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-13 Thread Masahiko Sawada
On Mon, Jan 13, 2025 at 1:22 AM vignesh C wrote: > > On Tue, 31 Dec 2024 at 10:15, Masahiko Sawada wrote: > > > > Hi all, > > > > Logical decoding (and logical replication) are available only when > > wal_level = logical. As the documentation says[1], Using the 'logical' > > level increases the W

Re: [PATCH] Add get_bytes() and set_bytes() functions

2025-01-13 Thread Dean Rasheed
On Mon, 13 Jan 2025 at 19:23, Alvaro Herrera wrote: > > But these don't show the acceptable range. We have these that do: > > #: utils/adt/varbit.c:1824 utils/adt/varbit.c:1882 > #, c-format > msgid "bit index %d out of valid range (0..%d)" > > #: utils/adt/varlena.c:3218 utils/adt/varlena.c:3285

Re: pgsql: Consolidate docs for vacuum-related GUCs in new subsection

2025-01-13 Thread Alvaro Herrera
On 2025-Jan-13, Melanie Plageman wrote: > Since I didn't hear back about this and I don't see an obvious > alternative reorganization in guc_tables.c, I plan to just push the > attached patch that updates only postgresql.conf.sample. Apologies, I was very unclear -- I didn't want to talk about th

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

2025-01-13 Thread Melanie Plageman
On Thu, Jan 9, 2025 at 1:24 PM Andres Freund wrote: > > On 2025-01-07 15:46:26 -0500, Melanie Plageman wrote: > > For table storage options, those related to vacuum but not autovacuum > > are in the main StdRdOptions struct. Of those, some are overridden by > > VACUUM command parameters which are

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-13 Thread Masahiko Sawada
On Mon, Jan 13, 2025 at 1:31 AM Ashutosh Bapat wrote: > > On Mon, Jan 13, 2025 at 2:52 PM vignesh C wrote: > > > > I felt that the only disadvantage with this approach is that we > > currently wait for all in-progress transactions to complete before > > enabling logical decoding. If a long-runnin

Re: AIO v2.2

2025-01-13 Thread Andres Freund
Hi, On 2025-01-13 15:43:46 -0500, Robert Haas wrote: > On Wed, Jan 8, 2025 at 7:26 PM Andres Freund wrote: > > 1) Shared memory representation of an IO, for the AIO subsystem internally > > > >Currently: PgAioHandle > > > > 2) A way for the issuer of an IO to reference 1), to attach informati

Re: Issues with ON CONFLICT UPDATE and REINDEX CONCURRENTLY

2025-01-13 Thread Michail Nikolaev
Hello, everyone! I have noticed tests are still flapping a little bit on FreeBSD. Now I have added some markers to isolation specs to avoid possible race conditions. Best regards, Mikhail. > v6-0003-Modify-the-infer_arbiter_indexes-function-to-also.patch Description: Binary data v6-0001-Spec

Re: Reduce TupleHashEntryData struct size by half

2025-01-13 Thread Jeff Davis
On Sun, 2025-01-12 at 14:54 +1300, David Rowley wrote: > While I do understand the desire to reduce Hash Agg's memory usage, > has this really been through enough performance testing to be getting > committed? Perhaps not. I'm going to revert it while we sort it out, and hopefully we can find a so

Re: Conflict detection for update_deleted in logical replication

2025-01-13 Thread Masahiko Sawada
On Sun, Jan 12, 2025 at 10:36 PM Amit Kapila wrote: > > On Fri, Jan 10, 2025 at 6:13 AM Masahiko Sawada wrote: > > > > 3. If the apply worker cannot catch up, it could enter to a bad loop; > > the publisher sends huge amount of data -> the apply worker cannot > > catch up -> it needs to wait for

Re: New GUC autovacuum_max_threshold ?

2025-01-13 Thread Nathan Bossart
On Mon, Jan 13, 2025 at 05:17:11PM -0600, Sami Imseih wrote: > I propose renaming the GUC from "autovacuum_max_threshold" to > "autovacuum_vacuum_max_threshold" to clarify that it applies only > to the vacuum operation performed by autovacuum, not to the analyze operation. > This will also align wi

Re: Some ExecSeqScan optimizations

2025-01-13 Thread Amit Langote
Hi Vladlen, On Fri, Jan 10, 2025 at 11:49 PM Vladlen Popolitov wrote: > Amit Langote писал(а) 2025-01-10 18:22: > > On Fri, Jan 10, 2025 at 7:36 PM David Rowley > > wrote: > >> On Fri, 10 Jan 2025 at 22:53, Vladlen Popolitov > >> wrote: > >> > In case of query > >> > select count(*) from test

Re: Some ExecSeqScan optimizations

2025-01-13 Thread Amit Langote
Hi Junwang, On Sat, Jan 11, 2025 at 7:39 PM Junwang Zhao wrote: > Hi Amit, > > On Fri, Jan 10, 2025 at 7:22 PM Amit Langote wrote: > > > > On Fri, Jan 10, 2025 at 7:36 PM David Rowley wrote: > > > On Fri, 10 Jan 2025 at 22:53, Vladlen Popolitov > > > wrote: > > > > In case of query > > > > s

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Peter Smith
Hi Sawada-San. Here are some cosmetic review comments for the patch v13-0001. == Commit message 1. This commit introduces an additional CLOG lookup to check the transaction status, so the logical decoding skips further change also when it doesn't touch system catalogs if the transaction is al

Re: Make pg_stat_io view count IOs as bytes instead of blocks

2025-01-13 Thread Michael Paquier
On Fri, Jan 10, 2025 at 08:23:46AM +, Bertrand Drouvot wrote: > On Fri, Jan 10, 2025 at 10:15:52AM +0300, Nazir Bilal Yavuz wrote: >> But I agree that having a macro has more benefits, >> also there already is a check for the 'io_op < IOOP_NUM_TYPES' in the >> pgstat_count_io_op() function. >

Docs for pg_basebackup needs v17 note for incremental backup

2025-01-13 Thread David G. Johnston
Hackers, Should the following paragraph in the docs be modified to point out minimum server version of v17 for incremental backups? pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and l

Re: WAL-logging facility for pgstats kinds

2025-01-13 Thread Michael Paquier
On Fri, Jan 10, 2025 at 01:46:53PM +0900, Michael Paquier wrote: > I'd rather use RecoveryInProgress() here even if XLogInsertAllowed() > is a synonym of that, minus the update of LocalXLogInsertAllowed for > the local process. I've applied v2-0002 for the new header as it is useful on its own. Re

Re: Issue with markers in isolation tester? Or not?

2025-01-13 Thread Noah Misch
On Mon, Jan 13, 2025 at 02:02:00AM +0100, Michail Nikolaev wrote: > While stabilizing tests for [0] I have noticed unclear (and confusing in my > opinion) behavior of markers in the isolation tester. > > I have attached a test with reproducer. > > There are two permutations in the test: > > perm

Re: Psql meta-command conninfo+

2025-01-13 Thread Alvaro Herrera
On 2025-Jan-10, Hunaid Sohail wrote: > IMO, we should continue with v35 and add all parameters from > PQparameterStatus, > as Sami suggested. The names themselves are informative enough. IMO we need more opinions. Maybe enough other people are not as unhappy about the three-table solution. --

Re: Changing shared_buffers without restart

2025-01-13 Thread Ashutosh Bapat
Hi Dmitry, On Tue, Dec 17, 2024 at 7:40 PM Ashutosh Bapat wrote: > > I could verify the memory mappings, their sizes etc. by looking at > /proc/PID/maps and /proc/PID/status but I did not find a way to verify > the amount of memory actually allocated and verify that it's actually > shrinking and

Re: Vacuum statistics

2025-01-13 Thread Andrei Zubkov
Hi Sami, Thank you for your attention to our patch and for your own work. On Sun, 2025-01-05 at 20:00 -0600, Sami Imseih wrote: > > You make valid points. I now think because track_vacuum_statistics is > optional, we should track total_time in 2 places. First place in the > new > view being prop

Re: Question about behavior of deletes with REPLICA IDENTITY NOTHING

2025-01-13 Thread Amit Kapila
On Mon, Jan 13, 2025 at 10:22 AM Robert Treat wrote: > > On Sun, Jan 12, 2025 at 11:00 PM Amit Kapila wrote: > > > > On Sat, Jan 11, 2025 at 7:28 PM Robert Treat wrote: > > > > > > Definitely couldn't hurt; Updated patch cleans that up a bit and > > > tweaks the link to alter table replica statu

Re: Documentation update of wal_retrieve_retry_interval to mention table sync worker

2025-01-13 Thread Shlok Kyal
On Mon, 13 Jan 2025 at 12:33, vignesh C wrote: > > On Mon, 6 Jan 2025 at 08:47, Peter Smith wrote: > > > > Hi Vignesh, > > > > Some review comments for your v2 patch. > > > > == > > doc/src/sgml/logical-replication.sgml > > > > AFAICT the only difference you made is changing: > > FROM "a spec

Re: Adjusting hash join memory limit to handle batch explosion

2025-01-13 Thread Tomas Vondra
On 1/13/25 17:32, Melanie Plageman wrote: > On Sat, Jan 11, 2025 at 7:42 PM Tomas Vondra wrote: >> >> I had a quiet evening yesterday, so I decided to take a stab at this and >> see how hard would it be, and how bad would the impact be. Attached is >> an experimental patch, doing the *bare* minimu

Re: Exists pull-up application with JoinExpr

2025-01-13 Thread Alena Rybakina
Hi! I have solved it. On 30.12.2024 11:24, Alena Rybakina wrote: Hi! Thank you for your interest to this subject! On 27.12.2024 15:53, Ilia Evdokimov wrote: Hi Alena, Thank you for your work on subqueries with JOIN. Have you considered the scenario where in subquery includes a qual like (t

Re: [PATCH] Add get_bytes() and set_bytes() functions

2025-01-13 Thread Dean Rasheed
On Mon, 13 Jan 2025 at 17:36, Aleksander Alekseev wrote: > > Besides fixing opr_sanity test I corrected error messages: > > -ERROR: bytea size 3 out of valid range, 0..2 > +ERROR: bytea out of valid range, ''..'\x' > "smallint out of range", "integer out of range" and "bigint out of range"

Re: Incorrect CHUNKHDRSZ in nodeAgg.c

2025-01-13 Thread Jeff Davis
On Fri, 2025-01-10 at 23:30 +1300, David Rowley wrote: > Since bump.c does not add headers to the palloc'd chunks, I think the > following code from hash_agg_entry_size() shouldn't be using > CHUNKHDRSZ anymore. Fixed. I also tried to account for the power-of-two allocations for the transition va

Re: explain analyze rows=%.0f

2025-01-13 Thread Ilia Evdokimov
On 11.01.2025 14:10, Ilia Evdokimov wrote: On 11.01.2025 12:15, Guillaume Lelarge wrote: Thanks for your patch, this looks like a very interesting feature that I'd like to see in a future release. It did a quick run: patch OK, make OK, make install OK, but make check fails quite a lot o

Re: New GUC autovacuum_max_threshold ?

2025-01-13 Thread Nathan Bossart
Here is a rebased version of the patch (commit ca9c6a5 adjusted the documentation for vacuum-related GUCs). -- nathan >From d78d621233734abc54d0273526983bf64b88e7ba Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 8 Jan 2025 13:11:52 -0600 Subject: [PATCH v5 1/1] Introduce autovacuum_max

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Andres Freund
Hi, On 2025-01-13 12:20:39 +, Bertrand Drouvot wrote: > On Fri, Jan 10, 2025 at 02:07:25PM -0500, Andres Freund wrote: > > There wasn't > > any code to treat receiving PMSIGNAL_XLOG_IS_SHUTDOWN in the wrong state as > > fatal. I think this needs to be treated the same way we treat not being

Re: pgsql: Consolidate docs for vacuum-related GUCs in new subsection

2025-01-13 Thread Melanie Plageman
On Mon, Jan 13, 2025 at 3:46 PM Alvaro Herrera wrote: > > On 2025-Jan-13, Melanie Plageman wrote: > > > Since I didn't hear back about this and I don't see an obvious > > alternative reorganization in guc_tables.c, I plan to just push the > > attached patch that updates only postgresql.conf.sample

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

2025-01-13 Thread Alena Rybakina
Hi! On 14.01.2025 00:46, Melanie Plageman wrote: On Thu, Jan 9, 2025 at 1:24 PM Andres Freund wrote: On 2025-01-07 15:46:26 -0500, Melanie Plageman wrote: For table storage options, those related to vacuum but not autovacuum are in the main StdRdOptions struct. Of those, some are overridden b

Re: pgbench error: (setshell) of script 0; execution of meta-command failed

2025-01-13 Thread Tom Lane
Nathan Bossart writes: > On Sat, Jan 11, 2025 at 02:04:13PM -0500, Tom Lane wrote: >> What I propose doing in the released branches is what's shown in >> the attached patch for v17: rename port/pqsignal.c's function to >> pqsignal_fe in frontend, but leave it as pqsignal in the backend. >> Leaving

Re: New GUC autovacuum_max_threshold ?

2025-01-13 Thread Sami Imseih
> Here is a rebased version of the patch (commit ca9c6a5 adjusted the > documentation for vacuum-related GUCs). I looked at the patch and have a few comments. I propose renaming the GUC from "autovacuum_max_threshold" to "autovacuum_vacuum_max_threshold" to clarify that it applies only to the vac

Re: Issue with markers in isolation tester? Or not?

2025-01-13 Thread Michail Nikolaev
Hello! In case if someone will look for a workaround - it is possible to use "notices N" mark.\ Instead of step before { SELECT injection_points_run('injection-points-wait-2'); } use something like: step before { DO $$ BEGIN PERFORM injection_points_run('injection

Re: pgsql: Consolidate docs for vacuum-related GUCs in new subsection

2025-01-13 Thread Alena Rybakina
Hi! On 14.01.2025 01:35, Melanie Plageman wrote: On Mon, Jan 13, 2025 at 3:46 PM Alvaro Herrera wrote: On 2025-Jan-13, Melanie Plageman wrote: Since I didn't hear back about this and I don't see an obvious alternative reorganization in guc_tables.c, I plan to just push the attached patch tha

Re: Virtual generated columns

2025-01-13 Thread Dean Rasheed
On Wed, 8 Jan 2025 at 16:14, Peter Eisentraut wrote: > > Here is a new patch version In expand_generated_columns_in_expr(): + RangeTblEntry *rte; + + rte = makeNode(RangeTblEntry); + rte->relid = RelationGetRelid(rel); + + node = expand_generated_columns_internal(node, re

SQLJSON: errmsg(" .. should ...") -> must

2025-01-13 Thread Alvaro Herrera
Hi There's a few recent SQL/JSON error messages in which we say something "should" be something else. We avoid this, so I think we shouldn't use it here either. (There's also wparser_def.c which uses "should" in that way, and I think it's there just because it's a dark unvisited corner that's be

Re: Pgoutput not capturing the generated columns

2025-01-13 Thread vignesh C
On Mon, 13 Jan 2025 at 14:57, Amit Kapila wrote: > > On Mon, Jan 13, 2025 at 5:25 AM Peter Smith wrote: > > > > Future -- there probably need to be further clarifications/emphasis to > > describe how the generated column replication feature only works for > > STORED generated columns (not VIRTUAL

Re: pgsql: Consolidate docs for vacuum-related GUCs in new subsection

2025-01-13 Thread Melanie Plageman
On Mon, Jan 13, 2025 at 10:22 AM Melanie Plageman wrote: > > Thanks to Álvaro for pointing this out. I didn't think of it. > > On Sun, Jan 12, 2025 at 2:21 PM Tom Lane wrote: > > > > Daniel Gustafsson writes: > > > On 11 Jan 2025, at 10:02, Alvaro Herrera wrote: > > >> and the GUC grouping in g

Re: connection establishment versus parallel workers

2025-01-13 Thread Nathan Bossart
On Thu, Dec 19, 2024 at 10:09:35AM -0600, Nathan Bossart wrote: > On Fri, Dec 13, 2024 at 03:56:00PM +1300, Thomas Munro wrote: >> 0001 patch is unchanged, 0002 patch sketches out a response to the >> observation a couple of paragraphs above. > > Both of these patches seem to improve matters quite

Re: InitControlFile misbehaving on graviton

2025-01-13 Thread Alexander Lakhin
Hello Christoph, 13.01.2025 21:04, Christoph Berg wrote: Bernd and I have been chasing a bug that happens when all of the following conditions are fulfilled: ... It smells like graviton's arm9 isn't as backwards compatible to arm8 as it should be. (But then I don't understand why disabling ope

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-13 Thread vignesh C
On Tue, 31 Dec 2024 at 10:15, Masahiko Sawada wrote: > > Hi all, > > Logical decoding (and logical replication) are available only when > wal_level = logical. As the documentation says[1], Using the 'logical' > level increases the WAL volume which could negatively affect the > performance. For tha

Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

2025-01-13 Thread Daniel Gustafsson
Circling back to wrap up this thread I retested with, and without, OpenSSL FIPS and tweaked the docs and code a little to ensure it detects the right function to use. The attached is what I propose we go ahead with. -- Daniel Gustafsson v7-0001-pgcrypto-Make-it-possible-to-disable-built-in-cry

Re: [PATCH] Hex-coding optimizations using SVE on ARM.

2025-01-13 Thread chiranmoy.bhattacha...@fujitsu.com
On Fri, Jan 10, 2025 at 09:38:14AM -0600, Nathan Bossart wrote: > Do you mean that the auto-vectorization worked and you observed no > performance improvement, or the auto-vectorization had no effect on the > code generated? Auto-vectorization is working now with the following addition on Graviton

Re: POC: track vacuum/analyze cumulative time per relation

2025-01-13 Thread Bertrand Drouvot
Hi, On Fri, Jan 10, 2025 at 04:26:05PM -0600, Sami Imseih wrote: > { > /* time is already in msec, just convert to double for presentation */ > PG_RETURN_FLOAT8((double) > pgstat_fetch_stat_checkpointer()->write_time); > } > > > + > > + PgStat_Counter total_vacuum_time; /* user initia

Re: pgsql: Consolidate docs for vacuum-related GUCs in new subsection

2025-01-13 Thread Melanie Plageman
Thanks to Álvaro for pointing this out. I didn't think of it. On Sun, Jan 12, 2025 at 2:21 PM Tom Lane wrote: > > Daniel Gustafsson writes: > > On 11 Jan 2025, at 10:02, Alvaro Herrera wrote: > >> and the GUC grouping in guc_tables.c/h? > > > I don't know what our policy around this is, and may

Re: [PATCH] SVE popcount support

2025-01-13 Thread Malladi, Rama
Here is the updated patch using pg_attribute_target("arch=armv8-a+sve") to compile the arch-specific function instead of using compiler flags. --- This looks good. Thanks Chiranmoy and team. Can you address any other feedback from Nathan or others here? Then we can pursue further reviews an

Re: pgbench error: (setshell) of script 0; execution of meta-command failed

2025-01-13 Thread Nathan Bossart
On Sat, Jan 11, 2025 at 02:04:13PM -0500, Tom Lane wrote: > After studying this more, I think what we should do in HEAD is > even more aggressive: let's make the real name of port/pqsignal.c's > function be either pqsignal_fe in frontend, or pqsignal_be in backend. > This positively ensures that th

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-13 Thread Ashutosh Bapat
On Mon, Jan 13, 2025 at 2:52 PM vignesh C wrote: > > I felt that the only disadvantage with this approach is that we > currently wait for all in-progress transactions to complete before > enabling logical decoding. If a long-running transaction exists and > the session enabling logical decoding fa

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Amit Kapila
On Tue, Jan 7, 2025 at 7:22 AM Peter Smith wrote: > > == > src/include/replication/reorderbuffer.h > > 6. > #define RBTXN_PREPARE 0x0040 > #define RBTXN_SKIPPED_PREPARE 0x0080 > #define RBTXN_HAS_STREAMABLE_CHANGE 0x0100 > +#define RBTXN_SENT_PREPARE 0x0200 > +#define RBTXN_I

Re: Psql meta-command conninfo+

2025-01-13 Thread Dean Rasheed
On Mon, 13 Jan 2025 at 08:44, Alvaro Herrera wrote: > > > IMO, we should continue with v35 and add all parameters from > > PQparameterStatus, > > as Sami suggested. The names themselves are informative enough. > > IMO we need more opinions. Maybe enough other people are not as unhappy > about the

Accept recovery conflict interrupt on blocked writing

2025-01-13 Thread Anthonin Bonnefoy
Hi, I have replicas that have regular transient bursts of replay lag (>5 minutes). The events have the following symptoms: - Replicas are using physical replication slot and hot_standby_feedback - The lag recovers by itself after at most 15 minutes - During the same timeframe, there's a query stuc

Re: why there is not VACUUM FULL CONCURRENTLY?

2025-01-13 Thread Michael Banck
Hi, On Sat, Jan 11, 2025 at 09:01:54AM -0500, Andrew Dunstan wrote: > On 2025-01-09 Th 8:35 AM, Alvaro Herrera wrote: > > Maybe we should have a new toplevel command. Some ideas that have been > > thrown around: > > > > - RETABLE (it's like REINDEX, but for tables) > > - ALTER TABLE SQUEEZE > >

Re: Pgoutput not capturing the generated columns

2025-01-13 Thread Amit Kapila
On Mon, Jan 13, 2025 at 5:25 AM Peter Smith wrote: > > Future -- there probably need to be further clarifications/emphasis to > describe how the generated column replication feature only works for > STORED generated columns (not VIRTUAL ones), but IMO it is better to > address that separately *aft

Re: Modern SHA2- based password hashes for pgcrypto

2025-01-13 Thread Alvaro Herrera
Hello I was passing by and I noticed that this needs badly pgindent, and some comments are enumerations that would lose formatting once through it. For example, this would happen which is not good: /* -* 1. Start digest A -* 2. Add the password string to digest A -* 3. Add the sal

Re: [PATCH] Hex-coding optimizations using SVE on ARM.

2025-01-13 Thread Nathan Bossart
On Mon, Jan 13, 2025 at 03:48:49PM +, chiranmoy.bhattacha...@fujitsu.com wrote: > There is a 30% improvement using auto-vectorization. It might be worth enabling auto-vectorization independently of any patches that use intrinsics, then. > Currently, it is assumed that all aarch64 machine sup

Re: per backend I/O statistics

2025-01-13 Thread Michael Paquier
On Fri, Jan 10, 2025 at 01:56:01PM +, Bertrand Drouvot wrote: > So now, I think that it is not a good idea for the per backend stats to > behave differently than the other stats kinds. I think we should get rid of > the "snapshot" for all of them or for none of them. Reading the above > comme

Re: Conflict Detection and Resolution

2025-01-13 Thread Amit Kapila
On Mon, Jan 13, 2025 at 11:01 PM Michail Nikolaev wrote: > > Just curious - are any plans to continue to work on this project? > Yes, but our intention is to reach a consensus/conclusion about how to deal with update_delete conflicts. The topic is discussed in the CF entry [1]. If you want to he

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Andres Freund
Hi, On 2025-01-13 12:20:39 +, Bertrand Drouvot wrote: > > We have > > multiple copies of code to go to FatalError, that doesn't seem great. > > + * FIXME: This should probably not have a copy fo the code to > + * transition into FatalError state. > + */ > + ereport(LOG, > + (errm

Re: [PATCH] Hex-coding optimizations using SVE on ARM.

2025-01-13 Thread Tom Lane
John Naylor writes: > We can do about as well simply by changing the nibble lookup to a byte > lookup, which works on every compiler and architecture: I didn't attempt to verify your patch, but I do prefer addressing this issue in a machine-independent fashion. I also like the brevity of the pat

RE: COPY performance on Windows

2025-01-13 Thread Ryohei Takahashi (Fujitsu)
Hi, I did more investigation on the COPY performance on Windows. By using community distributed binaries, the COPY performance of PG16.6 and PG17.0 is worse than PG16.4. However, by using the binaries build by myself, there are no difference. So, it is not the problem about the source code afte

Re: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-13 Thread Shubham Khanna
On Fri, Jan 10, 2025 at 6:56 AM Peter Smith wrote: > > Hi Shubham, > > Some review comments for patch v8=0001. > > == > Commit message. > > 1. > By ensuring 'max_slot_wal_keep_size' is -1, this patch prevents the potential > deletion of required WAL files on the publisher that could disrupt lo

RE: COPY performance on Windows

2025-01-13 Thread Ryohei Takahashi (Fujitsu)
Hi, I compared the performance between 94f9c08 (previous commit of 82a4edabd2) and 82a4edabd2 with nclient = 8. The performance is degraded by 8%. * 94f9c08 (previous commit of 82a4edabd2) 75sec * 82a4edabd2 81sec I looked the pg_stat_io view after COPY completed. * 94f9c08 (previous com

Re: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-13 Thread Shubham Khanna
On Fri, Jan 10, 2025 at 8:21 AM Peter Smith wrote: > > On Mon, Jan 6, 2025 at 1:29 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Shubham, > > > > Thanks for creating a patch. I have one comment about it. > > > > check_publisher() assumed that the SQL function > > `pg_catalog.current_setting('

Re: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-13 Thread Shubham Khanna
On Tue, Jan 14, 2025 at 10:10 AM Peter Smith wrote: > > Hi Shubham, > > In a previous review [1, comment #3] I was complaining about the lack > of output emitted by the pg_log_debug() because I wanted to see some > LOG evidence of the bad value of max_slot_wal_keep_size. > > FYI, I think that myst

Re: pg_createsubscriber TAP test wrapping makes command options hard to read.

2025-01-13 Thread Peter Smith
On Fri, Dec 13, 2024 at 12:17 AM Dagfinn Ilmari Mannsåker wrote: > > Dagfinn Ilmari Mannsåker writes: > > > Peter Smith writes: > > > >> On Thu, Dec 12, 2024 at 2:53 PM Michael Paquier > >> wrote: > >> ... > >> > >>> > So, AFAICT I can workaround the perltidy wrapping just by putting all > >>>

Re: Change COPY ... ON_ERROR ignore to ON_ERROR ignore_row

2025-01-13 Thread jian he
On Sat, Jan 11, 2025 at 5:54 PM Kirill Reshke wrote: > > On Fri, 10 Jan 2025 at 11:38, jian he wrote: > > I think there are three remaining issues that may need more attention > > 1. > > Table 27.42. pg_stat_progress_copy View > > (pg_stat_progress_copy) > > column pg_stat_progress_copy.tuples_sk

Re: Possible integer overflow in bringetbitmap()

2025-01-13 Thread Michael Paquier
On Fri, Jan 10, 2025 at 11:22:37AM -0800, James Hunter wrote: > Attached the proposed one-line fix. Thanks, applied and backpatched down to v13. While on it, I've checked all the other amgetbitmap callbacks and they all rely on an int64 for the result they return. So these parts are OK. -- Micha

Re: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-13 Thread Peter Smith
Hi Shubham, In a previous review [1, comment #3] I was complaining about the lack of output emitted by the pg_log_debug() because I wanted to see some LOG evidence of the bad value of max_slot_wal_keep_size. FYI, I think that mystery has finally been solved. AFAIK 2 separate problems were contri

Re: [PATCH] Hex-coding optimizations using SVE on ARM.

2025-01-13 Thread Michael Paquier
On Tue, Jan 14, 2025 at 12:27:30PM +0700, John Naylor wrote: > We can do about as well simply by changing the nibble lookup to a byte > lookup, which works on every compiler and architecture: > > select hex_encode_test(100, 1024); > master: > Time: 1158.700 ms > v2: > Time: 777.443 ms > > If

Re: [PATCH] Hex-coding optimizations using SVE on ARM.

2025-01-13 Thread John Naylor
On Sat, Jan 11, 2025 at 3:46 AM Nathan Bossart wrote: > > I was able to get auto-vectorization to take effect on Apple clang 16 with > the following addition to src/backend/utils/adt/Makefile: > > encode.o: CFLAGS += ${CFLAGS_VECTORIZE} -mllvm -force-vector-width=8 > > This gave the follow

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Amit Kapila
On Tue, Jan 14, 2025 at 7:32 AM Peter Smith wrote: > > Hi Sawada-San. > > Some review comments for patch v13-0002. > > == > > I think the v12 ambiguity of RBTXN_PREPARE versus RBTXN_SENT_PREPARE > was mostly addressed already by the improved comments for the macros > in patch 0001. > > Meanwhi

Prevent an error on attaching/creating a DSM/DSA from an interrupt handler.

2025-01-13 Thread Rahila Syed
Hi, If a DSM is created or attached from an interrupt handler while a transaction is being rolled back, it may result in the following error. "ResourceOwnerEnlarge called after release started" This was found during the testing of Enhancing Memory Context Reporting feature by Fujii Masao [1]. I p

Re: Psql meta-command conninfo+

2025-01-13 Thread Hunaid Sohail
Hi, On Mon, Jan 13, 2025 at 4:12 PM Dean Rasheed wrote: > I don't like the 3-table format either. I think it should be a single > table. > > The trouble with v35 is that it produces 1 row with lots of columns, > which is pretty unreadable unless expanded mode is used. So I think we > should just

Re: Conflict detection for update_deleted in logical replication

2025-01-13 Thread Amit Kapila
On Tue, Jan 14, 2025 at 7:14 AM Masahiko Sawada wrote: > > On Sun, Jan 12, 2025 at 10:36 PM Amit Kapila wrote: > > > > I don't think we can avoid accumulating garbage especially when the > > workload on the publisher is more. Consider the current case being > > discussed, on the publisher, we hav

Re: Conflict Detection and Resolution

2025-01-13 Thread Michail Nikolaev
Hello, everyone! Just curious - are any plans to continue to work on this project? Best regards, Mikhail.

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Masahiko Sawada
On Mon, Jan 13, 2025 at 3:07 AM Amit Kapila wrote: > > On Tue, Jan 7, 2025 at 7:22 AM Peter Smith wrote: > > > > == > > src/include/replication/reorderbuffer.h > > > > 6. > > #define RBTXN_PREPARE 0x0040 > > #define RBTXN_SKIPPED_PREPARE 0x0080 > > #define RBTXN_HAS_STREAMAB

Re: Skip collecting decoded changes of already-aborted transactions

2025-01-13 Thread Masahiko Sawada
On Mon, Jan 6, 2025 at 5:52 PM Peter Smith wrote: > > Hi Sawada-San. > > Here are some review comments for the patch v12-0001. Thank you for reviewing the patch! > > == > .../replication/logical/reorderbuffer.c > > ReorderBufferCheckAndTruncateAbortedTXN: > > 1. > +/* > + * Check the transac

InitControlFile misbehaving on graviton

2025-01-13 Thread Christoph Berg
Bernd and I have been chasing a bug that happens when all of the following conditions are fulfilled: * PG 15..18 (older PGs are ok) * gcc 14.2 on Debian unstable/testing (older Debians and Ubuntus are ok) * arm64 running on graviton (AWS EC2 c8g.2xlarge, ok on different arm64 host) * -O2 (ok with

Re: Adjusting hash join memory limit to handle batch explosion

2025-01-13 Thread Melanie Plageman
On Sat, Jan 11, 2025 at 7:42 PM Tomas Vondra wrote: > > I had a quiet evening yesterday, so I decided to take a stab at this and > see how hard would it be, and how bad would the impact be. Attached is > an experimental patch, doing the *bare* minimum for a simple query: > > 1) It defines a limit

Re: [PATCH] Add get_bytes() and set_bytes() functions

2025-01-13 Thread Aleksander Alekseev
Hi Michael, > v5-0001 includes the following output: > > --- a/src/test/regress/expected/opr_sanity.out > +++ b/src/test/regress/expected/opr_sanity.out > @@ -126,9 +126,12 @@ WHERE p1.oid < p2.oid AND > p1.proretset != p2.proretset OR > p1.provolatile != p2.provolatile OR > p1.p

Re: [PATCH] Add get_bytes() and set_bytes() functions

2025-01-13 Thread Alvaro Herrera
On 2025-Jan-13, Dean Rasheed wrote: > On Mon, 13 Jan 2025 at 17:36, Aleksander Alekseev > wrote: > > > > Besides fixing opr_sanity test I corrected error messages: > > > > -ERROR: bytea size 3 out of valid range, 0..2 > > +ERROR: bytea out of valid range, ''..'\x' > > "smallint out of rang

Re: InitControlFile misbehaving on graviton

2025-01-13 Thread Matthias van de Meent
On Mon, 13 Jan 2025 at 20:04, Christoph Berg wrote: > > Bernd and I have been chasing a bug that happens when all of the > following conditions are fulfilled: > > * PG 15..18 (older PGs are ok) > * gcc 14.2 on Debian unstable/testing (older Debians and Ubuntus are ok) > * arm64 running on graviton

Re: connection establishment versus parallel workers

2025-01-13 Thread Thomas Munro
On Tue, Jan 14, 2025 at 8:50 AM Nathan Bossart wrote: > On Thu, Dec 19, 2024 at 10:09:35AM -0600, Nathan Bossart wrote: > > On Fri, Dec 13, 2024 at 03:56:00PM +1300, Thomas Munro wrote: > >> 0001 patch is unchanged, 0002 patch sketches out a response to the > >> observation a couple of paragraphs

Re: AIO v2.2

2025-01-13 Thread Robert Haas
On Wed, Jan 8, 2025 at 7:26 PM Andres Freund wrote: > 1) Shared memory representation of an IO, for the AIO subsystem internally > >Currently: PgAioHandle > > 2) A way for the issuer of an IO to reference 1), to attach information to the >IO > >Currently: PgAioHandle* > > 3) A way for