Re: Logical replication timeout

2024-12-31 Thread Shlok Kyal
On Fri, 27 Dec 2024 at 09:41, vignesh C wrote: > > On Wed, 25 Dec 2024 at 13:55, Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Marc, > > > > > Thanks again for this new patch. > > > > > > Unfortunately it does not compile (17.2 source): > > > > Right, because of the reason I posted [1]. > > > > I

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Tom Lane
Thomas Munro writes: > NFS is at least supposed to tell the client that its cookie has been > invalidated with a cookie-invalidation-cookie called cookieverf. But > there isn't any specified way to recover. FreeBSD's client looks like > it might try to, but I'm not sure if that Linux's server ev

Re: Conflict detection for update_deleted in logical replication

2024-12-31 Thread Amit Kapila
On Thu, Dec 19, 2024 at 4:34 PM Zhijie Hou (Fujitsu) wrote: > > On Sunday, December 15, 2024 9:39 AM Amit Kapila > wrote: > > > > > > > 5. The apply worker needs to at least twice get the publisher status > > message to > > advance oldest_nonremovable_xid once. It then uses the remote_lsn of th

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Thomas Munro
On Wed, Jan 1, 2025 at 1:20 PM Kenneth Marshall wrote: > On Tue, Dec 31, 2024 at 06:58:14PM -0500, Tom Lane wrote: > > Larry Rosenman writes: > > > On 12/31/2024 5:37 pm, Tom Lane wrote: > > >> Do you know what its underlying file system is? > > > > > btrfs > Maybe there are some btrfs or nfs op

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Tom Lane
Larry Rosenman writes: > On 12/31/2024 5:37 pm, Tom Lane wrote: >> Do you know what its underlying file system is? > btrfs OK. My test was with XFS underneath the NFS service. regards, tom lane

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Larry Rosenman
On 12/31/2024 5:37 pm, Tom Lane wrote: Larry Rosenman writes: On 12/31/2024 5:24 pm, Thomas Munro wrote: The implementation-specific cookie scheme for encoding a sort of cursor position across readdir() calls has various different problems on various different OSes, NFS implementations and und

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-31 Thread Tomas Vondra
On 12/31/24 21:46, Jim Nasby wrote: > On Dec 30, 2024, at 7:05 PM, James Hunter wrote: >> >> On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: >>> >>> IMHO none of this will be very sane until we actually have cluster-level >>> limits. One sudden burst in active connections and you still OOM the

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Tom Lane
Larry Rosenman writes: > On 12/31/2024 5:24 pm, Thomas Munro wrote: >> The implementation-specific cookie scheme for encoding a sort of >> cursor position across readdir() calls has various different problems >> on various different OSes, NFS implementations and underlying local >> file systems (I

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Larry Rosenman
On 12/31/2024 5:24 pm, Thomas Munro wrote: On Wed, Jan 1, 2025 at 11:44 AM Tom Lane wrote: Larry Rosenman writes: > On 12/31/2024 12:22 pm, Larry Rosenman wrote: >> When I try to drop a database, PostgreSQL leaves files in the directory >> and does not even try to delete them. >> PostgreSQL 16

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Thomas Munro
On Wed, Jan 1, 2025 at 11:44 AM Tom Lane wrote: > Larry Rosenman writes: > > On 12/31/2024 12:22 pm, Larry Rosenman wrote: > >> When I try to drop a database, PostgreSQL leaves files in the directory > >> and does not even try to delete them. > >> PostgreSQL 16.6, FreeBSD 14.2, PGDATA mounted NFS

Adjusting hash join memory limit to handle batch explosion

2024-12-31 Thread Tomas Vondra
Hi, I've been once again reminded of the batch explosion issue in hashjoin, due to how it enforces the memory limit. This resurfaces every now and then, when a used gets strange OOM issues - see for example these threads from ~2019 for an example, and even some patches: [1] [2] [3] Let me restart

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Tom Lane
Larry Rosenman writes: > On 12/31/2024 12:22 pm, Larry Rosenman wrote: >> When I try to drop a database, PostgreSQL leaves files in the directory >> and does not even try to delete them. >> PostgreSQL 16.6, FreeBSD 14.2, PGDATA mounted NFS from UGreen NAS. FWIW, I couldn't reproduce such a probl

Re: add vacuum starttime columns

2024-12-31 Thread Sami Imseih
worth mentioning here that there is discussion for tracking vacuum history in this recent thread [1], and this includes both start_time and end_time of the vacuum from what I can tell. postgres=# select * from vacuum_history ; start_time | end_time | dbid | relid | is_autovacuum | . It will

Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)

2024-12-31 Thread Jim Nasby
On Dec 31, 2024, at 9:20 AM, Tomas Vondra wrote: > >> Speaking of retention, it would be nice if this feature allowed users to >> DELETE from the view that presented the data. That would allow for any >> kind of custom config that someone could dream up. > > I really don't intend / want to do th

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Bruce Momjian
On Tue, Dec 31, 2024 at 01:47:19PM -0700, David G. Johnston wrote: > On Tue, Dec 31, 2024 at 1:30 PM Bruce Momjian wrote: > > On Tue, Dec 31, 2024 at 03:19:25PM -0500, Roberto C. Sánchez wrote: > > > My thinking was "ask once, bump the thread once after 2 or 3 weeks just > > in case

Re: Document How Commit Handles Aborted Transactions

2024-12-31 Thread David G. Johnston
Thoughts? On Fri, Dec 20, 2024 at 9:02 AM David G. Johnston < david.g.johns...@gmail.com> wrote: > Hi, > > The commit reference page lacks an "Outputs" section even though it is > capable of outputting both "COMMIT" and "ROLLBACK". > > The attached adds this section, describes when each applies,

Re: Improve documentation regarding custom settings, placeholders, and the administrative functions

2024-12-31 Thread David G. Johnston
Thoughts? Anyone? On Sat, Oct 19, 2024 at 1:11 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > Hey! > > Motivated by recent user complaints regarding our documentation of > behavior in this area I propose the following to shore things up a bit. > > There may be other places that need

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread David G. Johnston
On Tue, Dec 31, 2024 at 1:30 PM Bruce Momjian wrote: > On Tue, Dec 31, 2024 at 03:19:25PM -0500, Roberto C. Sánchez wrote: > > > My thinking was "ask once, bump the thread once after 2 or 3 weeks just > > in case it got lost in the noise (this is a busy list), and after that > > let the matter re

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2024-12-31 Thread Jim Nasby
On Dec 30, 2024, at 7:05 PM, James Hunter wrote: > > On Sat, Dec 28, 2024 at 11:24 PM Jim Nasby wrote: >> >> IMHO none of this will be very sane until we actually have cluster-level >> limits. One sudden burst in active connections and you still OOM the >> instance. > > Fwiw, PG does support

Re: Strange issue with NFS mounted PGDATA on ugreen NAS

2024-12-31 Thread Larry Rosenman
On 12/31/2024 12:22 pm, Larry Rosenman wrote: When I try to drop a database, PostgreSQL leaves files in the directory and does not even try to delete them. PostgreSQL 16.6, FreeBSD 14.2, PGDATA mounted NFS from UGreen NAS. Truss of the create/delete attached. It does NOT seem to happen PG < 1

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Bruce Momjian
On Tue, Dec 31, 2024 at 03:19:25PM -0500, Roberto C. Sánchez wrote: > On Tue, Dec 31, 2024 at 02:46:37PM -0500, Bruce Momjian wrote: > > > > Good question. In a way, if the person who made the change sees your > > request and can answer it easily, it makes sense to ask. However, I > > don't know

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Roberto C . Sánchez
On Tue, Dec 31, 2024 at 02:46:37PM -0500, Bruce Momjian wrote: > > Good question. In a way, if the person who made the change sees your > request and can answer it easily, it makes sense to ask. However, I > don't know the odds of that happening. > > I would say ask, but don't take it personall

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Bruce Momjian
On Tue, Dec 31, 2024 at 12:14:37PM -0500, Roberto C. Sánchez wrote: > On Tue, Dec 31, 2024 at 11:50:45AM -0500, Bruce Momjian wrote: > > On Tue, Dec 31, 2024 at 10:50:10AM +0100, Christoph Berg wrote: > > > > Maybe, if we were doing an only-critical-fixes LTS release series, > > > > it'd be easier

Re: Non-text mode for pg_dumpall

2024-12-31 Thread Mahendra Singh Thalor
Hi all, With the help of Andrew and Dilip Kumar, I made a poc patch to dump all the databases in archive format and then restore them using pg_restore. Brief about the patch: new option to pg_dumpall: -F, --format=d|p (directory|plain) output file format (directory, plain text (default)) Ex: ./pg

Re: add vacuum starttime columns

2024-12-31 Thread Sami Imseih
> Also, the log files give you historical overview that the pg_stat views > simply cannot provide, > in addition to the actual details of what was vacuumed and why. Logfiles have the ability to provide more details and they have their place. However, one must also think about how much logging th

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Roberto C . Sánchez
On Tue, Dec 31, 2024 at 11:50:45AM -0500, Bruce Momjian wrote: > On Tue, Dec 31, 2024 at 10:50:10AM +0100, Christoph Berg wrote: > > > Maybe, if we were doing an only-critical-fixes LTS release series, > > > it'd be easier for downstream outfits to consume that instead of > > > cherry-picking secur

Re: Add XMLNamespaces to XMLElement

2024-12-31 Thread Pavel Stehule
čt 26. 12. 2024 v 14:46 odesílatel Jim Jones napsal: > Hi Umar, hi Pavel, > > Thanks for taking a look at this patch! > > On 26.12.24 05:15, Umar Hayat wrote: > > Hi, > > +1 for the enhancement. > > > > I haven't compiled and reviewed the full patch yet, please see a few > > comments from my side

Re: Proposal to Enable/Disable Index using ALTER INDEX (with patch)

2024-12-31 Thread Shayon Mukherjee
On Mon, Dec 30, 2024 at 3:48 PM Michail Nikolaev wrote: > Hello! > > One more thing (maybe I missed it in the patch, but anyway) - should we > add some migration to ensure what old databases will get enabled=true by > default after upgrade? > Hi! Thank you! I tested this by manually upgrading (

Re: Proposal to Enable/Disable Index using ALTER INDEX (with patch)

2024-12-31 Thread Shayon Mukherjee
On Mon, Dec 30, 2024 at 2:56 PM Sami Imseih wrote: > > Rebased with the latest master as well. > > Hi, > > This is a great, long needed feature. Thanks for doing this. > > I am late to this thread, but I took a look at the current patch > and have some comments as I continue to look. > > Thank yo

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Bruce Momjian
On Tue, Dec 31, 2024 at 10:50:10AM +0100, Christoph Berg wrote: > > Maybe, if we were doing an only-critical-fixes LTS release series, > > it'd be easier for downstream outfits to consume that instead of > > cherry-picking security fixes. I'm just speculating though. > > It's entirely possible tha

Re: Modern SHA2- based password hashes for pgcrypto

2024-12-31 Thread Bernd Helmle
Am Dienstag, dem 31.12.2024 um 17:06 +0100 schrieb Bernd Helmle: > I am going to add this patch to the upcoming january commitfest for > initial review. I see cfbot fails Debian Bookworm with autoconf and on macOS with meson. I will look into it.

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Bruce Momjian
On Tue, Dec 31, 2024 at 07:36:23AM -0500, Andrew Dunstan wrote: > On 2024-12-30 Mo 9:00 PM, Tom Lane wrote: > > Yeah, I can't see extending it, at least not under our current theory > > of back-patching (nearly) every bug fix to all supported branches. > > Could there be an intermediate state where

Re: Proposal: Progressive explain

2024-12-31 Thread Rafael Thofehrn Castro
Thanks Greg, Sami and Jian for the feedback so far. > Maybe track_explain instead? In the spirit of track_activity. That was the original name, and all other GUCs were following the track_activity_* logic. Changed to the name of the feature after discussion with colleagues at EDB. This is definit

Re: add vacuum starttime columns

2024-12-31 Thread Greg Sabino Mullane
On Tue, Dec 31, 2024 at 2:33 AM wenhui qiu wrote: > Of course, to observe the duration of vacuum operations, we can configure > the log_autovacuum_min_durationparameter, but if there are many tables in > the database, the vacuum entries in the logs might be quite numerous, > making it difficult t

Re: using index to speedup add not null constraints to a table

2024-12-31 Thread jian he
hi. In v1 I didn't do the `git add` for newly created isolation test related files. so the cfbot for isolation tests failed. v1 with index: create index t_idx_ab on t(a,b); we cannot fast add a not-null constraint for column b. with the attached v2 patch, now we can do that. v2, isolation test a

Modern SHA2- based password hashes for pgcrypto

2024-12-31 Thread Bernd Helmle
Hi Hackers, Some of you might already arrived 2025, so first a Happy New Year to everyone already there ;) Please find attached a patch to pgcrypto to add modern SHA-2 based password hashes sha256crypt (256 bit) and sha512crypt (512 bit) for crypt() and gen_salt() respectively. This is compatible

Re: add vacuum starttime columns

2024-12-31 Thread Sami Imseih
> However, if the log_autovacuum_min_duration parameter is not configured (as > I’ve encountered in many cases where this parameter is either not set or has > an inappropriate value), we cannot determine the trigger time and duration of > the operation. log_autovacuum_min_duration logs when the

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Tom Lane
Andrew Dunstan writes: > On 2024-12-30 Mo 9:00 PM, Tom Lane wrote: >> Could there be an intermediate state where older branches get only >> "critical" fixes? (Security and data-loss bugs only, IMV.) Another >> not-necessarily-exclusive idea is to designate only certain branches >> as LTS. We co

Re: SQLFunctionCache and generic plans

2024-12-31 Thread Pavel Stehule
Hi út 31. 12. 2024 v 16:36 odesílatel Alexander Pyhalov < a.pyha...@postgrespro.ru> napsal: > Hi. > > >> What should we do with "pre-parsed" SQL functions (when prosrc is > >> empty)? How should we create cached plans when we don't have raw > >> parsetrees? > >> Currently we can create cached pla

Re: FileFallocate misbehaving on XFS

2024-12-31 Thread Andres Freund
Hi, On 2024-12-19 17:47:13 +1100, Michael Harris wrote: > I have attached a file containing all the errors I collected. The > error is happening pretty regularly - over 400 times in a ~6 hour > period. The number of blocks being extended varies from ~9 to ~15, and > the statfs result shows plenty

Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)

2024-12-31 Thread Tomas Vondra
On 12/30/24 22:40, Jim Nasby wrote: > On Dec 25, 2024, at 11:25 AM, Tomas Vondra wrote: >> But maybe it'd be possible to just write the entries to a file. We don't >> need random access to past entries (unlike e.g. pg_stat_statements), and >> people won't query that very often either. > > Assumin

Re: PoC: history of recent vacuum/checkpoint runs (using new hooks)

2024-12-31 Thread Tomas Vondra
On 12/31/24 02:06, Michael Paquier wrote: > On Sat, Dec 28, 2024 at 02:25:16AM +0100, Tomas Vondra wrote: >> And the more I think about it the more I'm convinced we don't need to >> keep the data about past runs in memory, a file should be enough (except >> maybe for a small buffer). That would mea

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Andrew Dunstan
On 2024-12-30 Mo 9:00 PM, Tom Lane wrote: Michael Paquier writes: On Mon, Dec 30, 2024 at 04:58:26PM -0500, Bruce Momjian wrote: Is our five-year insufficient? FWIW, I'm already on the side that five-year support is quite good and I'd side with not extending that, even argue about reducing

Re: Test to dump and restore objects left behind by regression

2024-12-31 Thread Ashutosh Bapat
On Fri, Dec 27, 2024 at 6:17 PM Daniel Gustafsson wrote: > > > On 20 Dec 2024, at 11:01, Ashutosh Bapat > > wrote: > > On Wed, Dec 18, 2024 at 7:39 PM Daniel Gustafsson wrote: > >> > >>> On 18 Dec 2024, at 12:28, Ashutosh Bapat > >>> wrote: > > >> + if ( $ENV{PG_TEST_EXTRA} > >> + &

Re: WAL-logging facility for pgstats kinds

2024-12-31 Thread Bertrand Drouvot
Hi, On Tue, Dec 31, 2024 at 09:52:31AM +0900, Michael Paquier wrote: > On Fri, Dec 27, 2024 at 12:32:25PM +0900, Michael Paquier wrote: > > For clarity, the patch set has been split into several pieces, I hope > > rather edible: > > - 0001, a fix I've posted on a different thread [1], used in patc

Re: [PATCH] New predefined role pg_manage_extensions

2024-12-31 Thread Michael Banck
Hi, first, sorry for the late reply :-/ On Mon, Nov 18, 2024 at 11:26:40AM +0500, Kirill Reshke wrote: > On Fri, 1 Nov 2024 at 02:47, Michael Banck wrote: > > Even though there has not been a lot of discussion on this, here is a > > rebased patch. I have also added it to the upcoming commitfest

Re: Backport of CVE-2024-10978 fix to older pgsql versions (11, 9.6, and 9.4)

2024-12-31 Thread Christoph Berg
Re: Michael Paquier > > Is our five-year insufficient? > > FWIW, I'm already on the side that five-year support is quite good and > I'd side with not extending that, even argue about reducing it > (anti-tomato armor is now on). Backporting patches across up to 7 > branches can be really tedious d

RE: AIX support

2024-12-31 Thread Srirama Kucherlapati
> Meson appears to have AIX support, so it's worth trying out. Right, meson is supported on AIX, we have meson in our AIX toolbox. We are trying to build with meson as well. I shall update you. BTW, I would like to wish the entire Postgres community a Happy new year. Warm regards, Sriram.

Re: ERROR: corrupt MVNDistinct entry

2024-12-31 Thread Richard Guo
On Fri, Dec 27, 2024 at 1:16 PM Richard Guo wrote: > I'm wondering if we also need to strip out the nullingrels from the > expression in examine_variable(). I tried doing so and noticed a plan > diff in regression test join.sql. Here is an updated patch that implements this change, and also move