Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Daniil Davydov
On Thu, May 1, 2025 at 8:03 AM Masahiko Sawada wrote: > > As I understand it, we initially disabled parallel vacuum for > autovacuum because their objectives are somewhat contradictory. > Parallel vacuum aims to accelerate the process by utilizing additional > resources, while autovacuum is design

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Tom Lane
Jacob Champion writes: > On Fri, May 2, 2025 at 10:35 AM Tom Lane wrote: >> FWIW, on my Mac a meson build from HEAD goes through fine, with or >> without this patch. I'm getting openssl and libcurl from MacPorts >> not Homebrew, but I don't know why that would make any difference. > Do your an

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Nathan Bossart
On Fri, May 02, 2025 at 10:42:25AM -0700, Jacob Champion wrote: > On Fri, May 2, 2025 at 10:35 AM Tom Lane wrote: >> FWIW, on my Mac a meson build from HEAD goes through fine, with or >> without this patch. I'm getting openssl and libcurl from MacPorts >> not Homebrew, but I don't know why that w

Assert("vacrel->eager_scan_remaining_successes > 0")

2025-05-02 Thread Masahiko Sawada
Hi, I hit the assertion failure in the subject line with the following script: create table t (a int) with (autovacuum_enabled = off); insert into t select generate_series(1, 1_000_000); vacuum t; insert into t select generate_series(1, 1_000_000); set vacuum_freeze_min_age to 0; vacuum t; When

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

2025-05-02 Thread Peter Geoghegan
On Fri, May 2, 2025 at 2:22 PM Peter Geoghegan wrote: > A slight variant of my fuzzing Python script did in fact go on to > detect a couple of bugs. > > I'm attaching a compressed SQL file with repros for 2 different bugs. > The first bug was independently detected by some kind of fuzzing > perfor

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Tom Lane
Jacob Champion writes: > On Fri, May 2, 2025 at 11:26 AM Tom Lane wrote: >> Apropos of that: our fine manual claims that option is spelled >> --auto_features, but that fails for me. Is that a typo in our >> manual, or do some meson versions accept the underscore? > --auto_features doesn't work

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 11:56 AM Jacob Champion wrote: > -I/opt/homebrew/Cellar/openssl@3/3.5.0/include Except it _is_ right there. Oh, ha -- I'm not using Homebrew's Curl in this minimal build. Looks like it's coming from the sysroot. % ls -l /Library/Developer/CommandLineTools/SDKs/MacOSX

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 10:31 AM Nathan Bossart wrote: > Yup, thanks! Great, thanks. I'll push it soon. --Jacob

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 10:35 AM Tom Lane wrote: > FWIW, on my Mac a meson build from HEAD goes through fine, with or > without this patch. I'm getting openssl and libcurl from MacPorts > not Homebrew, but I don't know why that would make any difference. Do your and live in the same place? Mine

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-05-02 Thread Vitaly Davydov
Dear All, Thank you for the attention to the patch. I updated a patch with a better solution for the master branch which can be easily backported to the other branches as we agree on the final solution. Two tests are introduced which are based on Tomas Vondra's test for logical slots with inject

Re: SQL functions: avoid making a tuplestore unnecessarily

2025-05-02 Thread Tom Lane
I wrote: > Given the lack of field complaints over the many years it's been > like this, there doesn't seem to be a live problem. I think the > explanation for that is > (1) those mechanisms are never used to call set-returning functions, > (2) therefore, the tuplestore will not be called on to ho

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 11:26 AM Tom Lane wrote: > Yeah, they are both under /opt/local/include in a MacPorts setup. > But disabling NLS doesn't break it for me. I tried > > meson setup build --auto-features=disabled -Dlibcurl=enabled > > to make sure that /opt/local/include wasn't getting pulled

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Daniil Davydov
On Fri, May 2, 2025 at 11:58 PM Sami Imseih wrote: > > I am generally -1 on the idea of autovacuum performing parallel > index vacuum, because I always felt that the parallel option should > be employed in a targeted manner for a specific table. if you have a bunch > of large tables, some more imp

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Tom Lane
Jacob Champion writes: > Hm. If you clear out the build artifacts under > src/interfaces/libpq-oauth, and then build with > $ ninja -v src/interfaces/libpq-oauth/libpq-oauth.a > does that help surface anything interesting? $ rm -rf src/interfaces/libpq-oauth $ ninja -v src/interfaces/libpq-oa

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 11:52 AM Tom Lane wrote: > $ pkg-config --cflags libcurl > -I/opt/local/include -I/opt/local/libexec/openssl3/include > -I/opt/local/include > > I bet Homebrew's libcurl packaging doesn't do that. Nope, Homebrew breaks them out into smaller pieces: % PKG_CONFIG_PATH=/

Re: extension_control_path and "directory"

2025-05-02 Thread Peter Eisentraut
On 29.04.25 17:06, Matheus Alcantara wrote: On Tue, Apr 29, 2025 at 11:08 AM David E. Wheeler wrote: Right. My point is a minor one, but I thin you can use an if/ else there: ```c if (strcmp(piece, "$system") == 0) { /* Substitute the path macro if needed */ mangled = substit

Re: queryId constant squashing does not support prepared statements

2025-05-02 Thread Álvaro Herrera
On 2025-May-02, Michael Paquier wrote: > That depends. If we conclude that tracking this information through > the parser based on the start and end positions in a query string > for a set of values is more relevant, then we would be redesigning the > facility from the ground, so the old approach

Small fixes needed by high-availability tools

2025-05-02 Thread Andrey Borodin
Hi hackers! I want to revive attempts to fix some old edge cases of physical quorum replication. Please find attached draft patches that demonstrate ideas. These patches are not actually proposed code changes, I rather want to have a design consensus first. 1. Allow checking standby sync befo

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 8:11 AM Nathan Bossart wrote: > > After commit b0635bf, I'm seeing the following meson build failures on > macOS: Thanks for the report, and sorry for the breakage. > In file included from > ../postgresql/src/interfaces/libpq-oauth/oauth-curl.c:51: > ../po

Re: queryId constant squashing does not support prepared statements

2025-05-02 Thread Dmitry Dolgov
> On Fri, May 02, 2025 at 04:18:37PM GMT, Michael Paquier wrote: > On Fri, May 02, 2025 at 09:13:39AM +0200, Dmitry Dolgov wrote: > > Squashing constants was ment to be a first step towards doing the same > > for other types of queries (params, rte_values), reverting it to > > implement everything

Re: Fix slot synchronization with two_phase decoding enabled

2025-05-02 Thread Nisha Moond
On Wed, Apr 30, 2025 at 5:45 AM Masahiko Sawada wrote: > > On Tue, Apr 29, 2025 at 5:00 AM Nisha Moond wrote: > > Thank you for updating the patch! Here are some comments on v10. > Thanks for reviewing the patch! > --- > + > +# Also wait for two-phase to be enabled > +$subscriber1->poll_query_u

Re: fixing CREATEROLE

2025-05-02 Thread Robert Haas
On Thu, May 1, 2025 at 4:31 PM Tom Lane wrote: > In any case, I'd be happier about createrole_self_grant if it had > been a role property bit instead of a GUC. But we'd still need > to worry about whether it corrupts the results of dump/restore > (offhand I think it still would, if it results in

Re: extension_control_path and "directory"

2025-05-02 Thread Matheus Alcantara
On Fri, May 2, 2025 at 11:51 AM Peter Eisentraut wrote: > > Thanks, I have committed this. I did a bit of code reformatting and > adjusted the documentation a bit. It's good to get this in before beta1 > so that we don't have to change the valid values of > extension_control_path past beta1. >

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Nathan Bossart
After commit b0635bf, I'm seeing the following meson build failures on macOS: In file included from ../postgresql/src/interfaces/libpq-oauth/oauth-curl.c:51: ../postgresql/src/interfaces/libpq/libpq-int.h:70:10: fatal error: 'openssl/ssl.h' file not found 70 | #include

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
Thanks for raising this idea! I am generally -1 on the idea of autovacuum performing parallel index vacuum, because I always felt that the parallel option should be employed in a targeted manner for a specific table. if you have a bunch of large tables, some more important than others, a/c may end

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 8:59 AM Jacob Champion wrote: > libintl is already coming in via frontend_stlib_code, so that's fine. > So now I'm wondering if any other static clients of libpq-int.h (if > there are any) need the ssl dependency too, for correctness, or if > it's just me. Looks like it's j

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> On Fri, May 2, 2025 at 11:58 PM Sami Imseih wrote: > > > > I am generally -1 on the idea of autovacuum performing parallel > > index vacuum, because I always felt that the parallel option should > > be employed in a targeted manner for a specific table. if you have a bunch > > of large tables, s

Re: Parallel CREATE INDEX for GIN indexes

2025-05-02 Thread Tomas Vondra
On 4/30/25 14:39, Tomas Vondra wrote: > > On 4/18/25 03:03, Vinod Sridharan wrote: >> ... >> > > The patch seems fine to me - I repeated the tests with mailing list > archives, with MemoryContextStats() in _gin_parallel_merge, and it > reliably minimizes the memory usage. So that's fine. > > I w

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-02 Thread Sami Imseih
> Hmm. What are the workloads that you have seen as problematic? Do > these involve cursor names generated randomly, where most of them are > similar with a random factor for the name? postgres_fdw, as an example, in which cursor name get reused for different queries. Notice below "c1" and "c2"

Re: Issue attaching a table to a partitioned table with an auto-referenced foreign key

2025-05-02 Thread Tender Wang
Alvaro Herrera 于2025年5月1日周四 20:17写道: > Hello, > > I've been looking at this bug once again and I think I finally > understood what's going on and how to fix it. > > Ref 1: https://postgr.es/m/20230707175859.17c91538@karst >Re: Issue attaching a table to a partitioned table with an >

Re: queryId constant squashing does not support prepared statements

2025-05-02 Thread Dmitry Dolgov
> On Fri, May 02, 2025 at 07:10:19AM GMT, Michael Paquier wrote: > > I am really leaning towards that we should revert this feature as the > > limitation we have now with parameters is a rather large one and I think > > we need to go back and address this issue. > > I am wondering if this would not

Re: queryId constant squashing does not support prepared statements

2025-05-02 Thread Michael Paquier
On Fri, May 02, 2025 at 09:13:39AM +0200, Dmitry Dolgov wrote: > Squashing constants was ment to be a first step towards doing the same > for other types of queries (params, rte_values), reverting it to > implement everything at once makes very little sense to me. That depends. If we conclude tha

Re: Fix slot synchronization with two_phase decoding enabled

2025-05-02 Thread shveta malik
On Fri, May 2, 2025 at 12:57 PM Nisha Moond wrote: > > > Please find the v11 patch addressing the above points and all other > comments. I have also optimized the test by reducing the number of > subscriptions and slots. > Thanks for the patch. Few comments: 1) pg_upgrade/t/003_logical_slots.pl

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-05-02 Thread Xuneng Zhou
Yeh, tks for your clarification. I have a basic understanding of it now. I mean is this considered a bug or design defect in the codebase? If so, should we prevent it from occuring in general, not just for this specific test. vignesh C > > We have three processes involved in this scenario: > A

Re: PG 18 release notes draft committed

2025-05-02 Thread Bruce Momjian
On Fri, May 2, 2025 at 12:18:06PM -0400, Bruce Momjian wrote: > Finally, I see the big increases in this release as being the optimizer, > monitoring, and constraints. Also, and I am loving the chapter markers linking to gitweb. -- Bruce Momjian https://momjian.us EDB

Re: extension_control_path and "directory"

2025-05-02 Thread Christoph Berg
Re: Matheus Alcantara > > Thanks, I have committed this. I did a bit of code reformatting and > > adjusted the documentation a bit. It's good to get this in before beta1 > > so that we don't have to change the valid values of > > extension_control_path past beta1. > > Thanks Peter! And thanks e

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Jacob Champion
On Fri, May 2, 2025 at 8:46 AM Jacob Champion wrote: > Yeah, I wonder if libintl is being similarly "cheated" on the Meson side. libintl is already coming in via frontend_stlib_code, so that's fine. So now I'm wondering if any other static clients of libpq-int.h (if there are any) need the ssl de

Re: PG 18 release notes draft committed

2025-05-02 Thread Bruce Momjian
On Fri, May 2, 2025 at 01:00:57PM +0900, Amit Langote wrote: > 1. Speed up execution of cached plans by deferring locks on partitions > subject to pruning (Amit Langote) > (bb3ec16e1, d47cbf474, cbc127917, 525392d57) > > 2. Speed up child EquivalenceMember lookup in planner (Yuya Watari, > David

First-draft back-branch release notes are up

2025-05-02 Thread Tom Lane
Much less exciting than the v18 release notes, but we still gotta do 'em. See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=176877f461a8b55e921f597fb217f6ab89ee019f As usual, please send corrections by Sunday. regards, tom lane

Re: RFC: Additional Directory for Extensions

2025-05-02 Thread David E. Wheeler
On May 1, 2025, at 16:24, Peter Eisentraut wrote: > I see. I have committed it now describing the current state. Quick follow-up to tweak a couple of commas. --- a/doc/src/sgml/extend.sgml +++ b/doc/src/sgml/extend.sgml @@ -1813,8 +1813,8 @@ include $(PGXS) You can select a separat

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-05-02 Thread Tom Lane
vignesh C writes: > Due to the asynchronous nature of these processes, the ALTER > SUBSCRIPTION command may not be immediately observed by the apply > worker. Meanwhile, the walsender may process and decode an INSERT > statement. > If the insert targets a table (e.g., tab_3) that does not belong t

Re: PG 18 release notes draft committed

2025-05-02 Thread Jacob Champion
On Thu, May 1, 2025 at 7:44 PM Bruce Momjian wrote: > I will probably add markup in 1-3 weeks. Let the feedback begin. ;-) Thanks! > > Version 18 contains a number of changes that may affect compatibility > with previous releases. Observe the following incompatibilities: > >

Re: PG 18 release notes draft committed

2025-05-02 Thread Bruce Momjian
On Fri, May 2, 2025 at 08:24:42AM -0700, Jacob Champion wrote: > On Thu, May 1, 2025 at 7:44 PM Bruce Momjian wrote: > > I will probably add markup in 1-3 weeks. Let the feedback begin. ;-) > > Thanks! > > > > > Version 18 contains a number of changes that may affect compatibility > >

Re: First-draft back-branch release notes are up

2025-05-02 Thread Bruce Momjian
On Fri, May 2, 2025 at 12:39:15PM -0400, Tom Lane wrote: > Much less exciting than the v18 release notes, but we > still gotta do 'em. See > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=176877f461a8b55e921f597fb217f6ab89ee019f > > As usual, please send corrections by Sun

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Nathan Bossart
On Fri, May 02, 2025 at 10:05:26AM -0700, Jacob Champion wrote: > Nathan, if you get a chance, does the attached patch work for you? Yup, thanks! -- nathan

Re: [PoC] Federated Authn/z with OAUTHBEARER

2025-05-02 Thread Tom Lane
Jacob Champion writes: > Looks like it's just me. And using partial_dependency for the includes > seems like overkill, so I've kept the full ssl dependency object, but > moved it to the staticlib only, which is enough to solve the breakage > on my machine. > Nathan, if you get a chance, does the a

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Thu, May 1, 2025 at 4:04 PM Michael Paquier wrote: > > On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote: > > In light of these concerns, I've been contemplating alternative > > interface designs. One promising approach would involve registering > > custom copy formats via a C fun

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread David G. Johnston
On Thursday, May 1, 2025, Masahiko Sawada wrote: > > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This method woul

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 10:36 PM David G. Johnston wrote: > > On Thursday, May 1, 2025, Masahiko Sawada wrote: >> >> >> In light of these concerns, I've been contemplating alternative >> interface designs. One promising approach would involve registering >> custom copy formats via a C function dur

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 9:56 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 2 May 2025 21:38:32 -0700, > Masahiko Sawada wrote: > > >> How about requiring schema for all custom formats? > >> > >> Valid: > >> > >>

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 11:13 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > On Thu, May 1, 2025 at 8:03 AM Masahiko Sawada wrote: > > > > As I understand it, we initially disabled parallel vacuum for > > autovacuum because their objectives are somewhat contradictory. > > Parallel vacuum aims

Re: AIO v2.5

2025-05-02 Thread Noah Misch
On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote: > pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch > of other things, finds the oldest in-flight IO and waits for it. > > PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, >

Re: PG 18 release notes draft committed

2025-05-02 Thread Bruce Momjian
On Sat, May 3, 2025 at 01:46:29AM +0200, Jelte Fennema-Nio wrote: > On Fri, 2 May 2025 at 04:45, Bruce Momjian wrote: > > > > I have committd the first draft of the PG 18 release notes. The item > > count looks strong: > > Thanks for all the work. Some notes: > > 1. There's currently no mentio

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> I think it would more make > sense to maintain the existing autovacuum_max_workers parameter while > introducing a new parameter that would either control the maximum > number of parallel vacuum workers per autovacuum worker or set a > system-wide cap on the total number of parallel vacuum worker

Re: PG 18 release notes draft committed

2025-05-02 Thread Jelte Fennema-Nio
On Fri, 2 May 2025 at 04:45, Bruce Momjian wrote: > > I have committd the first draft of the PG 18 release notes. The item > count looks strong: Thanks for all the work. Some notes: 1. There's currently no mention that protocol version 3.2 was introduced in this release. I'm not sure where/how

Re: pgsql: Add function to log the memory contexts of specified backend pro

2025-05-02 Thread Fujii Masao
On 2025/05/02 14:58, torikoshia wrote: I confirmed that with this patch applied, the process no longer crashes even after applying the change Robert used to trigger the crash. a small nitpick: +    * requested  repeatedly and rapidly, potentially leading to infinite It looks li

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700, Masahiko Sawada wrote: > One of the primary considerations we need to address is the treatment > of the specified format name. The current patch set utilizes built-in > forma

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 15:52:49 -0700, Masahiko Sawada wrote: >> Hmm. How much should we care about the observability of the COPY >> format used by a given backend? Storing this information in a >> backend's Top

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 9:58 AM Sami Imseih wrote: > > > Once we have parallel heap vacuum, as discussed in thread[1], it would > > also likely be beneficial to incorporate it into autovacuum during > > aggressive vacuum or failsafe mode. > > IIRC, index cleanup is disabled by failsafe. Yes. My id

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 7:20 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Thu, 1 May 2025 12:15:30 -0700, > Masahiko Sawada wrote: > > > One of the primary considerations we need to address is the treatment > > of the

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700, Masahiko Sawada wrote: >> How about requiring schema for all custom formats? >> >> Valid: >> >> COPY ... TO ... (FORMAT 'text'); >> COPY ... TO ... (FORMAT 'my_schema.jso

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700, Masahiko Sawada wrote: > The progress view can know only numbers. We need to extend the > progress view infrastructure so that we can pass other data types. Sorry. Could you

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 11:20 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 2 May 2025 23:02:25 -0700, > Masahiko Sawada wrote: > > > The progress view can know only numbers. We need to extend the > > progress view

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread David G. Johnston
On Friday, May 2, 2025, Masahiko Sawada wrote: > > I'm concerned about allowing multiple 'text' format implementations > with identical names within the database, as this could lead to > considerable confusion. When users specify 'text', it would be more > logical to guarantee that the built-in '