Re: Some questions about gin index

2025-06-06 Thread Andy Fan
Tom Lane writes: > Andy Fan writes: >> When reading the code for Gin Index, there are lots of confusion about >> some concepts, so here are some questions in my mind. > > At least some of these things are explained in gin.sgml. I think the complexity of gin comes from the operators on the set o

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-06 Thread shveta malik
On Wed, Jun 4, 2025 at 3:40 PM shveta malik wrote: > > On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada wrote: > > > > On Tue, May 20, 2025 at 9:54 PM Amit Kapila wrote: > > > > > > Yeah, I find the idea that the presence of a logical slot will allow > > > the user to enable logical decoding/repli

Re: CHECKPOINT unlogged data

2025-06-06 Thread Christoph Berg
Re: Andres Freund > I'd add a 'mode' that can be set to an arbitrary string, which then can be > validated in C code. That seems more future proof. Changed in the attached v2, thanks. Christoph >From c8975cbd1dbe5e5cae18414ea27bc3f2c0e8 Mon Sep 17 00:00:00 2001 From: Christoph Berg Date: Fri

Re: Add enable_groupagg GUC parameter to control GroupAggregate usage

2025-06-06 Thread Tatsuro Yamada
Hi, # Query Execution Results (Average of 3 measurements) > - With parallel query: 39546 seconds > - With parallel query and enable_groupagg turned off: 1115 seconds > Oops, I made a mistake. The correct execution time is: - With parallel query:

Re: Conflict detection for update_deleted in logical replication

2025-06-06 Thread Amit Kapila
On Wed, Jun 4, 2025 at 4:12 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V33 patch set which includes the following changes: > Few comments: 1. + if (sub->enabled) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot set option %s for enabled subscription", +

Re: Enable data checksums by default

2025-06-06 Thread Peter Eisentraut
On 05.06.25 12:37, Christoph Berg wrote: Maybe it would be enough if the initdb options used to create a cluster would be stored in some file in the data dir. That probably wouldn't help if the default behavior changed, as in the current case.

Re: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

2025-06-06 Thread Ajin Cherian
On Wed, Jun 4, 2025 at 8:22 PM Ajin Cherian wrote: > > On Tue, Jun 3, 2025 at 4:25 PM Ajin Cherian wrote: > > > > On Tue, Jun 3, 2025 at 3:55 PM Amit Kapila wrote: > > > > > > > > > You haven't shared the exact test scenario, but I am assuming the > > > above tests are for very large transaction

Re: CHECKPOINT unlogged data

2025-06-06 Thread Fujii Masao
On 2025/06/06 19:03, Christoph Berg wrote: Re: Andres Freund I'd add a 'mode' that can be set to an arbitrary string, which then can be validated in C code. That seems more future proof. Changed in the attached v2, thanks. When I applied the patch and compiled it, I got the following warn

Re: Non-reproducible AIO failure

2025-06-06 Thread Konstantin Knizhnik
There is really essential difference in code generated by clang 15 (working) and 16 (not working). ``` pgaio_io_stage(PgAioHandle *ioh, PgAioOp op) { ... HOLD_INTERRUPTS();     ioh->op = op;     ioh->result = 0;     pgaio_io_update_state(ioh, PGAIO_HS_DEFINED);     ... } ``` c

Re: Missing program_XXX calling in pgbench tests

2025-06-06 Thread Fujii Masao
On 2025/06/05 16:44, Hayato Kuroda (Fujitsu) wrote: Dear Peter, Thanks for the comment. Patch 0001 looks very sensible. I don't think we need to bother we the other ones. pg_config works differently than the other programs anyway, because --version does not exit the program. And pg_bsd_i

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Robert Haas
On Thu, Jun 5, 2025 at 3:58 PM Andres Freund wrote: > There are a few related issues where we ourselves to blame. The most prominent > one is that we go around and delete PostmasterContext in child processes. That > however doesn't really save memory, as the memory is still needed in > postmaster,

Re: CHECKPOINT unlogged data

2025-06-06 Thread Nathan Bossart
On Fri, Jun 06, 2025 at 04:26:56PM +0200, Christoph Berg wrote: > Re: Fujii Masao >> Some users might want to trigger a spread checkpoint but not wait for >> it to finish, since it could take a long time? If that's a valid use case, >> maybe we should add a WAIT option to let users choose whether t

Re: CHECKPOINT unlogged data

2025-06-06 Thread Christoph Berg
Re: Nathan Bossart > I imagine the documentation will pretty clearly indicate that setting WAIT > to "false" will cause CHECKPOINT to not wait for it to finish. I can add it, it's easy enough... > I don't understand why we need to add both FAST and IMMEDIATE. We have both: =# checkpoint ; 2025-

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Nico Williams
On Fri, Jun 06, 2025 at 08:41:20AM -0700, Jacob Champion wrote: > I guess I'd be concerned that a hardware crypto provider might need > good-faith cleanup to work well. I understand they can't rely on > atexit in general, but there would be a big difference between "you > might have to clean up aft

Re: a couple of small cleanup patches for DSM-related code

2025-06-06 Thread Nathan Bossart
Committed. -- nathan

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Jacob Champion
On Fri, Jun 6, 2025 at 4:56 AM Peter Eisentraut wrote: > It seems weird to me that openssl spends so much effort tidying up its > memory allocations just before exiting. We could just skip that. > Looking through the code of OPENSSL_cleanup(), there might be one or two > cases of log or trace fil

Re: CHECKPOINT unlogged data

2025-06-06 Thread Nathan Bossart
On Fri, Jun 06, 2025 at 06:20:21PM +0200, Christoph Berg wrote: > Re: Nathan Bossart >> I don't understand why we need to add both FAST and IMMEDIATE. > > We have both: > > =# checkpoint ; > 2025-06-06 18:09:25.743 CEST [872379] LOG: checkpoint starting: immediate > force wait > > pg_basebacku

Re: strange perf regression with data checksums

2025-06-06 Thread Peter Geoghegan
On Wed, Jun 4, 2025 at 1:39 PM Peter Geoghegan wrote: > My current plan is to commit this in the next couple of days. Pushed. It would be great if we could also teach BufferGetLSNAtomic() to just call PageGetLSN() (at least on PG_HAVE_8BYTE_SINGLE_COPY_ATOMICITY platforms), but my sense is that

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Andres Freund
Hi, On 2025-06-06 08:41:20 -0700, Jacob Champion wrote: > On Fri, Jun 6, 2025 at 4:56 AM Peter Eisentraut wrote: > > It seems weird to me that openssl spends so much effort tidying up its > > memory allocations just before exiting. We could just skip that. > > Looking through the code of OPENSSL

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Christoph Berg
Re: Jacob Champion > I guess I'd be concerned that a hardware crypto provider might need > good-faith cleanup to work well. Hopefully not in every single backend. Christoph

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-06 Thread Sami Imseih
> Don't get me wrong, it would be an improvement to have some type of > mechanism that can move you from almost 100% to 100%, but the real > problem is how do you SAFELY get to almost 100% in the first place? This big use case is precisely the "almost 100% to 100%" confidence problem. Usually, us

Re: What's our minimum supported Python version?

2025-06-06 Thread Jacob Champion
On Fri, Jun 6, 2025 at 7:17 AM Tom Lane wrote: > Peter Eisentraut writes: > > Since we now require Python 3.6, we can also remove PL/Python test > > alternative expected files for earlier Python versions. See attached patch. > > +1. So nice to get rid of src/pl/plpython/expected/README. Awesom

Re: pg_dump --with-* options

2025-06-06 Thread Nathan Bossart
On Fri, Jun 06, 2025 at 09:14:32AM +0200, Peter Eisentraut wrote: > We have > > -a, --data-only dump only the data, not the schema or statistics > --no-datado not dump data > --with-data dump the data # this one is new > > (and there is also --section=data), and t

Re: Non-reproducible AIO failure

2025-06-06 Thread Andres Freund
Hi, On 2025-06-06 14:03:12 +0300, Konstantin Knizhnik wrote: > There is really essential difference in code generated by clang 15 (working) > and 16 (not working). There also are code gen differences between upstream clang 17 and apple's clang, which is based on llvm 17 as well (I've updated the

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Jacob Champion
On Fri, Jun 6, 2025 at 9:25 AM Nico Williams wrote: > I'd expect all subsystems to recover cleanly from unclean shutdowns. I > know, that's a lot to expect, but nowadays pretty much all filesystems > used in production do, for example. I guess, but if we stop cleaning up entirely, we will sudden

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Jacob Champion
On Fri, Jun 6, 2025 at 9:37 AM Andres Freund wrote: > It's not just crashes, e.g. the startup packet timeout is also handled by > _exit() - and it can be triggered remotely. Fair point... > ISTM that if crypto providers > can't handle _exit(), we have a bigger problem. ...so I guess I need to f

Re: Non-reproducible AIO failure

2025-06-06 Thread Tom Lane
Konstantin Knizhnik writes: > There is really essential difference in code generated by clang 15 > (working) and 16 (not working). It's a mistake to think that this is a compiler bug. The C standard explicitly allows compilers to use word-wide operations to access bit-field struct members. Suc

Re: Non-reproducible AIO failure

2025-06-06 Thread Andres Freund
Hi, On 2025-06-06 15:21:13 -0400, Tom Lane wrote: > Konstantin Knizhnik writes: > > There is really essential difference in code generated by clang 15 > > (working) and 16 (not working). > > It's a mistake to think that this is a compiler bug. The C standard > explicitly allows compilers to use

Proposal: Global Index for PostgreSQL

2025-06-06 Thread Dilip Kumar
PostgreSQL’s table partitioning allows large tables to be broken into smaller, more manageable pieces for better performance. However, a key limitation currently is the absence of global indexes, which restricts using partitioned tables, especially when you need unique constraints on columns that a

Re: Fix slot synchronization with two_phase decoding enabled

2025-06-06 Thread Nisha Moond
On Thu, Jun 5, 2025 at 3:26 PM Dilip Kumar wrote: > > On Thu, Jun 5, 2025 at 2:53 PM Dilip Kumar wrote: > > > > On Tue, Jun 3, 2025 at 11:05 AM Nisha Moond > > wrote: > > > > > > > > > Attached v17 patches. Added a top-up patch 0002 implementing the idea > > > suggested by Amit above. > > > > I

Re: Add SPLIT PARTITION/MERGE PARTITIONS commands

2025-06-06 Thread jian he
hi. in createTableConstraints + /* Add a pre-cooked default expression. */ + StoreAttrDefault(newRel, num, def, true); + + /* Store CHECK constraints. */ + StoreConstraints(newRel, cookedConstraints, false); Here, StoreConstraints last argument should be set to true? see also StoreAttrDefault. +

pg_dump --with-* options

2025-06-06 Thread Peter Eisentraut
I'm looking at the new in PG18 pg_dump --with-* options, and I'm having trouble understanding them. (I did not look into the source code or the git or mailing list history for this, to try to understand it as a user.) We have -a, --data-only dump only the data, not the schema or statis

Re: Proposal: Global Index for PostgreSQL

2025-06-06 Thread wenhui qiu
Hi Dilip Kumar Thank you for your working on this ,I remember six years ago there was talk about global index ,You can see if this mailing list has any references to ( https://www.postgresql.org/message-id/CALtqXTcurqy1PKXzP9XO%3DofLLA5wBSo77BnUnYVEZpmcA3V0ag%40mail.gmail.com ) Thanks On Fri

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2025-06-06 Thread Michael Paquier
On Mon, Nov 28, 2022 at 05:24:29PM -0500, Tom Lane wrote: > Andres Freund writes: >> And as you said earlier, the increased overhead inside the toast table / >> index >> is not relevant compared to the size of toasted datums. > > Perhaps not. (Replying to an old thread here, due to some work in

Re: synchronized_standby_slots used in logical replication

2025-06-06 Thread shveta malik
On Thu, Jun 5, 2025 at 3:10 PM Fabrice Chapuis wrote: > > Thank you very much for the detailed response. I will proceed with the native > implementation for synchronizing logical replication slots. Sure, thanks! > In a maintenance context, when standby is shutdown, it's possible to > temporar

Re: What's our minimum supported Python version?

2025-06-06 Thread Tom Lane
Peter Eisentraut writes: > Since we now require Python 3.6, we can also remove PL/Python test > alternative expected files for earlier Python versions. See attached patch. +1. So nice to get rid of src/pl/plpython/expected/README. regards, tom lane

Re: Enable data checksums by default

2025-06-06 Thread Bruce Momjian
On Fri, Jun 6, 2025 at 01:40:40PM +0200, Peter Eisentraut wrote: > On 05.06.25 12:37, Christoph Berg wrote: > > Maybe it would be enough if the initdb options used to create a > > cluster would be stored in some file in the data dir. > > That probably wouldn't help if the default behavior changed

Re: CHECKPOINT unlogged data

2025-06-06 Thread Christoph Berg
Re: Fujii Masao > utility.c:946:4: warning: label followed by a declaration is a C23 extension > [-Wc23-extensions] Thanks, my compiler didn't throw that. { } block added in v3. > RequestCheckpoint(CHECKPOINT_WAIT | > +

Add enable_groupagg GUC parameter to control GroupAggregate usage

2025-06-06 Thread Tatsuro Yamada
Hi hackers, When I measured the execution time of a certain query with parallel query enabled and disabled, I found that the execution time was slower when parallel query was enabled. To improve the performance of the parallel query, I considered adjusting the execution plan and attempted to swit

Re: [PATCH] Support for basic ALTER TABLE progress reporting.

2025-06-06 Thread jian he
On Mon, Jun 2, 2025 at 3:35 PM Jiří Kavalík wrote: > What I changed: > > `commands/tablecmds.c` > - start and end reporting inside `ATRewriteTables()` > - report blocks total, blocks and tuples scanned and possibly tuples written > in `ATRewriteTable` > - add at least phase info in `validateForei

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Peter Eisentraut
On 05.06.25 21:58, Andres Freund wrote: The reason for this difference is that by default openssl registers an atexit handler that frees a lot of memory that was initialized in postmaster. That in turn triggers page-faults due to the relevant pages now differing in child processes. Which a) isn't

Re: Fix slot synchronization with two_phase decoding enabled

2025-06-06 Thread Dilip Kumar
On Thu, Jun 5, 2025 at 3:59 PM Amit Kapila wrote: > > On Thu, Jun 5, 2025 at 2:53 PM Dilip Kumar wrote: > > > > On Tue, Jun 3, 2025 at 11:05 AM Nisha Moond > > wrote: > > > > > > > > > Attached v17 patches. Added a top-up patch 0002 implementing the idea > > > suggested by Amit above. > > > > I

Re: Add CHECK_FOR_INTERRUPTS in polling loop code path in XactLockTableWait

2025-06-06 Thread Xuneng Zhou
Hi, > Just idea, if XactLockTableWait() is expected to finish within a few seconds > after acquiring the lock, how about this approach: keep sleeping for 1ms > until the total sleep time reaches 10s (10s is just an example), > and after that, start doubling the sleep duration each cycle, up to > a

Re: autoprewarm_dump_now

2025-06-06 Thread Robert Haas
On Tue, Jun 3, 2025 at 2:58 PM Andres Freund wrote: > > I think the proposed patch should be committed and back-patched, after > > fixing it so that it's pgindent-clean and adding a comment. Does > > anyone have strong objection to that? > > No, seems like a thing that pretty obviously should be

Re: CREATE DATABASE command for non-libc providers

2025-06-06 Thread Daniel Verite
Jeff Davis wrote: > I have attached a patch 0001 that > fixes a misleading hint, but it's still not great. +1 for the patch > When using ICU or the builtin provider, it still requires coming up > with some valid locale name for LC_COLLATE and LC_CTYPE No, since the following invocation

Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

2025-06-06 Thread Nico Williams
On Fri, Jun 06, 2025 at 11:58:38AM -0700, Jacob Champion wrote: > > I'd expect all subsystems to recover cleanly from unclean shutdowns. I > > know, that's a lot to expect, but nowadays pretty much all filesystems > > used in production do, for example. > > I guess, but if we stop cleaning up ent

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-06 Thread Masahiko Sawada
On Fri, Jun 6, 2025 at 3:02 AM shveta malik wrote: > > On Wed, Jun 4, 2025 at 3:40 PM shveta malik wrote: > > > > On Wed, Jun 4, 2025 at 6:41 AM Masahiko Sawada > > wrote: > > > > > > On Tue, May 20, 2025 at 9:54 PM Amit Kapila > > > wrote: > > > > > > > > Yeah, I find the idea that the prese

Re: Batch TIDs lookup in ambulkdelete

2025-06-06 Thread Masahiko Sawada
On Tue, May 13, 2025 at 2:26 PM Matheus Alcantara wrote: > > Hi, > > On 30/04/25 19:36, Masahiko Sawada wrote: > > Here are the summary of each attached patch: > > > > 0001: Introduce TIdStoreIsMemberMulti() where we can do IsMember check > > for multiple TIDs in one function call. If the given TI

md5 password deprecation might cause problems with PgBouncer setups

2025-06-06 Thread Jelte Fennema-Nio
First of all, I'm definitely in favor of sunsetting md5 password auth myself. However, I would like to share a possible issue that users might run into while we're doing this: Apparently the overhead of scram-256 is much higher in some PgBouncer setups. I expect this to be mostly setups where ther

Re: md5 password deprecation might cause problems with PgBouncer setups

2025-06-06 Thread Andres Freund
Hi, On 2025-06-06 23:48:18 +0200, Jelte Fennema-Nio wrote: > First of all, I'm definitely in favor of sunsetting md5 password auth myself. > > However, I would like to share a possible issue that users might run > into while we're doing this: Apparently the overhead of scram-256 is > much higher i

Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData

2025-06-06 Thread Noah Misch
On Thu, Jun 05, 2025 at 04:22:48PM +0900, Michael Paquier wrote: > On Mon, Jun 02, 2025 at 06:48:46PM -0700, Noah Misch wrote: > > The wasShutdown case reaches consistency from the beginning, so I don't see > > that as an example of a time we benefit from reading pg_twophase before > > reaching con

Re: md5 password deprecation might cause problems with PgBouncer setups

2025-06-06 Thread Jelte Fennema-Nio
On Sat, 7 Jun 2025 at 00:12, Andres Freund wrote: > I assume this is due to the fairly high iteration count we use by default? Maybe... But looking closer at the PgBouncer code another option that seems pretty likely is that PgBouncer is regenerating a secret over and over again, if users store t

Re: CREATE DATABASE command for non-libc providers

2025-06-06 Thread Jeff Davis
On Fri, 2025-06-06 at 22:03 +0200, Daniel Verite wrote: > +1 for the patch Thank you, committed. > > Here we let 'locale' or 'lc_collate/lc_ctype' which is the same > thing, > defaulting from the template database. Right, in the normal case it's OK, but if anything goes wrong, it gets fairly co

Re: Remaining dependency on setlocale()

2025-06-06 Thread Jeff Davis
On Thu, 2025-06-05 at 22:15 -0700, Jeff Davis wrote: > To continue this thread, I did a symbol search in the meson build > directory like (patterns.txt attached): Attached a rough patch series which does what everyone seemed to agree on: * Change some trivial ASCII cases to use pg_ascii_* varia

Re: Batch TIDs lookup in ambulkdelete

2025-06-06 Thread Masahiko Sawada
On Sun, Jun 1, 2025 at 11:01 PM John Naylor wrote: > > On Thu, May 1, 2025 at 5:36 AM Masahiko Sawada wrote: > > > HEAD PATCHEDDIFF > > case-1: 3,021 ms 2.818 ms93.29% > > case-2: 5, 697 ms 5.545 ms97.34% > > case-3: 2,833 ms 2.790 m

Re: Batch TIDs lookup in ambulkdelete

2025-06-06 Thread Peter Geoghegan
On Fri, Jun 6, 2025 at 6:58 PM Masahiko Sawada wrote: > Agreed. Given the above test results, it's unlikely always sorting the > array helps speedups. Did you try specializing the sort? In my experience, it makes a big difference. -- Peter Geoghegan

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-06 Thread David Rowley
On Fri, 6 Jun 2025 at 14:32, Robert Treat wrote: > In production, you aren't watching to see what happen with > pg_stat_all_indexes, because you will first be watching pg_stat_activity to > see if the plans have flipped in some way that leads to an overloaded server > (extra latency, poor cachi

Re: Non-reproducible AIO failure

2025-06-06 Thread Alexander Lakhin
Hello Andres and Tom, 06.06.2025 22:37, Andres Freund wrote: On 2025-06-06 15:21:13 -0400, Tom Lane wrote: It's a mistake to think that this is a compiler bug. The C standard explicitly allows compilers to use word-wide operations to access bit-field struct members. Such accesses may fetch or

Re: Non-reproducible AIO failure

2025-06-06 Thread Nico Williams
On Fri, Jun 06, 2025 at 03:37:45PM -0400, Andres Freund wrote: > On 2025-06-06 15:21:13 -0400, Tom Lane wrote: > > So it's our code that is busted. No doubt, what is happening is > > that process A is fetching two fields, modifying one of them, > > and storing the word back (with the observed valu

Re: Non-reproducible AIO failure

2025-06-06 Thread Konstantin Knizhnik
On 06/06/2025 9:47 pm, Andres Freund wrote: Hi, On 2025-06-06 14:03:12 +0300, Konstantin Knizhnik wrote: There is really essential difference in code generated by clang 15 (working) and 16 (not working). There also are code gen differences between upstream clang 17 and apple's clang, which is

Re: Proposal: Global Index for PostgreSQL

2025-06-06 Thread Dilip Kumar
On Fri, Jun 6, 2025 at 1:01 PM wenhui qiu wrote: > > Hi Dilip Kumar >Thank you for your working on this ,I remember six years ago there was > talk about global index ,You can see if this mailing list has any references > to > (https://www.postgresql.org/message-id/CALtqXTcurqy1PKXzP9XO%3Dof