Address the bug in 041_checkpoint_at_promote.pl

2025-02-11 Thread Nitin Jadhav
While testing, I discovered an issue in 041_checkpoint_at_promote.pl. # Wait until the previous restart point completes on the newly-promoted # standby, checking the logs for that. my $checkpoint_complete = 0; foreach my $i (0 .. 10 * $PostgreSQL::Test::Utils::timeout_default) { if ($node_stan

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Shubham Khanna
On Wed, Feb 12, 2025 at 6:46 AM Hayato Kuroda (Fujitsu) wrote: > > Dear Shubham, > > Thanks for updating the patch! I feel the patch has good shape. Here is a > small comment. > > ``` > + /* Error if no databases were found on the source server */ > + if (num_rows == 0) > + { >

RE: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-11 Thread Zhijie Hou (Fujitsu)
On Wednesday, February 12, 2025 11:56 AM Amit Kapila wrote: > > On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart > wrote: > > > > On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote: > > > I find this proposed patch a bit strange and I feel it needs more > > > explanation. > > > > > >

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Shubham Khanna
On Wed, Feb 12, 2025 at 5:29 AM Peter Smith wrote: > > On Tue, Feb 11, 2025 at 9:16 PM Shubham Khanna > wrote: > > > > > #13. Unanswered question "How are tests expecting this even passing?". > > > Was a reason identified? IOW, how can we be sure the latest tests > > > don't have a similar proble

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-02-11 Thread Bertrand Drouvot
Hi, On Tue, Feb 11, 2025 at 02:11:10PM -0800, Masahiko Sawada wrote: > I've updated the patch that includes comment updates and bug fixes. Thanks! > The main idea of changing WAL level online is to decouple two aspects: > (1) the information included in WAL records and (2) the > functionalities

RE: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-02-11 Thread Hayato Kuroda (Fujitsu)
Dear Shubham, Thanks for updating the patch! Few comments. 01. pg_createsubscriber.sgml ``` + + Remove all existing publications on the subscriber node before creating + a subscription. + + ``` I think this is not accurate. Publications on databases which are not target

Re: [PATCH] pg_stat_activity: make slow/hanging authentication more visible

2025-02-11 Thread Michael Paquier
On Mon, Feb 10, 2025 at 11:05:32AM -0800, Jacob Champion wrote: > Bad regex escaping on my part; fixed in v8. Thanks for the report! > > While debugging, I also noticed that a poorly timed autovacuum could > also show up in my new pg_stat_activity query, so I've increased the > specificity. Now t

Re: Non-text mode for pg_dumpall

2025-02-11 Thread jian he
On Wed, Feb 12, 2025 at 1:17 AM Mahendra Singh Thalor wrote: > > > > > There are some tests per https://commitfest.postgresql.org/52/5495, I > > will check it later. hi. the cfbot failure is related to function _tocEntryRequired if (strcmp(te->desc, "DATABASE") == 0 || strcmp(te->desc, "DATA

Re: Fix punctuation errors in PostgreSQL documentation

2025-02-11 Thread John Naylor
On Mon, Feb 10, 2025 at 6:34 PM John Naylor wrote: > Thanks for the patch! I will push this after the upcoming minor releases. This is done. -- John Naylor Amazon Web Services

Re: Track the amount of time waiting due to cost_delay

2025-02-11 Thread Bertrand Drouvot
Hi, On Tue, Feb 11, 2025 at 04:42:26PM -0600, Nathan Bossart wrote: > On Tue, Feb 11, 2025 at 08:51:15AM +, Bertrand Drouvot wrote: > > On Mon, Feb 10, 2025 at 02:52:46PM -0600, Nathan Bossart wrote: > >> Off-list, I've asked Bertrand to gauge the feasibility of adding this > >> information to

EquivalenceClass and custom_read_write

2025-02-11 Thread Ashutosh Bapat
Hi All, In pathnodes.h typedef struct EquivalenceClass { pg_node_attr(custom_read_write, no_copy_equal, no_read, no_query_jumble) Because of custom_read_write attribute, I expect _outEquivalenceClass to be present in outfuncs.c and _readEquivalenceClass to be present in readfuncs.c. I find the fir

Re: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

2025-02-11 Thread Ajin Cherian
On Wed, Jan 29, 2025 at 9:31 AM Peter Smith wrote: Hi Ajin, Some review comments for patch v12-0001. == Commit message 1. Track transactions which have snapshot changes with a new flag RBTXN_HAS_SNAPSHOT_CHANGES ~ The commit message only says *what* it

Re: EvictUnpinnedBuffer and buffer free list

2025-02-11 Thread Ashutosh Bapat
Thanks a lot Melanie for a very detailed response, a good reference to pin. On Fri, Jan 31, 2025 at 8:20 PM Melanie Plageman wrote: > > I don't have an explicit issue with EvictUnpinnedBuffer() putting > buffers on the freelist -- it seems like that could be fine. But since > it is for testing/d

RE: pg_rewind with --write-recovery-conf option doesn't write dbname to primary_conninfo value.

2025-02-11 Thread Hayato Kuroda (Fujitsu)
Dear Sawada-san, Thanks for updating the patch! > I've attached the updated patch. I address all comments I got so far > and added a small regression test. > > It makes sense to me that we move GetDbnameFromConnectionOptions() to > recovery_gen.c since this function is currently used only with >

Re: Skip collecting decoded changes of already-aborted transactions

2025-02-11 Thread Peter Smith
Hi. Here are some minor comments for the v18* patch set. // Patch v18-0001 1.1. Commit message A previously reported typo still exists: /noticeble/noticeable/ // Patch v18-0002 2.1 +#define RBTXN_PREPARE_STATUS_FLAGS (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PR

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Shlok Kyal
On Tue, 11 Feb 2025 at 15:46, Shubham Khanna wrote: > > On Tue, Feb 11, 2025 at 6:30 AM Peter Smith wrote: > > > > Hi Shubham. > > > > Responding with a blanket statement "Fixed the given comments" makes > > it difficult to be sure what was done when I come to confirm the > > changes. Often embed

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Peter Smith
On Wed, Feb 12, 2025 at 10:58 AM Peter Smith wrote: > > On Tue, Feb 11, 2025 at 9:16 PM Shubham Khanna > wrote: > > > > > #13. Unanswered question "How are tests expecting this even passing?". > > > Was a reason identified? IOW, how can we be sure the latest tests > > > don't have a similar probl

TAP test command_fails versus command_fails_like

2025-02-11 Thread Peter Smith
Hi hackers, Recently, while writing some new TAP tests my colleague inadvertently called the command_fails() subroutine instead of command_fails_like() subroutine. Their parameters are almost the same but command_fails_like() also takes a pattern for checking in the logs. Notice if too many parame

Re: Showing applied extended statistics in explain Part 2

2025-02-11 Thread Tatsuro Yamada
Hi Tomas and ALL, >I wonder what Yamada-san thinks about these suggestions ... He's the one >actually developing the patch, so I'd like to know his opinions. I will state my thoughts on the two points of discussion. I often use SQL Server to compare execution plans generated by Postgre

Re: NOT ENFORCED constraint feature

2025-02-11 Thread Ashutosh Bapat
On Tue, Feb 11, 2025 at 9:09 PM Isaac Morland wrote: > > On Tue, 11 Feb 2025 at 08:36, Álvaro Herrera wrote: >> >> On 2025-Feb-10, Isaac Morland wrote: >> >> > I'm having a lot of trouble understanding the operational distinction >> > between your 'u' and 'U'. If it's not enforced, it cannot be a

Re: Test to dump and restore objects left behind by regression

2025-02-11 Thread Ashutosh Bapat
On Wed, Feb 12, 2025 at 5:25 AM Michael Paquier wrote: > > On Tue, Feb 11, 2025 at 12:19:33PM +0530, Ashutosh Bapat wrote: > > Sorry for replying late here. The refactored code in > > 002_compare_backups.pl has a potential to cause confusion even without > > this refactoring. The differences in ta

Re: Support POSITION with nondeterministic collations

2025-02-11 Thread Euler Taveira
On Mon, Dec 2, 2024, at 6:09 AM, Peter Eisentraut wrote: > On 26.08.24 08:09, Peter Eisentraut wrote: > > This patch allows using text position search functions with > > nondeterministic collations. These functions are > > > > - position, strpos > > - replace > > - split_part > > - string_to_arr

Re: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-11 Thread Amit Kapila
On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart wrote: > > On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote: > > I find this proposed patch a bit strange and I feel it needs more > > explanation. > > > > When this thread started, Bharath justified his patches saying that a > > slot th

Re: Small memory fixes for pg_createsubcriber

2025-02-11 Thread Michael Paquier
On Tue, Feb 11, 2025 at 01:32:32PM -0300, Euler Taveira wrote: > There is no bug. They are the same behind the scenes. I'm fine changing it. It > is a new code and it wouldn't cause a lot of pain to backpatch patches in the > future. On consistency grounds, and as this is documented in fe-exec.c a

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Julien Rouhaud
On Tue, Feb 11, 2025 at 08:57:46PM -0600, Sami Imseih wrote: > > Of course some people may want to keep the current behavior, if they have > > limited number of temp tables or similar, so I had a GUC for that. I don't > > think that the community would really welcome such GUC for core-postgres, >

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Sami Imseih
> Of course some people may want to keep the current behavior, if they have > limited number of temp tables or similar, so I had a GUC for that. I don't > think that the community would really welcome such GUC for core-postgres, > especially since it wouldn't be pg_stat_statements specific. FWIW,

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Greg Sabino Mullane
On Tue, Feb 11, 2025 at 7:08 PM Michael Paquier wrote: > On Mon, Feb 10, 2025 at 02:02:10PM -0600, Sami Imseih wrote: > > I am OK with moving away from "jumble" in-lieu of something else, but my > thoughts are we should actually call this process "fingerprint" > I agree fingerprint is the right

Re: Allow io_combine_limit up to 1MB

2025-02-11 Thread Thomas Munro
On Wed, Feb 12, 2025 at 3:22 PM Andres Freund wrote: > On 2025-02-12 13:59:21 +1300, Thomas Munro wrote: > > How about just maintaining it in a new variable > > effective_io_combine_limit, whenever either of them is assigned? > > Yea, that's probably the least bad way. > > I wonder if we should ju

Re: Allow io_combine_limit up to 1MB

2025-02-11 Thread Andres Freund
Hi, On 2025-02-12 13:59:21 +1300, Thomas Munro wrote: > How about just maintaining it in a new variable > effective_io_combine_limit, whenever either of them is assigned? Yea, that's probably the least bad way. I wonder if we should just name that variable io_combine_limit and have the GUC be _r

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Julien Rouhaud
On Wed, Feb 12, 2025 at 10:59:04AM +0900, Michael Paquier wrote: > On Wed, Feb 12, 2025 at 09:20:53AM +0800, Julien Rouhaud wrote: > > > > FTR my main motivation was to be able to deal with queries referencing > > temporary relations, as if your application creates a lot of those it > > basically

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Michael Paquier
On Wed, Feb 12, 2025 at 09:20:53AM +0800, Julien Rouhaud wrote: > On Wed, Feb 12, 2025 at 09:08:00AM +0900, Michael Paquier wrote: >> Wikipedia seems to agree with you that "fingerprint" would fit for >> this purpose, though: >> https://en.wikipedia.org/wiki/Fingerprint_(computing) >> >> Has anybod

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Julien Rouhaud
On Wed, Feb 12, 2025 at 09:08:00AM +0900, Michael Paquier wrote: > Wikipedia seems to agree with you that "fingerprint" would fit for > this purpose, though: > https://en.wikipedia.org/wiki/Fingerprint_(computing) > > Has anybody any comments about that? That would be a large renaming, > but in th

RE: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Hayato Kuroda (Fujitsu)
Dear Shubham, Thanks for updating the patch! I feel the patch has good shape. Here is a small comment. ``` + /* Error if no databases were found on the source server */ + if (num_rows == 0) + { + pg_log_error("no databases found on the source server"); +

Re: Confine vacuum skip logic to lazy_scan_skip

2025-02-11 Thread Melanie Plageman
On Thu, Feb 6, 2025 at 1:06 PM Melanie Plageman wrote: > > On Wed, Feb 5, 2025 at 5:26 PM Melanie Plageman > wrote: > > > > Yes, looking at these results, I also feel good about it. I've updated > > the commit metadata in attached v14, but I could use a round of review > > before pushing it. > >

Re: Allow io_combine_limit up to 1MB

2025-02-11 Thread Thomas Munro
On Wed, Feb 12, 2025 at 1:03 PM Andres Freund wrote: > On 2025-02-11 13:12:17 +1300, Thomas Munro wrote: > > I was also > > anticipating future code that would need to multiply that number by other > > terms to allocate shared memory, but after some off-list discussion, that > > seems OK: such cod

Re: WAL-logging facility for pgstats kinds

2025-02-11 Thread Michael Paquier
On Mon, Feb 10, 2025 at 11:43:30AM -0500, Andres Freund wrote: > Because I saw this being moved to the new CF: I continue to *strenuously* > object to this design. As outlined upthread, I think it's going into the > completely wrong direction. Right. FWIW, I'm not sure that we can absolutely just

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Michael Paquier
On Mon, Feb 10, 2025 at 02:02:10PM -0600, Sami Imseih wrote: > I am OK with moving away from "jumble" in-lieu of something else, but > my thoughts are we should actually call this process "fingerprint" > ( a term we already use in the queryjumblefuncs.c comment ). > A fingerprint consists of all th

Re: Allow io_combine_limit up to 1MB

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 13:12:17 +1300, Thomas Munro wrote: > Tomas queried[1] the limit of 256kB (or really 32 blocks) for > io_combine_limit. Yeah, I think we should increase it and allow > experimentation with larger numbers. Note that real hardware and > protocols have segment and size limits that

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-11 Thread Peter Smith
On Tue, Feb 11, 2025 at 9:16 PM Shubham Khanna wrote: > > > #13. Unanswered question "How are tests expecting this even passing?". > > Was a reason identified? IOW, how can we be sure the latest tests > > don't have a similar problem? > > > > In the v4-0001 patch [1], the tests were mistakenly usi

Re: [PATCH] Optionally record Plan IDs to track plan changes for a query

2025-02-11 Thread Michael Paquier
On Mon, Feb 10, 2025 at 02:14:09PM -0600, Sami Imseih wrote: > Another thought that I have is that If we mention that extensions can use > these jumbling ( or whatever the final name is ) functions outside of > core, it makes > sense to actually show an example of this. What do you think? Not sure

Re: pg_rewind with --write-recovery-conf option doesn't write dbname to primary_conninfo value.

2025-02-11 Thread Masahiko Sawada
On Mon, Feb 3, 2025 at 12:36 PM Masahiko Sawada wrote: > > On Sun, Feb 2, 2025 at 9:50 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Sawada-san, > > > > > I think it's a good idea to support it at least on HEAD. I've attached > > > a patch for that. > > > > +1. I've confirmed that pg_rewind an

Re: Test to dump and restore objects left behind by regression

2025-02-11 Thread Michael Paquier
On Tue, Feb 11, 2025 at 12:19:33PM +0530, Ashutosh Bapat wrote: > Sorry for replying late here. The refactored code in > 002_compare_backups.pl has a potential to cause confusion even without > this refactoring. The differences in tablespace paths are adjusted in > compare_files() and not in the ac

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-02-11 Thread Michail Nikolaev
Hello! I realize proposed solution does not guarantee absent of false negative cases... It happens because I am looking just at XID values, but them have nothing with transaction commitment order in the common case. I'll look for some other option. Best regards, Mikhail.

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 17:55:39 -0500, Tom Lane wrote: > Andres Freund writes: > > So the issue would actually be that we're currently doing set_max_safe_fds() > > too late, not too early :/ > > Well, we'd rather set_max_safe_fds happen after semaphore creation, > so that it doesn't have to be explic

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Tomas Vondra writes: > On 2/11/25 21:18, Tom Lane wrote: >> I think what we actually would like to know is how often we have to >> close an open FD in order to make room to open a different file. >> Maybe that's the same thing you mean by "cache miss", but it doesn't >> seem like quite the right t

Re: describe special values in GUC descriptions more consistently

2025-02-11 Thread Nathan Bossart
On Tue, Feb 11, 2025 at 11:41:51PM +0100, Daniel Gustafsson wrote: >> On 11 Feb 2025, at 19:11, Nathan Bossart wrote: > >> I thought about this one a bit, and I actually came to the opposite >> conclusion. IMHO it's reasonably obvious that an empty string means that >> the file isn't loaded, so

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Andres Freund writes: > In the current version of the AIO patchset, the creation of those io_uring > instances does happen as part of an shmem init callback, as the io uring > creation also sets up queues visible in shmem. Hmm. > So the issue would actually be that we're currently doing set_max_

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tomas Vondra
On 2/11/25 22:14, Andres Freund wrote: > Hi, > > On 2025-02-11 21:04:25 +0100, Tomas Vondra wrote: >> I agree the defaults may be pretty low for current systems, but do we >> want to get into the business of picking a value and overriding whatever >> value is set by the sysadmin? I don't think

Re: Track the amount of time waiting due to cost_delay

2025-02-11 Thread Nathan Bossart
On Tue, Feb 11, 2025 at 08:51:15AM +, Bertrand Drouvot wrote: > On Mon, Feb 10, 2025 at 02:52:46PM -0600, Nathan Bossart wrote: >> Off-list, I've asked Bertrand to gauge the feasibility of adding this >> information to the autovacuum logs and to VACUUM/ANALYZE (VERBOSE). IMHO >> those are natu

Re: describe special values in GUC descriptions more consistently

2025-02-11 Thread Daniel Gustafsson
> On 11 Feb 2025, at 19:11, Nathan Bossart wrote: > I thought about this one a bit, and I actually came to the opposite > conclusion. IMHO it's reasonably obvious that an empty string means that > the file isn't loaded, so there's not much point in stating it in the GUC > description. Instead,

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tomas Vondra
On 2/11/25 21:18, Tom Lane wrote: > Tomas Vondra writes: >> I did run into bottlenecks due to "too few file descriptors" during a >> recent experiments with partitioning, which made it pretty trivial to >> get into a situation when we start trashing the VfdCache. I have a >> half-written draft

Re: describe special values in GUC descriptions more consistently

2025-02-11 Thread Peter Smith
I don't have an opinion about the ssl_crl stuff. Everything else looks good to me. == Kind Regards, Peter Smith. Fujitsu Australia.

Re: PATCH: Disallow a netmask of zero unless the IP is also all zeroes

2025-02-11 Thread Greg Sabino Mullane
On Tue, Feb 11, 2025 at 3:25 PM Tom Lane wrote: > More generally, should we reject if the netmask causes *any* nonzero > IP bits to be ignored? Our CIDR type already imposes that rule: > Yeah, I like that idea a lot. That's a great DETAIL message. Cheers, Greg -- Crunchy Data - https://www.cr

Re: Proposal: allow non-masked IPs inside of pg_hba.conf

2025-02-11 Thread Greg Sabino Mullane
> > This too would work better if hba.c were sharing cidr_in's logic: +1, a two-for-one solution. Cheers, Greg -- Crunchy Data - https://www.crunchydata.com Enterprise Postgres Software Products & Tech Support

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 16:18:37 -0500, Tom Lane wrote: > Andres Freund writes: > > And when using something like io_uring for AIO, it'd allow to > > max_files_per_process in addition to the files requires for the io_uring > > instances. > > Not following? Surely we'd not be configuring that so early

Re: Skip collecting decoded changes of already-aborted transactions

2025-02-11 Thread Masahiko Sawada
On Mon, Feb 3, 2025 at 10:41 AM Masahiko Sawada wrote: > > On Wed, Jan 29, 2025 at 11:12 PM Peter Smith wrote: > > > > On Tue, Jan 28, 2025 at 9:26 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Jan 27, 2025 at 7:01 PM Peter Smith wrote: > > > > > > ... > > > > > > To be honest, I didn't un

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-02-11 Thread Masahiko Sawada
On Fri, Jan 24, 2025 at 11:16 AM Masahiko Sawada wrote: > > On Thu, Jan 23, 2025 at 3:24 AM Ashutosh Bapat > wrote: > > > > On Thu, Jan 23, 2025 at 6:16 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Jan 10, 2025 at 12:33 AM Masahiko Sawada > > > wrote: > > > > > > > > On Thu, Jan 9, 2025

Re: Proposal: "query_work_mem" GUC, to distribute working memory to the query's individual operators

2025-02-11 Thread Jeff Davis
On Tue, 2025-02-11 at 10:39 -0800, James Hunter wrote: > * The Path would store "nbytes" (= the optimizer's estimate of how > much working memory a given Path will use), to allow for future > optimizer logic to consider memory usage when choosing the best Path. > > * The Plan would store a copy of

Re: pgbench with partitioned tables

2025-02-11 Thread Melanie Plageman
On Tue, Feb 11, 2025 at 2:29 PM Melanie Plageman wrote: > > I was testing with this with the intent to commit it and noticed that > it does change behavior in one way -- previously if you created an > unlogged table with the same schema as one of the pgbench tables and > then used client-side data

RE: Improve CRC32C performance on SSE4.2

2025-02-11 Thread Devulapalli, Raghuveer
Hello, Attached v3 which is same as v2 with the added PCLMULQDQ runtime CPUID check. > > I ran the same benchmark drive_crc32c with the postgres infrastructure and > found that your v2 sse42 version from corsix is slower than > pg_comp_crc32c_sse42 in master branch when buffer is < 128 bytes. >

Re: AIO v2.3

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 11:48:38 +1300, Thomas Munro wrote: > Would the API be better like this?: When you want to create a batch > of I/Os submitted together, you wrap the work in pgaio_begin_batch() > and pgaio_submit_batch(), eg the loop in read_stream_lookahead(). One annoying detail is that an AP

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
On Mon, 10 Feb 2025 at 19:15, Burd, Greg wrote: > > Apologies for not being clear, this preserves the current behavior for > summarizing indexes allowing for HOT updates while also updating the index. > No degradation here that I’m aware of, indeed the tests that ensure that > behavior are unc

Re: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-11 Thread Peter Smith
On Wed, Feb 12, 2025 at 12:36 AM Nisha Moond wrote: > > On Tue, Feb 11, 2025 at 8:49 AM Peter Smith wrote: > > > > Hi Nisha. > > > > Some review comments about v74-0001 > > > > == > > src/backend/replication/slot.c > > > > 1. > > /* Maximum number of invalidation causes */ > > -#define RS_IN

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Andres Freund writes: > My suggestion would be to redefine max_files_per_process as the number of > files we try to be able to open in backends. I.e. set_max_safe_fds() would > first count the number of already open fds (since those will largely be > inherited by child processes) and then check if

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
On Mon, 10 Feb 2025 at 20:11, Burd, Greg wrote: > > On Feb 10, 2025, at 12:17 PM, Matthias van de Meent > > wrote: > > > >> > >> I have a few concerns with the patch, things I’d greatly appreciate your > >> thoughts on: > >> > >> First, I pass an EState along the update path to enable running t

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 21:04:25 +0100, Tomas Vondra wrote: > I agree the defaults may be pretty low for current systems, but do we > want to get into the business of picking a value and overriding whatever > value is set by the sysadmin? I don't think a high hard limit should be > seen as an implicit p

Re: PATCH: Disallow a netmask of zero unless the IP is also all zeroes

2025-02-11 Thread Daniel Gustafsson
> On 11 Feb 2025, at 21:25, Tom Lane wrote: > I'm a bit distressed to realize that hba.c isn't using cidr_in. > Maybe we should try to share code instead of duplicating yet more. +1. I have a note along these lines on my never-shrinking TODO, I think it would be great if we took a stab at that.

Re: explain analyze rows=%.0f

2025-02-11 Thread Tom Lane
Robert Haas writes: > On Tue, Feb 11, 2025 at 12:14 PM Andrei Lepikhov wrote: >> I support the idea in general, but I believe it should be expanded to >> cover all cases of parameterised plan nodes. Each rescan iteration may >> produce a different number of tuples, and rounding can obscure import

Re: Proposal: allow non-masked IPs inside of pg_hba.conf

2025-02-11 Thread Tom Lane
Greg Sabino Mullane writes: > Currently, all IP addresses in the address field of records in the > pg_hba.conf file require a CIDR mask. > ... > I'd like to relax that requirement, such that a lack of an explicit mask > defaults to a /32 mask, allowing that first example to work as most people > w

Re: Separate GUC for replication origins

2025-02-11 Thread Euler Taveira
On Wed, Feb 5, 2025, at 9:49 PM, Masahiko Sawada wrote: > On Wed, Feb 5, 2025 at 4:39 PM Euler Taveira wrote: > > > > On Wed, Feb 5, 2025, at 1:56 AM, Amit Kapila wrote: > > > > On Wed, Feb 5, 2025 at 8:17 AM Euler Taveira wrote: > > > > > > Under reflection, an accurate name is > > > max_replic

Re: PATCH: Disallow a netmask of zero unless the IP is also all zeroes

2025-02-11 Thread Tom Lane
Greg Sabino Mullane writes: > I say "of course" but few people (even tech ones) know the distinction. > (Nor should they have to! But that's for a nearby thread). This patch aims > to prevent this very bad footgun by only allowing a /0 if the IP consists > of only zeroes. It works for ipv4 and ipv

Re: explain analyze rows=%.0f

2025-02-11 Thread Andrei Lepikhov
On 12/2/2025 00:41, Robert Haas wrote: On Tue, Feb 11, 2025 at 12:14 PM Andrei Lepikhov wrote: Moreover, switching to a floating-point type for row explanations in each parameterised node would provide a more comprehensive view and add valuable information about the parameterisation of the node

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Tomas Vondra writes: > I did run into bottlenecks due to "too few file descriptors" during a > recent experiments with partitioning, which made it pretty trivial to > get into a situation when we start trashing the VfdCache. I have a > half-written draft of a blog post about that somewhere. > But

Re: pgbench with partitioned tables

2025-02-11 Thread Melanie Plageman
On Tue, Feb 11, 2025 at 2:29 PM Melanie Plageman wrote: > > On Tue, Feb 11, 2025 at 3:10 AM Sergey Tatarintsev > wrote: > > > > 08.02.2025 17:32, Álvaro Herrera пишет: > > > On 2025-Feb-07, Melanie Plageman wrote: > > > > > >> Okay, I've stared at this a bit, and it seems basically fine the way >

PATCH: Disallow a netmask of zero unless the IP is also all zeroes

2025-02-11 Thread Greg Sabino Mullane
I ran into this alarming mistake again the other day. Luckily it was on a dev system. Someone sees an entry in a pg_hba.conf that looks like this: host all all 0.0.0.0/0 md5 They are gobsmacked when they learn this means to let everyone in. So they fix it by adding new entries that look like this

Proposal: allow non-masked IPs inside of pg_hba.conf

2025-02-11 Thread Greg Sabino Mullane
Currently, all IP addresses in the address field of records in the pg_hba.conf file require a CIDR mask. This means that rather than the intuitive way to add a single IP address: host postgres alice 10.128.0.10 scram-sha-256 One must do: host postgres alice 10.128.0.10/32 scram-sha-256 I'd li

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tomas Vondra
On 2/11/25 20:20, Tom Lane wrote: > Jelte Fennema-Nio writes: >> The default open file limit of 1024 is extremely low, given modern >> resources and kernel architectures. The reason that this hasn't changed >> change is because doing so would break legacy programs that use the >> select(2) system

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Andres Freund writes: > I don't see a downside to just increasing the soft limit for > pgbench. Agreed, that end of the patch seems relatively harmless. regards, tom lane

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
I wrote: > Maybe a sanity limit on how high we'll try to raise the ulimit > would help. Oh, I'd forgotten that we already have one: max_files_per_process. Since that's only 1000 by default, this patch doesn't actually have any effect (on Linux anyway) unless the DBA raises max_files_per_process.

Re: injection points for hash aggregation

2025-02-11 Thread Jeff Davis
On Mon, 2025-02-03 at 15:54 -0800, Jeff Davis wrote: > Attaching > v2. I committed just the injection points part. It seemed simple and low- risk. Regards, Jeff Davis

Re: Fix assert failure when decoding XLOG_PARAMETER_CHANGE on primary

2025-02-11 Thread Masahiko Sawada
On Mon, Feb 10, 2025 at 4:11 AM Amit Kapila wrote: > > On Fri, Feb 7, 2025 at 11:30 PM Masahiko Sawada wrote: > > > > On Thu, Feb 6, 2025 at 9:30 PM Amit Kapila wrote: > > > > > > True but it sounds like there is more harm than benefit. It seems > > > reasonable to do this on HEAD. Shall we thin

Re: explain analyze rows=%.0f

2025-02-11 Thread Ilia Evdokimov
On 11.02.2025 20:41, Robert Haas wrote: On Tue, Feb 11, 2025 at 12:14 PM Andrei Lepikhov wrote: I support the idea in general, but I believe it should be expanded to cover all cases of parameterised plan nodes. Each rescan iteration may produce a different number of tuples, and rounding can ob

Re: explain analyze rows=%.0f

2025-02-11 Thread Alena Rybakina
Hi! Thank you for your valuable work on this! On 11.02.2025 22:18, Ilia Evdokimov wrote: On 11.02.2025 20:41, Robert Haas wrote: On Tue, Feb 11, 2025 at 12:14 PM Andrei Lepikhov wrote: I support the idea in general, but I believe it should be expanded to cover all cases of parameterised plan

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 19:52:34 +0100, Jelte Fennema-Nio wrote: > So this starts bumping postmaster and pgbench its soft open file limit > to the hard open file limit. Not sure that's quite the right thing to do for postmaster. What I'd start with is to increase the soft limit to "already used files

Re: pgbench with partitioned tables

2025-02-11 Thread Melanie Plageman
On Tue, Feb 11, 2025 at 3:10 AM Sergey Tatarintsev wrote: > > 08.02.2025 17:32, Álvaro Herrera пишет: > > On 2025-Feb-07, Melanie Plageman wrote: > > > >> Okay, I've stared at this a bit, and it seems basically fine the way > >> it is (I might add a bit more whitespace, clean up the commit message

Re: Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Tom Lane
Jelte Fennema-Nio writes: > The default open file limit of 1024 is extremely low, given modern > resources and kernel architectures. The reason that this hasn't changed > change is because doing so would break legacy programs that use the > select(2) system call in hard to debug ways. So instead p

Re: Statistics Import and Export

2025-02-11 Thread Corey Huinker
> > > > Comments on v45-0002: > Assuming you meant 47 > > * Why is generate_old_dump() passing optionally passing --no-statistics > to pg_dumpall along with --globals-only? If --globals-only is > specified, no stats are dumped anyway, right? > Removed. > > * The tag is still wrong: it is "STA

Re: pg_stat_statements and "IN" conditions

2025-02-11 Thread Dmitry Dolgov
> On Tue, Feb 11, 2025 at 07:51:43PM GMT, Álvaro Herrera wrote: > On 2025-Feb-11, Dmitry Dolgov wrote: > > > > On Tue, Feb 11, 2025 at 10:49:59AM GMT, Sami Imseih wrote: > > > I have only looked at 0001, but I am wondering why > > > query_id_const_merge is a pg_stat_statements GUC > > > rather than

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

2025-02-11 Thread Melanie Plageman
On Tue, Feb 11, 2025 at 11:31 AM Andres Freund wrote: > > On 2025-02-05 12:23:29 -0500, Melanie Plageman wrote: > > Attached v16 implements the logic to not count pages we failed to > > freeze because of cleanup lock contention as eager freeze failures. > > That looks good to me. Cool. Committed

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-02-11 Thread Andres Freund
Hi, On 2025-02-11 18:45:13 +0100, Jelte Fennema-Nio wrote: > On Tue, 11 Feb 2025 at 17:19, Andres Freund wrote: > > Yes, at least initially: > > Ah, then I understand your point of view much better. Still I think we > could easily frame it as: If you enable io_uring, you also get these > additio

Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

2025-02-11 Thread Jelte Fennema-Nio
The default open file limit of 1024 is extremely low, given modern resources and kernel architectures. The reason that this hasn't changed change is because doing so would break legacy programs that use the select(2) system call in hard to debug ways. So instead programs that want to opt-in to a hi

Re: pg_stat_statements and "IN" conditions

2025-02-11 Thread Álvaro Herrera
On 2025-Feb-11, Sami Imseih wrote: > I have only looked at 0001, but I am wondering why > query_id_const_merge is a pg_stat_statements GUC > rather than a core GUC? I was wondering the same thing and found the explanation here: https://postgr.es/m/ztmuctymis3n3...@paquier.xyz > Other extensions

Re: pg_stat_statements and "IN" conditions

2025-02-11 Thread Álvaro Herrera
On 2025-Feb-11, Dmitry Dolgov wrote: > > On Tue, Feb 11, 2025 at 10:49:59AM GMT, Sami Imseih wrote: > > I have only looked at 0001, but I am wondering why > > query_id_const_merge is a pg_stat_statements GUC > > rather than a core GUC? > > It was moved from being a core GUC into a pg_stat_stateme

Re: read stream on amcheck

2025-02-11 Thread Matheus Alcantara
Hi, Em ter., 11 de fev. de 2025 às 03:39, jian he escreveu: > hi. some minor issue i found. > > +#include "storage/block.h" > no need, since "#include "storage/bufmgr.h" already included it. > Fixed > do we need to add ``CHECK_FOR_INTERRUPTS()`` in heapam_read_stream_next_block? > The current co

Re: Proposal: "query_work_mem" GUC, to distribute working memory to the query's individual operators

2025-02-11 Thread James Hunter
On Tue, Feb 11, 2025 at 10:00 AM Jeff Davis wrote: > > On Mon, 2025-02-10 at 19:09 -0800, James Hunter wrote: > > I think it makes sense to split the work into two parts: one part > > that > > improves SQL execution, and a second part that improves the > > optimizer, > > to reflect the improvement

Re: pg_stat_statements and "IN" conditions

2025-02-11 Thread Dmitry Dolgov
> On Tue, Feb 11, 2025 at 10:49:59AM GMT, Sami Imseih wrote: > I have only looked at 0001, but I am wondering why > query_id_const_merge is a pg_stat_statements GUC > rather than a core GUC? It was moved from being a core GUC into a pg_stat_statements GUC on the request from the reviewers. Communi

Re: pg_stat_statements and "IN" conditions

2025-02-11 Thread Álvaro Herrera
On 2025-Feb-11, Sami Imseih wrote: > I do not have an explanation from the patch yet, but I have a test > that appears to show unexpected results. I only tested a few datatypes, > but from what I observe, some merge as expected and others do not; > i.e. int columns merge correctly but bigint do no

Re: describe special values in GUC descriptions more consistently

2025-02-11 Thread Nathan Bossart
Thank you for the reviews. On Mon, Feb 10, 2025 at 05:25:42PM -0700, David G. Johnston wrote: > On Mon, Feb 10, 2025 at 4:53 PM Peter Smith wrote: >> {"shared_memory_size_in_huge_pages", PGC_INTERNAL, PRESET_OPTIONS, >> gettext_noop("Shows the number of huge pages needed for the main >> share

Re: Proposal: "query_work_mem" GUC, to distribute working memory to the query's individual operators

2025-02-11 Thread Jeff Davis
On Mon, 2025-02-10 at 19:09 -0800, James Hunter wrote: > I think it makes sense to split the work into two parts: one part > that > improves SQL execution, and a second part that improves the > optimizer, > to reflect the improvements to execution. I like the idea to store the value of work_mem in

Re: Expanding HOT updates for expression and partial indexes

2025-02-11 Thread Matthias van de Meent
On Tue, 11 Feb 2025 at 00:20, Nathan Bossart wrote: > > On Mon, Feb 10, 2025 at 06:17:42PM +0100, Matthias van de Meent wrote: > > I have serious doubts about the viability of any proposal working to > > implement PHOT/WARM in PostgreSQL, as they seem to have an inherent > > nature of fundamentall

  1   2   >