Fsync (flush) all inserted WAL records

Hi Hackers, I use async commits. At some moment, I would like to make sure that all inserted WAL records are fsync-ed. I can use XLogFlush function but I have some doubts which LSN to specify. There is a number of functions which return write or insert LSNs but they are not applicable. I can'

Re: BlastRADIUS mitigation

On Mon, Aug 05, 2024 at 05:41:21PM +0300, Heikki Linnakangas wrote: > On 05/08/2024 15:43, Thomas Munro wrote: >> Since PostgreSQL v12 and v13 don't have the modern "common/hmac.h" >> API, I came up with a cheap kludge: locally #define those interfaces >> to point directly to the OpenSSL HMAC API,

Re: pg_verifybackup: TAR format backup verification

On Mon, Aug 5, 2024 at 10:29 PM Robert Haas wrote: > > On Fri, Aug 2, 2024 at 7:43 AM Amul Sul wrote: > > Please consider the attached version for the review. > > Thanks. I committed 0001-0003. The only thing that I changed was that > in 0001, you forgot to pgindent, which actually mattered quite

Instability with incremental backup tests (pg_combinebackup, 003_timeline.pl)

Hi all, dikkop has reported a failure with the regression tests of pg_combinebackup: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dikkop&dt=2024-08-04%2010%3A04%3A51 That's in the test 003_timeline.pl, from dc212340058b: # Failed test 'incremental backup from node1' # at t/003_time

Re: Do we still need parent column in pg_backend_memory_context?

On Wed, 31 Jul 2024 at 13:27, Tom Lane wrote: > David Rowley writes: > > One thing we could do is remove it and see if anyone complains. If we > > did that today, there's about a year-long window for people to > > complain where we could still reverse the decision. > > Seems like a plausible com

Re: Remove support for old realpath() API

On Mon, Aug 05, 2024 at 10:08:04AM -0400, Tom Lane wrote: > Heikki Linnakangas writes: >> We don't seem to have any mentions of POSIX or SuS in docs, in the >> installation sections. There are a few mentions of POSIX-1.2008 and >> POSIX-1.2001 it in the commit log, though, where we require featu

Re: [HACKERS] make async slave to wait for lsn to be replayed

On Tue, Aug 06, 2024 at 05:17:10AM +0300, Alexander Korotkov wrote: > The 0001 patch is intended to improve this situation. Actually, it's > not right to just put RecoveryInProgress() after > GetXLogReplayRecPtr(), because more wal could be replayed between > these calls. Instead we need to reche

Re: Logical Replication of sequences

On Tue, Aug 6, 2024 at 9:54 AM Amit Kapila wrote: > > On Tue, Aug 6, 2024 at 8:49 AM shveta malik wrote: > > > > > > > I was reviewing the design of patch003, and I have a query. Do we need > > > > > to even start an apply worker and create replication slot when > > > > > subscription created is

Re: Logical Replication of sequences

On Tue, Aug 6, 2024 at 8:49 AM shveta malik wrote: > > On Mon, Aug 5, 2024 at 5:28 PM Amit Kapila wrote: > > > > On Mon, Aug 5, 2024 at 2:36 PM vignesh C wrote: > > > > > > On Fri, 2 Aug 2024 at 14:24, shveta malik wrote: > > > > > > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik > > > > wro

Re: Fix memory counter update in reorderbuffer

On Sat, Aug 3, 2024 at 1:21 AM Masahiko Sawada wrote: > > I found a bug in the memory counter update in reorderbuffer. It was > introduced by commit 5bec1d6bc5e, so pg17 and master are affected. > > In ReorderBufferCleanupTXN() we zero the transaction size and then > free the transaction entry as

Re: Logical Replication of sequences

On Tue, Aug 6, 2024 at 8:49 AM shveta malik wrote: > Do we need some kind of coordination between table sync and sequence sync for internally generated sequences? Lets say we have an identity column with a 'GENERATED ALWAYS' sequence. When the sequence is synced to subscriber, subscriber can also

2024-08-08 update release announcement draft

Hi, Attached is the release announcement draft for the 2024-08-08 release. Please review for accuracy and omissions, and provide feedback no later than 2024-08-08 12:00 UTC. Thanks! Jonathan The PostgreSQL Global Development Group has released an update to all supported versions of PostgreSQ

Re: Logical Replication of sequences

On Mon, Aug 5, 2024 at 5:28 PM Amit Kapila wrote: > > On Mon, Aug 5, 2024 at 2:36 PM vignesh C wrote: > > > > On Fri, 2 Aug 2024 at 14:24, shveta malik wrote: > > > > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik > > > wrote: > > > > > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C wrote:

Re: [HACKERS] make async slave to wait for lsn to be replayed

On Sat, Aug 3, 2024 at 6:07 AM Alexander Korotkov wrote: > On Sat, Aug 3, 2024 at 3:45 AM Kevin Hale Boyes wrote: > > In the for loop in WaitForLSNReplay, shouldn't the check for in-recovery be > > moved up above the call to GetXLogReplayRecPtr? > > If we get promoted while waiting for the timeo

Re: SQL:2011 application time

On Fri, Aug 2, 2024 at 1:09 AM Paul Jungwirth wrote: > > On 7/25/24 08:52, Paul Jungwirth wrote: > > Here is a patch moving the not-empty check into > > check_exclusion_or_unique_constraint. That is a more > > logical place for it than ExecConstraints, since WITHOUT OVERLAPS is part > > of the i

Re: Missing reflection in nls.mk from recent basebackup changes

On Tue, Aug 06, 2024 at 10:21:23AM +0900, Kyotaro Horiguchi wrote: > After a recent commit f80b09bac8, make update-po fails with "missing > required files". The commit moved some files to fe_utils but this > change was not reflected in pg_basebackup's nls.mk. The attached patch > fixes this issue.

Missing reflection in nls.mk from recent basebackup changes

Hello. After a recent commit f80b09bac8, make update-po fails with "missing required files". The commit moved some files to fe_utils but this change was not reflected in pg_basebackup's nls.mk. The attached patch fixes this issue. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >F

Re: Make tuple deformation faster

On Tue, 16 Jul 2024 at 00:13, Matthias van de Meent wrote: > > On Tue, 2 Jul 2024 at 02:23, David Rowley wrote: > > I'm happy to keep going with this version of the patch > > +1, go for it. I've attached an updated patch series which are a bit more polished than the last set. I've attempted to d

Re: BlastRADIUS mitigation

On Tue, Aug 6, 2024 at 2:41 AM Heikki Linnakangas wrote: > Seems that on linux or freebsd, you'd plow ahead even if the binary is > not found, and fail later, while on macOS you'd skip the tests. I think > we should always error out if the dependencies are not found. If you > make an effort to add

Re: POC, WIP: OR-clause support for indexes

Ok, thank you for your work) I think we can leave only the two added libraries in the first patch, others are superfluous. On 05.08.2024 22:48, Alexander Korotkov wrote: On Mon, Jul 29, 2024 at 5:36 AM Alexander Korotkov wrote: On Sun, Jul 28, 2024 at 12:59 PM Alena Rybakina Because of the

Re: Restart pg_usleep when interrupted

v4-0001-vaccum_delay-with-absolute-time-nanosleep.patch Description: Binary data > yeah, we already have a few macros that access the .ticks, so maybe we could > add > 2 new ones, say: > > 1. INSTR_TIME_ADD_MS(t1, msec) > 2. INSTR_TIME_IS_GREATER(t1, t2) > > I think the less operations is don

Re: POC, WIP: OR-clause support for indexes

On Mon, Jul 29, 2024 at 5:36 AM Alexander Korotkov wrote: > On Sun, Jul 28, 2024 at 12:59 PM Alena Rybakina > > Because of these reasons, I tried to save this and that transformation > > together for each column and try to analyze for each expr separately > > which method would be optimal. > > Yes

Re: BlastRADIUS mitigation

Heikki Linnakangas writes: > On 05/08/2024 15:43, Thomas Munro wrote: >> The response requirement can be enabled by radiusrequirema=1 in >> pg_hba.conf. For example, Debian stable is currently shipping >> FreeRADIUS 3.2.1 which doesn't yet send the MA in its responses, but >> FreeBSD and Debian "

Re: Is *fast* 32-bit support still important?

On Mon, Jul 29, 2024 at 4:41 PM Joel Jacobson wrote: > To me, it's non-obvious whether introducing `#if SIZEOF_DATUM < 8` with > separate 32-bit and 64-bit code paths is worthwhile to maintain performance > for both. I feel like it's probably mostly up to the patch author. Reviewers and/or the ev

Re: pg_verifybackup: TAR format backup verification

On Fri, Aug 2, 2024 at 7:43 AM Amul Sul wrote: > Please consider the attached version for the review. Thanks. I committed 0001-0003. The only thing that I changed was that in 0001, you forgot to pgindent, which actually mattered quite a bit, because astreamer is one character shorter than bbstrea

Re: Detailed release notes

On Mon, Aug 5, 2024, at 10:33 AM, Daniel Gustafsson wrote: >> On 5 Aug 2024, at 13:16, Marcos Pegoraro wrote: >> >> Em seg., 5 de ago. de 2024 às 07:54, jian he >> escreveu: >>> >>> [commitId_link1, commitId_link2]: Change functions to use a safe >>> search_path during maintenance operations (

Re: Support specify tablespace for each merged/split partition

Hi Amul, Thanks for your review. On Mon, Aug 5, 2024 at 8:38 PM Amul Sul wrote: > > On Mon, Jul 15, 2024 at 11:19 AM Junwang Zhao wrote: > > > > In [1], it is suggested that it might be a good idea to support > > specifying the tablespace for each merged/split partition. > > > > We can do the f

Re: [PATCH] Add crc32(text) & crc32(bytea)

On Mon, Aug 05, 2024 at 04:19:45PM +0300, Aleksander Alekseev wrote: > Thanks, PFA patch v3. This looks pretty good to me. The only point that I think deserves more discussion is the return type. Does bytea make the most sense here? Or should we consider int/bigint? -- nathan

Re: WIP: parallel GiST index builds

Hi, Here's an updated patch using GetFakeLSNForUnloggedRel() instead of the atomic counter. I think this looks much nicer and less invasive, as it simply uses XLogCtl shared memory (instead of having to pass a new pointer everywhere). We still need to pass the is_parallel flag, though. I wonder i

Re: psql client does not handle WSAEWOULDBLOCK on Windows

I have not reproduce your test scenario, looking at code please see following comments: If you check the function definition of pqsecure_raw_read() it actually do set errno like bellow SOCK_ERRNO_SET(result_errno); where result_errno = SOCK_ERRNO Means anybody using those function pqsecure_raw

Re: Amcheck verification of GiST and GIN

Hi, I've spent a bit more time looking at the GiST part as part of my "parallel GiST build" patch nearby, and I think there's some sort of memory leak. Consider this: create table t (a text); insert into t select md5(i::text) from generate_series(1,2500) s(i); create index on t u

Re: BlastRADIUS mitigation

On 05/08/2024 15:43, Thomas Munro wrote: The response requirement can be enabled by radiusrequirema=1 in pg_hba.conf. For example, Debian stable is currently shipping FreeRADIUS 3.2.1 which doesn't yet send the MA in its responses, but FreeBSD and Debian "testing" have started shipping FreeRADIU

Re: Remove support for old realpath() API

Heikki Linnakangas writes: > We don't seem to have any mentions of POSIX or SuS in docs, in the > installation sections. There are a few mentions of POSIX-1.2008 and > POSIX-1.2001 it in the commit log, though, where we require features > specified by those. Can we rely on everything from POSIX

Re: meson "experimental"?

Hi, > Agreed. I've pushed your patch for that: > > > Here are corresponding patches. > > The ID is new in v17, so I also renamed it like > s/installation-notes-visual/installation-notes-visual-studio/. > > (I didn't examine or push your other patch, which was about $SUBJECT.) Thanks. Re-attachi

Re: Remove obsolete RECHECK keyword completely

Hi, > I propose to remove the obsolete RECHECK keyword completely. This used > to be part of CREATE OPERATOR CLASS and ALTER OPERATOR FAMILY, but it > has done nothing (except issue a NOTICE) since PostgreSQL 8.4. Commit > 30e7c175b81 removed support for dumping from pre-9.2 servers, so this no

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

On 8/5/24 09:14, Tom Lane wrote: Joe Conway writes: I know I am way late to this thread, and I have only tried a cursory skim of it given the length, but have we made any kind of announcement (packagers at least?) that we intend to not support Postgres 18 with ssl on RHEL 7.9 and derivatives?

Re: Detailed release notes

> On 5 Aug 2024, at 13:16, Marcos Pegoraro wrote: > > Em seg., 5 de ago. de 2024 às 07:54, jian he > escreveu: >> >> [commitId_link1, commitId_link2]: Change functions to use a safe >> search_path during maintenance operations (Jeff Davis) > > I don't like t

Re: [PATCH] Add crc32(text) & crc32(bytea)

Hi, > I'm curious why we need to do this instead of only using the macros: > > INIT_TRADITIONAL_CRC32(crc); > COMP_TRADITIONAL_CRC32(crc, VARDATA_ANY(in), len); > FIN_TRADITIONAL_CRC32(crc); > > + * IDENTIFICATION > + *src/backend/utils/adt/hashfuncs.c > > Perhaps these would fit i

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

Joe Conway writes: > I know I am way late to this thread, and I have only tried a cursory > skim of it given the length, but have we made any kind of announcement > (packagers at least?) that we intend to not support Postgres 18 with ssl > on RHEL 7.9 and derivatives? Yes, RHEL 7 just passed EO

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

Hi Joe, On Mon, 2024-08-05 at 08:38 -0400, Joe Conway wrote: > I know I am way late to this thread, and I have only tried a cursory > skim of it given the length, but have we made any kind of announcement > (packagers at least?) that we intend to not support Postgres 18 with > ssl on RHEL 7.9 an

Re: SQL Property Graph Queries (SQL/PGQ)

Hi Imran, On Sun, Aug 4, 2024 at 12:32 PM Imran Zaheer wrote: > > Hi > I am attaching a new patch for a minor feature addition. > > - Adding support for 'Labels and properties: EXCEPT list' Do you intend to support EXCEPT in the label expression as well or just properties? > > Please let me kno

Re: Is *fast* 32-bit support still important?

On Mon, Aug 5, 2024, at 14:24, Aleksander Alekseev wrote: >> Could we add a text message that is displayed to a user, >> when compiling PostgreSQL on a 32-bit platform? > > What would be actionable items depending on the results? Option A: > someone claims to use PostgreSQL on 32-bit hardware. Opti

BlastRADIUS mitigation

Hi, (This discussion started on the security@ list, but was deemed suitable for the -hackers@ list.) PostgreSQL supports RADIUS authentication[1]. It's probably not widely used, and generally any security technology based on MD5 alone should by now be considered weak and obsolete, but there are

Re: Support specify tablespace for each merged/split partition

On Mon, Jul 15, 2024 at 11:19 AM Junwang Zhao wrote: > > In [1], it is suggested that it might be a good idea to support > specifying the tablespace for each merged/split partition. > > We can do the following after this feature is supported: > > CREATE TABLESPACE tblspc LOCATION '/tmp/tblspc'; >

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

On 7/15/24 17:42, Daniel Gustafsson wrote: On 14 Jul 2024, at 14:03, Peter Eisentraut wrote: On 12.07.24 21:42, Daniel Gustafsson wrote: On 11 Jul 2024, at 23:22, Peter Eisentraut wrote: The 0001 patch removes the functions pgtls_init_library() and pgtls_init() but keeps the declarations in l

Re: Logical Replication of sequences

On Mon, Aug 5, 2024 at 11:04 AM vignesh C wrote: > > On Wed, 31 Jul 2024 at 14:39, shveta malik wrote: > > > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C wrote: > > > > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul wrote: > > > > > > > > > > > > > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C wro

Re: Optimize mul_var() for var1ndigits >= 8

On Tue, Jul 30, 2024, at 00:31, Tom Lane wrote: > Dean Rasheed writes: >> On Mon, 29 Jul 2024 at 21:39, Joel Jacobson wrote: >>> I think it's non-obvious if the separate code paths for 32-bit and 64-bit, >>> using `#if SIZEOF_DATUM < 8`, to get *fast* 32-bit support, outweighs >>> the benefits of

Re: Is *fast* 32-bit support still important?

Hi, > > Pretty much any embedded system that uses just a few GB of memory may > > win from using a 32-bit processor (not necessarily in terms of > > performance, maybe in terms of price). Think of WiFi-routers, smart > > TVs, 3D printers, etc. > > Thanks for feedback! > > Do we know of any such pr

Re: Is *fast* 32-bit support still important?

On Tue, Jul 30, 2024, at 11:06, Aleksander Alekseev wrote: > Hi Joel, > > Here are my two cents. > >> 1. Who are the current users of 32-bit PostgreSQL? > > Pretty much any embedded system that uses just a few GB of memory may > win from using a 32-bit processor (not necessarily in terms of > perfo

Re: Logical Replication of sequences

On Mon, Aug 5, 2024 at 2:36 PM vignesh C wrote: > > On Fri, 2 Aug 2024 at 14:24, shveta malik wrote: > > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik wrote: > > > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C wrote: > > > > > > > > Thanks for reporting this, these issues are fixed in the a

Re: Is *fast* 32-bit support still important?

On Tue, Jul 30, 2024, at 09:25, Heikki Linnakangas wrote: > On 29/07/2024 23:40, Joel Jacobson wrote: >> To me, it's non-obvious whether introducing `#if SIZEOF_DATUM < 8` with >> separate 32-bit and 64-bit code paths is worthwhile to maintain performance >> for both. >> >> Knowing more about $sub

Re: Adding OLD/NEW support to RETURNING

On Fri, Aug 2, 2024 at 6:13 PM Dean Rasheed wrote: > > On Fri, 2 Aug 2024 at 08:25, jian he wrote: > > > > if (resultRelInfo->ri_projectReturning && (processReturning || saveOld)) > > { > > } > > > > "saveOld" imply "resultRelInfo->ri_projectReturning" > > we can simplified it as > >

Re: scalability bottlenecks with (many) partitions (and more)

Hi, On 6/25/24 12:04, Tomas Vondra wrote: On 6/24/24 17:05, Robert Haas wrote: On Sun, Jan 28, 2024 at 4:57 PM Tomas Vondra wrote: For NUM_LOCK_PARTITIONS this is pretty simple (see 0001 patch). The LWLock table has 16 partitions by default - it's quite possible that on machine with many co

Re: Speed up JSON escape processing with SIMD plus other optimisations

On Sun, 4 Aug 2024 at 02:11, David Rowley wrote: > I did some more testing on this on a few different machines; apple M2 > Ultra, AMD 7945HX and with a Raspberry Pi 4. I did some more testing on this patch today as I wanted to see what Intel CPUs thought about it. The only modern Intel CPU I ha

Re: Detailed release notes

Em seg., 5 de ago. de 2024 às 07:54, jian he escreveu: > > [commitId_link1, commitId_link2]: Change functions to use a safe > search_path during maintenance operations (Jeff Davis) > I don't like that prefix dirtying each item. I think having just a link after every item would be better. Firstly

Re: [BUG?] check_exclusion_or_unique_constraint false negative

Hello! > Right, but we are extending this functionality to detect and resolve > such conflicts [1][2]. I am hoping after that such updates won't be > missed. Yes, this is a nice feature. However, without the DirtySnapshot index scan fix, it will fail in numerous instances, especially in master-ma

Re: Detailed release notes

On Fri, Jul 26, 2024 at 10:11 PM Tom Lane wrote: > > Daniel Gustafsson writes: > >> On 26 Jul 2024, at 15:00, Marcos Pegoraro wrote: > >> But why is that just a hidden comment and not a visible link for us ? > > > That's likely the wrong level of detail for the overwhelming majority of > > relea

Re: Conflict detection and logging in logical replication

On Sun, Aug 4, 2024 at 1:22 PM Zhijie Hou (Fujitsu) wrote: > > On Friday, August 2, 2024 7:03 PM Amit Kapila wrote: > > > > Here is the V11 patch set which addressed above and Kuroda-san[1] comments. > A few design-level points: * @@ -525,10 +602,33 @@ ExecSimpleRelationInsert(ResultRelInfo *re

Re: [BUG?] check_exclusion_or_unique_constraint false negative

On Fri, Aug 2, 2024 at 10:38 PM Michail Nikolaev wrote: > > > I think it is rather less likely or not possible in a parallel apply > > case because such conflicting updates (updates on the same tuple) > > should be serialized at the publisher itself. So one of the updates > > will be after the com

Re: Logical Replication of sequences

On Fri, 2 Aug 2024 at 14:33, shveta malik wrote: > > On Fri, Aug 2, 2024 at 2:24 PM shveta malik wrote: > > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik wrote: > > > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C wrote: > > > > > > > > Thanks for reporting this, these issues are fixed in th

Re: Logical Replication of sequences

On Fri, 2 Aug 2024 at 14:24, shveta malik wrote: > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik wrote: > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C wrote: > > > > > > Thanks for reporting this, these issues are fixed in the attached > > > v20240730_2 version patch. > > > > > I was reviewin

Re: Fixed small typo in bufpage.h

On Mon, Aug 5, 2024 at 6:54 AM Senglee Choi wrote: > > Fixed a minor typo in src/include/storage/bufpage.h. > LGTM. BTW, we do use two spaces before the new sentence. See comments in the nearby code. -- With Regards, Amit Kapila.

Re: Remove support for old realpath() API

On 05/08/2024 09:12, Peter Eisentraut wrote: The now preferred way to call realpath() is by passing NULL as the second argument and get a malloc'ed result.  We still supported the old way of providing our own buffer as a second argument, for some platforms that didn't support the new way yet.  Th

Re: Set query_id for query contained in utility statement

I've realised my initial approach was wrong, calling the post parse for all nested queries in analyze.c prevents extension like pgss to correctly track the query's nesting level. I've changed the approach to replicate what's done in ExplainQuery to both CreateTableAs and DeclareCursor: Jumble the

Re: MAINTAIN privilege -- what do we need to un-revert it?

On Fri, 2 Aug 2024 16:13:01 +0900 Yugo NAGATA wrote: > On Thu, 01 Aug 2024 11:31:53 -0700 > Jeff Davis wrote: > > > On Wed, 2024-07-31 at 18:20 +0900, Yugo NAGATA wrote: > > > I agree that it might not be important, but I think adding the flag > > > would be > > > also helpful for improving cod