Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Kyotaro Horiguchi
At Tue, 01 Aug 2023 15:28:54 +0900 (JST), Kyotaro Horiguchi wrote in > While we will not agree, we could establish a defalut behavior where > an OOM during recovery immediately triggers an ERROR. Then, we could > introduce a *GUC* that causes recovery to regard OOM as an > end-of-recovery error.

Pgoutput not capturing the generated columns

2023-07-31 Thread Rajendra Kumar Dangwal
Hi PG Users. We are using Debezium to capture the CDC events into Kafka. With decoderbufs and wal2json plugins the connector is able to capture the generated columns in the table but not with pgoutput plugin. We tested with the following example: CREATE TABLE employees ( id SERIAL PRIMARY K

Re: Fix compilation warnings when CFLAGS -Og is specified

2023-07-31 Thread Kyotaro Horiguchi
At Tue, 1 Aug 2023 04:51:55 +, "Hayato Kuroda (Fujitsu)" wrote in > Dear hackers, > > # Background > > Based on [1], I did configure and build with options: > (I used Meson build system, but it could be reproduced by Autoconf/Make) > > ``` > $ meson setup -Dcassert=true -Ddebug=true -Dc_a

Re: Simplify some logical replication worker type checking

2023-07-31 Thread Alvaro Herrera
On 2023-Aug-01, Peter Smith wrote: > PSA a small patch making those above-suggested changes. The 'make > check' and TAP subscription tests are all passing OK. I think the code ends up more readable with this style of changes, so +1. I do wonder if these calls should appear in a proc_exit callbac

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-31 Thread vignesh C
On Tue, 1 Aug 2023 at 09:44, Peter Smith wrote: > > On Fri, Jul 28, 2023 at 5:22 PM Peter Smith wrote: > > > > Hi Melih, > > > > BACKGROUND > > -- > > > > We wanted to compare performance for the 2 different reuse-worker > > designs, when the apply worker is already busy handling other >

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Kyotaro Horiguchi
At Tue, 1 Aug 2023 14:03:36 +0900, Michael Paquier wrote in > On Tue, Aug 01, 2023 at 01:51:13PM +0900, Kyotaro Horiguchi wrote: > > I believe a database server is not supposed to be executed under such > > a memory-constrained environment. > > I don't really follow this argument. The backend

Fix pg_stat_reset_single_table_counters function

2023-07-31 Thread Masahiro Ikeda
Hi, My colleague, Mitsuru Hinata (in CC), found the following issue. The documentation of pg_stat_reset_single_table_counters() says pg_stat_reset_single_table_counters ( oid ) → void Resets statistics for a single table or index in the current database or shared across all databases in the cl

Re: Simplify some logical replication worker type checking

2023-07-31 Thread Peter Smith
On Tue, Aug 1, 2023 at 12:59 PM Zhijie Hou (Fujitsu) wrote: > > > About 2,3,4, it seems you should use "if (am_leader_apply_worker())" instead > of > "if (!am_leader_apply_worker())" because only leader apply worker should > invoke > this function. > Hi Hou-san, Thanks for your review! Oops.

Re: Faster "SET search_path"

2023-07-31 Thread Jeff Davis
On Sat, 2023-07-29 at 12:44 -0400, Isaac Morland wrote: > Essentially, "just" observe efficiently (somehow) that no change is > needed, and skip changing it? I gave this a try and it speeds things up some more. There might be a surprise factor with an optimization like that, though. If someone be

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 01:51:13PM +0900, Kyotaro Horiguchi wrote: > I believe a database server is not supposed to be executed under such > a memory-constrained environment. I don't really follow this argument. The backend and the frontends are reliable on OOM, where we generate ERRORs or even F

Fix compilation warnings when CFLAGS -Og is specified

2023-07-31 Thread Hayato Kuroda (Fujitsu)
Dear hackers, # Background Based on [1], I did configure and build with options: (I used Meson build system, but it could be reproduced by Autoconf/Make) ``` $ meson setup -Dcassert=true -Ddebug=true -Dc_args=-Og ../builder $ cd ../builder $ ninja ``` My gcc version is 4.8.5, and ninja is 1.10

Re: Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Kyotaro Horiguchi
At Tue, 1 Aug 2023 12:43:21 +0900, Michael Paquier wrote in > A colleague, Ethan Mertz (in CC), has discovered that we don't handle > correctly WAL records that are failing because of an OOM when > allocating their required space. In the case of Ethan, we have bumped > on the failure after an a

Extract numeric filed in JSONB more effectively

2023-07-31 Thread Andy Fan
Hi: Currently if we want to extract a numeric field in jsonb, we need to use the following expression: cast (a->>'a' as numeric). It will turn a numeric to text first and then turn the text to numeric again. See jsonb_object_field_text and JsonbValueAsText. However the binary format of numeric i

Re: Extension Enhancement: Buffer Invalidation in pg_buffercache

2023-07-31 Thread Palak Chaturvedi
Hii, Thanks for your feedback. We have decided to add a role for the extension to solve that problem. And concerning to pg_unwarm table I think we can create a new function to do that but I think a general user would not require to clear a table from buffercache. We can use bufferid and where state

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-31 Thread Peter Smith
On Fri, Jul 28, 2023 at 5:22 PM Peter Smith wrote: > > Hi Melih, > > BACKGROUND > -- > > We wanted to compare performance for the 2 different reuse-worker > designs, when the apply worker is already busy handling other > replications, and then simultaneously the test table tablesyncs are >

Incorrect handling of OOM in WAL replay leading to data loss

2023-07-31 Thread Michael Paquier
Hi all, A colleague, Ethan Mertz (in CC), has discovered that we don't handle correctly WAL records that are failing because of an OOM when allocating their required space. In the case of Ethan, we have bumped on the failure after an allocation failure on XLogReadRecordAlloc(): "out of memory whi

Re: Support to define custom wait events for extensions

2023-07-31 Thread Andres Freund
Hi, On 2023-08-01 12:14:49 +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: > > Thanks for committing the main patch. > > > > In my understanding, the rest works are > > * to support WaitEventExtensionMultiple() > > * to replace WAIT_EVENT_EXTENSION

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 11:51:35AM +0900, Masahiro Ikeda wrote: > Thanks for committing the main patch. > > In my understanding, the rest works are > * to support WaitEventExtensionMultiple() > * to replace WAIT_EVENT_EXTENSION to custom wait events > > Do someone already works for them? If not,

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 5:04 PM Tomas Vondra wrote: > > On 7/31/23 11:25, Amit Kapila wrote: > > On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra > > wrote: > >> > >> On 7/28/23 14:44, Ashutosh Bapat wrote: > >>> On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra > >>> wrote: > > Anyway, I was t

RE: Simplify some logical replication worker type checking

2023-07-31 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 1, 2023 9:36 AM Peter Smith > PROBLEM / SOLUTION > > During recent reviews, I noticed some of these conditions are a bit unusual. Thanks for the patch. > > == > worker.c > > 1. apply_worker_exit > > /* > * Reset the last-start time for this apply worker so that the lau

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 19:22, Michael Paquier wrote: I am not sure that any of that is necessary. Anyway, I have applied v11 to get the basics done. Thanks for committing the main patch. In my understanding, the rest works are * to support WaitEventExtensionMultiple() * to replace WAIT_EVENT_EXTENSION

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
On Tue, Aug 01, 2023 at 11:37:49AM +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 10:22:09AM +0800, Julien Rouhaud wrote: > > Looking at the rest of the ignored patterns, the only remaining one would be > > DEALLOCATE, which AFAICS doesn't have a query_jumble_ignore tag for now. > > This o

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 10:22:09AM +0800, Julien Rouhaud wrote: > Looking at the rest of the ignored patterns, the only remaining one would be > DEALLOCATE, which AFAICS doesn't have a query_jumble_ignore tag for now. This one seems to be simple as well with a location field, looking quickly at it

Re: Inaccurate comments in ReorderBufferCheckMemoryLimit()

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 8:46 PM Masahiko Sawada wrote: > > While reading the code, I realized that the following code comments > might not be accurate: > > /* > * Pick the largest transaction (or subtransaction) and evict it from > * memory by streaming, if possible. Oth

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
On Tue, Aug 01, 2023 at 11:00:32AM +0900, Michael Paquier wrote: > On Tue, Aug 01, 2023 at 09:28:08AM +0800, Julien Rouhaud wrote: > > > FTR we had to entirely ignore all those statements in powa years ago to try > > to > > make the tool usable in such case for some users who where using 2pc, it

Re: [PATCH] Add support function for containment operators

2023-07-31 Thread Laurenz Albe
On Sat, 2023-07-08 at 08:11 +0200, Kim Johan Andersson wrote: > On 07-07-2023 13:20, Laurenz Albe wrote: > > I wrote: > > > You implement both "SupportRequestIndexCondition" and > > > "SupportRequestSimplify", > > > but when I experimented, the former was never called.  That does not > > > surpris

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
On Tue, Aug 01, 2023 at 09:28:08AM +0800, Julien Rouhaud wrote: > Having an application relying on 2pc leads to pg_stat_statements being > virtually unusable on the whole instance, so +1 for the patch. Cool, thanks for the feedback! > FTR we had to entirely ignore all those statements in powa yea

Simplify some logical replication worker type checking

2023-07-31 Thread Peter Smith
Hi hackers, BACKGROUND There are 3 different types of logical replication workers. 1. apply leader workers 2. parallel apply workers 3. tablesync workers And there are a number of places where the current worker type is checked using the am_ inline functions. PROBLEM / SOLUTION During recent r

Re: Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Julien Rouhaud
Hi, On Tue, Aug 01, 2023 at 09:38:14AM +0900, Michael Paquier wrote: > > 31de7e6 has silenced savepoint names in the query jumbling, and > something similar can be done for 2PC transactions once the GID is > ignored in TransactionStmt. This leads to the following grouping in > pg_stat_statements,

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-31 Thread Andres Freund
Hi, On 2023-07-27 20:53:16 +1200, David Rowley wrote: > To summarise, REL_15_STABLE can run this benchmark in 526.014 ms on my > AMD 3990x machine. Today's REL_16_STABLE takes 530.344 ms. We're > talking about another patch to speed up the pg_strtoint functions > which gets this down to 483.790 m

PostgreSQL 16 Beta 3 release date

2023-07-31 Thread Jonathan S. Katz
Hi, The release date for PostgreSQL 16 Beta 3 is August 10, 2023, alongside the regular update release[1]. Please be sure to commit any open items[2] for the Beta 3 release before August 6, 2023 0:00 AoE[3] to give them enough time to work through the buildfarm. Thanks, Jonathan [1] https

Re: pg_upgrade fails with in-place tablespace

2023-07-31 Thread Michael Paquier
On Sat, Jul 29, 2023 at 11:10:22PM +0800, Rui Zhao wrote: > 2) Only check the tablespace with an absolute path in pg_upgrade. > There are also other solutions, such as supporting the creation of > relative-path tablespace in function CreateTableSpace. But do we > really need relative-path table

Re: [feature]COPY FROM enable FORCE_NULL/FORCE_NOT_NULL on all columns

2023-07-31 Thread Zhang Mingli
On Aug 1, 2023, at 03:35, Andrew Dunstan wrote:I was hoping it be able to get to it today but that's not happening. If you want to submit a revised patch as above that will be good. I hope to get to it later this week.HI, Andrew Patch rebased and updated like above, thanks. v4-0001-COPY-FROM-enab

Ignore 2PC transaction GIDs in query jumbling

2023-07-31 Thread Michael Paquier
Hi all, 31de7e6 has silenced savepoint names in the query jumbling, and something similar can be done for 2PC transactions once the GID is ignored in TransactionStmt. This leads to the following grouping in pg_stat_statements, for instance, which is something that matters with workloads that heav

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > On 2023-07-31 19:11:38 -0400, Tom Lane wrote: >> Huh. Maybe worth reporting as a FreeBSD bug? > Yea. Hoping our local freebsd developer has a suggestion as to which component > to report it to, or even fix it :). You already have a reproducer using just tcl, so I'd sugge

Re: Getting rid of OverrideSearhPath in namespace.c

2023-07-31 Thread Noah Misch
On Mon, Jul 17, 2023 at 05:11:46PM +0300, Aleksander Alekseev wrote: > > As a follow-up for the CVE-2023-2454 fix, I think that it makes sense to > > completely remove unsafe functions > > PushOverrideSearchPath()/PopOverrideSearchPath(), which are not used in the > > core now. > > Please look at t

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 19:11:38 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > >> (And could it be that we had one in the predecessor 13.1 image?) > > > No, I checked, and it's not in there either... It looks like the difference > > is > > that 13.

Re: Avoid possible memory leak (src/common/rmtree.c)

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 08:10:55PM -0300, Ranier Vilela wrote: > Thanks for the commit, Michael. Sorry for the lack of update here. For the sake of the archives, this is f1e9f6b. -- Michael signature.asc Description: PGP signature

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 21:25:06 +, José Neves wrote: > Ok, if I understood you correctly, I start to see where my logic is faulty. > Just to make sure that I got it right, taking the following example again: > > T-1 > INSERT LSN1-1000 > UPDATE LSN2-2000 > UPDATE LSN3-3000 > COMMIT LSN4-4000 > > T

Re: Avoid possible memory leak (src/common/rmtree.c)

2023-07-31 Thread Ranier Vilela
Em sex, 28 de jul de 2023 11:54 PM, Michael Paquier escreveu: > On Tue, Jul 25, 2023 at 04:45:22PM +0200, Daniel Gustafsson wrote: > > Skimming the tree there doesn't seem to be any callers which aren't > exiting or > > ereporting on failure so the real-world impact seems low. That being > said,

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > On 2023-07-31 18:33:37 -0400, Tom Lane wrote: >> (And could it be that we had one in the predecessor 13.1 image?) > No, I checked, and it's not in there either... It looks like the difference is > that 13.1 reads the UTC zoneinfo in that case, whereas 13.2 doesn't. Huh.

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > Andres Freund writes: > > I saw that CI image builds for freebsd were failing, because 13.1, used > > until > > now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against > > 13.2 > > - unfortunately that failed. In pltcl of all

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Tom Lane
Andres Freund writes: > I saw that CI image builds for freebsd were failing, because 13.1, used until > now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against 13.2 > - unfortunately that failed. In pltcl of all places. I tried to replicate this in a freshly-installed 13.2 VM, an

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 13:17 -0400, Joe Conway wrote: > But the analysis of the issue needs to go one step further. Even if > the > search_path does not change from the originally intended one, a newly > created function can shadow the intended one based on argument > coercion > rules. There are

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 12:53 -0400, Robert Haas wrote: > I agree. I think there are actually two interrelated problems here. > > One is that virtually all code needs to run with the originally > intended search_path rather than some search_path chosen at another > time and maybe by a different user

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Andres, thanks for your reply. Ok, if I understood you correctly, I start to see where my logic is faulty. Just to make sure that I got it right, taking the following example again: T-1 INSERT LSN1-1000 UPDATE LSN2-2000 UPDATE LSN3-3000 COMMIT LSN4-4000 T-2 INSERT LSN1-500 UPDATE LSN2-1500

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Jeff Davis
On Mon, 2023-07-31 at 16:06 -0400, Robert Haas wrote: > if you > include in your search_path a schema to which some other user can > write, you are pretty much agreeing to execute code provided by that > user. Agreed on all counts here. I don't think it's reasonable for us to try to make such a se

Re: Correct the documentation for work_mem

2023-07-31 Thread Tristen Raab
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Hello, I've reviewed and built the documentation for the updated

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 14:16:22 +, José Neves wrote: > Hi Amit, thanks for the reply. > > In our worker (custom pg replication client), we care only about INSERT, > UPDATE, and DELETE operations, which - sure - may be part of the issue. That seems likely. Postgres streams out changes in commit ord

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Robert Haas
On Mon, Jul 31, 2023 at 1:18 PM Joe Conway wrote: > But the analysis of the issue needs to go one step further. Even if the > search_path does not change from the originally intended one, a newly > created function can shadow the intended one based on argument coercion > rules. Yeah, this is a co

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 12:15:10 -0700, Andres Freund wrote: > It sure looks like freebsd 13.2 tcl is just busted. Notably it can't even > parse what it generates: > > echo 'puts [clock scan [clock format [clock seconds] -format "%Y/%m/%d"] > -format "%Y/%m/%d"]'|tclsh8.6 > > Which works on 13.1 (and

Re: [feature]COPY FROM enable FORCE_NULL/FORCE_NOT_NULL on all columns

2023-07-31 Thread Andrew Dunstan
On 2023-07-26 We 03:03, Zhang Mingli wrote: HI, I've looked at this patch and it looks mostly fine, though I do not intend to commit it myself; perhaps Andrew will. HI, Amit, thanks for review. A few minor things to improve: +  If * is specified, it will be applied in all columns.

pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, I saw that CI image builds for freebsd were failing, because 13.1, used until now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against 13.2 - unfortunately that failed. In pltcl of all places. https://api.cirrus-ci.com/v1/artifact/task/5275616266682368/testrun/build/testrun/pl

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Euler, thank you for your reply. Your output is exactly how mine doesn't look like, I don't have such an order - that is - not only under heavy load. Conditions in which this occurs make it challenging to provide detailed information, and will also take a while to trigger. I've sent a previo

Re: should frontend tools use syncfs() ?

2023-07-31 Thread Nathan Bossart
On Mon, Jul 31, 2023 at 10:51:38AM -0700, Nathan Bossart wrote: > Here is a new version of the patch with documentation updates and a couple > other small improvements. I just realized I forgot to update the --help output for these utilities. I'll do that in the next version of the patch. -- Nat

Re: add timing information to pg_upgrade

2023-07-31 Thread Nathan Bossart
On Mon, Jul 31, 2023 at 11:34:57AM +0530, Bharath Rupireddy wrote: > Either of "Checking for \"aclitem\" data type usage" or "Checking for > \"aclitem\" data type in user tables" seems okay to me, however, I > prefer the second wording. Okay. I used the second wording for all the data type check

Re: should frontend tools use syncfs() ?

2023-07-31 Thread Nathan Bossart
On Sat, Jul 29, 2023 at 02:40:10PM -0700, Nathan Bossart wrote: > I was about to start a new thread, but I found this one with some good > preliminary discussion. I came to the same conclusion about introducing a > new option instead of using syncfs() by default wherever it is available. > The att

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Joe Conway
On 7/31/23 12:53, Robert Haas wrote: On Fri, Jun 30, 2023 at 3:41 AM Jeff Davis wrote: I'm not sure that everyone in this thread realizes just how broken it is to depend on search_path in a functional index at all. And doubly so if it depends on a schema other than pg_catalog in the search_path

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
On Mon, Jul 31, 2023 at 12:24 PM Alena Rybakina wrote: > I noticed that you are going to add CNF->DNF transformation at the index > construction stage. If I understand correctly, you will rewrite > restrictinfo node, > change boolean "AND" expressions to "OR" expressions, but would it be > possibl

Re: Faster "SET search_path"

2023-07-31 Thread Robert Haas
On Sat, Jul 29, 2023 at 11:59 AM Jeff Davis wrote: > Unfortunately, adding a "SET search_path" clause to functions slows > them down. The attached patches close the performance gap > substantially. I haven't reviewed the code but I like the concept a lot. This is badly needed. -- Robert Haas ED

Re: pgsql: Fix search_path to a safe value during maintenance operations.

2023-07-31 Thread Robert Haas
On Fri, Jun 30, 2023 at 3:41 AM Jeff Davis wrote: > I'm not sure that everyone in this thread realizes just how broken it > is to depend on search_path in a functional index at all. And doubly so > if it depends on a schema other than pg_catalog in the search_path. > > Let's also not forget that l

Re: POC, WIP: OR-clause support for indexes

2023-07-31 Thread Alena Rybakina
Hi! I think it really helps to speed up the search with similar deep filtering compared to cluster indexes, but do we have cases where we don't use this algorithm because it takes longer than an usual index? I thought about the situation with wide indexes (with a lot of multiple columns) and hav

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
On Thu, Jul 27, 2023 at 10:00 AM Matthias van de Meent wrote: > My idea is not quite block nested loop join. It's more 'restart the > index scan at the location the previous index scan ended, if > heuristics say there's a good chance that might save us time'. I'd say > it is comparable to the fast

Re: Request for comment on setting binary format output per session

2023-07-31 Thread Dave Cramer
Dave Cramer On Mon, 10 Jul 2023 at 03:56, Daniel Gustafsson wrote: > > On 25 Apr 2023, at 16:47, Dave Cramer wrote: > > > Patch attached with comments removed > > This patch no longer applies, please submit a rebased version on top of > HEAD. > Rebased see attached > > -- > Daniel Gustafss

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Alena Rybakina
Hi, all! CNF -> DNF conversion = Like many great papers, the MDAM paper takes one core idea, and finds ways to leverage it to the hilt. Here the core idea is to take predicates in conjunctive normal form (an "AND of ORs"), and convert them into disjunctive normal form (an

Re: proposal: psql: show current user in prompt

2023-07-31 Thread Jelte Fennema
On Mon, 24 Jul 2023 at 21:16, Pavel Stehule wrote: > I don't understand how it can be possible to do it without. I need to > process possible errors, and then I need to read and synchronize protocol. I > didn't inject > this feature to some oher flow, so I need to implement a complete process.

Re: pg_upgrade fails with in-place tablespace

2023-07-31 Thread Junwang Zhao
On Mon, Jul 31, 2023 at 5:36 PM Rui Zhao wrote: > > Hello postgres hackers, > Recently I encountered an issue: pg_upgrade fails when dealing with in-place > tablespace. As we know, pg_upgrade uses pg_dumpall to dump objects and > pg_restore to restore them. The problem seems to be that pg_dumpal

Inaccurate comments in ReorderBufferCheckMemoryLimit()

2023-07-31 Thread Masahiko Sawada
Hi all, While reading the code, I realized that the following code comments might not be accurate: /* * Pick the largest transaction (or subtransaction) and evict it from * memory by streaming, if possible. Otherwise, spill to disk. */ if (ReorderBuffer

RE: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi Amit, thanks for the reply. In our worker (custom pg replication client), we care only about INSERT, UPDATE, and DELETE operations, which - sure - may be part of the issue. I can only replicate this with production-level load, not easy to get a real example, but as I'm understanding the issue

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Euler Taveira
On Sat, Jul 29, 2023, at 8:07 PM, José Neves wrote: > I'm attempting to develop a CDC on top of Postgres, currently using 12, the > last minor, with a custom client, and I'm running into issues with data loss > caused by out-of-order logical replication messages. Can you provide a test case to s

Re: Improve join_search_one_level readibilty (one line change)

2023-07-31 Thread Julien Rouhaud
Hi, On Wed, Jun 07, 2023 at 11:02:09AM +0800, 謝東霖 wrote: > Thank you, Julien, for letting me know that cfbot doesn't test txt files. > Much appreciated! Thanks for posting this v2! So unsurprisingly the cfbot is happy with this patch, since it doesn't change the behavior at all. I just have som

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 3:06 PM José Neves wrote: > > Hi there, hope to find you well. > > I'm attempting to develop a CDC on top of Postgres, currently using 12, the > last minor, with a custom client, and I'm running into issues with data loss > caused by out-of-order logical replication messa

Re: Adding a LogicalRepWorker type field

2023-07-31 Thread Amit Kapila
On Mon, Jul 31, 2023 at 3:25 PM Bharath Rupireddy wrote: > > On Mon, Jul 31, 2023 at 7:17 AM Peter Smith wrote: > > > > PROBLEM: > > > > IMO, deducing the worker's type by examining multiple different field > > values seems a dubious way to do it. This maybe was reasonable enough > > when there w

Re: [BUG] Fix DETACH with FK pointing to a partitioned table fails

2023-07-31 Thread Alvaro Herrera
On 2023-Jul-05, Jehan-Guillaume de Rorthais wrote: > ALTER TABLE r ATTACH PARTITION r_1 FOR VALUES IN (1); > > The old sub-FKs (below 18289) created in this table to enforce the action > triggers on referenced partitions are not deleted when the table becomes a > partition. Because of this, we

Re: New PostgreSQL Contributors

2023-07-31 Thread Matthias van de Meent
On Fri, 28 Jul 2023 at 17:29, Christoph Berg wrote: > > The PostgreSQL contributors team has been looking over the community > activity and, over the first half of this year, has been recognizing > new contributors to be listed on > > https://www.postgresql.org/community/contributors/ > > New Post

Re: stats test intermittent failure

2023-07-31 Thread Masahiko Sawada
Hi, On Tue, Jul 11, 2023 at 3:35 AM Melanie Plageman wrote: > > Hi, > > Jeff pointed out that one of the pg_stat_io tests has failed a few times > over the past months (here on morepork [1] and more recently here on > francolin [2]). > > Failing test diff for those who prefer not to scroll: > > +

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread John Naylor
On Mon, Jul 31, 2023 at 5:57 PM Tom Lane wrote: > > John Naylor writes: > > Works for me, so done that way for both forward and reverse variants. Since > > the return value is no longer checked in any builds, I thought about > > removing the variable containing it, but it seems best to leave it

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Tomas Vondra
On 7/31/23 11:25, Amit Kapila wrote: > On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra > wrote: >> >> On 7/28/23 14:44, Ashutosh Bapat wrote: >>> On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra >>> wrote: Anyway, I was thinking about this a bit more, and it seems it's not as difficult

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread Tom Lane
John Naylor writes: > On Sun, Jul 30, 2023 at 9:45 PM Tom Lane wrote: >> That's basically equivalent to the existing Assert(non_zero). >> I think it'd be okay to drop that one and instead have >> the same Assert condition as other platforms, but having both >> would be redundant. > Works for me,

Re: Support to define custom wait events for extensions

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 3:54 PM Michael Paquier wrote: > > On Mon, Jul 31, 2023 at 05:10:21PM +0900, Kyotaro Horiguchi wrote: > > +/* > > + * Return the name of an wait event ID for extension. > > + */ > > +static const char * > > +GetWaitEventExtensionIdentifier(uint16 eventId) > > > > This looks

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 05:10:21PM +0900, Kyotaro Horiguchi wrote: > +/* > + * Return the name of an wait event ID for extension. > + */ > +static const char * > +GetWaitEventExtensionIdentifier(uint16 eventId) > > This looks inconsistent. Shouldn't it be GetWaitEventExtentionName()? This is an i

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 01:37:49PM +0530, Bharath Rupireddy wrote: > Do you think it's worth adding a note here in the docs about an > external module defining more than one custom wait event? A pseudo > code if possible or just a note? Also, how about a XXX comment atop > WaitEventExtensionNew and

Missing comments/docs about custom scan path

2023-07-31 Thread Etsuro Fujita
Hi, While working on [1], I noticed $SUBJECT: commit e7cb7ee14 failed to update comments for the CustomPath struct in pathnodes.h, and commit f49842d1e failed to update docs about custom scan path callbacks in custom-scan.sgml, IIUC. Attached are patches for updating these, which I created separa

Re: Adding a LogicalRepWorker type field

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 7:17 AM Peter Smith wrote: > > PROBLEM: > > IMO, deducing the worker's type by examining multiple different field > values seems a dubious way to do it. This maybe was reasonable enough > when there were only 2 types, but as more get added it becomes > increasingly complica

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-31 Thread John Naylor
On Thu, Jul 27, 2023 at 7:17 AM David Rowley wrote: > > It would be really good if someone with another a newish intel CPU > could test this too. I ran the lotsaints test from last email on an i7-10750H (~3 years old) and got these results (gcc 13.1 , turbo off): REL_15_STABLE: latency average =

回复:pg_rewind fails with in-place tablespace

2023-07-31 Thread Rui Zhao
Sorry for the delay in responding to this matter as I have been waiting for another similar subject to approved by a moderator. Upon review, I am satisfied with the proposed solution and believe that checking absolute path is better than hard coding with "pg_tblspc/". I think we have successfull

CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread José Neves
Hi there, hope to find you well. I'm attempting to develop a CDC on top of Postgres, currently using 12, the last minor, with a custom client, and I'm running into issues with data loss caused by out-of-order logical replication messages. The problem is as follows: postgres streams A, B, D, G,

pg_upgrade fails with in-place tablespace

2023-07-31 Thread Rui Zhao
Hello postgres hackers, Recently I encountered an issue: pg_upgrade fails when dealing with in-place tablespace. As we know, pg_upgrade uses pg_dumpall to dump objects and pg_restore to restore them. The problem seems to be that pg_dumpall is dumping in-place tablespace as relative path, which

Re: logical decoding and replication of sequences, take 2

2023-07-31 Thread Amit Kapila
On Sat, Jul 29, 2023 at 5:53 PM Tomas Vondra wrote: > > On 7/28/23 14:44, Ashutosh Bapat wrote: > > On Wed, Jul 26, 2023 at 8:48 PM Tomas Vondra > > wrote: > >> > >> Anyway, I was thinking about this a bit more, and it seems it's not as > >> difficult to use the page LSN to ensure sequences don't

Re: postgres_fdw: wrong results with self join + enable_nestloop off

2023-07-31 Thread Richard Guo
On Fri, Jul 28, 2023 at 4:56 PM Etsuro Fujita wrote: > Cool! I pushed the first patch after polishing it a little bit, so > here is a rebased version of the second patch, in which I modified the > ForeignPath and CustomPath cases in reparameterize_path_by_child() to > reflect the new members fdw

Re: Support to define custom wait events for extensions

2023-07-31 Thread Kyotaro Horiguchi
At Mon, 31 Jul 2023 16:28:16 +0900, Michael Paquier wrote in > Attaching a v11 based on Bharath's feedback and yours, for now. I > have also applied the addition of the two masking variables in > wait_event.c separately with 7395a90. +/* + * Return the name of an wait event ID for extension. +

Re: Support to define custom wait events for extensions

2023-07-31 Thread Bharath Rupireddy
On Mon, Jul 31, 2023 at 12:58 PM Michael Paquier wrote: > > > Attaching a v11 based on Bharath's feedback and yours, for now. I > have also applied the addition of the two masking variables in > wait_event.c separately with 7395a90. +uint32 WaitEventExtensionNew(void) + + Next, each process

Re: Avoid undefined behavior with msvc compiler (src/include/port/pg_bitutils.h)

2023-07-31 Thread John Naylor
On Sun, Jul 30, 2023 at 9:45 PM Tom Lane wrote: > > John Naylor writes: > > It seems that we should have "Assert(word != 0);" at the top, which matches > > the other platforms anyway, so I'll add that. > > That's basically equivalent to the existing Assert(non_zero). > I think it'd be okay to dro

Re: Support to define custom wait events for extensions

2023-07-31 Thread Masahiro Ikeda
On 2023-07-31 16:28, Michael Paquier wrote: On Mon, Jul 31, 2023 at 03:53:27PM +0900, Masahiro Ikeda wrote: /* This should only be called for user-defined wait event. */ if (eventId < NUM_BUILTIN_WAIT_EVENT_EXTENSION) ereport(ERROR,

Re: Support to define custom wait events for extensions

2023-07-31 Thread Michael Paquier
On Mon, Jul 31, 2023 at 03:53:27PM +0900, Masahiro Ikeda wrote: > I think the order in which they are mentioned should be matched. I mean that > - so an LWLock or Extension wait > + so an Extension or LWLock wait Makes sense. > /* This should only be called for user-defined wait eve