Re: Detaching a partition with a FK on itself is not possible

2022-03-21 Thread Jehan-Guillaume de Rorthais
Hi, On Thu, 17 Mar 2022 17:58:04 + Arne Roland wrote: > I don't think this a bug, but a feature request. I therefore think hackers > would be more appropriate. +1 I changed the list destination > I don't see how an additional syntax to modify the constraint should help. Me neiher. > If

Re: vacuum -vs reltuples on insert only index

2020-11-09 Thread Jehan-Guillaume de Rorthais
On Wed, 4 Nov 2020 18:44:03 -0800 Peter Geoghegan wrote: > On Mon, Nov 2, 2020 at 10:03 AM Peter Geoghegan wrote: > > Actually, it seems better to always count num_index_tuples the old way > > during cleanup-only index VACUUMs, despite the inaccuracy that that > > creates with posting list tuple

Re: [PATCH] Disable bgworkers during servers start in pg_upgrade

2021-01-27 Thread Jehan-Guillaume de Rorthais
Hi, On Wed, 27 Jan 2021 11:25:11 +0100 Denis Laxalde wrote: > Andres Freund a écrit : > > On 2021-01-21 16:23:58 +0100, Denis Laxalde wrote: > > > We found an issue in pg_upgrade on a cluster with a third-party > > > background worker. The upgrade goes fine, but the new cluster is then in > >

Re: [PATCH] Disable bgworkers during servers start in pg_upgrade

2021-01-27 Thread Jehan-Guillaume de Rorthais
Oh, I forgot another point before sending my previous email. Maybe it might worth adding some final safety checks in pg_upgrade itself? Eg. checking controldata and mxid files coherency between old and new cluster would have catch the inconsistency here.

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
Hi all, First, sorry to step in this discussion this late. I didn't noticed it before :( I did some work about these compatibility issues in late 2020 to use PostgresNode in the check_pgactivity TAP tests. See https://github.com/ioguix/check_pgactivity/tree/tests/t/lib PostgresNode.pm, TestLib.

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 11:54:36 -0400 Andrew Dunstan wrote: > On 4/7/21 10:37 AM, Jehan-Guillaume de Rorthais wrote: > > Hi all, > > > > First, sorry to step in this discussion this late. I didn't noticed it > > before :( > > > > I did some work about th

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 09:08:31 -0700 Mark Dilger wrote: > > On Apr 7, 2021, at 7:37 AM, Jehan-Guillaume de Rorthais > > And here is a demo test file: > > https://github.com/ioguix/check_pgactivity/blob/tests/t/01-streaming_delta.t > > > > My limited set of tests are

Re: why pg_walfile_name() cannot be executed during recovery?

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Fri, 2 Apr 2021 08:22:09 -0400 Robert Haas wrote: > On Fri, Apr 2, 2021 at 4:23 AM SATYANARAYANA NARLAPURAM > wrote: > > Why pg_walfile_name() can't be executed under recovery? > > I believe the issue is that the backend executing the function might > not have an accurate idea about which

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 12:51:55 -0400 Alvaro Herrera wrote: > On 2021-Apr-07, Jehan-Guillaume de Rorthais wrote: > > > When I'm creating a new node, I'm using the "pgaTester" factory class. It > > relies on PATH to check the major version using pg_config

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 13:36:31 -0400 Alvaro Herrera wrote: > On 2021-Apr-07, Jehan-Guillaume de Rorthais wrote: > > > Yes, it would be much saner to make PostgresNode the factory class. Plus, > > some more logic could be injected there to either auto-detect the version > >

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 10:50:26 -0700 Mark Dilger wrote: > > On Apr 7, 2021, at 10:36 AM, Alvaro Herrera wrote: > > > >> Yes, it would be much saner to make PostgresNode the factory class. Plus, > >> some more logic could be injected there to either auto-detect the version > >> (current behavior)

Re: multi-install PostgresNode fails with older postgres versions

2021-04-07 Thread Jehan-Guillaume de Rorthais
On Wed, 7 Apr 2021 13:38:39 -0400 Andrew Dunstan wrote: > On 4/7/21 1:19 PM, Jehan-Guillaume de Rorthais wrote: > > On Wed, 7 Apr 2021 12:51:55 -0400 > > Alvaro Herrera wrote: > > > >> On 2021-Apr-07, Jehan-Guillaume de Rorthais wrote: > >> > >&

Re: multi-install PostgresNode fails with older postgres versions

2021-04-12 Thread Jehan-Guillaume de Rorthais
Hi, On Wed, 7 Apr 2021 20:07:41 +0200 Jehan-Guillaume de Rorthais wrote: [...] > > > Let me know if it worth that I work on an official patch. > > > > Let's give it a try ... > > OK So, as promised, here is my take to port my previous work on PostgreS

Re: multi-install PostgresNode fails with older postgres versions

2021-04-12 Thread Jehan-Guillaume de Rorthais
On Mon, 12 Apr 2021 09:52:24 -0400 Andrew Dunstan wrote: > On 4/12/21 8:59 AM, Jehan-Guillaume de Rorthais wrote: > > Hi, > > > > On Wed, 7 Apr 2021 20:07:41 +0200 > > Jehan-Guillaume de Rorthais wrote: > > [...] > >>>> Let me k

Re: Retry in pgbench

2021-04-13 Thread Jehan-Guillaume de Rorthais
Hi, On Tue, 13 Apr 2021 16:12:59 +0900 (JST) Tatsuo Ishii wrote: > [...] > [...] > [...] > > Thanks for the pointer. It seems we need to resume the discussion. By the way, I've been playing with the idea of failing gracefully and retry indefinitely (or until given -T) on SQL error AND

Re: [patch] demote

2020-08-04 Thread Jehan-Guillaume de Rorthais
to conclude this patch set should keep growing to see were it goes, I wonder if/when I should add it to the commitfest. Advice? Opinion? Regards, >From da3c4575f8ea40c089483b9cfa209db4993148ff Mon Sep 17 00:00:00 2001 From: Jehan-Guillaume de Rorthais Date: Fri, 31 Jul 2020 10:58:40 +0200 Sub

Re: [patch] demote

2020-08-18 Thread Jehan-Guillaume de Rorthais
Hi, Please find in attachment v5 of the patch set rebased on master after various conflicts. Regards, On Wed, 5 Aug 2020 00:04:53 +0200 Jehan-Guillaume de Rorthais wrote: > Demote now keeps backends with no active xid alive. Smart mode keeps all > backends: it waits for them to finish

Re: [patch] demote

2020-09-01 Thread Jehan-Guillaume de Rorthais
On Tue, 18 Aug 2020 17:41:31 +0200 Jehan-Guillaume de Rorthais wrote: > Hi, > > Please find in attachment v5 of the patch set rebased on master after various > conflicts. > > Regards, > > On Wed, 5 Aug 2020 00:04:53 +0200 > Jehan-Guillaume de Rorthais wrote: >

Re: Make description of heap records more talkative for flags

2018-04-24 Thread Jehan-Guillaume de Rorthais
Hi all, Bellow a 1¢ on feedback from a side project about this. On Mon, 23 Apr 2018 12:37:20 -0300 Alvaro Herrera wrote: > Michael Paquier wrote: > > On Thu, Apr 12, 2018 at 08:49:03PM -0700, Andres Freund wrote: > > > OTOH, that also kinda bloats the output noticeably... I'm somewhat > > > i

Re: Fetching timeline during recovery

2019-09-06 Thread Jehan-Guillaume de Rorthais
On Wed, 4 Sep 2019 00:32:03 +0900 Fujii Masao wrote: > On Mon, Jul 29, 2019 at 7:26 PM Jehan-Guillaume de Rorthais > wrote: > > > > On Fri, 26 Jul 2019 18:22:25 +0200 > > Jehan-Guillaume de Rorthais wrote: > > > > > On Fri, 26 Jul 2019 10:02:58 +0200 &

Re: Fetching timeline during recovery

2019-09-26 Thread Jehan-Guillaume de Rorthais
On Mon, 9 Sep 2019 19:44:10 +0900 Fujii Masao wrote: > On Sat, Sep 7, 2019 at 12:06 AM Jehan-Guillaume de Rorthais > wrote: > > > > On Wed, 4 Sep 2019 00:32:03 +0900 > > Fujii Masao wrote: > > [...] > Thanks for updating the patch! Thank you for your review!

Re: [HACKERS] Restricting maximum keep segments by repslots

2019-10-02 Thread Jehan-Guillaume de Rorthais
On Tue, 30 Jul 2019 21:30:45 +0900 (Tokyo Standard Time) Kyotaro Horiguchi wrote: > Thanks for reviewing! > > At Thu, 27 Jun 2019 16:22:56 +0200, Jehan-Guillaume de Rorthais > wrote in <20190627162256.4f4872b8@firost> > > Hi all, > > > > Being interested b

Logical replication dead but synching

2019-10-10 Thread Jehan-Guillaume de Rorthais
Hello, While giving assistance to some customer with their broker procedure, I found a scenario where the subscription is failing but the table are sync'ed anyway. Here is bash script to reproduce it with versions 10, 11 and 12 (make sure to set PATH correctly): # env PUB=/tmp/pub SUB=/tmp/

segmentation fault when cassert enabled

2019-10-25 Thread Jehan-Guillaume de Rorthais
Hi list, When investigating for the bug reported in thread "logical replication - negative bitmapset member not allowed", I found a way to seg fault postgresql only when cassert is enabled. See the scenario in attachment. When executed against binaries compiled with --enable-cassert, I have the

Re: PostgreSQL pollutes the file system

2019-03-20 Thread Jehan-Guillaume de Rorthais
On Wed, 20 Mar 2019 13:56:55 -0400 Tom Lane wrote: > Julien Rouhaud writes: > > On Wed, Mar 20, 2019 at 6:25 PM Euler Taveira > > wrote: > >> createdb, dropdb, createuser, dropuser, reindexdb are binaries that > >> confuse most newbies. Which tool is theses binaries from? The names > >> does

Fetching timeline during recovery

2019-07-23 Thread Jehan-Guillaume de Rorthais
8-E7F5-486F-92C8-A6959040DEB6%40yandex-team.ru >From 9d0fb73d03c6e7e06f2f8be62abab4e54cf01117 Mon Sep 17 00:00:00 2001 From: Jehan-Guillaume de Rorthais Date: Tue, 23 Jul 2019 17:28:44 +0200 Subject: [PATCH] Support pg_walfile_name on standby Support executing both SQL functions pg

Re: Fetching timeline during recovery

2019-07-23 Thread Jehan-Guillaume de Rorthais
On Tue, 23 Jul 2019 16:00:29 -0400 David Steele wrote: > On 7/23/19 2:59 PM, Andrey Borodin wrote: > > > >> 23 июля 2019 г., в 21:05, Jehan-Guillaume de Rorthais > >> написал(а): > >> > >> Fetching the timeline from a standby could be useful in va

Re: Fetching timeline during recovery

2019-07-24 Thread Jehan-Guillaume de Rorthais
Hello Michael, On Wed, 24 Jul 2019 09:49:05 +0900 Michael Paquier wrote: > On Tue, Jul 23, 2019 at 06:05:18PM +0200, Jehan-Guillaume de Rorthais wrote: > > Please, find in attachment a first trivial patch to support > > pg_walfile_name() and pg_walfile_name_offset() on a standb

Re: pg_receivewal documentation

2019-07-24 Thread Jehan-Guillaume de Rorthais
chment. Regards, >From 01a7de92dbaae5a61d5ec7bd04bef1553467f29d Mon Sep 17 00:00:00 2001 From: Jehan-Guillaume de Rorthais Date: Wed, 24 Jul 2019 14:58:41 +0200 Subject: [PATCH] Add doc details for pg_receivewal with remote_apply --- doc/src/sgml/ref/pg_receivewal.sgml | 7 ++- 1 file changed, 6

Re: pg_receivewal documentation

2019-07-25 Thread Jehan-Guillaume de Rorthais
On Thu, 25 Jul 2019 16:58:17 +0900 Michael Paquier wrote: > On Wed, Jul 24, 2019 at 03:03:04PM +0200, Jehan-Guillaume de Rorthais wrote: > > Unless I am missing something, another solution might be to use a dedicated > > role to pg_receive{xlog|wal} with synchronous_co

Re: Fetching timeline during recovery

2019-07-25 Thread Jehan-Guillaume de Rorthais
Hello, On Wed, 24 Jul 2019 14:33:27 +0200 Jehan-Guillaume de Rorthais wrote: > On Wed, 24 Jul 2019 09:49:05 +0900 > Michael Paquier wrote: > > > On Tue, Jul 23, 2019 at 06:05:18PM +0200, Jehan-Guillaume de Rorthais > > wrote: [...] > > I think that there a

Re: Fetching timeline during recovery

2019-07-26 Thread Jehan-Guillaume de Rorthais
On Fri, 26 Jul 2019 16:49:53 +0900 (Tokyo Standard Time) Kyotaro Horiguchi wrote: > Hi. > > At Thu, 25 Jul 2019 19:38:08 +0200, Jehan-Guillaume de Rorthais > wrote in <20190725193808.1648ddc8@firost> > > On Wed, 24 Jul 2019 14:33:27 +0200 > > Jehan-Guillaume de Ro

Re: pg_walfile_name_offset can return inconsistent values

2019-07-26 Thread Jehan-Guillaume de Rorthais
On Fri, 26 Jul 2019 17:21:20 +0900 (Tokyo Standard Time) Kyotaro Horiguchi wrote: > Hello. > > While looking [1], I noticed that pg_walfile_name_offset behaves > somewhat oddly at segment boundary. > > select * from (values ('0/16ff'), ('0/1700'), ('0/1701')) as > t(lsn), lateral pg

Re: Fetching timeline during recovery

2019-07-26 Thread Jehan-Guillaume de Rorthais
On Fri, 26 Jul 2019 10:02:58 +0200 Jehan-Guillaume de Rorthais wrote: > On Fri, 26 Jul 2019 16:49:53 +0900 (Tokyo Standard Time) > Kyotaro Horiguchi wrote: [...] > > We have an LSN reporting function each for several objectives. > > > > pg_current_wal_lsn > &

Re: Fetching timeline during recovery

2019-07-29 Thread Jehan-Guillaume de Rorthais
On Fri, 26 Jul 2019 18:22:25 +0200 Jehan-Guillaume de Rorthais wrote: > On Fri, 26 Jul 2019 10:02:58 +0200 > Jehan-Guillaume de Rorthais wrote: > > > On Fri, 26 Jul 2019 16:49:53 +0900 (Tokyo Standard Time) > > Kyotaro Horiguchi wrote: > [...] > > > We have a

Re: pg_waldump erroneously outputs newline for FPWs, and another minor bug

2019-11-04 Thread Jehan-Guillaume de Rorthais
On Wed, 30 Oct 2019 09:26:21 +0900 Michael Paquier wrote: > On Tue, Oct 29, 2019 at 04:42:07PM -0700, Peter Geoghegan wrote: > > The same commit from Heikki omitted one field from that record, for no > > good reason. I backpatched a bugfix to the output format for nbtree > > page splits a few wee

Re: segmentation fault when cassert enabled

2019-11-05 Thread Jehan-Guillaume de Rorthais
On Fri, 25 Oct 2019 12:28:38 -0400 Tom Lane wrote: > Jehan-Guillaume de Rorthais writes: > > When investigating for the bug reported in thread "logical replication - > > negative bitmapset member not allowed", I found a way to seg fault > > postgresql only when ca

Re: segmentation fault when cassert enabled

2019-11-12 Thread Jehan-Guillaume de Rorthais
On Mon, 28 Oct 2019 16:47:02 +0900 (JST) Kyotaro Horiguchi wrote: > At Fri, 25 Oct 2019 12:28:38 -0400, Tom Lane wrote in > > Jehan-Guillaume de Rorthais writes: > > > When investigating for the bug reported in thread "logical replication - > > > negative

Re: segmentation fault when cassert enabled

2019-11-25 Thread Jehan-Guillaume de Rorthais
On Wed, 6 Nov 2019 14:34:38 +0100 Peter Eisentraut wrote: > On 2019-11-05 17:29, Jehan-Guillaume de Rorthais wrote: > > My best bet so far is that logicalrep_relmap_invalidate_cb is not called > > after the DDL on the subscriber so the relmap cache is not invalidated. So > >

Re: Explain: Duplicate key "Workers" in JSON format

2019-12-05 Thread Jehan-Guillaume de Rorthais
"Actual Startup Time": 1468.256, > > "Actual Total Time": 2012.744, > > "Actual Rows": 2684443, > > "Actual Loops": 1 > > } > > ], > > Yes, that's r

Re: segmentation fault when cassert enabled

2019-12-16 Thread Jehan-Guillaume de Rorthais
On Fri, 13 Dec 2019 12:10:07 +0530 vignesh C wrote: > On Fri, Dec 6, 2019 at 5:30 PM Amit Kapila wrote: > > > > On Mon, Nov 25, 2019 at 8:25 PM Jehan-Guillaume de Rorthais > > wrote: > > > > > > On Wed, 6 Nov 2019 14:34:38 +0100 > > > Peter Eis

Re: segmentation fault when cassert enabled

2019-12-16 Thread Jehan-Guillaume de Rorthais
On Mon, 16 Dec 2019 13:27:43 +0100 Peter Eisentraut wrote: > On 2019-12-16 11:11, Amit Kapila wrote: > > I agree that this is a timing issue. I also don't see a way to write > > a reproducible test for this. However, I could reproduce it via > > debugger consistently by following the below step

Re: segmentation fault when cassert enabled

2019-12-19 Thread Jehan-Guillaume de Rorthais
On Wed, 18 Dec 2019 08:46:01 +0530 Amit Kapila wrote: > On Tue, Dec 17, 2019 at 6:01 PM vignesh C wrote: > > > > On Tue, Dec 17, 2019 at 10:09 AM Amit Kapila > > wrote: > > > > > > Attached patch with updated commit message based on suggestions. I am > > > planning to commit this tomorrow un

Re: Fetching timeline during recovery

2019-12-19 Thread Jehan-Guillaume de Rorthais
On Wed, 11 Dec 2019 14:20:02 +0900 Michael Paquier wrote: > On Thu, Sep 26, 2019 at 07:20:46PM +0200, Jehan-Guillaume de Rorthais wrote: > > If this solution is accepted, some other function of the same family might > > be good candidates as well, for the sake

Re: Fetching timeline during recovery

2019-12-19 Thread Jehan-Guillaume de Rorthais
On Fri, 13 Dec 2019 16:12:55 +0900 Michael Paquier wrote: > On Wed, Dec 11, 2019 at 10:45:25AM -0500, Stephen Frost wrote: > > I'm confused- wouldn't the above approach be a function that's returning > > only one row, if you had a bunch of columns and then had NULL values for > > those cases that

Re: How is this possible "publication does not exist"

2019-12-20 Thread Jehan-Guillaume de Rorthais
On Thu, 19 Dec 2019 19:19:56 +0100 Peter Eisentraut wrote: > On 2019-12-19 19:15, Dave Cramer wrote: > > It seems that if you drop the publication on an existing slot it needs > > to be recreated. Is this expected behaviour > > A publication is not associated with a slot. Only a subscription

Re: Fetching timeline during recovery

2019-12-20 Thread Jehan-Guillaume de Rorthais
On Fri, 20 Dec 2019 13:41:25 +0900 (JST) Kyotaro Horiguchi wrote: > At Fri, 20 Dec 2019 00:35:19 +0100, Jehan-Guillaume de Rorthais > wrote in > > On Fri, 13 Dec 2019 16:12:55 +0900 > > Michael Paquier wrote: > > The first one; > > > > I mentioned

Re: Fetching timeline during recovery

2019-12-23 Thread Jehan-Guillaume de Rorthais
On Mon, 23 Dec 2019 12:36:56 +0900 Michael Paquier wrote: > On Fri, Dec 20, 2019 at 11:14:28AM +0100, Jehan-Guillaume de Rorthais wrote: > > Yes, that would be great but sadly, it would introduce a regression on > > various tools relying on them. At least, the one doing &qu

Re: Fetching timeline during recovery

2020-01-03 Thread Jehan-Guillaume de Rorthais
Hi, On Mon, 23 Dec 2019 15:38:16 +0100 Jehan-Guillaume de Rorthais wrote: [...] > My idea would be to return a row from pg_stat_get_wal_receiver() as soon as > a wal receiver has been replicating during the uptime of the standby, no > matter if there's one currently working or

Re: [HACKERS] Restricting maximum keep segments by repslots

2020-01-22 Thread Jehan-Guillaume de Rorthais
Hi, First, it seems you did not reply to Alvaro's concerns in your new set of patch. See: https://www.postgresql.org/message-id/20190917195800.GA16694%40alvherre.pgsql On Tue, 24 Dec 2019 21:26:14 +0900 (JST) Kyotaro Horiguchi wrote: [...] > > Indeed, "loosing" is a better match for this state.

Re: Fetching timeline during recovery

2020-01-23 Thread Jehan-Guillaume de Rorthais
On Tue, 07 Jan 2020 15:57:29 +0900 (JST) Kyotaro Horiguchi wrote: > At Mon, 23 Dec 2019 15:38:16 +0100, Jehan-Guillaume de Rorthais > wrote in > > 1. we could decide to remove this filter to expose the data even when no > > wal receiver is active. It's

Re: block-level incremental backup

2019-04-10 Thread Jehan-Guillaume de Rorthais
Hi, On Tue, 9 Apr 2019 11:48:38 -0400 Robert Haas wrote: > Several companies, including EnterpriseDB, NTT, and Postgres Pro, have > developed technology that permits a block-level incremental backup to > be taken from a PostgreSQL server. I believe the idea in all of those > cases is that non-r

Re: block-level incremental backup

2019-04-10 Thread Jehan-Guillaume de Rorthais
Hi, First thank you for your answer! On Wed, 10 Apr 2019 12:21:03 -0400 Robert Haas wrote: > On Wed, Apr 10, 2019 at 10:57 AM Jehan-Guillaume de Rorthais > wrote: > > My idea would be create a new tool working on archived WAL. No burden > > server side. Basic concept i

Re: block-level incremental backup

2019-04-10 Thread Jehan-Guillaume de Rorthais
On Wed, 10 Apr 2019 14:38:43 -0400 Robert Haas wrote: > On Wed, Apr 10, 2019 at 2:21 PM Jehan-Guillaume de Rorthais > wrote: > > In my current design, the scan is done backward from end to start and I > > keep all the records appearing after the last occurrence of their

Re: block-level incremental backup

2019-04-10 Thread Jehan-Guillaume de Rorthais
On Wed, 10 Apr 2019 11:55:51 -0700 Andres Freund wrote: > Hi, > > On 2019-04-10 14:38:43 -0400, Robert Haas wrote: > > On Wed, Apr 10, 2019 at 2:21 PM Jehan-Guillaume de Rorthais > > wrote: > > > In my current design, the scan is done backward from end to

Re: Using old master as new replica after clean switchover

2018-10-25 Thread Jehan-Guillaume de Rorthais
On Thu, 25 Oct 2018 02:57:18 -0400 Nikolay Samokhvalov wrote: ... > My research shows that some people already rely on the following when > planned failover (aka switchover) procedure, doing it in production: > > 1) shutdown the current master > 2) ensure that the "master candidate" replica has

Re: Using old master as new replica after clean switchover

2018-10-25 Thread Jehan-Guillaume de Rorthais
On Thu, 25 Oct 2018 20:45:57 +0900 Michael Paquier wrote: > On Thu, Oct 25, 2018 at 11:15:51AM +0200, Jehan-Guillaume de Rorthais wrote: > > On Thu, 25 Oct 2018 02:57:18 -0400 > > Nikolay Samokhvalov wrote: > >> My research shows that some people already re

Re: Memory leak from ExecutorState context?

2023-03-01 Thread Jehan-Guillaume de Rorthais
Hi Justin, On Tue, 28 Feb 2023 12:25:08 -0600 Justin Pryzby wrote: > On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote: > > Hello all, > > > > A customer is facing out of memory query which looks similar to this > > situa

Re: Memory leak from ExecutorState context?

2023-03-01 Thread Jehan-Guillaume de Rorthais
Hi Tomas, On Tue, 28 Feb 2023 20:51:02 +0100 Tomas Vondra wrote: > On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote: > > * HashBatchContext goes up to 1441MB after 240s then stay flat until the end > > (400s as the last record) > > That's interesting. We

Re: Memory leak from ExecutorState context?

2023-03-01 Thread Jehan-Guillaume de Rorthais
On Wed, 1 Mar 2023 18:48:40 +0100 Jehan-Guillaume de Rorthais wrote: ... > You'll find some intermediate stats I already collected in attachment: > > * break 1, 2 and 3 are from AllocSetAlloc, break 4 is from AllocSetFree. > * most of the non-free'd chunk are allocated si

Re: Memory leak from ExecutorState context?

2023-03-01 Thread Jehan-Guillaume de Rorthais
On Wed, 1 Mar 2023 20:34:08 +0100 Tomas Vondra wrote: > On 3/1/23 19:09, Jehan-Guillaume de Rorthais wrote: > > On Wed, 1 Mar 2023 18:48:40 +0100 > > Jehan-Guillaume de Rorthais wrote: > > ... > >> You'll find some intermediate stats I already collected in

Re: Memory leak from ExecutorState context?

2023-03-01 Thread Jehan-Guillaume de Rorthais
Hi, On Wed, 1 Mar 2023 20:29:11 +0100 Tomas Vondra wrote: > On 3/1/23 18:48, Jehan-Guillaume de Rorthais wrote: > > On Tue, 28 Feb 2023 20:51:02 +0100 > > Tomas Vondra wrote: > >> On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote: > >>> * HashBatchConte

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 01:30:27 +0100 Tomas Vondra wrote: > On 3/2/23 00:18, Jehan-Guillaume de Rorthais wrote: > >>> ExecHashIncreaseNumBatches was really chatty, having hundreds of thousands > >>> of calls, always short-cut'ed to 1048576, I guess because of the &g

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
Hi! On Thu, 2 Mar 2023 13:44:52 +0100 Tomas Vondra wrote: > Well, yeah and no. > > In principle we could/should have allocated the BufFiles in a different > context (possibly hashCxt). But in practice it probably won't make any > difference, because the query will probably run all the hashjoins

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 19:15:30 +0100 Jehan-Guillaume de Rorthais wrote: [...] > For what it worth, these two patches seems really interesting to me. Do you > need any help to revive it? To avoid confusion, the two patches I meant were: * 0001-move-BufFile-stuff-into-separate-context.patch

Re: Memory leak from ExecutorState context?

2023-03-02 Thread Jehan-Guillaume de Rorthais
On Thu, 2 Mar 2023 19:53:14 +0100 Tomas Vondra wrote: > On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote: ... > > There was some thoughts about how to make a better usage of the memory. As > > memory is exploding way beyond work_mem, at least, avoid to waste it with > >

Re: Memory leak from ExecutorState context?

2023-03-10 Thread Jehan-Guillaume de Rorthais
Hi, > So I guess the best thing would be to go through these threads, see what > the status is, restart the discussion and propose what to do. If you do > that, I'm happy to rebase the patches, and maybe see if I could improve > them in some way. OK! It took me some time, but I did it. I'll try t

Re: Memory leak from ExecutorState context?

2023-03-17 Thread Jehan-Guillaume de Rorthais
Hi there, On Fri, 10 Mar 2023 19:51:14 +0100 Jehan-Guillaume de Rorthais wrote: > > So I guess the best thing would be to go through these threads, see what > > the status is, restart the discussion and propose what to do. If you do > > that, I'm happy to rebase the patc

Re: Memory leak from ExecutorState context?

2023-03-20 Thread Jehan-Guillaume de Rorthais
On Mon, 20 Mar 2023 09:32:17 +0100 Tomas Vondra wrote: > >> * Patch 1 could be rebased/applied/backpatched > > > > Would it help if I rebase Patch 1 ("move BufFile stuff into separate > > context")? > > Yeah, I think this is something we'd want to do. It doesn't change the > behavior, but it

Re: Memory leak from ExecutorState context?

2023-03-27 Thread Jehan-Guillaume de Rorthais
Hi, On Mon, 20 Mar 2023 15:12:34 +0100 Jehan-Guillaume de Rorthais wrote: > On Mon, 20 Mar 2023 09:32:17 +0100 > Tomas Vondra wrote: > > > >> * Patch 1 could be rebased/applied/backpatched > > > > > > Would it help if I rebase Patch 1 ("mo

Re: Memory leak from ExecutorState context?

2023-03-28 Thread Jehan-Guillaume de Rorthais
On Tue, 28 Mar 2023 00:43:34 +0200 Tomas Vondra wrote: > On 3/27/23 23:13, Jehan-Guillaume de Rorthais wrote: > > Please, find in attachment a patch to allocate bufFiles in a dedicated > > context. I picked up your patch, backpatch'd it, went through it and did > > s

Re: Memory leak from ExecutorState context?

2023-03-28 Thread Jehan-Guillaume de Rorthais
Hi, Sorry for the late answer, I was reviewing the first patch and it took me some time to study and dig around. On Thu, 23 Mar 2023 08:07:04 -0400 Melanie Plageman wrote: > On Fri, Mar 10, 2023 at 1:51 PM Jehan-Guillaume de Rorthais > wrote: > > > So I guess the best thing

Re: Memory leak from ExecutorState context?

2023-03-31 Thread Jehan-Guillaume de Rorthais
80 total in 3256 blocks; 970128 free (38180 chunks); 27635552 used HashBatchContext: 960544 total in 23 blocks; 7928 free (0 chunks); 952616 used Regards, >From 6814994fa0576a8ba6458412ac5f944135fc3813 Mon Sep 17 00:00:00 2001 From: Jehan-Guillaume de Rorthais D

Re: Memory leak from ExecutorState context?

2023-04-11 Thread Jehan-Guillaume de Rorthais
On Sat, 8 Apr 2023 02:01:19 +0200 Jehan-Guillaume de Rorthais wrote: > On Fri, 31 Mar 2023 14:06:11 +0200 > Jehan-Guillaume de Rorthais wrote: > > [...] > > After rebasing Tomas' memory balancing patch, I did some memory measures > to answer some of my questions.

Re: OOM in hash join

2023-04-14 Thread Jehan-Guillaume de Rorthais
On Fri, 14 Apr 2023 13:21:05 +0200 Matthias van de Meent wrote: > On Fri, 14 Apr 2023 at 12:59, Konstantin Knizhnik wrote: > > > > Hi hackers, > > > > Too small value of work_mem cause memory overflow in parallel hash join > > because of too much number batches. > > There is the plan: > > [..

[BUG] FK broken after DETACHing referencing part

2023-04-20 Thread Jehan-Guillaume de Rorthais
Hi, Considering two partitionned tables with a FK between them: DROP TABLE IF EXISTS p, c, c_1 CASCADE; -- -- Parent table + partition + data CREATE TABLE p ( id bigint PRIMARY KEY ) PARTITION BY list (id); CREATE TABLE p_1 PARTITION OF p FOR VA

Re: Commitfest 2023-03 starting tomorrow!

2023-04-21 Thread Jehan-Guillaume de Rorthais
Hi, After catching up with this thread, where pending bugs are listed and discussed, I wonder if the current patches trying to lower the HashJoin memory explosion[1] could be added to the "Older bugs affecting stable branches" list of https://wiki.postgresql.org/wiki/PostgreSQL_16_Open_Items as I

Re: Memory leak from ExecutorState context?

2023-05-04 Thread Jehan-Guillaume de Rorthais
Hi, On Fri, 21 Apr 2023 16:44:48 -0400 Melanie Plageman wrote: > On Fri, Apr 7, 2023 at 8:01 PM Jehan-Guillaume de Rorthais > wrote: > > > > On Fri, 31 Mar 2023 14:06:11 +0200 > > Jehan-Guillaume de Rorthais wrote: > > > > > > [...] > > >

Re: Memory leak from ExecutorState context?

2023-05-10 Thread Jehan-Guillaume de Rorthais
llCxt"--indicating it > is for the memory required for spilling to permanent storage while > executing hash joins. "Spilling" seems fair and a large enough net to grab everything around temp files and accessing them. > I discuss this more in my code review below. > > &

Re: Unlinking Parallel Hash Join inner batch files sooner

2023-05-10 Thread Jehan-Guillaume de Rorthais
Hi, Thanks for working on this! On Wed, 10 May 2023 15:11:20 +1200 Thomas Munro wrote: > One complaint about PHJ is that it can, in rare cases, use a > surprising amount of temporary disk space where non-parallel HJ would > not. When it decides that it needs to double the number of batches to

Re: Query execution in Perl TAP tests needs work

2023-11-06 Thread Jehan-Guillaume de Rorthais
On Wed, 18 Oct 2023 18:25:01 +0200 Alvaro Herrera wrote: > On 2023-Oct-18, Robert Haas wrote: > > > Without FFI::Platypus, we have to write Perl code that can speak the > > wire protocol directly. Basically, we're writing our own PostgreSQL > > driver for Perl, though we might need only a subset

Sorting regression of text function result since commit 586b98fdf1aae

2023-12-11 Thread Jehan-Guillaume de Rorthais
Hi, A customer found what looks like a sort regression while testing his code from v11 on a higher version. We hunt this regression down to commit 586b98fdf1aae, introduced in v12. Consider the following test case: createdb -l fr_FR.utf8 -T template0 reg psql reg <<<" BEGIN; CREATE TABLE

Re: Sorting regression of text function result since commit 586b98fdf1aae

2023-12-12 Thread Jehan-Guillaume de Rorthais
On Mon, 11 Dec 2023 15:43:12 -0500 Tom Lane wrote: > Jehan-Guillaume de Rorthais writes: > > It looks like since 586b98fdf1aae, the result type collation of > > "convert_from" is forced to "C", like the patch does for type "name", > >

Re: Detoasting optionally to make Explain-Analyze less misleading

2023-09-13 Thread Jehan-Guillaume de Rorthais
Hi Stepan & all, On Tue, 12 Sep 2023 17:16:00 +0200 stepan rutz wrote: ... > Attached a new patch. Hoping for feedback, Nice addition to EXPLAIN! On the feature front, what about adding the actual detoasting/serializing time in the explain output? That could be: => explain (analyze,serial

Re: Transparent column encryption

2022-11-24 Thread Jehan-Guillaume de Rorthais
On Wed, 23 Nov 2022 19:45:06 +0100 Peter Eisentraut wrote: > On 28.10.22 12:16, Jehan-Guillaume de Rorthais wrote: [...] > >* I wonder if encryption related fields in ParameterDescription and > > RowDescription could be optional somehow? The former might be quite >

Re: Retry in pgbench

2021-04-16 Thread Jehan-Guillaume de Rorthais
On Fri, 16 Apr 2021 10:28:48 +0900 (JST) Tatsuo Ishii wrote: > > By the way, I've been playing with the idea of failing gracefully and retry > > indefinitely (or until given -T) on SQL error AND connection issue. > > > > It would be useful to test replicating clusters with a (switch|fail)over >

Re: multi-install PostgresNode fails with older postgres versions

2021-04-19 Thread Jehan-Guillaume de Rorthais
On Mon, 19 Apr 2021 07:43:58 -0700 Mark Dilger wrote: > > On Apr 19, 2021, at 5:11 AM, Andrew Dunstan wrote: > > > > I think therefore I'm inclined for now to do nothing for old version > > compatibility. > > I agree with waiting until the v15 development cycle. Agree.

Re: multi-install PostgresNode fails with older postgres versions

2021-04-19 Thread Jehan-Guillaume de Rorthais
On Mon, 19 Apr 2021 12:37:08 -0400 Andrew Dunstan wrote: > > On 4/19/21 10:43 AM, Mark Dilger wrote: > > > >> On Apr 19, 2021, at 5:11 AM, Andrew Dunstan wrote: > >> > >> I think therefore I'm inclined for now to do nothing for old version > >> compatibility. > > I agree with waiting until the

Re: multi-install PostgresNode fails with older postgres versions

2021-04-19 Thread Jehan-Guillaume de Rorthais
On Mon, 19 Apr 2021 10:35:39 -0700 Mark Dilger wrote: > > On Apr 19, 2021, at 10:25 AM, Jehan-Guillaume de Rorthais > > wrote: > > > > On Mon, 19 Apr 2021 12:37:08 -0400 > > Andrew Dunstan wrote: > > > >> > >> On 4/19/21 10:43 AM, Mark

Re: when the startup process doesn't

2021-04-20 Thread Jehan-Guillaume de Rorthais
On Tue, 20 Apr 2021 15:04:28 +0200 Magnus Hagander wrote: [...] > Yeah, I think we should definitely limit this to local access, one way > or another. Realistically using pg_hba is going to require catalog > access, isn't it? And we can't just go ignore those rows in pg_hba > that for example refe

Re: when the startup process doesn't

2021-04-21 Thread Jehan-Guillaume de Rorthais
On Wed, 21 Apr 2021 12:36:05 -0700 Andres Freund wrote: > [...] > > I don't think that concern equally applies for what I am proposing > here. For one, we already have minRecoveryPoint in ControlData, and we > already use it for the purpose of determining where we need to recover > to, albeit

Re: [BUG] parenting a PK constraint to a self-FK one (Was: Self FK oddity when attaching a partition)

2022-09-30 Thread Jehan-Guillaume de Rorthais
handle missing index without paying attention to related constraints? Regards, On Wed, 24 Aug 2022 12:49:13 +0200 Alvaro Herrera wrote: > On 2022-Aug-24, Jehan-Guillaume de Rorthais wrote: > > > I was naively wondering about such a patch, but was worrying about potential > >

Re: [BUG] parenting a PK constraint to a self-FK one (Was: Self FK oddity when attaching a partition)

2022-10-03 Thread Jehan-Guillaume de Rorthais
On Fri, 30 Sep 2022 16:11:09 -0700 Zhihong Yu wrote: > On Fri, Sep 30, 2022 at 3:30 PM Jehan-Guillaume de Rorthais > wrote: ... > > +* Self-Foreign keys are ignored as the index was preliminary > created > > preliminary created -> primarily created Than

Re: Transparent column encryption

2022-10-28 Thread Jehan-Guillaume de Rorthais
Hi, I did a review of the documentation and usability. # Applying patch The patch applied on top of f13b2088fa2 without trouble. Notice a small warning during compilation: colenccmds.c:134:27: warning: ‘encval’ may be used uninitialized A simple fix could be: +++ b/src/bac

Commitfest documentation

2022-10-31 Thread Jehan-Guillaume de Rorthais
Hi, In the commitfest application, I was wondering today what was the exact meaning and difference between open/closed status (is it only for the current commitfest?) and between «waiting for author» and «Returned with feedback». I couldn't find a clear definition searching the wiki, the mailing

Re: Commitfest documentation

2022-10-31 Thread Jehan-Guillaume de Rorthais
Hi Aleksander, Thank you for your help! On Mon, 31 Oct 2022 16:51:23 +0300 Aleksander Alekseev wrote: [...] > > In the commitfest application, I was wondering today what was the exact > > meaning and difference between open/closed status (is it only for the > > current commitfest?) > > Close

Re: psql: Add command to use extended query protocol

2022-11-02 Thread Jehan-Guillaume de Rorthais
Hi, On Fri, 28 Oct 2022 08:52:51 +0200 Peter Eisentraut wrote: > This adds a new psql command \gp that works like \g (or semicolon) but > uses the extended query protocol. Parameters can also be passed, like > > SELECT $1, $2 \gp 'foo' 'bar' As I wrote in my TCE review, would it be possi

Re: psql: Add command to use extended query protocol

2022-11-02 Thread Jehan-Guillaume de Rorthais
On Wed, 02 Nov 2022 16:04:02 +0100 "Daniel Verite" wrote: > Jehan-Guillaume de Rorthais wrote: > > > As I wrote in my TCE review, would it be possible to use psql vars to set > > some named parameters for the prepared query? This would looks like: > >

Re: [BUG] parenting a PK constraint to a self-FK one (Was: Self FK oddity when attaching a partition)

2022-11-04 Thread Jehan-Guillaume de Rorthais
On Thu, 3 Nov 2022 20:44:16 +0100 Alvaro Herrera wrote: > On 2022-Oct-05, Alvaro Herrera wrote: > > > I've been giving the patches a look and it caused me to notice two > > additional bugs in the same area: > > > > - FKs in partitions are sometimes marked NOT VALID. This is because of > > mi

Re: Fix proposal for comparaison bugs in PostgreSQL::Version

2022-11-04 Thread Jehan-Guillaume de Rorthais
On Thu, 3 Nov 2022 13:11:18 -0500 Justin Pryzby wrote: > On Tue, Jun 28, 2022 at 06:17:40PM -0400, Andrew Dunstan wrote: > > Nice catch, but this looks like massive overkill. I think we can very > > simply fix the test in just a few lines of code, instead of a 190 line > > fix and a 130 line TAP

  1   2   >