Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-22 Thread Matthias van de Meent
On Tue, 22 Feb 2022 at 07:39, Nitin Jadhav wrote: > > > > Thank you for sharing the information. 'triggering backend PID' (int) > > > - can be stored without any problem. 'checkpoint or restartpoint?' > > > (boolean) - can be stored as a integer value like > > > PROGRESS_CHECKPOINT_TYPE_CHECKPOIN

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-23 Thread Matthias van de Meent
On Wed, 23 Feb 2022 at 15:24, Nitin Jadhav wrote: > > > At least for pg_stat_progress_checkpoint, storing only a timestamp in > > the pg_stat storage (instead of repeatedly updating the field as a > > duration) seems to provide much more precise measures of 'time > > elapsed' for other sessions if

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-24 Thread Matthias van de Meent
OGRESS_CHECKPOINT_PHASE_INIT 0 Generally, enum-like values in a stat_progress field are 1-indexed, to differentiate between empty/uninitialized (0) and states that have been set by the progress reporting infrastructure. Kind regards, Matthias van de Meent

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Matthias van de Meent
On Fri, 25 Feb 2022 at 17:35, Julien Rouhaud wrote: > > On Fri, Feb 25, 2022 at 08:53:50PM +0530, Nitin Jadhav wrote: > > > > > > I'm not sure what Matthias meant, but as far as I know there's no > > > fundamental > > > difference between checkpoint with and without the CHECKPOINT_IMMEDIATE > >

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-28 Thread Matthias van de Meent
On Sun, 27 Feb 2022 at 16:14, Bharath Rupireddy wrote: > 3) Why do we need this extra calculation for start_lsn? > Do you ever see a negative LSN or something here? > +('0/0'::pg_lsn + ( > +CASE > +WHEN (s.param3 < 0) THEN pow((2)::numeric, (64)::numeric) > +ELS

Re: New Table Access Methods for Multi and Single Inserts

2022-03-04 Thread Matthias van de Meent
On Mon, 19 Apr 2021 at 06:52, Bharath Rupireddy wrote: > > On Mon, Apr 5, 2021 at 9:49 AM Bharath Rupireddy > wrote: > > > > On Wed, Mar 10, 2021 at 10:21 AM Bharath Rupireddy > > wrote: > > > Attaching the v4 patch set. Please review it further. > > > > Attaching v5 patch set after rebasing ont

Re: New Table Access Methods for Multi and Single Inserts

2022-03-07 Thread Matthias van de Meent
On Sun, 6 Mar 2022 at 12:12, Bharath Rupireddy wrote: > > On Fri, Mar 4, 2022 at 8:07 PM Matthias van de Meent > wrote: > > > Another rebase due to conflicts in 0003. Attaching v6 for review. > > > > I recently touched the topic of multi_insert, and I remembered

Re: Lowering the ever-growing heap->pd_lower

2022-03-10 Thread Matthias van de Meent
On Wed, 16 Feb 2022 at 21:14, Matthias van de Meent wrote: > > On Wed, 16 Feb 2022 at 20:54, Peter Geoghegan wrote: > > > > On Tue, Feb 15, 2022 at 10:48 AM Matthias van de Meent > > wrote: > > > Peter Geoghegan asked for good arguments for the two change

Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-11 Thread Matthias van de Meent
his patch, I've also added a comment to the XLogRecord spec to document that xl_tot_len should not be larger than 1GB - 1B; and why that limit exists. Kind regards, Matthias van de Meent From a72121b8fccbbf8c289e71a2c76801a1004f5353 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Fri

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-14 Thread Matthias van de Meent
Thank you all for the feedback. Please find attached v2 of the patchset, which contains updated comments and applies the suggested changes. On Sat, 12 Mar 2022 at 02:03, Andres Freund wrote: > > Hi, > > On 2022-03-11 22:42:42 +0200, Heikki Linnakangas wrote: > > Have you been able to create a tes

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
hich would > allow us to e.g. get away from needing Heap2. Which would aestethically be > pleasing. That would be interesting; though out of scope for this bug I'm trying to fix. > On 2022-03-14 17:57:23 +0100, Matthias van de Meent wrote: > > I'm not sure whether or no

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-03-15 Thread Matthias van de Meent
Apart from registering this in CF 2022-07 last Friday, I've also just added this issue to the Open Items list for PG15 under "Older bugs affecting stable branches"; as a precaution to not lose track of this issue in the buzz of the upcoming feature freeze. -Matthias

Re: Probable CF bot degradation

2022-03-20 Thread Matthias van de Meent
or CF entries, so that "bad" commits on master won't impact CFbot results as easy? Kind regards, Matthias van de Meent

Re: Hardening heap pruning code (was: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum)

2022-03-20 Thread Matthias van de Meent
On Sun, 20 Mar 2022 at 04:48, Peter Geoghegan wrote: > > Attached is v6, which goes a bit further than v5 in using local state > that we build up-front to describe the state of the page being pruned > (rather than rereading the page itself). I didn't test the code; so these comments are my genera

Re: get_constraint_index() and conindid

2020-12-08 Thread Matthias van de Meent
On Mon, 7 Dec 2020 at 11:09, Peter Eisentraut wrote: > > get_constraint_index() does its work by going through pg_depend. It was > added before pg_constraint.conindid was added, and some callers are > still not changed. Are there reasons for that? Probably not. The > attached patch changes get

Re: [PATCH] Simple progress reporting for COPY command

2021-01-05 Thread Matthias van de Meent
and also disambiguates the meaning in case of e.g. COPY FROM (format CSV), as multiline CSV fields do exist, and we're not necessarily counting lines from or to a file. With regards, Matthias van de Meent

Re: [PATCH] Simple progress reporting for COPY command

2021-01-07 Thread Matthias van de Meent
#x27;s strings delimited by newlines (which I believe is not exactly what we're counting). One common user of COPY is the pg_dump tool, and that uses binary format by default (iirc). Additionally, all comments surrounding the *LINES_PROCESSED updates only mention '

Re: [PATCH] Simple progress reporting for COPY command

2021-01-08 Thread Matthias van de Meent
On Thu, 7 Jan 2021 at 23:00, Josef Šimánek wrote: > > čt 7. 1. 2021 v 22:37 odesílatel Tomas Vondra > napsal: > > > > I'm not particularly attached to the "lines" naming, it just seemed OK > > to me. So if there's consensus to rename this somehow, I'm OK with it. > > The problem I do see here is

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
ct notice to changes in the view. With regards, Matthias van de Meent. From f7d761f6774753d4914d0dbc80effbb1ab09b58e Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Mon, 8 Feb 2021 17:36:00 +0100 Subject: [PATCH v4 4/6] Add a command column to the copy progress view This allows filteri

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
On Fri, 12 Feb 2021 at 12:23, Matthias van de Meent wrote: > > On Thu, 11 Feb 2021 at 15:44, Bharath Rupireddy > wrote: > > > > > > On Thu, Feb 11, 2021, 8:08 PM Josef Šimánek wrote: > >> I have split it since it should be the start of progress reportin

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
On Fri, 12 Feb 2021 at 13:40, Bharath Rupireddy wrote: > > On Fri, Feb 12, 2021 at 5:40 PM Matthias van de Meent > wrote: > > > > On Fri, 12 Feb 2021 at 12:23, Matthias van de Meent > > wrote: > > > > > > On Thu, 11 Feb 2021 at 15:44, Bharath Rupir

Re: PostgreSQL <-> Babelfish integration

2021-02-12 Thread Matthias van de Meent
te the benefits or would provide a guide for implementing such frontend, I see no reason for the postgres project to put work into such a feature. With regards, Matthias van de Meent [0] I believe this is an optimistic guess, based on the changes that were (and are yet still) required for the zedstore and/or zheap tableam, but am happy to be proven wrong.

Re: SSL SNI

2021-02-15 Thread Matthias van de Meent
#x27;1one.example.com' is a valid hostname, but would fail this trivial test, and thus would not have SNI enabled on its connection. With regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/37846a5e-bb5e-0c4f-3ee8-54fb4bd02fab%40gmx.de

Re: progress reporting for partitioned REINDEX

2021-02-16 Thread Matthias van de Meent
ing worked on. This field needs a pgstat_progress_start_command because CIC / RiC reindexes all indexes concurrently at the same time (and not grouped by e.g. table), so we must re-start reporting for each index in each new phase in which we report data to get the heapId reported correctly for that index. With regards, Matthias van de Meent

Re: Improvements and additions to COPY progress reporting

2021-02-18 Thread Matthias van de Meent
ually the *source*. Yes, I was also not quite happy with this, but couldn't find a better one at the point of writing the initial patchset. Would "io_operations", "io_port", "operates_through" or "through" maybe be better? With regards, Matthias van

Re: Improvements and additions to COPY progress reporting

2021-02-21 Thread Matthias van de Meent
Thank you, that's way better! PFA what I believe is a finalized patchset v9, utilizing io_type terminology instead of io_target. With regards, Matthias van de Meent From 5c0c9658e7c50a2a2d74a738b7bdbfcaa6362e24 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Fri, 12 Feb 2021 14:0

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Matthias van de Meent
On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy wrote: > > On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent > wrote: > > > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > > wrote: > > > > > > For COPY TO the name "source_type" co

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
changes I believe that that was the extent of the not-yet-resolved comments and suggestions. With regards, Matthias van de Meent. From 7831549452b6d95c3db9060333750256675ff322 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Josef=20=C5=A0im=C3=A1nek?= Date: Tue, 9 Feb 2021 13:06:45 +0100 Subject: [PATCH v11

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
quot; is, a.f.a.i.k., correct English. Adding 'will' would indeed still be correct, but it would in my opinion also be decremental to the readability of the section due to the repeated use of the same template sentence structure. I think that keeping it as-is is just fine. With regards, Matthias van de Meent.

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
On Thu, 4 Mar 2021 at 17:29, Justin Pryzby wrote: > > On Thu, Mar 04, 2021 at 05:19:18PM +0100, Matthias van de Meent wrote: > > > > "Backends running [...] report progress to [...] instead" is, > > a.f.a.i.k., correct English. Adding 'will' would ind

Re: Improvements and additions to COPY progress reporting

2021-03-08 Thread Matthias van de Meent
rent thread, but I'm not sure that would make it into pg14. With regards, Matthias van de Meent. From 94876abe0ab9c28a6f4b0ac006f356251ca4746c Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Mon, 8 Mar 2021 16:08:24 +0100 Subject: [PATCH v12 3/3] Adapt COPY progress reporting to report

Re: non-HOT update not looking at FSM for large tuple update

2021-03-08 Thread Matthias van de Meent
quot;It doesn't remove unused line pointers! Please don't change this." (Oct. 2000), but I can't seem to find the documentation / conversations on the implications that such shrinking would have. With regards, Matthias van de Meent.

Lowering the ever-growing heap->pd_lower

2021-03-09 Thread Matthias van de Meent
he space reserved for ItemIds on a page, keeping valid linepointers intact. Enjoy, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/cad21aod0ske11fmw4jd4renawbmcw1wasvnwpjvw3tvqpoq...@mail.gmail.com [1] https://www.postgresql.org/message-id/CAEze2W

Re: Lowering the ever-growing heap->pd_lower

2021-03-09 Thread Matthias van de Meent
recovery to determine if that may touch pages that have shrunken line pointer arrays, or if those situations won't exist due to never using dirtied pages in recovery) and backwards table scans on non-MVCC snapshots (which would be fixed in the attached patch). With regards, Matthias

Re: Improvements and additions to COPY progress reporting

2021-03-10 Thread Matthias van de Meent
On Tue, 9 Mar 2021 at 06:34, Michael Paquier wrote: > > On Mon, Mar 08, 2021 at 05:33:40PM +0100, Matthias van de Meent wrote: > > Seems reasonable. PFA updated patches. I've renamed the previous 0003 > > to 0002 to keep git-format-patch easy. > > Thanks for updat

Re: Lowering the ever-growing heap->pd_lower

2021-03-10 Thread Matthias van de Meent
On Tue, 9 Mar 2021 at 22:35, Tom Lane wrote: > > Matthias van de Meent writes: > > The only two existing mechanisms that I could find (in the access/heap > > directory) that possibly could fail on shrunken line pointer arrays; > > being xlog recovery (I do not have enou

Re: non-HOT update not looking at FSM for large tuple update

2021-03-11 Thread Matthias van de Meent
len + saveFreeSpace > MaxHeapTupleSize) + if (len + saveFreeSpace > maxPaddedFsmRequest) ... - targetFreeSpace = Max(len, MaxHeapTupleSize - (MaxHeapTupleSize * 2 / 100)); + targetFreeSpace = Max(len, maxPaddedFsmRequest); ... Other than this, I think this is a good fix. With regards, Matthias van de Meent.

Re: Self-join optimisation

2021-03-11 Thread Matthias van de Meent
ng them as the > same table. I think the same would be true for self-joins on any non-null > columns covered by a uniqueness constraint. > > If this is considered a desirable change, I'd be keen to work on it (with > some guidance). There's currently a p

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Matthias van de Meent
On Thu, 11 Mar 2021 at 17:31, Robert Haas wrote: > > On Tue, Mar 9, 2021 at 3:35 PM Peter Geoghegan wrote: > > Speaking of line pointer bloat (and "irreversible" bloat), I came > > across something relevant today. I believe that this recent patch from > > Mat

Re: non-HOT update not looking at FSM for large tuple update

2021-03-12 Thread Matthias van de Meent
On Thu, 11 Mar 2021 at 16:16, John Naylor wrote: > > On Thu, Mar 11, 2021 at 9:46 AM Matthias van de Meent > wrote: > > > Regarding the 2% slack logic, could we change it to use increments of > > line pointers instead? That makes it more clear what problem this > &

Re: Improvements and additions to COPY progress reporting

2021-03-15 Thread Matthias van de Meent
On Mon, 15 Mar 2021 at 05:53, Michael Paquier wrote: > > On Wed, Mar 10, 2021 at 09:35:10AM +0100, Matthias van de Meent wrote: > > There are examples in which pg_stat_progress_* -views report > > inaccurate data. I think it is fairly reasonable to at least validate > > s

Re: libpq compression

2020-11-02 Thread Matthias van de Meent
On Mon, 2 Nov 2020 at 15:03, Konstantin Knizhnik wrote: > > It seems to be very important to be able to measure network traffic > between client and server, especially in case of using compression. > Althought there are a lot of toll for monitoring network traffix for > Linux and other OSes, I di

Re: libpq compression

2020-11-05 Thread Matthias van de Meent
On Mon, 2 Nov 2020 at 20:20, Konstantin Knizhnik wrote: > > On 02.11.2020 19:53, Matthias van de Meent wrote: > > This is the result of network traffic of two backends one with enabled > >> compression and another with disable compression > >> after execution of &q

Re: libpq compression

2020-11-05 Thread Matthias van de Meent
On Thu, 5 Nov 2020 at 17:01, Konstantin Knizhnik wrote: > > Sorry, I do not understand your point. > This view reports network traffic from server's side. > But client's traffic information is "mirror" of this statistic: > server_tx=client_rx and visa versa. > > Yes, first few bytes exchanged by

[patch] CLUSTER blocks scanned progress reporting

2020-11-20 Thread Matthias van de Meent
van de Meent From 12f7617f4999b7bf85cc2c9ef4c3b96aac3b26e8 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Fri, 20 Nov 2020 16:23:59 +0100 Subject: [PATCH v1] Fix CLUSTER progress reporting of number of blocks scanned. The heapScan need not start at block 0, so heapScan->rs_cblock ne

Re: [patch] CLUSTER blocks scanned progress reporting

2020-11-24 Thread Matthias van de Meent
On Tue, 24 Nov 2020 at 15:05, Fujii Masao wrote: > > On 2020/11/21 2:32, Matthias van de Meent wrote: > > Hi, > > > > The pg_stat_progress_cluster view can report incorrect > > heap_blks_scanned values when synchronize_seqscans is enabled, because > > it al

Re: [patch] CLUSTER blocks scanned progress reporting

2020-11-25 Thread Matthias van de Meent
On Wed, 25 Nov 2020 at 10:35, Fujii Masao wrote: > > On 2020/11/25 0:25, Matthias van de Meent wrote: > > > > I noticed that with my proposed patch it is still possible to go to > > the next phase while heap_blks_scanned != heap_blks_total. This can > > happen whe

Re: [patch] CLUSTER blocks scanned progress reporting

2020-11-26 Thread Matthias van de Meent
hen the table scan finishes. > PFA a patch with updated message and comment. I've reworded the messages to specifically mention empty pages and the need for setting heap_blks_scanned = total_heap_blks explicitly. Feel free to update the specifics, other than that I think this is a co

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-18 Thread Matthias van de Meent
y through VACUUM (and optimistic ALL_DEAD detection). According to my limited knowledge of RI, it requires VACUUM to not run on the table during the initial index build process (which is currently guaranteed through the use of a snapshot). Regards, Matthias van de Meent

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-19 Thread Matthias van de Meent
On Mon, 18 Jan 2021, 21:25 Álvaro Herrera, wrote: > > On 2021-Jan-18, Matthias van de Meent wrote: > > > Example: > > > > 1.) RI starts > > 2.) PHASE 2: filling the index: > > 2.1.) scanning the heap (live tuple is cached) > > < tuple is deleted &g

postgres_fdw: IMPORT FOREIGN SCHEMA ... LIMIT TO (partition)

2021-01-21 Thread Matthias van de Meent
locks on the foreign root partition during queries, and less suprising behaviour for LIMIT TO ("table that happens to be a partition"). Regards, Matthias van de Meent From f69b961d743b3fdd647b761d6069d8fa9163e2c7 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent Date: Thu, 21 Jan 202

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-21 Thread Matthias van de Meent
On Tue, 19 Jan 2021 at 21:59, Matthias van de Meent wrote: > > On Mon, 18 Jan 2021, 21:25 Álvaro Herrera, wrote: > > > > On 2021-Jan-18, Matthias van de Meent wrote: > > > > > Example: > > > > > > 1.) RI starts > > > 2.) PHASE 2: fil

Improvements and additions to COPY progress reporting

2021-02-08 Thread Matthias van de Meent
005 is that it reports the io_target for the logical replications' initial tablesyncs' internal COPY. This would otherwise be measured, but no knowledge about the type of copy (or its origin) would be available on the worker's side. I'm not married to this patch 0005, but I bel

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread Matthias van de Meent
On Tue, 9 Feb 2021 at 08:12, Bharath Rupireddy wrote: > > On Tue, Feb 9, 2021 at 12:06 AM Matthias van de Meent > wrote: > > With [0] we got COPY progress reporting. Before the column names of > > this newly added view are effectively set in stone with the release of &g

Re: Improvements and additions to COPY progress reporting

2021-02-11 Thread Matthias van de Meent
On Wed, 10 Feb 2021 at 07:43, Bharath Rupireddy wrote: > > On Tue, Feb 9, 2021 at 6:02 PM Matthias van de Meent > wrote: > > > Also, you can add this to the current commitfest. > > > > See https://commitfest.postgresql.org/32/2977/ > > > > On Tue,

Re: generalized conveyor belt storage

2021-12-15 Thread Matthias van de Meent
re ForkNumber etc. are defined) that allows one more fork per relation. Is that too on the missing features list, or did I misunderstand what you meant with "relation fork"? Kind regards, Matthias van de Meent

Re: non-HOT update not looking at FSM for large tuple update

2021-03-18 Thread Matthias van de Meent
On Wed, 17 Mar 2021 at 21:52, John Naylor wrote: > > On Fri, Mar 12, 2021 at 8:45 AM Matthias van de Meent > wrote: > > > > If this case isn't added, the lower else branch will fail to find > > fitting pages for len > maxPaddedFsmRequest tuples; potentially &g

Re: non-HOT update not looking at FSM for large tuple update

2021-03-19 Thread Matthias van de Meent
On Fri, 19 Mar 2021 at 19:16, John Naylor wrote: > > On Thu, Mar 18, 2021 at 5:30 PM Matthias van de Meent > wrote: > > > > This is slightly more verbose, but I think this clarifies the > > reasoning why we need this a bit better. Feel free to reject or adapt > &

Re: postgres_fdw: IMPORT FOREIGN SCHEMA ... LIMIT TO (partition)

2021-03-24 Thread Matthias van de Meent
On Mon, 22 Mar 2021 at 21:16, Bernd Helmle wrote: > > Hi, > > I took a look at this patch. Thanks! > Patch applies on current master. > > Documentation and adjusted regression tests included. > Regression tests passes without errors. > > The patch changes IMPORT FOREIGN SCHEMA to explicitely all

Re: Calendar support in localization

2021-03-29 Thread Matthias van de Meent
sei", as the era changed in 2019. ICU releases have since implemented this and other corrections; this specific change was implemented in the batched release of ICU versions on 2019-04-12. With regards, Matthias van de Meent

Re: Calendar support in localization

2021-03-29 Thread Matthias van de Meent
On Mon, 29 Mar 2021 at 14:33, Daniel Verite wrote: > > Matthias van de Meent wrote: > > > The results for the Japanese locale should be "0003 Reiwa" instead of > > "0033 Heisei", as the era changed in 2019. ICU releases have since > > implem

Re: Lowering the ever-growing heap->pd_lower

2021-03-31 Thread Matthias van de Meent
On Wed, 31 Mar 2021 at 05:35, Peter Geoghegan wrote: > > On Wed, Mar 10, 2021 at 6:01 AM Matthias van de Meent > wrote: > > > The case I was concerned about back when is that there are various bits of > > > code that may visit a page with a predetermined TID in mind

Re: New IndexAM API controlling index vacuum strategies

2021-04-05 Thread Matthias van de Meent
d aggressive scans have a chance to > skip > + index cleanup. This documentation doesn't quite make it clear what its relationship is with autovacuum_freeze_max_age. How about the following: "... >VACUUM< will use the higher of this value and 105% of >guc-autovacuum-freeze-max-age<, so that only ...". It's only slightly easier to read, but at least it conveys that values lower than 105% of autovacuum_freeze_max_age are not considered. The same can be said for the multixact guc documentation. With regards, Matthias van de Meent

Re: postgres_fdw: IMPORT FOREIGN SCHEMA ... LIMIT TO (partition)

2021-04-06 Thread Matthias van de Meent
t; Attached is the updated version of the patch. Barring any objection, > I'm thinking to commit this. Thanks, this was on my to-do list for today, but you were faster. No objections on my part, and thanks for picking this up. With regards, Matthias van de Meent

Re: New IndexAM API controlling index vacuum strategies

2021-04-06 Thread Matthias van de Meent
On Tue, 6 Apr 2021 at 05:13, Peter Geoghegan wrote: > > On Mon, Apr 5, 2021 at 2:44 PM Matthias van de Meent > wrote: > > This assertion is false, and should be a guarding if-statement. HOT > > redirect pointers are not updated if the tuple they're pointing to is >

Re: 2019-03 CF now in progress

2021-04-08 Thread Matthias van de Meent
t with regards to number of committed patches registered to one CF. > Today and tomorrow I'll be checking the Waiting on Author patches to > determine which should be moved to the next CF and which should be > returned with feedback. The rest will be moved to the next CF when this > one is closed. I noticed that there is an old patch stuck at 'Needs review' in CF 2020-09. Could you also update the state of that patch, because I don't think I am the right person to do that. With regards, Matthias van de Meent

Re: RFC: compression dictionaries for JSONB

2021-10-11 Thread Matthias van de Meent
well. > > Thoughts? I think that an 'universal dictionary encoder' would be useful, but that a data type might also have good reason to implement their replacement methods by themselves for better overall performance (such as maintaining partial detoast support in dictionaried items, or overall lower memory footprint, or ...). As such, I'd really appreciate it if Option 1 is not ruled out by any implementation of Option 2. Kind regards, Matthias van de Meent

Re: RFC: compression dictionaries for JSONB

2021-10-13 Thread Matthias van de Meent
On Wed, 13 Oct 2021 at 11:48, Aleksander Alekseev wrote: > > Hi Matthias, > > > Assuming this above is option 1. If I understand correctly, this > > option was 'adapt the data type so that it understands how to handle a > > shared dictionary, decreasing storage requirements'. > > [...] > > Assumin

Re: Adding CI to our tree

2021-10-21 Thread Matthias van de Meent
latform.) With the provided arguments I won't object to the addition of these config files, but I would appreciate it if a clear policy could be provided on the inclusion of configurations for external tools that are not expected to be used by all users of the repository, such as CI, e

Re: Next Steps with Hash Indexes

2021-10-27 Thread Matthias van de Meent
x27;t think it happens all the time, I do think it can happen often enough on e.g. an HTAP system to make it a noteworthy test case. Kind regards, Matthias van de Meent

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-05 Thread Matthias van de Meent
ccordingly. I've tested this patch, and it correctly fixes the issue of blocking xmin from advancing, and also fixes an issue of retreating the observed *_oldest_nonremovable in XidHorizons through parallel workers. There are still some other soundness issues with xmin handling (see [0]), but that should not prevent this patch from landing in the relevant branches. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/flat/17257-1e46de26bec11433%40postgresql.org

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Matthias van de Meent
On Wed, 10 Nov 2021 at 11:51, Amit Kapila wrote: > > On Fri, Nov 5, 2021 at 8:16 PM Matthias van de Meent > wrote: > > AFAICU, in the thread referred by you, it seems that the main reported > issue will be resolved by this patch but there is a discussion about > xmin moving

Re: [PATCH] Partial foreign key updates in referential integrity triggers

2021-12-01 Thread Matthias van de Meent
ve, and DDL operations now growing linearly with the amount of tenants in the system. Kind regards, Matthias van de Meent

Re: the s_lock_stuck on perform_spin_delay

2024-01-10 Thread Matthias van de Meent
lf shouldn't be needed in places where we need to make sure that we don't hold any spinlocks, and miscadmin.h already holds things related to "System interrupt and critical section handling", which seems quite related. Kind regards, Matthias van de Meent

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-15 Thread Matthias van de Meent
many cycles. > +-- > +-- Here we don't remember the scan's array keys before processing a page, > only > +-- after processing a page (which is implicit, it's just the scan's current > +-- keys). So when we move the scan backwards we think that the top-level > scan > +-- should terminate, when in reality it should jump backwards to the leaf > page > +-- that we last visited. I notice this adds a complex test case that outputs many rows. Can we do with less rows if we build the index after data insertion, and with a lower (non-default) fillfactor? Note: I did not yet do any in-depth review of the planner changes in indxpath.c/selfuncs.c. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: cleanup patches for incremental backup

2024-01-15 Thread Matthias van de Meent
> Thanks, committed. Off-list I was notified that the new WAL summarizer process was not yet added to the glossary, so PFA a patch that does that. In passing, it also adds "incremental backup" to the glossary, and updates the documented types of backends in monitoring.sgml with the new

Re: cleanup patches for incremental backup

2024-01-16 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 16:39, Robert Haas wrote: > > On Mon, Jan 15, 2024 at 3:31 PM Matthias van de Meent > wrote: > > Off-list I was notified that the new WAL summarizer process was not > > yet added to the glossary, so PFA a patch that does that. > > In passing

Re: cleanup patches for incremental backup

2024-01-17 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 21:49, Robert Haas wrote: > > On Tue, Jan 16, 2024 at 3:22 PM Matthias van de Meent > wrote: > + A special base > backup > + that for some WAL-logged relations only contains the pages that were > + modified since a previous backup, a

Re: cleanup patches for incremental backup

2024-01-18 Thread Matthias van de Meent
On Wed, 17 Jan 2024 at 21:10, Robert Haas wrote: > > On Wed, Jan 17, 2024 at 1:42 PM Matthias van de Meent > wrote: > > Sure, added in attached. > > I think this mostly looks good now but I just realized that I think > this needs rephrasing: > > + To restor

Re: Sequence Access Methods, round two

2024-01-18 Thread Matthias van de Meent
s it would allow a sequence to be backed by an Linear Congruential Generator directly, rather than the implementation of our own transactional random_sequence table. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Increasing IndexTupleData.t_info from uint16 to uint32

2024-01-18 Thread Matthias van de Meent
eaderData.pd_pagesize_version). Next, the only non-internal use of IndexTuple is in IndexOnlyScans. However, here the index may fill the scandesc->xs_hitup with a heap tuple instead, which has a length stored in uint32, too. So, I don't quite see why this would be required for all indexes. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-18 Thread Matthias van de Meent
On Tue, 16 Jan 2024 at 03:03, Peter Geoghegan wrote: > > On Mon, Jan 15, 2024 at 2:32 PM Matthias van de Meent > wrote: > > Can you pull these planner changes into their own commit(s)? > > As mentioned upthread, it's a significant change in behavior that > > sh

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2024-01-22 Thread Matthias van de Meent
On Fri, 19 Jan 2024 at 23:42, Peter Geoghegan wrote: > Thank you for your replies so far. > On Thu, Jan 18, 2024 at 11:39 AM Matthias van de Meent > wrote: > > I would agree with you if this was about new APIs and features, but > > here existing APIs are being repurposed

Re: btree: implement dynamic prefix truncation (was: Improving btree performance through specializing by key shape, take 2)

2024-01-24 Thread Matthias van de Meent
On Fri, 19 Jan 2024 at 05:55, Dilip Kumar wrote: > > On Wed, Nov 1, 2023 at 2:42 AM Matthias van de Meent > wrote: > > > > Hi, > > > > Currently, nbtree code compares each and every column of an index > > tuple during the binary search on the index page

Re: Add bump memory context type and use it for tuplesorts

2024-01-25 Thread Matthias van de Meent
On Mon, 6 Nov 2023 at 19:54, Matthias van de Meent wrote: > > On Tue, 11 Jul 2023 at 01:51, David Rowley wrote: >> >> On Tue, 27 Jun 2023 at 21:19, David Rowley wrote: >>> I've attached the bump allocator patch and also the script I used to >>> gather th

Re: Reducing output size of nodeToString

2024-01-30 Thread Matthias van de Meent
On Tue, 9 Jan 2024, 09:23 Peter Eisentraut, wrote: > > On 04.01.24 00:23, Matthias van de Meent wrote: > > Something like the attached? It splits out into the following > > 0001: basic 'omit default values' > > /* Write an integer field (anything writt

Re: Reducing output size of nodeToString

2024-01-31 Thread Matthias van de Meent
On Wed, 31 Jan 2024, 09:16 Peter Eisentraut, wrote: > On 30.01.24 12:26, Matthias van de Meent wrote: > >> Most of the other defaults I'm doubtful about. First, we are colliding > >> here between the goals of minimizing the storage size and making the > >> d

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-02-22 Thread Matthias van de Meent
On Wed, 22 Feb 2023 at 13:15, Tomas Vondra wrote: > > On 2/20/23 19:15, Matthias van de Meent wrote: > > Thanks. Based on feedback, attached is v2 of the patch, with as > > significant changes: > > > > - We don't store the columns we mention in predicat

Re: PATCH: Using BRIN indexes for sorted output

2023-02-23 Thread Matthias van de Meent
f the patches are marked WIP. Could you share a status on those, because currently that status is unknown: Are these patches you don't plan on including, or are these patches only (or mostly) included for debugging? Kind regards, Matthias van de Meent

Re: PATCH: Using BRIN indexes for sorted output

2023-02-23 Thread Matthias van de Meent
On Thu, 23 Feb 2023 at 16:22, Tomas Vondra wrote: > > On 2/23/23 15:19, Matthias van de Meent wrote: >> Comments on 0001, mostly comments and patch design: One more comment: >>> + range->min_value = bval->bv_values[0]; >>> + range->max_value = bval

Re: PATCH: Using BRIN indexes for sorted output

2023-02-24 Thread Matthias van de Meent
k = 250 >heapBlk = (250 + 10) % 255 = 5 > > and we don't do that loop. Or did I get this wrong, somehow? The result is off-by-one due to (u)int8 overflows being mod-256, but apart from that your result is accurate. The condition only stops the loop when we wrap around or when we go past the last block, but no earlier than that. Kind regards, Matthias van de Meent

Re: PATCH: Using BRIN indexes for sorted output

2023-02-24 Thread Matthias van de Meent
On Thu, 23 Feb 2023 at 19:48, Tomas Vondra wrote: > > On 2/23/23 17:44, Matthias van de Meent wrote: > > On Thu, 23 Feb 2023 at 16:22, Tomas Vondra > > wrote: > >> > >> On 2/23/23 15:19, Matthias van de Meent wrote: > >>> Comments on 0001, mo

Re: PATCH: Using BRIN indexes for sorted output

2023-02-27 Thread Matthias van de Meent
On Fri, 24 Feb 2023, 20:14 Tomas Vondra, wrote: > > On 2/24/23 19:03, Matthias van de Meent wrote: > > On Thu, 23 Feb 2023 at 19:48, Tomas Vondra > >> Yeah, that sounds like a bug. Also a sign the tests should have some > >> by-ref data types (presumably there are

Re: PATCH: Using BRIN indexes for sorted output

2023-02-27 Thread Matthias van de Meent
Hi, On Thu, 23 Feb 2023 at 17:44, Matthias van de Meent wrote: > > I'll see to further reviewing 0004 and 0005 when I have additional time. Some initial comments on 0004: > +/* > + * brin_minmax_ranges > + *Load the BRIN ranges and sort them. > + */ > +D

Re: pgsql: Harden new test case against force_parallel_mode = regress.

2023-03-03 Thread Matthias van de Meent
On Fri, 3 Mar 2023 at 17:16, Robert Haas wrote: > > On Thu, Mar 2, 2023 at 5:47 PM Tom Lane wrote: > > Harden new test case against force_parallel_mode = regress. > > > > Per buildfarm: worker processes can't see a role created in > > the current transaction. > > Now why would that happen? Surely

Re: Add pg_walinspect function with block info columns

2023-03-06 Thread Matthias van de Meent
n processing on pg_walinspect where they want or need it, without us having to decide what the user wants, and without having to associate blocks with the main xlog record data through the joining of several (fairly expensive) xlog decoding passes. The basic idea is to create a single entrypoint to all relevant data from DecodedXLogRecord in SQL, not multiple. Kind regards, Matthias van de Meent

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-06 Thread Matthias van de Meent
n a system that has experienced timeline switches. It is entirely possible that the first file (as sorted by the default collation) is not the first record you can inspect, or even in your timeline's history. Kind regards, Matthias van de Meent

Re: Combine pg_walinspect till_end_of_wal functions with others

2023-03-06 Thread Matthias van de Meent
On Mon, 6 Mar 2023 at 16:37, Bharath Rupireddy wrote: > > On Mon, Mar 6, 2023 at 8:52 PM Matthias van de Meent > wrote: > > > > On Mon, 6 Mar 2023 at 16:06, Bharath Rupireddy > > wrote: > > > If we try to make these functions figure out the oldest WAl file an

Re: Add pg_walinspect function with block info columns

2023-03-07 Thread Matthias van de Meent
On Tue, 7 Mar 2023 at 01:34, Michael Paquier wrote: > > On Mon, Mar 06, 2023 at 04:08:28PM +0100, Matthias van de Meent wrote: > > On Mon, 6 Mar 2023 at 15:40, Bharath Rupireddy > >> IMO, pg_get_wal_records_extended_info as proposed doesn't look good to > >> m

Re: Ignoring BRIN for HOT updates (was: -udpates seems broken)

2023-03-08 Thread Matthias van de Meent
On Wed, 22 Feb 2023 at 14:14, Matthias van de Meent wrote: > > On Wed, 22 Feb 2023 at 13:15, Tomas Vondra > wrote: > > > > On 2/20/23 19:15, Matthias van de Meent wrote: > > > Thanks. Based on feedback, attached is v2 of the patch, with as > > > significan

  1   2   3   4   5   6   7   8   >