Re: [PATCH] Compression dictionaries for JSONB

2022-06-05 Thread Matthias van de Meent
that would allow for an overall better user experience and better performance due to decreased need for fully decompressed type casting. Kind regards, Matthias van de Meent.

Re: better page-level checksums

2022-06-09 Thread Matthias van de Meent
ffer manager also touch / reserve space in the special areas seems like a break of abstraction: Quoting from bufpage.h: * AM-generic per-page information is kept in PageHeaderData. * * AM-specific per-page data (if any) is kept in the area marked "special * space"; each AM has an "opaque" structure defined somewhere that is * stored as the page trailer. an access method should always * initialize its pages with PageInit and then set its own opaque * fields. I'd rather we keep this contract: am-generic stuff belongs in PageHeaderData, with the rest of the page fully available for the AM to use (including the special area). Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-06-11 Thread Matthias van de Meent
On Sat, 11 Jun 2022 at 01:32, David Zhang wrote: > > Hi, > > > > MaxAllocSize is pretty easy: > > > SELECT pg_logical_emit_message(false, long, long) FROM repeat(repeat(' ', > > > 1024), 1024*1023) as l(long); > > > > > > on a standby: > > > > > > 2022-03-11 16:41:59.336 PST [3639744][startup][1/

Re: better page-level checksums

2022-06-13 Thread Matthias van de Meent
On Fri, 10 Jun 2022 at 15:58, Robert Haas wrote: > > On Thu, Jun 9, 2022 at 8:00 PM Matthias van de Meent > wrote: > > Why so? We already dole out per-page space in 4-byte increments > > through pd_linp, and I see no reason why we can't reserve some line > > poin

Re: better page-level checksums

2022-06-14 Thread Matthias van de Meent
On Tue, 14 Jun 2022 at 14:56, Robert Haas wrote: > > On Mon, Jun 13, 2022 at 5:14 PM Matthias van de Meent > wrote: > > It's not that I disagree with (or dislike the idea of) increasing the > > resilience of checksums, I just want to be very careful that we do

Re: CREATE TABLE ( .. STORAGE ..)

2022-06-16 Thread Matthias van de Meent
On Wed, 15 Jun 2022 at 16:51, Aleksander Alekseev wrote: > > Hi hackers, > > I noticed that cfbot is not entirely happy with the patch, so I rebased it. > > > I see that COMPRESSION and STORAGE now are handled slightly > > differently in the grammar. Maybe we could standardize that a bit > > more;

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-06-20 Thread Matthias van de Meent
On Mon, 20 Jun 2022 at 07:02, Michael Paquier wrote: > > On Sat, Jun 11, 2022 at 09:25:48PM +0200, Matthias van de Meent wrote: > > Did you initiate a new cluster or otherwise skip the invalid record > > you generated when running the instance based on master? It seems to >

Re: some aspects of our qsort might not be ideal

2022-06-23 Thread Matthias van de Meent
On Thu, 23 Jun 2022 at 15:52, Robert Haas wrote: > > On Thu, Jun 23, 2022 at 6:13 AM John Naylor > wrote: > > Here is a *rough* first pass at dual-pivot quicksort. I haven't > > changed the regression tests to adjust for qsort being an unstable > > sort, ... > > Hmm. I thought we had some reasons

Re: some aspects of our qsort might not be ideal

2022-06-23 Thread Matthias van de Meent
On Thu, 23 Jun 2022 at 17:04, Peter Geoghegan wrote: > > On Thu, Jun 23, 2022 at 7:51 AM Matthias van de Meent > wrote: > > I think that mostly has to do with reliability / stability of ORDER BY > > in combination with LIMIT and OFFSET, even though right now we cannot >

Pre-installed index access methods cannot be manually installed.

2022-06-24 Thread Matthias van de Meent
of the supplied relation's access method, which will not match the expected value when manually installed. Is this expected? Would we accept patches that remove or reduce the impact of these artificial limitations? Kind regards, Matthias van de Meent PS. I noticed this when checking the sortsupport

Re: Pre-installed index access methods cannot be manually installed.

2022-06-24 Thread Matthias van de Meent
> PS. I noticed this when checking the sortsupport code for some active > patches, and after playing around a bit while trying to run PG > extensions that are not system-registered. The attached test case > fails, where I'd expect it to succeed, or at least I expected it not > to fail at "This AM's

Re: making relfilenodes 56 bits

2022-06-28 Thread Matthias van de Meent
o be common practise, but this workload increases the number of dbOid+spcOid combinations to 100s (even while using only a single tablespace), which in my opinion requires some more thought than just handwaving it into an smgr array and/or checkpoint records. Kind regards, Matthias van de Meent

Re: making relfilenodes 56 bits

2022-06-29 Thread Matthias van de Meent
On Wed, 29 Jun 2022 at 14:41, Simon Riggs wrote: > > On Tue, 28 Jun 2022 at 19:18, Matthias van de Meent > wrote: > > > I will be the first to admit that it is quite unlikely to be common > > practise, but this workload increases the number of dbOid+spcOid > > co

Re: Pluggable toaster

2022-07-01 Thread Matthias van de Meent
he correct set of patches to apply once downloaded. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/message-id/CA%2BTgmoZBgNipyKuQAJzNw2w7C9z%2B2SMC0SAHqCnc_dG1nSLNcw%40mail.gmail.com

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-01 Thread Matthias van de Meent
On Tue, 21 Jun 2022 at 03:45, Michael Paquier wrote: > + /* > +* Ensure that xlogreader.c can read the record by ensuring that the > +* data section of the WAL record can be allocated. > +*/ > + if (unlikely(!AllocSizeIsValid(total_len))) > + XLogErrorDataLimitExceeded(); > >

Re: [PATCH] Compression dictionaries for JSONB

2022-07-04 Thread Matthias van de Meent
tion with different TableAMs, etc), I > seriously doubt that this particular part of "pluggable TOASTer" will > end-up in the core. Yes, and that's why I think that this where this dictionary infrastructure could provide value, as an alternative or extension to the proposed jsonb toaster in the 'pluggable toaster' thread. Kind regards, Matthias van de Meent

Re: doc: Fix description of how the default user name is chosen

2022-07-06 Thread Matthias van de Meent
not connect to the database named after the user's username"). The right-ness of the systematic default doesn't matter in this context (that's something to put in the comments of that code or discuss on -hackers), and the right-ness of the contextual default was already proven to be wrong in this configuration of server and client, by the context of failing to connect to that defaulted database. Kind regards, Matthias van de Meent

Re: Custom tuplesorts for extensions

2022-07-06 Thread Matthias van de Meent
hat it is used for, and how to use it. > +struct Tuplesortstate > +{ > +TuplesortOps ops; This field needs a comment too. 0006: Needs a commit message, but otherwise seems fine. Kind regards, Matthias van de Meent

Re: [RFC] Improving multi-column filter cardinality estimation using MCVs and HyperLogLog

2022-07-06 Thread Matthias van de Meent
ned bitmap-based statistics would not allow this, because the index of a sampled row will likely shift between runs, even assuming that a row is shared in the sample. Summary: We shouldn't use HLL, (compressed) bitmaps will work fine for an initial implementation of combined sketch-based MCV

Re: doc: Move enum storage commentary to top of section

2022-07-06 Thread Matthias van de Meent
On Thu, 9 Jun 2022 at 18:12, David G. Johnston wrote: > > Per suggestion over on -docs: > > https://www.postgresql.org/message-id/bl0pr06mb4978f6c0b69f3f03aebed0fbb3...@bl0pr06mb4978.namprd06.prod.outlook.com I agree with moving the size of an enum into the top, but I don't think that the label l

Re: Make name optional in CREATE STATISTICS

2022-07-06 Thread Matthias van de Meent
pdated section that details the expected grammar for CREATE STATISTICS, as you seem to have overlooked that copy of grammar documentation. Apart from these two small issues, this passes tests and seems complete. Kind regards, Matthias van de Meent

Re: Make name optional in CREATE STATISTICS

2022-07-07 Thread Matthias van de Meent
tat types)] > Patch v4 attached Thanks for working on this! Kind regards, Matthias van de Meent

Re: Reducing Memory Consumption (aset and generation)

2022-07-11 Thread Matthias van de Meent
ving them to MEMORY_CONTEXT_CHECKING context. I like this patch over 001 due to allowing less corruption to occur in the memory context code. This allows for detecting some issues in 003, as opposed to none in 001. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-11 Thread Matthias van de Meent
reviously, > > "postgres=# SELECT pg_logical_emit_message(false, long, long) FROM > repeat(repeat(' ', 1024), 1024*1023) as l(long); > ERROR: too much WAL data" > > 3) without this v5 patch, the same test will cause the standby crash > like below, and the st

Re: Commitfest Update

2022-07-11 Thread Matthias van de Meent
On Fri, 8 Jul 2022, 23:41 Jacob Champion, wrote: > > On 3/31/22 07:37, Tom Lane wrote: >> Robert Haas writes: >>> On Thu, Mar 31, 2022 at 10:11 AM Tom Lane wrote: ... Would it be feasible or reasonable to drop reviewers if they've not commented in the thread in X amount of time? >

Re: MERGE and parsing with prepared statements

2022-07-15 Thread Matthias van de Meent
popped up earlier, and I think it's not something we've planned to fix in backbranches. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-15 Thread Matthias van de Meent
use that space for rmgr-specific flags, as opposed to stealing bits from xl_info? Kind regards, Matthias van de Meent

Re: Make name optional in CREATE STATISTICS

2022-07-20 Thread Matthias van de Meent
On Wed, 13 Jul 2022 at 08:07, Simon Riggs wrote: > > On Thu, 7 Jul 2022 at 11:58, Matthias van de Meent > wrote: > > A more correct version would be > > > > + *CREATE STATISTICS [ [IF NOT EXISTS] stats_name ] > > [(stat types)] > > There you

Re: StrategyAM for IndexAMs

2022-07-22 Thread Matthias van de Meent
27;t > clear). > > Comments welcome on this still WIP patch. I think this is a great step in the right direction, fixing one of the issues with core index AMs, issues I also complained about earlier [0]. Thanks, Matthias van de Meent [0] https://www.postgresql.org/message-id/CAEze2Wg8QhpOnHoqPNB-AaexGX4Zaij%3D4TT0kaMhF_6T5FXxmQ%40mail.gmail.com

Re: Pluggable toaster

2022-07-22 Thread Matthias van de Meent
ch. --- > Currently I'm working on an update to the default Toaster (some internal > optimizations, not affecting functionality) > and readme files explaining Pluggable TOAST. That would be greatly appreciated - 0006 does not cover why we need vtable, nor how it's expect

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-25 Thread Matthias van de Meent
On Wed, 13 Jul 2022 at 07:54, Michael Paquier wrote: > > On Mon, Jul 11, 2022 at 02:26:46PM +0200, Matthias van de Meent wrote: > > Thanks for reviewing. > > I think that v6 is over-engineered because there should be no need to > add a check in xlogreader.c as long as the ori

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-26 Thread Matthias van de Meent
ta, so that we check against integer overflows in the registerdata code instead of only an assert in XLogRecordAssemble where it might be too late. - Kept the inline static elog-ing function (as per Andres' suggestion on 2022-03-14; this decreases binary sizes) - Dropped any changes in xlogreader.h/c Kind regards, Matthias van de Meent v8-0001-Add-protections-in-xlog-record-APIs-against-large.patch Description: Binary data

Re: [PATCH] Compression dictionaries for JSONB

2022-07-27 Thread Matthias van de Meent
his dictionary compression can instead be added using the proposed extensible toasting mechanism at relatively low overhead. Kind regards, Matthias van de Meent

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-07-27 Thread Matthias van de Meent
On Wed, 27 Jul 2022 at 11:09, Michael Paquier wrote: > > On Tue, Jul 26, 2022 at 06:58:02PM +0200, Matthias van de Meent wrote: > > - Retained the check in XLogRegisterData, so that we check against > > integer overflows in the registerdata code instead of only an assert > &g

Re: Maximize page freezing

2022-07-28 Thread Matthias van de Meent
; So +1 for this idea. > > Updated patch attached. Great idea, yet this patch seems to only freeze those tuples that are located after the first to-be-frozen tuple. It should probably re-visit earlier live tuples to potentially freeze those as well. Kind regards, Matthias van de Meent

Re: ci: update to freebsd 13.1 / remove minor versions from image names

2022-07-28 Thread Matthias van de Meent
esting() was not seen. # Looks like your test exited with 29 just after 1. t/033_replay_tsp_drops.pl .... Dubious, test returned 29 (wstat 7424, 0x1d00) All 1 subtests passed --- Kind regards, Matthias van de Meent [0] https://cirrus-ci.com/task/5147001137397760?logs=test_world#L2631-

Re: ci: update to freebsd 13.1 / remove minor versions from image names

2022-07-28 Thread Matthias van de Meent
On Thu, 28 Jul 2022 at 19:31, Andres Freund wrote: > > Hi, > > On July 28, 2022 7:29:43 PM GMT+02:00, Matthias van de Meent > wrote: > >On Thu, 28 Jul 2022 at 11:57, Andres Freund wrote: > >> > >> Hi, > >> > >> Freebsd 13.0, so far used

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Matthias van de Meent
d. See bottom for examples (all 3 builds for different patches). Kind regards, Matthias van de Meent. [1] https://cirrus-ci.com/task/4960990331666432?logs=test_world#L2631-L2662 [2] https://cirrus-ci.com/task/5012678384025600?logs=test_world#L2631-L2662 [3] https://cirrus-ci.com/task/5147001137397760?logs=test_world#L2631-L2662

Re: Maximize page freezing

2022-07-29 Thread Matthias van de Meent
On Fri, 29 Jul 2022 at 16:38, Simon Riggs wrote: > > On Thu, 28 Jul 2022 at 14:55, Matthias van de Meent > wrote: > > Great idea, yet this patch seems to only freeze those tuples that are > > located after the first to-be-frozen tuple. It should probably > > re-v

Re: Clarifying Commitfest policies

2022-08-03 Thread Matthias van de Meent
f the main PostgreSQL repository receives a decent amount of pull request traffic. When a project has a CONTRIBUTING.md -file at the top level people writing the pull request message will be pointed to those contributing guidelines. This could Thank you for raising this to a topical thread. Kind regards, Matthias van de Meent [0] https://www.postgresql.org/developer/

Re: postgres_fdw: batch inserts vs. before row triggers

2022-08-03 Thread Matthias van de Meent
ered on that statement, regardless of the FOR EACH ROW specifier. The sections for DELETE are 15.7 (6) and 15.7 (7); and for UPDATE 15.13(7) and 15.13(9) respectively. I don't know about the semantics of triggers in the latest SQL standard versions, but based on that sample it seems like we're non-compliant on BEFORE trigger behaviour, and it doesn't seem like it's documented in the trigger documentation. I seem to recall a mail on this topic (changes in trigger execution order with respect to the DML it is triggered by in the newest SQL spec) but I can't seem to find that thread. Kind regards, Matthias van de Meent

Re: postgres_fdw: batch inserts vs. before row triggers

2022-08-03 Thread Matthias van de Meent
On Wed, 3 Aug 2022 at 23:57, Tom Lane wrote: > > Matthias van de Meent writes: > > I don't have a current version of the SQL spec, but one preliminary > > version of SQL:2012 I retrieved via the wiki details that all BEFORE > > triggers on INSERT/UPDATE/DELETE state

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
t the count is incorrect. if (scanned_tuples > old_rel_tuples) return some_weighted_scanned_tuples; Kind regards, Matthias van de Meent

Re: Clarifying Commitfest policies

2022-08-08 Thread Matthias van de Meent
On Thu, 4 Aug 2022 at 20:38, Jacob Champion wrote: > > On Wed, Aug 3, 2022 at 2:05 PM Matthias van de Meent > wrote: > > On Wed, 3 Aug 2022 at 20:04, Jacob Champion wrote: > > > Is that enough, or should we do more? > > > > "The CF Checklist" see

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 17:26, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 8:14 AM Matthias van de Meent > wrote: > > I do not have intimate knowledge of this code, but shouldn't we also > > add some sefety guarantees like the following in these blocks?

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-08 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 17:49, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 8:33 AM Matthias van de Meent > wrote: > > For example, if currently the measured 2% of the pages contains more > > than 100% of the previous count of tuples, or with your patch the last > &

Re: Remaining case where reltuples can become distorted across multiple VACUUM operations

2022-08-11 Thread Matthias van de Meent
On Mon, 8 Aug 2022 at 18:48, Peter Geoghegan wrote: > > On Mon, Aug 8, 2022 at 9:17 AM Matthias van de Meent > wrote: > > Because if a subset of the pages of a relation contains more tuples > > than your current total expected tuples in the table, you should > &g

Re: 64 bit TID?

2021-09-13 Thread Matthias van de Meent
beit some may be used for rolled-back and thus dead tuples). > Are there any plans in the works to widen the TID? This was recently discussed here [0] as well, but to the best of my knowledge no material proposal to update the APIs has been suggested as of yet. Kind regards, Matthias van de

Re: CLUSTER on partitioned index

2021-09-23 Thread Matthias van de Meent
LE clstrpart; I believe that this cleanup should not be fully removed, but moved to before '-- Test CLUSTER with external tuplesorting', as the table is not used after that line. Kind regards, Matthias van de Meent

Re: Map WAL segment files on PMEM as WAL buffers

2021-10-07 Thread Matthias van de Meent
On Wed, 30 Jun 2021 at 06:53, Takashi Menjo wrote: > > Rebased. Thanks for these patches! I recently took a dive into the WAL subsystem, and got to this patchset through the previous ones linked from [0]. This patchset seems straightforward to understand, so thanks! I've looked over the patches

Re: RFC: compression dictionaries for JSONB

2021-10-08 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 11:47, Aleksander Alekseev wrote: > This is a follow-up thread to `Add ZSON extension to /contrib/` [1]. > The ZSON extension introduces a new type called ZSON, which is 100% > compatible with JSONB but uses a shared dictionary of strings most > frequently used by given JSONB

Re: RFC: compression dictionaries for JSONB

2021-10-08 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 17:19, Alvaro Herrera wrote: > > On 2021-Oct-08, Matthias van de Meent wrote: > > > On Fri, 8 Oct 2021 at 11:47, Aleksander Alekseev > > wrote: > > > > In order to do this, the SQL syntax should be modified. The proposed > > > synta

Re: RFC: compression dictionaries for JSONB

2021-10-09 Thread Matthias van de Meent
On Fri, 8 Oct 2021 at 21:21, Alvaro Herrera wrote: > > On 2021-Oct-08, Matthias van de Meent wrote: > > > That's a good point, but if we're extending this syntax to allow the > > ability of including other types, then I'd instead extend the syntax > &

Re: ATTACH PARTITION locking documentation for DEFAULT partitions

2021-07-27 Thread Matthias van de Meent
On Tue, 27 Jul 2021 at 08:02, David Rowley wrote:\> > On Tue, 13 Jul 2021 at 02:30, Matthias van de Meent > wrote: > > The algoritm as described in your patch implies that this recursive > > locking is conditional on _only_ the check-constraints of the topmost > > p

Re: Lowering the ever-growing heap->pd_lower

2021-08-03 Thread Matthias van de Meent
On Tue, 3 Aug 2021 at 08:57, Simon Riggs wrote: > > On Tue, 18 May 2021 at 20:33, Peter Geoghegan wrote: > > > > On Tue, May 18, 2021 at 12:29 PM Matthias van de Meent > > wrote: > > > PFA the updated version of this patch. Apart from adding

Re: Lowering the ever-growing heap->pd_lower

2021-08-03 Thread Matthias van de Meent
On Tue, 3 Aug 2021 at 20:37, Simon Riggs wrote: > > On Tue, 3 Aug 2021 at 17:15, Matthias van de Meent > wrote: > > > and further future optimizations might include > > > > - Full-page WAL logging of empty pages produced in the checkpointer > > could potential

Re: Lowering the ever-growing heap->pd_lower

2021-08-04 Thread Matthias van de Meent
ad for more than just this one vacuum run, but the chances on fully clearing the page are indeed very, very slim. Kind regards, Matthias van de Meent

Re: Lowering the ever-growing heap->pd_lower

2021-08-04 Thread Matthias van de Meent
calculating free space', then I would argue 'yes, but isn't it better then to also actually fully mark that space as unused'. All in all, I'd just rather remove the distinction between once-used pages and fresh pages completely by truncating the LP-array to 0 than to leave this bloating behaviour in the system. Kind regards, Matthias van de Meent.

Re: NAMEDATALEN increase because of non-latin languages

2021-08-19 Thread Matthias van de Meent
r attlen >= 0. This keeps the whole structure in 8 bytes, whilst also being useable in both tuple forming and deforming. I hope this can is useful, otherwise sorry for the noise. Kind regards, Matthias van de Meent From 3dd3f470ab78b8811015c9f374d0bdc44ffef531 Mon Sep 17 00:00:00 2001 From: Mat

Re: NAMEDATALEN increase because of non-latin languages

2021-08-20 Thread Matthias van de Meent
On Thu, 19 Aug 2021 at 14:58, Andres Freund wrote: > > Hi, > > On 2021-08-19 14:47:42 +0200, Matthias van de Meent wrote: > > I tried to implement this 'compact attribute access descriptor' a few > > months ago in my effort to improve btree index performance. &g

Re: PostgreSQL <-> Babelfish integration

2021-08-25 Thread Matthias van de Meent
es [0][1][2] also do not appear to have seen an update. The last mention of babelfish in a thread here on -hackers also only seem to date back to late March. Kind regards, Matthias van de Meent [0] https://babelfish-for-postgresql.github.io/babelfish-for-postgresql/ [1] https://aws.amazon.com/

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-06 Thread Matthias van de Meent
On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > > On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent > wrote: >> >> On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wrote: >>> >>> Why not use it too, if not binary_upgrade? >> >> Because in th

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Matthias van de Meent
On Fri, 7 Jun 2024 at 10:28, Dilip Kumar wrote: > > On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent > wrote: >> >> On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: >>> >>> On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent >>> wrote:

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-11 Thread Matthias van de Meent
On Tue, 11 Jun 2024 at 04:01, Nathan Bossart wrote: > > On Fri, Jun 07, 2024 at 11:10:25AM +0200, Matthias van de Meent wrote: > > My primary concern isn't the IO, but the O(shared_buffers) that we > > have to go through during a checkpoint. As I mentioned upthread, it is

Re: SQL:2011 application time

2024-06-12 Thread Matthias van de Meent
On Wed, 5 Jun 2024 at 22:57, Paul Jungwirth wrote: > > On Thu, May 9, 2024 at 5:44 PM Matthias van de Meent > wrote: > > Additionally, because I can't create my own non-constraint-backing > > unique GIST indexes, I can't pre-create my unique constraints >

Re: RFC: adding pytest as a supported test framework

2024-06-17 Thread Matthias van de Meent
may have given off-hand mentions that [other language] would've helped in certain cases, sure, but I'd hardly call that clamoring. Kind regards, Matthias van de Meent

Re: [PATCH] Improve error message when trying to lock virtual tuple.

2024-06-17 Thread Matthias van de Meent
, why does the assertion in heapam_tuple_lock that checks for a BufferHeapTupleTableSlot not fire before this `block 0` error? If the error is not in the heapam code, could you show an example of the code that breaks with that error code? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: [PATCH] Improve error message when trying to lock virtual tuple.

2024-06-18 Thread Matthias van de Meent
On Tue, 18 Jun 2024 at 09:32, Sven Klemm wrote: > > On Mon, Jun 17, 2024 at 10:25 PM Matthias van de Meent > wrote: > > > I think you're solving the wrong problem here, as I can't think of a > > place where both virtual tuple slots and tuple locking are allowed

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Matthias van de Meent
t; > connects to each database (including template0) and updates a bunch of > > pg_class rows, and we probably want those on disk before we start copying > > the files to create all the user's databases. Good catch, I hadn't thought of that. > Here is an updated patch.

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Matthias van de Meent
e future that'll change if pg_upgrade and pg_dump are made smart enough to resume their restore progress across forceful disconnects, but for now this patch seems like a nice performance improvement. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Table AM Interface Enhancements

2024-06-21 Thread Matthias van de Meent
Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Improve EXPLAIN output for multicolumn B-Tree Index

2024-06-24 Thread Matthias van de Meent
ve slown down quite significantly, so if the testset is the same then this doesn't bode well for your patchset. Kind regards, Matthias van de Meent

Re: Improve EXPLAIN output for multicolumn B-Tree Index

2024-06-24 Thread Matthias van de Meent
On Mon, 24 Jun 2024 at 11:58, Jelte Fennema-Nio wrote: > > +1 for the idea. > > On Mon, 24 Jun 2024 at 11:11, Matthias van de Meent > wrote: > > I think this is too easy to confuse with the pre-existing 'Filter' > > condition, which you'll find on indexe

Re: Improve EXPLAIN output for multicolumn B-Tree Index

2024-06-24 Thread Matthias van de Meent
On Mon, 24 Jun 2024 at 14:42, Jelte Fennema-Nio wrote: > > On Mon, 24 Jun 2024 at 13:02, Matthias van de Meent > wrote: > > It does not really behave similar: index scan keys (such as the > > id3=101 scankey) don't require visibility checks in the btree code, >

Re: add a new explain option including_query for include query string inside the json plan output

2024-06-25 Thread Matthias van de Meent
t;}, + >{+ > "Plan": { + If we were to add the query to the explain output, I think it should be a top-level key in the same JSON object that holds the "Plan", Triggers, and "Execution Time" keys. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Adminpack removal

2024-06-27 Thread Matthias van de Meent
t glance doesn't currently seem to depend on adminpack: it is not included in the control file as dependency, and has not been included as a dependency since the creation of that file. Where else would you expect us to search for dependencies? Kind regards, Matthias van de Meent

Re: Improve EXPLAIN output for multicolumn B-Tree Index

2024-06-28 Thread Matthias van de Meent
hrough the planner, adding overhead everywhere which is only used for btree- and btree-compatible indexes. I think the better choice would be adding an IndexAmRoutine->amexplain support function, which would get called in e.g. explain.c's ExplainIndexScanDetails to populate a new "Index Scan Details" (name to be bikeshed) subsection of explain plans. This would certainly be possible, as the essentials for outputting things to EXPLAIN are readily available in the explain.h header. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Linux likely() unlikely() for PostgreSQL

2024-06-30 Thread Matthias van de Meent
will likely correctly predict this branch correctly after a small amount of time (as hook null-ness is generally approximately constant for the duration of a backend lifetime); while if this code is cold, the benefit is not worth the additional binary size overhead of the out-of-lined code section. Kind regards, Matthias van de Meent Neon (https://neon.tech/)

Re: Make tuple deformation faster

2024-07-01 Thread Matthias van de Meent
ble series of compare and jump. +1, that's something I'd missed in my patches, and is probably the largest contributor to the speedup. > I'll stick this in the July CF. It would be good to get some feedback > on the idea and feedback on whether more work on this is worthwhile.

Re: Make tuple deformation faster

2024-07-01 Thread Matthias van de Meent
On Mon, 1 Jul 2024 at 12:49, David Rowley wrote: > > On Mon, 1 Jul 2024 at 22:07, Matthias van de Meent > wrote: > > One thing I'm slightly concerned about is that this allocates another > > 8 bytes for each attribute in the tuple descriptor. While that's not a

Re: Use generation memory context for tuplestore.c

2024-07-02 Thread Matthias van de Meent
On Fri, 31 May 2024 at 05:26, David Rowley wrote: > > On Sat, 4 May 2024 at 03:51, Matthias van de Meent > wrote: > > > > On Fri, 3 May 2024 at 15:55, David Rowley wrote: > > > master @ 8f0a97dff > > > Storage: Memory Maximum Storage: 16577kB > > &g

Re: Incorrect Assert in BufFileSize()?

2024-07-03 Thread Matthias van de Meent
objections or additional feedback, I'll push this patch soon. Finding this thread after reviewing [0], this does explain why the Assert was changed in that patch. LGTM. Kind regards, Matthias van de Meent Neon (https://neon.tech) [0] https://www.postgresql.org/message-id/flat/CAApHDvp5Py9g4Rjq7_inL3-MCK1Co2CRt_YWFwTU2zfQix0p4A%40mail.gmail.com

Re: Parallel CREATE INDEX for GIN indexes

2024-07-03 Thread Matthias van de Meent
typedef struct GinTuple I think this needs some more care: currently, each GinTuple is at least 36 bytes in size on 64-bit systems. By using int instead of Size (no normal indexable tuple can be larger than MaxAllocSize), and packing the fields better we can shave off 10 bytes; or 12 bytes if GinTuple.keylen is further adjusted to (u)int16: a key needs to fit on a page, so we can probably safely assume that the key size fits in (u)int16. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: First draft of PG 17 release notes

2024-07-05 Thread Matthias van de Meent
C (Tristan Partin)", or "Allow psql connections to be canceled with control-C while psql is still connecting (Tristan Partin)". Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Parallel CREATE INDEX for GIN indexes

2024-07-08 Thread Matthias van de Meent
On Sun, 7 Jul 2024, 18:11 Tomas Vondra, wrote: > > On 7/3/24 20:36, Matthias van de Meent wrote: >> On Mon, 24 Jun 2024 at 02:58, Tomas Vondra >> wrote: >>> So I've switched this to use the regular data-type comparisons, with >>> SortSupport etc. There&#

Re: Parallel CREATE INDEX for GIN indexes

2024-07-08 Thread Matthias van de Meent
On Sun, 7 Jul 2024, 18:26 Tomas Vondra, wrote: > > On 7/5/24 21:45, Matthias van de Meent wrote: >> On Wed, 3 Jul 2024 at 20:36, Matthias van de Meent >> wrote: >>> >>> On Mon, 24 Jun 2024 at 02:58, Tomas Vondra >>> wrote: >>>> So I&

Re: Thoughts on NBASE=100000000

2024-07-08 Thread Matthias van de Meent
his is worth implementing for general usage, it could be worth exploring for the larger numeric values, where the relative overhead of the larger representation is lower. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Parallel CREATE INDEX for GIN indexes

2024-07-08 Thread Matthias van de Meent
On Mon, 8 Jul 2024, 13:38 Tomas Vondra, wrote: > > On 7/8/24 11:45, Matthias van de Meent wrote: > > As to the GIN TS code itself; yes it's more complicated, mainly > > because it uses several optimizations to reduce unnecessary > > allocations and (de)serialization

Re: PATCH: Using BRIN indexes for sorted output

2023-07-10 Thread Matthias van de Meent
On Fri, 7 Jul 2023 at 20:26, Tomas Vondra wrote: > > Hi, > > I finally had time to look at this patch again. There's a bit of bitrot, > so here's a rebased version (no other changes). Thanks! > On 2/27/23 16:40, Matthias van de Meent wrote: >

Re: PATCH: Using BRIN indexes for sorted output

2023-07-10 Thread Matthias van de Meent
On Mon, 10 Jul 2023 at 13:43, Tomas Vondra wrote: > On 7/10/23 12:22, Matthias van de Meent wrote: >> On Fri, 7 Jul 2023 at 20:26, Tomas Vondra >> wrote: >>> However, it's not quite clear to me is what you mean by the weight- and >>>

Re: PATCH: Using BRIN indexes for sorted output

2023-07-10 Thread Matthias van de Meent
On Mon, 10 Jul 2023 at 17:09, Tomas Vondra wrote: > On 7/10/23 14:38, Matthias van de Meent wrote: >> Kind of. For single-dimensional opclasses (minmax, minmax_multi) we >> only need to extract the normal min/max values for ASC/DESC sorts, >> which are readily available in

Re: PATCH: Using BRIN indexes for sorted output

2023-07-11 Thread Matthias van de Meent
On Mon, 10 Jul 2023 at 22:04, Tomas Vondra wrote: > On 7/10/23 18:18, Matthias van de Meent wrote: >> On Mon, 10 Jul 2023 at 17:09, Tomas Vondra >> wrote: >>> On 7/10/23 14:38, Matthias van de Meent wrote: >>>>> I haven't really thought about geometr

Re: [PATCH]Add a tip to the check mode

2023-07-11 Thread Matthias van de Meent
ch doesn't return) - it shouldn't hit the abort() clause. If it did hit the abort() clause, that'd probably be a problem on its own, right? -- Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: [Question] Can someone provide some links related to the MemoryContext?

2023-07-11 Thread Matthias van de Meent
in advance! You should take a look at the README in the mmgr directory; at src/backend/utils/mmgr/README. I think that this README provides a lot of the requested information. -- Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Parallel CREATE INDEX for BRIN indexes

2023-07-11 Thread Matthias van de Meent
On Thu, 6 Jul 2023 at 16:13, Tomas Vondra wrote: > > On 7/5/23 16:33, Matthias van de Meent wrote: > > ... > > > >> Maybe. I wasn't that familiar with what parallel tuplesort can and can't > >> do, and the little I knew I managed to forget since I wr

Potential memory leak in contrib/intarray's g_intbig_compress

2023-07-13 Thread Matthias van de Meent
s the issue. It looks like this bug has existed since the code was first committed, so backpatching would go back to 11 if this is an active issue. Kind regards, Matthias van de Meent 0001-Fix-memory-leak-in-g_intbig_compress.patch Description: Binary data

Re: Potential memory leak in contrib/intarray's g_intbig_compress

2023-07-13 Thread Matthias van de Meent
On Thu, 13 Jul 2023 at 17:20, Tom Lane wrote: > > Matthias van de Meent writes: > > My collegue Konstantin (cc-ed) noticed that the GiST code of intarray > > may leak memory in certain index operations: > > Can you demonstrate an actual problem here, that is a query-lif

Re: PATCH: Using BRIN indexes for sorted output

2023-07-14 Thread Matthias van de Meent
On Fri, 14 Jul 2023 at 16:21, Tomas Vondra wrote: > > > > > On 7/11/23 13:20, Matthias van de Meent wrote: >> On Mon, 10 Jul 2023 at 22:04, Tomas Vondra >> wrote: >>> Approximation in what sense? My understanding was we'd get a range of >>> di

Re: Parallel CREATE INDEX for BRIN indexes

2023-07-14 Thread Matthias van de Meent
On Fri, 14 Jul 2023 at 15:57, Tomas Vondra wrote: > > On 7/11/23 23:11, Matthias van de Meent wrote: >> On Thu, 6 Jul 2023 at 16:13, Tomas Vondra >> wrote: >>> >>> One thing I was wondering about is whether it might be better to allow >>> the worker

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-26 Thread Matthias van de Meent
t of memory potentially needed to execute the statement. So, this begs the question: does this patch have the same issue? Does it fail with OOM, does it gracefully fall back to the old behaviour when the clauses are too complex to linearize/compose/fold into the btree ordering clauses, or are scan keys dynamically constructed using just-in-time- or generator patterns? Kind regards, Matthias van de Meent Neon (https://neon.tech/)

<    1   2   3   4   5   6   7   8   >