Re: incremental-checkopints

2023-07-26 Thread Matthias van de Meent
age protection at first update after checkpoint" we'll probably always have higher-than-average FPI load just after a new checkpoint. Kind regards, Matthias van de Meent Neon (https://neon.tech/)

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-26 Thread Matthias van de Meent
On Wed, 26 Jul 2023 at 15:42, Peter Geoghegan wrote: > > On Wed, Jul 26, 2023 at 5:29 AM Matthias van de Meent > wrote: > > Considering that it caches/reuses the page across SAOP operations, can > > (or does) this also improve performance for index scans on the outer >

Re: incremental-checkopints

2023-07-26 Thread Matthias van de Meent
On Wed, 26 Jul 2023 at 20:58, Tomas Vondra wrote: > > > > On 7/26/23 15:16, Matthias van de Meent wrote: > > On Wed, 26 Jul 2023 at 14:41, Alvaro Herrera > > wrote: > >> > >> Hello > >> > >> On 2023-Jul-26, Thomas wen wrote: >

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Matthias van de Meent
On Wed, 26 Jul 2023 at 15:42, Peter Geoghegan wrote: > > On Wed, Jul 26, 2023 at 5:29 AM Matthias van de Meent > > I'm not sure I understand. MDAM seems to work on an index level to > > return full ranges of values, while "skip scan" seems to try to allow >

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Matthias van de Meent
On Thu, 27 Jul 2023 at 06:14, Peter Geoghegan wrote: > > On Wed, Jul 26, 2023 at 12:07 PM Matthias van de Meent > wrote: > > We could cache the last accessed leaf page across amrescan operations > > to reduce the number of index traversals needed when the join key of > &

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-28 Thread Matthias van de Meent
On Thu, 27 Jul 2023 at 16:01, Peter Geoghegan wrote: > > On Thu, Jul 27, 2023 at 7:59 AM Matthias van de Meent > wrote: > > > Basically, the patch that added that feature had to revise the index > > > AM API, in order to support a mode of operation where scans return

Re: Let's make PostgreSQL multi-threaded

2023-07-28 Thread Matthias van de Meent
his can be significantly more work. Kind regards, Matthias van de Meent Neon (https://neon.tech/)

Re: New PostgreSQL Contributors

2023-07-31 Thread Matthias van de Meent
ith > Vigneshwaran C > > New PostgreSQL Major Contributors: > > Julien Rouhaud > Stacey Haysler > Steve Singer > > Thank you all for contributing to PostgreSQL to make it such a great project! +1, thank you all, and congratulations! Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Extract numeric filed in JSONB more effectively

2023-08-01 Thread Matthias van de Meent
gain. Why wouldn't you use cast(a->'a' as numeric), or ((a->'a')::numeric)? We have a cast from jsonb to numeric (jsonb_numeric in jsonb.c) that does not require this additional (de)serialization through text. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Potential memory leak in contrib/intarray's g_intbig_compress

2023-08-02 Thread Matthias van de Meent
On Fri, 14 Jul 2023 at 07:57, Michael Paquier wrote: > > On Thu, Jul 13, 2023 at 06:28:39PM +0200, Matthias van de Meent wrote: > > There are similar pfree calls in the _int_gist.c file's g_int_compress > > function, which made me think we do need to clean up after us

Re: Adding a pg_servername() function

2023-08-03 Thread Matthias van de Meent
ocal user's hostname can be very different from the hostname used in the url that connects to that PostgreSQL instance. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Extract numeric filed in JSONB more effectively

2023-08-03 Thread Matthias van de Meent
On Wed, 2 Aug 2023 at 03:05, Andy Fan wrote: > > Hi Matthias: > > On Wed, Aug 2, 2023 at 7:33 AM Andy Fan wrote: >> >> >> >> On Tue, Aug 1, 2023 at 7:03 PM Matthias van de Meent >> wrote: >>> >>> On Tue, 1 Aug 2023 at 06:39, Andy Fan

Re: [question] multil-column range partition prune

2023-08-10 Thread Matthias van de Meent
< 6, B: any value - A = 6, B < 6 As you can see, each partition contains a set of rows that may have any value for B, and thus these partitions cannot be pruned based on the predicate. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Disabling Heap-Only Tuples

2023-08-24 Thread Matthias van de Meent
On Fri, 7 Jul 2023 at 12:18, Tomas Vondra wrote: > > On 7/7/23 11:55, Matthias van de Meent wrote: >> On Fri, 7 Jul 2023 at 06:53, Dilip Kumar wrote: >>> >>> >>> So IIUC, this parameter we can control that instead of putting the new >>> version

Re: Disabling Heap-Only Tuples

2023-08-24 Thread Matthias van de Meent
, unique (id, num_updates) ) would be assumed to remain bloated if I'd set a parameter named something_hot_something, as all updates would be non-hot and thus should not be influenced by the GUC/parameter. How about 'local_update_limit'? Kind regards, Matthias van de Meent

Re: broken master regress tests

2023-08-24 Thread Matthias van de Meent
ry)' > It looks like the error message matcher doesn't account for the localized version of "No such file or directory", might that be the issue? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Disabling Heap-Only Tuples

2023-08-28 Thread Matthias van de Meent
On Wed, 19 Jul 2023 at 14:58, Laurenz Albe wrote: > > On Thu, 2023-07-06 at 22:18 +0200, Matthias van de Meent wrote: > > On Wed, 5 Jul 2023 at 19:55, Thom Brown wrote: > > > > > > On Wed, 5 Jul 2023 at 18:05, Matthias van de Meent > > > wrote: > >

Re: Disabling Heap-Only Tuples

2023-08-28 Thread Matthias van de Meent
On Mon, 28 Aug 2023 at 17:14, Robert Haas wrote: > > On Mon, Aug 28, 2023 at 10:52 AM Matthias van de Meent > wrote: > > In this new patch, I've updated a few comments to get mostly within > > line length limits; the name of the storage parameter is now > &

Cleanup: remove unused fields from nodes

2024-04-22 Thread Matthias van de Meent
fields, as nobody seems to be using them. /cc Tom Lane and Etsuro Fujita: 2 and 4 were introduced with your commit afb9249d in 2015. /cc Amit Kapila: 0003 was introduced with your spearheaded commit 6185c973 this year. Kind regards, Matthias van de Meent 0001 removes two old fields that are not in

Re: Cleanup: remove unused fields from nodes

2024-04-22 Thread Matthias van de Meent
On Mon, 22 Apr 2024 at 17:41, Tom Lane wrote: > > Matthias van de Meent writes: > > 0002/0004 remove fields in ExecRowMark which were added for FDWs to > > use, but there are no FDWs which use this: I could only find two FDWs > > who implement RefetchForeignRow, one bei

Re: Statistics Import and Export

2024-04-23 Thread Matthias van de Meent
'll happen when the tables grow larger with the current stats. As for other planner inputs: table size is relatively easy to shim with sparse files; cumulative statistics can be copied from a donor replica if needed, and btree indexes only really really need to contain their highest and lowest values (and need their height set correctly). Kind regards, Matthias van de Meent

Re: Cleanup: remove unused fields from nodes

2024-04-24 Thread Matthias van de Meent
On Wed, 24 Apr 2024 at 09:28, Michael Paquier wrote: > > On Tue, Apr 23, 2024 at 11:03:40PM -0400, Tom Lane wrote: > > Hah. Seems like the comment for isall needs to explain that it > > exists for this purpose, so we don't make this mistake again. > > How about something like the attached? LGTM.

Re: Statistics Import and Export

2024-04-24 Thread Matthias van de Meent
On Wed, 24 Apr 2024 at 21:31, Bruce Momjian wrote: > > On Tue, Apr 23, 2024 at 06:33:48PM +0200, Matthias van de Meent wrote: > > I've heard of use cases where dumping stats without data would help > > with production database planner debugging on a non-prod system. >

Re: Extend ALTER DEFAULT PRIVILEGES for large objects

2024-04-26 Thread Matthias van de Meent
ues to be stored in table's attributes, we'll have to figure out how we're going to transfer those larger values to (and from) clients. For large objects, this is much less of an issue because the IO operations are already chunked by design, but this may not work well for types

Re: Possible to get LIMIT in an index access method?

2024-04-29 Thread Matthias van de Meent
> the index. Sorry, but I don't think we can know in advance how many tuples are going to be extracted from an index scan. Kind regards, Matthias van de Meent.

Re: Parallel CREATE INDEX for GIN indexes

2024-05-02 Thread Matthias van de Meent
the tuplesort, we just compress the TIDs. See note on 0002: Could we do this in the tuplesort writeback, rather than by moving the data around multiple times? [...] > So 0007 does something similar - it tracks if the TID value goes > backward in the callback, and if it does it dumps the state into the > tuplesort before processing the first tuple from the beginning of the > table. Which means we end up with two separate "narrow" TID list, not > one very wide one. See note above: We may still need a merge phase, just to make sure we handle all TAM parallel scans correctly, even if that merge join phase wouldn't get hit in vanilla PostgreSQL. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Why is FOR ORDER BY function getting called when the index is handling ordering?

2024-05-02 Thread Matthias van de Meent
ut an expression over that column) the executor will execute that projection. This happens regardless of it's use in downstream nodes due to planner or executor limitations. See also Heikki's thread over at [0], and a comment of me about the same issue over at pgvector's issue board

Re: Use generation memory context for tuplestore.c

2024-05-03 Thread Matthias van de Meent
nd use a bump context if we require rewind capabilities (i.e. where _trim is never effectively executed)? > master @ 8f0a97dff > Storage: Memory Maximum Storage: 16577kB > > patched: > Storage: Memory Maximum Storage: 8577kB Those are some impressive numbers. Kind regards, Matth

Expand applicability of aggregate's sortop optimization

2024-05-08 Thread Matthias van de Meent
exed paths that append the SORT BY paths to the aggregate's sort operator, but that'd significantly increase complexity here. Kind regards, Matthias van de Meent Neon (https://neon.tech) From 215600d12164f214aae8f0de94b16373bd202d69 Mon Sep 17 00:00:00 2001 From: Matthias van de Meent

Re: Parallel CREATE INDEX for GIN indexes

2024-05-09 Thread Matthias van de Meent
uired for the patch, Agreed. > and I don't think I'll have time to work on this > anytime soon. The patch is not extremely complex, but it's not trivial > either. But if someone wants to take a stab at extending tuplesort to > allow this, I won't object ... Same here:

Re: SQL:2011 application time

2024-05-09 Thread Matthias van de Meent
#x27;t support a CONCURRENTLY modifier, nor an INVALID modifier. This means temporal unique constraints have much less administrative wiggle room than normal unique constraints, and I think that's not great. Kind regards, Matthias van de Meent.

Re: Use generation memory context for tuplestore.c

2024-05-10 Thread Matthias van de Meent
On Sat, 4 May 2024 at 04:02, David Rowley wrote: > > On Sat, 4 May 2024 at 03:51, Matthias van de Meent > wrote: > > Was a bump context considered? If so, why didn't it make the cut? > > If tuplestore_trim is the only reason why the type of context in patch > >

Re: SQL:2011 application time

2024-05-12 Thread Matthias van de Meent
On Sun, 12 May 2024 at 05:26, Paul Jungwirth wrote: > On 5/9/24 17:44, Matthias van de Meent wrote: > > I haven't really been following this thread, but after playing around > > a bit with the feature I feel there are new gaps in error messages. I > > also t

Re: Comments about TLS (no SSLRequest) and ALPN

2024-05-12 Thread Matthias van de Meent
want new options, you're free to stay at older versions. > Allow the user to specify ALPN > > I don't think this is particularly controversial or nuanced, so I don't have > much to say here - most TLS tools allow the user to specify ALPN for the same > reason they allow specifying the port number - either for privacy, > security-by-obscurity, or navigating some form of application or user routing. As I mentioned above, I don't see any value, and only demerits, in the psql client reporting support for anything other than the protocol that PostgreSQL supports, i.e. the alpn identifier "postgresql". Kind regards, Matthias van de Meent

Re: Allowing additional commas between columns, and at the end of the SELECT clause

2024-05-13 Thread Matthias van de Meent
;` in pg_stat_statements? Why, or why not? Overall, I don't think unlimited commas is a good feature. A trailing comma in the select list would be less problematic, but I'd still want to follow the standard first and foremost. Kind regards, Matthias van de Meent Neon (https://neon.tech)

WAL_LOG CREATE DATABASE strategy broken for non-standard page layouts

2024-05-13 Thread Matthias van de Meent
a system crash. I've not looked for any such AMs and am unaware of any that would have this issue, but it's better to fix this. PFA a patch that fixes this issue, by assuming that all pages in the source database utilize a non-standard page layout. Kind regards, Matthias van de Meent

Re: WAL_LOG CREATE DATABASE strategy broken for non-standard page layouts

2024-05-13 Thread Matthias van de Meent
On Mon, 13 May 2024 at 16:13, Tom Lane wrote: > > Matthias van de Meent writes: > > PFA a patch that fixes this issue, by assuming that all pages in the > > source database utilize a non-standard page layout. > > Surely that cure is worse than the disease? I don'

Re: commitfest.postgresql.org is no longer fit for purpose

2024-05-20 Thread Matthias van de Meent
e it. To start addressing this limitation, I've just created a wiki page about the CFA [0], with a handbook section. Feel free to extend or update the information as appropriate; I've only added that information the best of my knowledge, so it may contain wrong, incomplete and/or inaccurate information. Kind regards, Matthias van de Meent [0] https://wiki.postgresql.org/wiki/CommitFest_App

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-05 Thread Matthias van de Meent
shared buffers. For initdb the database size is known in advance and shared_buffers is approximately empty, but the same is not (always) true when we're doing binary upgrades. Kind regards, Matthias van de Meent

Re: First draft of the PG 15 release notes

2022-08-27 Thread Matthias van de Meent
aut) > [...] > to match the operating system collation version. DETAILS? > > Kind regards, Matthias van de Meent

Re: effective_multixact_freeze_max_age issue

2022-08-29 Thread Matthias van de Meent
blem we're trying to solve. I understand that the use of < is pre-existing, but since we're touching this code shouldn't we try to get this new code up to current standards? Kind regards, Matthias van de Meent

Re: Tracking last scan time

2022-08-31 Thread Matthias van de Meent
h I think is a waste of time (heh) even in VDSO-enabled systems. Apart from the above, I don't have any other meaningful opinion on this patch - it might be a good addition, but I don't consume stats often enough to make a good cost / benefit comparison. Kind regards, Matthias van de Meent

Re: Tracking last scan time

2022-09-01 Thread Matthias van de Meent
On Wed, 31 Aug 2022 at 20:56, Andres Freund wrote: > > Hi, > > On 2022-08-31 19:52:49 +0200, Matthias van de Meent wrote: > > As for having a lower granularity and preventing the > > one-syscall-per-Relation issue, can't we reuse the query_start or > > sta

Re: Different compression methods for FPI

2022-09-05 Thread Matthias van de Meent
| COMPRESS_ZSTD)? Kind regards, Matthias van de Meent

Issue in GIN fast-insert: XLogBeginInsert + Read/LockBuffer ordering

2022-09-08 Thread Matthias van de Meent
ifies this issue, by moving the XLogBeginInsert() down to where 1.) we have all relevant buffers pinned and locked, and 2.) we're in a critical section, making that part of the code consistent with the general scheme for XLog insertion. Kind regards, Matthias van de Meent [0] access/tran

Adding CommandID to heap xlog records

2022-09-08 Thread Matthias van de Meent
tch)insert, update and delete records. It is based on the changes we made in the fork we maintain for Neon. Kind regards, Matthias van de Meent [0] https://github.com/neondatabase/neon/#neon [1] https://github.com/neondatabase/neon/blob/main/docs/core_changes.md#add-t_cid-to-heap-wal-records [2

Re: Tuples inserted and deleted by the same transaction

2022-09-13 Thread Matthias van de Meent
(e.g.) the existence of another tuple with the same XID but with a newer CID. That last one would not be impossible, but probably not worth the extra cost of command id tracking. Kind regards, Matthias van de Meent

Re: Tuples inserted and deleted by the same transaction

2022-09-13 Thread Matthias van de Meent
This means that this needs no specific handling in the HTSVH code that Laurenz asked about. Kind regards, Matthias van de Meent

Re: Tuples inserted and deleted by the same transaction

2022-09-13 Thread Matthias van de Meent
.) I hadn't realized that triggers indeed consume command ids but might not be visible to the outer query (that might still be running). That invalidates the "or (e.g.) the existence of another tuple with the same XID but with a newer CID" claim I made earlier, so thanks for clarifying. Kind regards, Matthias van de Meent

Re: cataloguing NOT NULL constraints

2022-09-19 Thread Matthias van de Meent
"tab_b_not_null" CHECK (b IS NOT NULL) "tab_c_not_null" CHECK (c IS NOT NULL) "tab_d_not_null" CHECK (d IS NOT NULL) "tab_e_not_null" CHECK (e IS NOT NULL) But the performance of repeated row-casting would probably not be as goo

Re: Pruning never visible changes

2022-09-19 Thread Matthias van de Meent
. (same row) > > > COMMIT; > > > > Didn't we just have this discussion in another thread? > > Well. not "just" :) This recent thread [0] mentioned the same, and I mentioned it in [1] too last year. Kind regards, Matthias

Re: Proposal to use JSON for Postgres Parser format

2022-09-20 Thread Matthias van de Meent
for JSON that is only 2 bytes for each (up to 3 when you consider separators, plus potential extra overhead for escaped values that are unlikely to appear our catalogs). Some numbers can be stored more efficiently in JSONB, but only large numbers and small fractions that we're unlikely t

Re: Proposal to use JSON for Postgres Parser format

2022-09-21 Thread Matthias van de Meent
On Tue, 20 Sept 2022 at 17:29, Alvaro Herrera wrote: > > On 2022-Sep-20, Matthias van de Meent wrote: > > > Allow me to add: compressability > > > > In the thread surrounding [0] there were complaints about the size of > > catalogs, and specifically the templa

Re: cutting down the TODO list thread

2023-05-15 Thread Matthias van de Meent
opg, pg_qualstats and some manual glue to get suggested indexes in the current database - but none of these are available in the main distribution. I'm not asking this to be part of the main PostgreSQL binary, but I don't think that the idea of 'automated suggestions' should be moved to

Re: cutting down the TODO list thread

2023-05-16 Thread Matthias van de Meent
On Mon, 15 May 2023 at 20:51, Robert Haas wrote: > > On Mon, May 15, 2023 at 2:05 PM Matthias van de Meent > wrote: > > > Add SET PERFORMANCE_TIPS option to suggest INDEX, VACUUM, VACUUM ANALYZE, > > > and CLUSTER > > > -> There are external too

Re: cutting down the TODO list thread

2023-05-16 Thread Matthias van de Meent
On Tue, 16 May 2023 at 14:27, Robert Haas wrote: > > On Tue, May 16, 2023 at 8:18 AM Matthias van de Meent > wrote: > > Agreed; and that's why I'm not against removing the specific wording > > of the item. This may not have been clearly described in my previous >

Inconsistent behavior with locale definition in initdb/pg_ctl init

2023-05-17 Thread Matthias van de Meent
set to "UTF8". The default text search configuration will be set to "english". Notably; if initdb chooses the C locale from --no-locale, it uses SQL_ASCII through libc, but when the C locale is specified through --locale=C, it somehow defaults to the ICU locale en-US and uses UTF8 as encoding. In my view that's very unexpected behaviour. Kind regards, Matthias van de Meent Neon (neon.tech)

Re: RFI: Extending the TOAST Pointer

2023-05-18 Thread Matthias van de Meent
7;t think we have no disk representations left, nor do I think we'll need to add another option to the ToastCompressionId enum. As an example, we can add another VARTAG option for dictionary-enabled external toast; like what the pluggable toast patch worked on. I think we can salvage some ideas from that patch, even if the main idea got stuck. Kind regards, Matthias van de Meent Neon Inc.

Re: RFI: Extending the TOAST Pointer

2023-05-18 Thread Matthias van de Meent
nyway (see VARSIZE_EXTERNAL(ptr)). By not using the varatt_external struct currently in use, we could be able to get down to <18B toast pointers as well, though I'd consider that unlikely. Kind regards, Matthias van de Meent Neon, Inc.

XLog size reductions: smaller XLRec block header for PG17

2023-05-18 Thread Matthias van de Meent
er than this one. Kind regards, Matthias van de Meent Neon, Inc. [0] https://wiki.postgresql.org/wiki/Updating_the_WAL_infrastructure [1] https://www.postgresql.org/message-id/flat/CAEze2Wjd3jY_UhhOGdGGnC6NO%3D%2BNmtNOmd%3DJaYv-v-nwBAiXXA%40mail.gmail.com#17a51d83923f4390d8f407d0d6c5da07

Re: Order changes in PG16 since ICU introduction

2023-05-18 Thread Matthias van de Meent
lent to `--locale=C` it now also overrides `--locale-provider=libc`, resulting in undocumented behaviour. Kind regards, Matthias van de Meent Neon, Inc. [0] https://www.postgresql.org/message-id/CAEze2WiZFQyyb-DcKwayUmE4rY42Bo6kuK9nBjvqRHYxUYJ-DA%40mail.gmail.com

Re: PG 16 draft release notes ready

2023-05-18 Thread Matthias van de Meent
s didn't exceed those limits. This change is immediately user-visible through a change in behaviour of `pg_logical_emit_message(true, repeat('_', 2^30 - 10), repeat('_', 2^30 - 10))`, and extensions that implement their own rmgrs could also see a change in behavior from this change. Kind regards, Matthias van de Meent Neon, Inc.

Re: XLog size reductions: smaller XLRec block header for PG17

2023-05-18 Thread Matthias van de Meent
On Thu, 18 May 2023 at 18:22, Heikki Linnakangas wrote: > > On 18/05/2023 17:59, Matthias van de Meent wrote: > Perhaps we should introduce a few generic inline functions to do varint > encoding. That could be useful in many places, while this scheme is very > tailored for XLogRe

Re: RFI: Extending the TOAST Pointer

2023-05-22 Thread Matthias van de Meent
fra is built around 4 uint32 fields in the toast pointer; but with this change in place we can devise a new toast pointer that uses varint encoding on the length-indicating fields to reduce the footprint of 18B to an expected 14 bytes. Kind regards, Matthias van de Meent Neon, Inc.

Re: RFI: Extending the TOAST Pointer

2023-05-22 Thread Matthias van de Meent
e. But this is data we store in the original tuple, and moving that around is relatively expensive. Reducing the aligned size of the toast pointer can help reduce the size of the heap tuple, thus increasing the efficiency of the primary data table. Kind regards, Matthias van de Meent Neon, Inc.

Re: RFI: Extending the TOAST Pointer

2023-05-24 Thread Matthias van de Meent
On Tue, 23 May 2023 at 18:34, Robert Haas wrote: > > On Thu, May 18, 2023 at 8:06 AM Matthias van de Meent > wrote: > > This enum still has many options to go before it exceeds the maximum > > of the uint8 va_tag field. Therefore, I don't think we have no disk > >

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Matthias van de Meent
ed address space, it'd be easier to do any cleanup necessary for dynamically increasing/decreasing its size. Same with parallel workers - if we have a shared address space, the workers can pass any sized objects around without being required to move the tuples through DSM and waiting for the leader process to empty that buffer when it gets full. Sure, most of that is probably possible with DSM as well, it's just that I see a lot more issues that you need to take care of when you don't have a shared address space (such as the pointer translation we do in dsa_get_address). Kind regards, Matthias van de Meent Neon, Inc.

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Matthias van de Meent
On Thu, 8 Jun 2023 at 14:44, Hannu Krosing wrote: > > On Thu, Jun 8, 2023 at 2:15 PM Matthias van de Meent > wrote: > > > > On Thu, 8 Jun 2023 at 11:54, Hannu Krosing wrote: > > > > > > This part was touched in the "AMA with a Linux Kernale Hacker&qu

Re: Let's make PostgreSQL multi-threaded

2023-06-08 Thread Matthias van de Meent
performance benefits for older kernels, benefits which we would get if we were to implement threading. I'm not on board with a policy of us twiddling thumbs and waiting for the OS to fix our architectural performance issues. Sure, the kernel could optimize for our usage pattern, but I think that's n

Re: Let's make PostgreSQL multi-threaded

2023-06-09 Thread Matthias van de Meent
s not force long down > times I agree that we should improve our upgrade process (and we had a great discussion on the topic at the PGCon Unconference last week), but in my view that's not relevant to this discussion. Kind regards, Matthias van de Meent Neon, Inc.

Re: trying again to get incremental backup

2023-06-14 Thread Matthias van de Meent
hat WAL-logged changes in the FSM fork are actually not recoverable from backup, regardless of the type of contents, we should still keep track of the changes in the FSM fork and include the fork in our backups or only exclude those FSM updates that we know are safe to ignore. Kind regards, Matthias van de Meent Neon, Inc.

Re: bgwriter doesn't flush WAL stats

2023-06-21 Thread Matthias van de Meent
x27;re not included in the patch and the current master branch doesn't emit object=wal, so I can't really check that the patch works as intended. But on the topic of reporting the WAL stats in bgwriter; that seems like a good idea to fix, yes. +1 Kind regards, Matthias van de Meent Neon, Inc.

Re: New function to show index being vacuumed

2023-06-22 Thread Matthias van de Meent
he other thread yet, but what was the reason we couldn't store that oid in a column in the pg_s_p_vacuum-view? Could you summarize the other solutions that were considered for this issue? Kind regards, Matthias van de Meent Neon, Inc.

Re: Improving btree performance through specializing by key shape, take 2

2023-06-23 Thread Matthias van de Meent
On Fri, 23 Jun 2023 at 11:26, Dilip Kumar wrote: > > On Fri, Jun 23, 2023 at 2:21 AM Matthias van de Meent > wrote: > > > > > == Dynamic prefix truncation (0001) > > The code now tracks how many prefix attributes of the scan key are > > already considered eq

Extensible storage manager API - SMGR hook Redux

2023-06-30 Thread Matthias van de Meent
options Looking forward to any comments, suggestions and reviews. Kind regards, Matthias van de Meent Neon (https://neon.tech/) [0] https://www.postgresql.org/message-id/CAP4vRV6JKXyFfEOf%3Dn%2Bv5RGsZywAQ3CTM8ESWvgq%2BS87Tmgx_g%40mail.gmail.com [1] https://www.postgresql.org/message-id/d36

Re: Why is DATESTYLE, ordering ignored for output but used for input ?

2023-07-03 Thread Matthias van de Meent
hackers archives don't seem to have any mails from that time period, so I couldn't find much info on the precise rationale around this behavior. Kind regards, Matthias van de Meent Neon (https://neon.tech/) PS. That was some interesting digging into the history of the date formatting module.

Re: Parallel CREATE INDEX for BRIN indexes

2023-07-04 Thread Matthias van de Meent
to be assigned to each process? I think that would be the most elegant solution that would require relatively little effort: table_block_parallelscan_nextpage already does parallel management of multiple chunk sizes, and I think this modification would fit quite well in that code. Kind regards, Matthias van de Meent

Re: Parallel CREATE INDEX for BRIN indexes

2023-07-05 Thread Matthias van de Meent
On Wed, 5 Jul 2023 at 00:08, Tomas Vondra wrote: > > > > On 7/4/23 23:53, Matthias van de Meent wrote: > > On Thu, 8 Jun 2023 at 14:55, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> Here's a WIP patch allowing parallel CREATE

Re: Disabling Heap-Only Tuples

2023-07-05 Thread Matthias van de Meent
uple if at all possible, so disabling HOT wouldn't solve the issue of tuples residing in the tail of your table - at least not while there is still empty space in those pages. Kind regards, Matthias van de Meent Neon (https://neon.tech/)

Re: Disabling Heap-Only Tuples

2023-07-05 Thread Matthias van de Meent
On Wed, 5 Jul 2023 at 13:03, Thom Brown wrote: > > On Wed, 5 Jul 2023 at 11:57, Matthias van de Meent > wrote: > > > > On Wed, 5 Jul 2023 at 12:45, Thom Brown wrote: > > > Heap-Only Tuple (HOT) updates are a significant performance > > > enhancement,

Re: Parallel CREATE INDEX for BRIN indexes

2023-07-05 Thread Matthias van de Meent
On Wed, 5 Jul 2023 at 00:08, Tomas Vondra wrote: > > > > On 7/4/23 23:53, Matthias van de Meent wrote: > > On Thu, 8 Jun 2023 at 14:55, Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> Here's a WIP patch allowing parallel CREATE

Re: [PATCH] Add GitLab CI to PostgreSQL

2023-07-05 Thread Matthias van de Meent
experience with this tool", which is good, but not exclusive to Cirrus CI - clearly there are also people here who have experience with (or are interested in) maintaining GitLab CI configurations. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Disabling Heap-Only Tuples

2023-07-05 Thread Matthias van de Meent
On Wed, 5 Jul 2023 at 14:39, Thom Brown wrote: > > On Wed, 5 Jul 2023 at 13:12, Matthias van de Meent > wrote: > > > > On Wed, 5 Jul 2023 at 13:03, Thom Brown wrote: > > > > > > On Wed, 5 Jul 2023 at 11:57, Matthias van de Meent > > > wrote: &

HOT readme missing documentation on summarizing index handling

2023-07-06 Thread Matthias van de Meent
ot sure if such internal documentation is relevant for backpatching, but I also don't think it woudl hurt to have this included in the REL_16_STABLE branch. Kind regards, Matthias van de Meent Neon (https://neon.tech/) v1-0001-Add-documentation-in-README.HOT-for-handling-summ.patch Description: Binary data

Re: UUID v7

2023-07-06 Thread Matthias van de Meent
g process usually takes, but it seems like the WG considers the text final, so unless this would take months I wouldn't mind keeping this patch around as "waiting for external process to complete". Sure, it's earlier than the actual release of the standard, but that wasn'

Re: Disabling Heap-Only Tuples

2023-07-06 Thread Matthias van de Meent
On Wed, 5 Jul 2023 at 19:55, Thom Brown wrote: > > On Wed, 5 Jul 2023 at 18:05, Matthias van de Meent > wrote: > > So what were you thinking of? A session GUC? A table option? > > Both. Here's a small patch implementing a new table option max_local_update (name very

Re: Disabling Heap-Only Tuples

2023-07-07 Thread Matthias van de Meent
On Fri, 7 Jul 2023 at 06:53, Dilip Kumar wrote: > > On Fri, Jul 7, 2023 at 1:48 AM Matthias van de Meent > wrote: > > > > On Wed, 5 Jul 2023 at 19:55, Thom Brown wrote: > > > > > > On Wed, 5 Jul 2023 at 18:05, Matthias van de Meent > > > wrote:

Re: HOT readme missing documentation on summarizing index handling

2023-07-07 Thread Matthias van de Meent
text was really about on/off, and I'm not quite sure the > part about "exception" makes this very clear. Agreed on both points. Attached an updated version which incorporates your points. Kind regards, Matthias van de Meent Neon (https://neon.tech) v2-0001-Add-documentation-in-README.HOT-for-handling-summ.patch Description: Binary data

Re: HOT readme missing documentation on summarizing index handling

2023-07-07 Thread Matthias van de Meent
On Fri, 7 Jul 2023 at 19:06, Tomas Vondra wrote: > > On 7/7/23 18:34, Matthias van de Meent wrote: > > On Fri, 7 Jul 2023 at 00:14, Tomas Vondra > > wrote: > >> The original text was really about on/off, and I'm not quite sure the > >> par

Re: Introduce new multi insert Table AM and improve performance of various SQL commands with it for Heap AM

2024-08-26 Thread Matthias van de Meent
th. Is support for RETURN-clauses planned for this API? In a previous iteration, the flush operation was capable of returning a TTS, but that seems to have been dropped, and I can't quite figure out why. > Note: I believe this API will extend naturally to updates and deletes, > as well. I have the same concern about UPDATE ... RETURNING not fitting with this callback-based design. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Parallel CREATE INDEX for GIN indexes

2024-08-27 Thread Matthias van de Meent
ostly be allocated to just BuildAccumulator if we can dump the data into the tuplesort directly, as it'd reduce the overall number of operations and memory allocations for the tuplesort. I think that once we correctly account for memory allocations (and an improved write path) we'll be able to see a meaningfully larger performance improvement. > So my option is if we can have agreement on 0008, then we can final > review/test on the existing code (including 0009), and leave further > improvement as a dedicated patch. As mentioned above, I think I could update the patch for a btree implementation that also has immediate benefits, if so desired? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Introduce new multi insert Table AM and improve performance of various SQL commands with it for Heap AM

2024-08-27 Thread Matthias van de Meent
On Tue, 27 Aug 2024 at 07:42, Jeff Davis wrote: > > On Mon, 2024-08-26 at 23:59 +0200, Matthias van de Meent wrote: > > Specifically, I'm having trouble seeing how this could be used to > > implement ```INSERT INTO ... SELECT ... RETURNING ctid``` as I see no > >

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-08-27 Thread Matthias van de Meent
On Fri, 16 Aug 2024 at 00:34, Peter Geoghegan wrote: > > On Thu, Aug 15, 2024 at 5:47 PM Matthias van de Meent > wrote: > > > > I'm asking, because > > > > I'm not very convinced that 'primitive scans' are a useful metric > > > &g

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-08-27 Thread Matthias van de Meent
On Tue, 27 Aug 2024 at 23:40, Peter Geoghegan wrote: > > On Tue, Aug 27, 2024 at 5:03 PM Matthias van de Meent > wrote: > > If the counter was put into the BTScanOpaque, rather than the > > IndexScanDesc, then this could trivially be used in an explain AM > > cal

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-08-27 Thread Matthias van de Meent
On Wed, 28 Aug 2024 at 01:42, Peter Geoghegan wrote: > > On Tue, Aug 27, 2024 at 7:22 PM Matthias van de Meent > wrote: > > On Tue, 27 Aug 2024 at 23:40, Peter Geoghegan wrote: > > > Right, "trivial". Except in that it requires inventing a whole new &g

Re: Parallel CREATE INDEX for GIN indexes

2024-08-27 Thread Matthias van de Meent
On Wed, 28 Aug 2024 at 02:38, Andy Fan wrote: > > Matthias van de Meent writes: > > tuplesort_performsort usually only needs to flush the last elements of > > ... In certain rare > > cases it may take a longer time as it may have to merge sorted runs, > > but th

Re: Reading all tuples in Index Access Method

2024-08-28 Thread Matthias van de Meent
ded indexes currently have a metapage at block 0 of the main data fork, which contains metadata and bookkeeping info about the index's structure, and you're free to do the same for your index. I hope that helps? Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Showing primitive index scan count in EXPLAIN ANALYZE (for skip scan and SAOP scans)

2024-08-30 Thread Matthias van de Meent
ecProcNodeInstr, but it'd need some more special-casing to make sure it only addresses (index) relation scan nodes. By pulling the stats directly from Relation->pgstat_info, no catalog index scans are counted if they aren't also the index which is subject to that [Bitmap]Index[Only]Scan. In effect, this would need no changes in AM code, as this would "just" pull the data from those statistics which are already being updated in AM code. Kind regards, Matthias van de Meent Neon (https://neon.tech)

Re: Avoiding superfluous buffer locking during nbtree backwards scans

2024-08-30 Thread Matthias van de Meent
On Mon, 19 Aug 2024 at 13:43, Matthias van de Meent wrote: > > On Sun, 11 Aug 2024 at 21:44, Peter Geoghegan wrote: > > > > On Tue, Aug 6, 2024 at 6:31 PM Matthias van de Meent > > wrote: > > > +1, LGTM. > > > > > > This change

Re: Avoiding superfluous buffer locking during nbtree backwards scans

2024-09-02 Thread Matthias van de Meent
On Fri, 30 Aug 2024 at 21:43, Matthias van de Meent wrote: > > On Mon, 19 Aug 2024 at 13:43, Matthias van de Meent > wrote: > > > > On Sun, 11 Aug 2024 at 21:44, Peter Geoghegan wrote: > > > > > > On Tue, Aug 6, 2024 at 6:31 PM Matthias van

<    1   2   3   4   5   6   7   8   >