Re: confusing / inefficient "need_transcoding" handling in copy

On Fri, Dec 13, 2024 at 12:27:37PM +0900, Sutou Kouhei wrote: > Oh, sorry... I attached wrong patch... > I attach the v4 patch that includes this case. Sounds fair to me as a beginning for the code paths setting need_transcoding. Note that using "test" as table name for the tests is not a good id

Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

On Thu, Jun 13, 2024 at 2:37 AM Andres Freund wrote: > > It's hard to understand, but a nonetheless helpful page is > https://users.ece.cmu.edu/~koopman/crc/crc32.html which lists properties for > crc32c: > https://users.ece.cmu.edu/~koopman/crc/c32/0x8f6e37a0_len.txt > which lists > (0x8f6e37a0;

Re: Fix early elog(FATAL)

On Fri, Dec 13, 2024 at 10:14:43PM -0600, Nathan Bossart wrote: > On Fri, Dec 13, 2024 at 07:15:05PM -0800, Noah Misch wrote: >> I don't feel strongly either way. I did write it the option-1 way >> originally. >> Then I started thinking about changes at a distance causing the other >> InitProcess

Re: Fix early elog(FATAL)

On Fri, Dec 13, 2024 at 07:15:05PM -0800, Noah Misch wrote: > On Thu, Dec 12, 2024 at 10:07:00AM -0600, Nathan Bossart wrote: >> On Wed, Dec 11, 2024 at 07:34:14PM -0800, Noah Misch wrote: >> > On Tue, Dec 10, 2024 at 04:18:19PM -0600, Nathan Bossart wrote: >> >> FWIW I'd probably vote for option 1

Re: Track the amount of time waiting due to cost_delay

I spent some time preparing v12 for commit and made the following larger changes: * I renamed the column to delay_time and changed it back to repoting milliseconds to match other stats views like pg_stat_io. * I optimized the code in vacuum_delay_point a bit. Notably, we're now just storing

Re: Hot standby queries see transient all-zeros pages

On Fri, Dec 13, 2024 at 09:34:21PM -0500, Andres Freund wrote: > On 2024-12-13 16:38:05 -0800, Noah Misch wrote: > > On Fri, Dec 13, 2024 at 05:41:15PM -0500, Andres Freund wrote: > > > Hm. Leaving RBM_ZERO_AND_LOCK aside, is it actually always safe to do > > > RestoreBlockImage() into a buffer tha

Re: Fix early elog(FATAL)

On Thu, Dec 12, 2024 at 10:07:00AM -0600, Nathan Bossart wrote: > On Wed, Dec 11, 2024 at 07:34:14PM -0800, Noah Misch wrote: > > On Tue, Dec 10, 2024 at 04:18:19PM -0600, Nathan Bossart wrote: > >> FWIW I'd probably vote for option 1. That keeps the initialization of the > >> globals together, re

Re: Exceptional md.c paths for recovery and zero_damaged_pages

Hi, On 2024-12-13 19:06:16 -0500, Tom Lane wrote: > Andres Freund writes: > > We have a fair number of special paths in md.c that are specific to > > recovery. E.g. in mdreadv() we do: > > ... > > As far as I can tell, nearly all - including the above - InRecovery paths in > > md.c are basically

Re: typo in a comment of restrictinfo.c

On Sat, Dec 14, 2024 at 8:41 AM Ryo Kanbayashi wrote: > > Hi everyone, > > I found a typo in comment of restrictinfo.c line 99. > ( > https://github.com/postgres/postgres/blob/master/src/backend/optimizer/util/restrictinfo.c#L99 > ) > > Not 'construcitng' but 'constructing' ? Pushed, thanks! --

Re: Hot standby queries see transient all-zeros pages

Hi, On 2024-12-14 13:41:40 +1300, Thomas Munro wrote: > On Sat, Dec 14, 2024 at 11:41 AM Andres Freund wrote: > > On 2024-05-12 10:16:58 -0700, Noah Misch wrote: > > > The use of RBM_ZERO_AND_LOCK is incompatible with that. See a similar > > > argument at https://postgr.es/m/flat/5101.1328219...

Re: Hot standby queries see transient all-zeros pages

Hi, On 2024-12-13 16:38:05 -0800, Noah Misch wrote: > On Fri, Dec 13, 2024 at 05:41:15PM -0500, Andres Freund wrote: > > Afaics we didn't do anything about this issue? > > postgr.es/c/e656657 fixed this. I thought this was longstanding, but it > turned out to have started on 2024-04-02. Ah, tha

"collation" or "collation oder"

Currently in the docs both "collation" and "collation order" are used. I think this is confusing because IMO "collation" implies "order", and "collation order" is redundant. Can we change "collation order" to "collation"? git grep shows followings: amcheck.sgml: collation order between a prim

typo in a comment of restrictinfo.c

Hi everyone, I found a typo in comment of restrictinfo.c line 99. ( https://github.com/postgres/postgres/blob/master/src/backend/optimizer/util/restrictinfo.c#L99 ) Not 'construcitng' but 'constructing' ? -- Best regards, Ryo Kanbayashi kanbayashi@gmail.com https://github.com/ryogrid diff -

Re: Query regarding pg_prewarm extension

On Fri, 13 Dec 2024 16:16:16 +0530 Ayush Vatsa wrote: > How can I decide which range of pages to prewarm? > I assume that it is related to hot pages in the relation, > but how can I identify which pages are likely to be hot > before they are even in the buffer cache? > Additionally, since tuples

Re: checkpointer: PANIC: could not fsync file: No such file or directory

On Sat, Dec 14, 2024 at 9:55 AM Andres Freund wrote: > But since the above change checkpointer doesn't use EXTENSION_DONT_CHECK_SIZE > anymore. > > Seems we should remove this code? +1

Re: Hot standby queries see transient all-zeros pages

On Sat, Dec 14, 2024 at 11:41 AM Andres Freund wrote: > On 2024-05-12 10:16:58 -0700, Noah Misch wrote: > > The use of RBM_ZERO_AND_LOCK is incompatible with that. See a similar > > argument at https://postgr.es/m/flat/5101.1328219...@sss.pgh.pa.us that led > > me to the cause. Adding a 10ms sle

Re: Hot standby queries see transient all-zeros pages

On Fri, Dec 13, 2024 at 05:41:15PM -0500, Andres Freund wrote: > Afaics we didn't do anything about this issue? postgr.es/c/e656657 fixed this. I thought this was longstanding, but it turned out to have started on 2024-04-02. > Hm. Leaving RBM_ZERO_AND_LOCK aside, is it actually always safe to d

Re: Recovering from detoast-related catcache invalidations

On 13/12/2024 17:30, Tom Lane wrote: Heikki Linnakangas writes: CatalogCacheCreateEntry() can accept catcache invalidations when it opens the toast table, and it now has recheck logic to detect the case that the tuple it's processing (ntp) is invalidated. However, isn't it also possible that it

Re: Exceptional md.c paths for recovery and zero_damaged_pages

Andres Freund writes: > We have a fair number of special paths in md.c that are specific to > recovery. E.g. in mdreadv() we do: > ... > As far as I can tell, nearly all - including the above - InRecovery paths in > md.c are basically unreachable. And have been for quite a while. > XLogReadBuffer

Re: Added schema level support for publication.

[ reviving one aspect of an old thread ] vignesh C writes: > On Mon, Jul 19, 2021 at 9:32 AM tanghy.f...@fujitsu.com < > tanghy.f...@fujitsu.com> wrote: >> I tested your v12 patch and found a problem in the following case. >> >> Step 1: >> postgres=# create schema s1; >> CREATE SCHEMA >> postgre

Exceptional md.c paths for recovery and zero_damaged_pages

Hi, We have a fair number of special paths in md.c that are specific to recovery. E.g. in mdreadv() we do: v = _mdfd_getseg(reln, forknum, blocknum, false, EXTENSION_FAIL | EXTENSION_CREATE_RECOVERY); and

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

On Thu, Nov 7, 2024 at 10:42 AM Andres Freund wrote: > > Hi, Thanks for the review! Attached v2 should address your feedback and also fixes a few bugs with v1. I've still yet to run very long-running benchmarks. I did start running more varied benchmark scenarios -- but all still under two hours

Re: Hot standby queries see transient all-zeros pages

Hi, Just found this thread because I was looking for discussions for some behaviour of XLogReadBufferExtended()... Afaics we didn't do anything about this issue? I just tried to repro this, without success so far. But it very well might be sufficiently timing dependent that it just happens to n

Re: bt_index_parent_check and concurrently build indexes

On Mon, Dec 9, 2024 at 3:51 PM Michail Nikolaev wrote: > After some time, I managed to find a way to reproduce the issue. It turns out > that bt_index_parent_check is not suitable for validating indexes built > concurrently. Good catch. > I’ve attached a patch that reproduces the issue (it inc

Re: Track the amount of time waiting due to cost_delay

Hello! +/* + * In case of parallel workers, the last time the delay has been reported to + * the leader. + * We assume this initializes to zero. + */ +static instr_time last_report_time; Maybe last_report_time would be better named worker_last_report_time? (It is not clear to me from the comment

Re: checkpointer: PANIC: could not fsync file: No such file or directory

Hi, On 2019-12-13 17:41:56 +1300, Thomas Munro wrote: > From 9609c9a153232fb2de169bf76158781d354c633b Mon Sep 17 00:00:00 2001 > From: Thomas Munro > Date: Fri, 13 Dec 2019 17:12:42 +1300 > Subject: [PATCH] Don't use _mdfd_getseg() in mdsyncfiletag(). > > _mdfd_getseg() opens all segments up to

Re: COPY performance on Windows

Ryohei Takahashi (Fujitsu) писал(а) 2024-12-11 15:18: Hi, I continuously investigate the performance problem of COPY on Windows. I noticed that not only PG17.0 but also PG16.6 have performance problem compared to PG16.4. The performance is 2.5%-5.8% worse, especially when the number of clien

Re: pg_attribute_noreturn(), MSVC, C11

Hi, On 2024-12-13 14:10:13 -0500, Andres Freund wrote: > I just encountered another > warning C4715: 'XYZ: not all control paths return a value > > with msvc in CI in a case where it should be trivial for the compiler to > recognize that the function return isn't reachable. > > Which made me c

pg_attribute_noreturn(), MSVC, C11

Hi, I just encountered another warning C4715: 'XYZ: not all control paths return a value with msvc in CI in a case where it should be trivial for the compiler to recognize that the function return isn't reachable. Which made me check if these days msvc has something like gcc's __attribute__((n

Re: fixing tsearch locale support

On Fri, 2024-12-13 at 07:16 +0100, Peter Eisentraut wrote: > isdigit() and isspace() in particular are widely used throughout the > backend code without such concerns.  I think the assumption is that > this > is not a problem in practice: For multibyte encodings, these > functions > would only b

Re: OLEDB provider for PostgreSQL

Hi pá 13. 12. 2024 v 15:22 odesílatel Zaid Shabbir napsal: > Hello, > > I am looking for an OLEDB provider for PostgreSQL 16 or later. I found one > link, but it seems that it does not support PostgreSQL 16+ and, > additionally, it is not free. > > PostgreSQL: PostgreSQL Native OLEDB Provider (P

Re: Proposals for EXPLAIN: rename ANALYZE to EXECUTE and extend VERBOSE

> I've pushed the main patch. Woohoo! And thank you. I've already seen quite a lot of positivity around the commit on Twitter[1][2][3]. > I'm not planning on pushing the auto_explain.log_buffers default change unless there's a bit more discussion about it. Much like Guillaume, I'd also be in fav

Re: OLEDB provider for PostgreSQL

On 2024-12-13 Fr 9:25 AM, Zaid Shabbir wrote: Hello, I am looking for an OLEDB provider for PostgreSQL 16 or later. I found one link, but it seems that it does not support PostgreSQL 16+ and, additionally, it is not free. PostgreSQL: PostgreSQL Native OLEDB Provider (PGNP) 1.3.0 32/64-bit

Re: Allow FDW extensions to support MERGE command via CustomScan

On 12/13/24 16:03, Önder Kalacı wrote: > Hi Alvaro, all > >> IMO this is a bad plan.  It'll become _the_ way to run MERGE on foreign >> tables, which will become a selling point for proprietary FDWs, and >> nobody will be motivated to write the code to implement the long-term >> plan you were d

Re: Recovering from detoast-related catcache invalidations

Heikki Linnakangas writes: > CatalogCacheCreateEntry() can accept catcache invalidations when it > opens the toast table, and it now has recheck logic to detect the case > that the tuple it's processing (ntp) is invalidated. However, isn't it > also possible that it accepts an invalidation mess

Re: Allow FDW extensions to support MERGE command via CustomScan

Hi Alvaro, all > IMO this is a bad plan. It'll become _the_ way to run MERGE on foreign > tables, which will become a selling point for proprietary FDWs, and > nobody will be motivated to write the code to implement the long-term > plan you were describing. > > In short, -1 from me. > I see your

Re: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

> On 11/21/24 14:59, Tomas Vondra wrote: > > I don't have a great idea how to improve this. It seems wrong for > ReplicationSlotsComputeRequiredLSN() to calculate the LSN using values > from dirty slots, so maybe it should simply retry if any slot is dirty? > Or retry on that one slot? But various

Re: Crash: invalid DSA memory alloc request

On 12/12/2024 22:49, Matthias van de Meent wrote: On Thu, 12 Dec 2024 at 22:28, Andreas 'ads' Scherbaum wrote: Hello, I'm running a couple of large tests, and in this particular test I have a few million tables more. At some point it fails, and I gathered the following trace: 2024-12-12

OLEDB provider for PostgreSQL

Hello, I am looking for an OLEDB provider for PostgreSQL 16 or later. I found one link, but it seems that it does not support PostgreSQL 16+ and, additionally, it is not free. PostgreSQL: PostgreSQL Native OLEDB Provider (PGNP) 1.3.0 32/64-bit released!

Re: Recovering from detoast-related catcache invalidations

On 25/09/2024 00:20, Noah Misch wrote: On Sun, Jan 14, 2024 at 12:14:11PM -0800, Noah Misch wrote: On Fri, Jan 12, 2024 at 03:47:13PM -0500, Tom Lane wrote: Oh! After nosing around a bit more I remembered systable_recheck_tuple, which is meant for exactly this purpose. So v4 attached. systa

Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

On Fri, 13 Dec 2024 at 00:14, Nathan Bossart wrote: > > On Thu, Dec 12, 2024 at 10:45:29AM -0500, Andres Freund wrote: > > Frankly, we should just move away from using CRCs. They're good for cases > > where short runs of bit flips are much more likely than other kinds of > > errors > > and where

Re: Allow subfield references without parentheses

On Thu, Dec 12, 2024 at 5:54 PM Peter Eisentraut wrote: > > This patch allows subfield references in column references without > parentheses, subject to certain condition. This implements (hopes to, > anyway) the rules from the SQL standard (since SQL99). > > This has been requested a number of t

Re: [PATCH] Fixed creation of empty .log files during log rotation

Hi, Tomas. 08.12.2024 01:06, Tomas Vondra wrote: > Yeah, creating all those files seems rather unnecessary. But wouldn't it > be easier to do the check in logfile_rotate_dest(), instead of for each > call? I think something like this would do the trick: > > @@ -1292,6 +1292,9 @@ logfile_rotate_d

Re: Allow FDW extensions to support MERGE command via CustomScan

On 2024-Dec-13, Önder Kalacı wrote: > Matheus Alcantara wrote: > > I didn't understand what is a "custom scan node" on the fdw context at > > first place (I don't know if it is an already know word on this > > context), but from what I've understood so far, to a fdw extension > > support MERGE i

Re: Introduce XID age and inactive timeout based replication slot invalidation

On Wed, Dec 11, 2024 at 8:14 AM Peter Smith wrote: > > Hi Nisha. > > Here are some review comments for patch v54-0002. > == > src/test/recovery/t/043_invalidate_inactive_slots.pl > > 5. > +# Wait for slot to first become idle and then get invalidated > +sub wait_for_slot_invalidation > +{ > +

Re: Introduce XID age and inactive timeout based replication slot invalidation

On Thu, Dec 12, 2024 at 9:42 AM vignesh C wrote: > > > Now that we support idle_replication_slot_timeout in milliseconds, we > can set this value from 1s to 1ms or 10millseconds and change sleep to > usleep, this will bring down the test execution time significantly: +1 v55 implements the test us

Re: [PoC] Reducing planning time when tables have many partitions

Hello, On 2024-Dec-13, Yuya Watari wrote: > Thank you for your reply, and I'm sorry if my previous emails caused > confusion or made it seem like I was ignoring more important issues. Not to worry! > > Memory usage and planning time in production builds [are] important. > > You can better spend

Query regarding pg_prewarm extension

Hi PostgreSQL Community, I have a question regarding the use of the pg_prewarm() function [1] in the pg_prewarm extension. The function requires a relation name and a range of pages (e.g., pages 10 to 50) to be warmed by shifting them from disk to the buffer cache. How can I decide which range of

Re: Difference in dump from original and restored database due to NOT NULL constraints on children

On 2024-Dec-12, Ashutosh Bapat wrote: > Attached updated patch. Once we commit this patch, I will be able to > proceed with the dump/restore test at [1]. Thanks, I have pushed it. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "You don't solve a bad join with S

Re: Adding a '--two-phase' option to 'pg_createsubscriber' utility.

On Fri, Dec 13, 2024 at 2:39 PM vignesh C wrote: > > On Fri, 13 Dec 2024 at 13:01, Shubham Khanna > wrote: > > > > On Fri, Dec 13, 2024 at 12:20 PM Peter Smith wrote: > > > > > > Hi Shubham. > > > > > > Here are my review comments for v6-0001. > > > > > > == > > > 1. > > > +# Verify that the

Re: Managing IO workers in postmaster's state machine

Hi Andres, Thanks for sharing these detailed questions and insights. Below are some comments and additional thoughts that might help, particularly regarding bgworkers. On 09/12/2024 22:42, Andres Freund wrote: Hi, For AIO, when using the worker process based "AIO", we need to manage a set

Re: Remaining dependency on setlocale()

On Fri, Dec 13, 2024 at 8:22 AM Jeff Davis wrote: > On Wed, 2024-08-14 at 12:00 -0700, Jeff Davis wrote: > > On Wed, 2024-08-14 at 14:31 +1200, Thomas Munro wrote: > > > 1. The process global locale is always "C". If you ever call > > > uselocale(), it can only be for short stretches, and you ha

Re: DOCS: pg_createsubscriber wrong link?

On Fri, 13 Dec 2024 at 10:58, Peter Smith wrote: > > Hi, > > While reviewing the pg_createsubscriber [1] docs I found a potentially > wrong linkend. > > This sentence: > "For smaller databases, initial data synchronization is recommended." > > links to [2] ("29.4.5. Initial Data Synchronization").

Re: per backend I/O statistics

Hi, On Fri, Dec 13, 2024 at 11:02:53AM +0900, Michael Paquier wrote: > On Thu, Dec 12, 2024 at 02:02:38PM +, Bertrand Drouvot wrote: > > Anyway, isn't it possible that this lookup loop finishes by finding > nothing depending on concurrent updates of other beentries? It sounds > to me that th

Re: Adding a '--two-phase' option to 'pg_createsubscriber' utility.

On Fri, 13 Dec 2024 at 13:01, Shubham Khanna wrote: > > On Fri, Dec 13, 2024 at 12:20 PM Peter Smith wrote: > > > > Hi Shubham. > > > > Here are my review comments for v6-0001. > > > > == > > 1. > > +# Verify that the subtwophase is enabled ('e') in the pg_subscription > > catalog > > +$node

Re: Detect buffer underflow in get_th()

Hello, ping? 24.09.2024 17:52, Alexander Kuznetsov wrote: Hello, is there anything else we can help with or discuss in order to apply this fix? 24.07.2024 18:53, Alexander Kuznetsov пишет: 24.07.2024 18:39, Peter Eisentraut wrote: If it can't happen in practice, maybe an assertion would be

Re: [PATCH] Check for TupleTableSlot nullness before dereferencing

Hello, ping. What do you think about reasoning below? Maybe we should consider proposing different patch for removing redundant check there? 09.10.2024 18:23, Alexander Kuznetsov wrote: 03.10.2024 12:48, Daniel Gustafsson wrote:  From a quick reading we can only reach there after evaluating an

Re: [PoC] Reducing planning time when tables have many partitions

Hello Alvaro, Thank you for your reply, and I'm sorry if my previous emails caused confusion or made it seem like I was ignoring more important issues. On Thu, Dec 12, 2024 at 9:09 PM Alvaro Herrera wrote: > > I'm repeating myself, but I disagree that this is something we should > spend _any_ ti

Re: on_error table, saving error info to a table

On Wed, Dec 11, 2024 at 7:41 PM Nishant Sharma wrote: > > Thanks for the v3 patch! > > Please find review comments on v3:- > > 1) I think no need to change the below if condition, we can keep > it the way it was before i.e with > "cstate->opts.on_error != COPY_ON_ERROR_STOP" and we > add a new err