Re: Purpose of wal_init_zero

2025-01-17 Thread Andres Freund
ut it really isn't a whole lot. I've in the past tried to get the linux filesytem developers to add an fallocate mode that doesn't utilize the "unwritten extents" "optimization", but didn't have luck with that. The block layer in linux actually does have support for zeroing out regions of blocks without having to do actually write the data, but it's only used in some narrow cases (don't remember the details). Greetings, Andres Freund

Re: Accept recovery conflict interrupt on blocked writing

2025-01-17 Thread Andres Freund
Hi, On 2025-01-17 13:03:35 -0500, Andres Freund wrote: > > Previously, all interrupts except process dying were ignored while a > > process was blocked writing to a socket. If the connection to the client > > was broken (no clean FIN nor RST), a process sending results to the &

Re: Accept recovery conflict interrupt on blocked writing

2025-01-17 Thread Andres Freund
Hi, On 2025-01-17 13:03:35 -0500, Andres Freund wrote: > I don't see anything implementing the promotion of ERRORs to FATAL? You're > preventing the error message being sent to the client, but I don't think that > causes the connection to be terminated. The pre-existing

Re: Accept recovery conflict interrupt on blocked writing

2025-01-17 Thread Andres Freund
{ > + if (whereToSendOutput == DestRemote) > + whereToSendOutput = DestNone; > + ProcessRecoveryConflictInterrupts(); > + } > + } > This largely duplicates the existing code in ProcessClientWriteInterrupt(). Perhaps it's worth flipping the order of branches so that we first check that we're neither blocked nor have InterruptHoldoffCount == 0 && CritSectionCount == 0? Greetings, Andres Freund

Re: Remove XLogRecGetFullXid() in xlogreader.c?

2025-01-17 Thread Andres Freund
Hi, On 2025-01-17 14:00:49 +0900, Michael Paquier wrote: > XLogRecGetFullXid() has been introduced in 67b9b3ca3283 back in 2019, > but as far as I can see this has never been used in the code and this > is used nowhere in the core code. > > I have looked at Debian's codesearch and also looked at t

Re: per backend WAL statistics

2025-01-16 Thread Andres Freund
Hi, On 2025-01-16 17:11:09 +, Bertrand Drouvot wrote: > On Thu, Jan 16, 2025 at 11:38:47AM -0500, Andres Freund wrote: > > Hi, > > > > On 2025-01-16 15:59:31 +, Bertrand Drouvot wrote: > > > On Wed, Jan 15, 2025 at 03:11:32PM +0900, Michael Paquier wrote: &

Re: per backend WAL statistics

2025-01-16 Thread Andres Freund
allowed to just opt out of the restriction of not allowing memory allocations in critical sections? The only cases where we can somewhat safely allocate memory in critical sections is when using memory contexts with pre-reserved memory, where there's a pretty low bound on how much memory is going to be needed. E.g. logging a message inside a critical section, where elog.c can reset ErrorContext afterwards. Greetings, Andres Freund

Re: per backend I/O statistics

2025-01-16 Thread Andres Freund
case again. And for something like injection points (not really convinced that it's a good thing to have stats for, but ...), allocation the shared memory stats object outside of the critical section and not having any pending stats seems to be the right approach. Greetings, Andres Freund

Re: Increase NUM_XLOGINSERT_LOCKS

2025-01-16 Thread Andres Freund
gets considerably worse. IIRC I saw something like an 8x regression on a large machine in the past, but I couldn't find the actual numbers anymore, so I wouldn't want to bet on it. Greetings, Andres Freund

Re: Purpose of wal_init_zero

2025-01-15 Thread Andres Freund
m metadata changes. Note that not all filesystems can benefit from wal_init_zero=1, e.g. ZFS or BTRFS won't benefit, because they always allocate new disk space for each write. With the associated overheads. Greetings, Andres Freund

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-14 Thread Andres Freund
Hi, On 2025-01-14 13:06:31 +, Bertrand Drouvot wrote: > On Tue, Jan 14, 2025 at 12:58:44AM -0500, Andres Freund wrote: > > The current code has the weird behaviour of going through PM_WAIT_BACKENDS. > > I > > left it like that for now. In fact more paths do so now, becau

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Andres Freund
irectly jump to PM_WAIT_DEAD_END in fatal situations: Attached patch is implemented that way. Greetings, Andres Freund >From 833ee00ea2a1341b2e20e88c96d8824c09189df2 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Fri, 10 Jan 2025 11:11:40 -0500 Subject: [PATCH v4 1/7] checkpointer: Req

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-13 Thread Andres Freund
Hi, On 2025-01-13 12:20:39 +, Bertrand Drouvot wrote: > On Fri, Jan 10, 2025 at 02:07:25PM -0500, Andres Freund wrote: > > There wasn't > > any code to treat receiving PMSIGNAL_XLOG_IS_SHUTDOWN in the wrong state as > > fatal. I think this needs to be treated the s

Re: AIO v2.2

2025-01-13 Thread Andres Freund
Hi, On 2025-01-13 15:43:46 -0500, Robert Haas wrote: > On Wed, Jan 8, 2025 at 7:26 PM Andres Freund wrote: > > 1) Shared memory representation of an IO, for the AIO subsystem internally > > > >Currently: PgAioHandle > > > > 2) A way for the issuer of

Re: llvm dependency and space concerns

2025-01-11 Thread Andres Freund
Hi, On 2025-01-11 13:22:39 -0800, Jeremy Schneider wrote: > It's a cleaner solution if JIT works more like an extension, and we can > run a single build and split JIT into a separate package. It does work like that. Only llvmjit.so has the llvm dependency, the main postgres binary doesn't link to

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-10 Thread Andres Freund
hat the first and crash recovery scenarios are more similar. Thomas, what do you think? Separately, is it a problem that somewhere in process_pm_* we did ConfigurePostmasterWaitSet(false) but then the event loop in ServerLoop() continues to process "old" WL_SOCKET_ACCEPT events? Greetings, An

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-10 Thread Andres Freund
Hi, Thanks for the reviews! On 2025-01-09 12:01:05 +, Bertrand Drouvot wrote: > On Wed, Jan 08, 2025 at 02:26:15PM -0500, Andres Freund wrote: > === 2 > > + PM_WAIT_XLOG_ARCHIVAL, /* waiting for archiver and walsenders to > > > I don't love PM_WAIT_XLOG_A

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-10 Thread Andres Freund
Hi, On 2025-01-09 16:50:45 +0300, Nazir Bilal Yavuz wrote: > On Wed, 8 Jan 2025 at 22:26, Andres Freund wrote: > === 0005 > > > I think this is much better than before. I don't love PM_WAIT_XLOG_ARCHIVAL, > > but I can't think of anything better. > > I

Re: AIO v2.0

2025-01-09 Thread Andres Freund
Hi, On 2025-01-09 20:10:24 +0200, Ants Aasma wrote: > On Thu, 9 Jan 2025 at 18:25, Andres Freund wrote: > > > I'm curious about this because the checksum code should be fast enough > > > to easily handle that throughput. > > > > It seems to top out at ab

Re: Eagerly scan all-visible pages to amortize aggressive vacuum

2025-01-09 Thread Andres Freund
n't think this makes it worse in any meaningful way. It's not really different from having other heap-specific params like freeze_min_age in VacuumParams. Greetings, Andres Freund

Re: AIO v2.0

2025-01-09 Thread Andres Freund
Hi, On 2025-01-09 10:59:22 +0200, Ants Aasma wrote: > On Wed, 8 Jan 2025 at 22:58, Andres Freund wrote: > > master: ~18 GB/s > > patch, buffered:~20 GB/s > > patch, direct, worker: ~28 GB/s > > patch, direct, uring: ~35 GB/s > > &g

Re: AIO v2.2

2025-01-08 Thread Andres Freund
Hi, On 2025-01-07 14:59:58 -0500, Robert Haas wrote: > On Tue, Jan 7, 2025 at 11:11 AM Andres Freund wrote: > > The difference between a handle and a reference is useful right now, to have > > some separation between the functions that can be called by anyone (taking a > >

Re: AIO v2.2

2025-01-08 Thread Andres Freund
Hi, On 2025-01-07 22:09:56 +0200, Heikki Linnakangas wrote: > On 07/01/2025 18:11, Andres Freund wrote: > > > I didn't quite understand the point of the prepare callbacks. For example, > > > when AsyncReadBuffers() calls smgrstartreadv(), the > > > shared_buff

Re: use a non-locking initial test in TAS_SPIN on AArch64

2025-01-08 Thread Andres Freund
Hi, On 2025-01-08 16:01:19 -0600, Nathan Bossart wrote: > On Wed, Jan 08, 2025 at 03:07:24PM -0500, Andres Freund wrote: > > Out of curiosity, have you measured whether this has a positive effect > > without > > pg_stat_statements? I think it'll e.g. also affect

Re: AIO v2.0

2025-01-08 Thread Andres Freund
Hi, On 2025-01-08 15:04:39 +0100, Jakub Wartak wrote: > On Mon, Jan 6, 2025 at 5:28 PM Andres Freund wrote: > > I didn't think that pg_stat_* was quite the right namespace, given that it > > shows not stats, but the currently ongoing IOs. I am going with pg_aios for

Re: use a non-locking initial test in TAS_SPIN on AArch64

2025-01-08 Thread Andres Freund
Hi, On 2025-01-08 12:12:19 -0600, Nathan Bossart wrote: > On Tue, Oct 22, 2024 at 02:54:57PM -0500, Nathan Bossart wrote: > > My colleague Salvatore Dipietro (CC'd) sent me a couple of profiles that > > showed an enormous amount of s_lock() time going to the > > __sync_lock_test_and_set() call in

Re: using PGOPTIONS in meson

2025-01-08 Thread Andres Freund
ooks at environment variables. Which means you just need to set an environment variable, which can be done without make: PGOPTIONS="..." meson test ... or export PGOPTIONS="..."; mesoin test ... Greetings, Andres Freund

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-08 Thread Andres Freund
Hi, On 2025-01-08 14:48:21 +, Bertrand Drouvot wrote: > On Wed, Jan 08, 2025 at 03:10:03PM +0200, Heikki Linnakangas wrote: > > On 08/01/2025 04:10, Andres Freund wrote: > > > 0002: Make logging of postmaster signalling child processes more > > > consistent &g

Re: Reorder shutdown sequence, to flush pgstats later

2025-01-08 Thread Andres Freund
Hi, On 2025-01-08 15:10:03 +0200, Heikki Linnakangas wrote: > On 08/01/2025 04:10, Andres Freund wrote: > > I instead opted to somewhat rearrange the shutdown sequence: > > - I don't love the naming of the various PMState values (existing and new), > >but a larger r

Reorder shutdown sequence, to flush pgstats later

2025-01-07 Thread Andres Freund
0004, the reason for this thread. Comments? Better ideas? Greetings, Andres Freund [1] A buffered write completes and RememberSyncRequest() fails, leading to the IO workers performing the flush itself. >From 5978a67e17b2750b766d959b4529c09a8d5b3023 Mon Sep 17 00:00:00 2001 From: And

Re: AIO v2.2

2025-01-07 Thread Andres Freund
{"io_method", PGC_POSTMASTER, RESOURCES_MEM, > > + gettext_noop("Selects the method of asynchronous I/O to > > use."), > > + NULL > > + }, > > + &io_method, > > + DEFAULT_IO_METHOD, io_method_options, > > + NULL, assign_io_method, NULL > > + }, > > + > > The description is a bit funny because synchronous I/O is one of the > possible methods. Hah. How about: "Selects the method of, potentially asynchronous, IO execution."? Greetings, Andres Freund

Re: AIO v2.2

2025-01-07 Thread Andres Freund
Hi, On 2025-01-07 17:09:58 +0200, Heikki Linnakangas wrote: > On 01/01/2025 06:03, Andres Freund wrote: > > Hi, > > > > Attached is a new version of the AIO patchset. > > I haven't gone through it all yet, but some comments below. Thanks! > > The bigg

Re: allow changing autovacuum_max_workers without restarting

2025-01-06 Thread Andres Freund
Hi, On January 6, 2025 5:15:25 PM EST, Nathan Bossart wrote: >On Mon, Jan 06, 2025 at 03:50:24PM -0600, Nathan Bossart wrote: >> I'm obviously biased, but I think it would be unfortunate to block features >> like this one because of low settings that would otherwise be unsuitable >> for any rea

Re: A few patches to clarify snapshot management

2025-01-06 Thread Andres Freund
mmand ids, but they're not suitable for general purpose queries. So I > think we should turn that into an error, per attached patch. Hm. I'm not sure it's a good idea to forbid this. Couldn't there be sane C code in an output functions calling GetTransactionSnapshot() or such to do some internal lookups? Greetings, Andres Freund

Re: AIO v2.2

2025-01-06 Thread Andres Freund
Hi, On 2025-01-06 10:52:20 -0800, Noah Misch wrote: > Patches 1 and 2 are still Ready for Committer. I feel somewhat weird about pushing 0002 without a user, but I guess it's still exercised, so it's probably fine... > On Tue, Dec 31, 2024 at 11:03:33PM -0500, Andres Freund

Re: AIO v2.0

2025-01-06 Thread Andres Freund
Hi, On 2024-12-19 17:29:12 -0500, Andres Freund wrote: > > Not about patch itself, but questions about related stack functionality: > > > > > > > > 7. Is pg_stat_

Re: Autovacuum giving up on tables after crash because of lack of stats

2025-01-03 Thread Andres Freund
ething here a few weeks ago, as well, and only flushing the stats > at redo was not enough while being full of holes. Good then that I was not at all proposing to only flush the stats at redo? Greetings, Andres Freund

Re: Autovacuum giving up on tables after crash because of lack of stats

2025-01-03 Thread Andres Freund
data size), and insert records associated to > their stats kind. As mentioned in the other thread, I don't think we need this - nearly all the information is already and in the WAL for the actual action that the stats track. We should re-emit the stats changes during recovery. Greetings, Andres Freund

Re: WAL-logging facility for pgstats kinds

2025-01-02 Thread Andres Freund
on the WAL record of the "underlying object". Do you have counter-examples? Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2025-01-02 Thread Andres Freund
Hi, On 2024-12-20 11:39:42 -0500, Andres Freund wrote: > On 2024-12-19 17:47:13 +1100, Michael Harris wrote: > > This is a different system to those I previously provided logs from. > > It is also running RHEL8 with a similar configuration to the other > > system. > &g

Re: FileFallocate misbehaving on XFS

2025-01-02 Thread Andres Freund
Hi, On 2025-01-02 11:41:56 +0100, Andrea Gelmini wrote: > Il giorno mar 31 dic 2024 alle ore 16:31 Andres Freund > ha scritto: > > 2024-12-19 04:47:04 CET [2646363]: ERROR: could not extend file > > "pg_tblspc/107724/PG_16_202307071/465960/3232056651.2" by 11 blo

Re: System views for versions reporting

2025-01-01 Thread Andres Freund
On 2025-01-02 10:36:48 +0800, jian he wrote: > https://cirrus-ci.com/github/postgresql-cfbot/postgresql/cf%2F5318 > shows lots of failures, but it doesn't seem to tell you about doc build > failure. It does: https://cirrus-ci.com/task/6472750665039872?logs=docs_build#L0 [15:26:26.443] time make -

Re: FileFallocate misbehaving on XFS

2024-12-31 Thread Andres Freund
en messages, whereas previous they were more frequent b) f_bfree increased substantially. I assume that somewhere around 2AM some script prunes old partitions? Greetings, Andres Freund

Re: AIX support

2024-12-30 Thread Andres Freund
Hi, On December 30, 2024 10:29:53 AM GMT+01:00, Peter Eisentraut wrote: >On 25.12.24 11:22, Heikki Linnakangas wrote: >> Thanks for the links. It's disappointing there isn't a standard way to do >> this. It's nice to see the comments in cpython's makeexp_aix script >> explaining the "hidden tr

Re: AIO v2.0

2024-12-20 Thread Andres Freund
Hi, On 2024-12-20 18:27:13 +0100, Jelte Fennema-Nio wrote: > On Fri, 20 Dec 2024 at 01:54, Andres Freund wrote: > > Arguably the configuration *did* tell us, by having a higher hard limit... > > > > But opting into a higher rlimit, while obviously adhering to the hard limit

Re: FileFallocate misbehaving on XFS

2024-12-20 Thread Andres Freund
ould probably reduce fragmentation, and the extra space would be quickly used in workloads where we extend by a bunch of blocks at once, anyway. Greetings, Andres Freund

Re: AIO v2.0

2024-12-19 Thread Andres Freund
Hi, On 2024-12-19 17:34:29 -0500, Tom Lane wrote: > Andres Freund writes: > > My current thoughts around this are that we should generally, independent of > > io_uring, increase the FD limit ourselves. > > I'm seriously down on that, because it amounts to an assumption

Re: AIO v2.0

2024-12-19 Thread Andres Freund
Hi, Sorry for loosing track of your message for this long, I saw it just now because I was working on posting a new version. On 2024-11-18 13:19:58 +0100, Jakub Wartak wrote: > On Fri, Sep 6, 2024 at 9:38 PM Andres Freund wrote: > Thank You for worth admiring persistence on this. Ple

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2024-12-19 Thread Andres Freund
On 2024-12-19 09:31:14 -0600, Nathan Bossart wrote: > On Thu, Dec 19, 2024 at 02:44:53PM +0900, Michael Paquier wrote: > > I've been double-checking the code to refresh myself with the problem, > > and I don't see a reason to not apply something like the attached set > > down to v13 for all these r

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2024-12-18 Thread Andres Freund
ink() - but at the end of a checkpoint we do that operations hundreds of times in a row on a busy server. And that's just after potentially doing lots of write IO during a checkpoint, filling up drive write caches / eating up IOPS/bandwidth disk quots. Greetings, Andres Freund

Re: Regression tests fail on OpenBSD due to low semmns value

2024-12-18 Thread Andres Freund
Hi, On 2024-12-18 12:00:48 -0500, Tom Lane wrote: > Andres Freund writes: > > Maybe we should consider switching those platforms to unnamed posix > > semaphores? > > I already looked into that. OpenBSD still doesn't have cross-process > posix semaphores, at le

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-18 Thread Andres Freund
Hi, On 2024-12-18 15:53:39 +, Bertrand Drouvot wrote: > On Wed, Dec 18, 2024 at 10:35:45AM -0500, Andres Freund wrote: > > Hi, > > > > On 2024-12-18 15:14:07 +, Bertrand Drouvot wrote: > > > While working on [1], it has been noticed that pgstat_flush_io()

Re: Regression tests fail on OpenBSD due to low semmns value

2024-12-18 Thread Andres Freund
id/3010886.1634950831%40sss.pgh.pa.us * netbsd, 2022: https://www.postgresql.org/message-id/20220828013914.5hzc7kvcpum5h2yn%40awork3.anarazel.de But TBH, nobody uses openbsd and netbsd if performance matters even one iota. And considering a bunch of postgres changes to deal with idiotic default sysv limits doesn't feal like a sensible thing to do in 2024. Greetings, Andres Freund

Re: Removing the pgstat_flush_io() call from the walwriter

2024-12-18 Thread Andres Freund
$SUBJECT, until the > walwriter is part of the I/O stats tracking system. I don't really see the point of this change? What do we gain by moving stuff around like you did? Greetings, Andres Freund

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Andres Freund
Hi, On 2024-12-17 17:07:34 -0500, Peter Geoghegan wrote: > On Tue, Dec 17, 2024 at 4:46 PM Andres Freund wrote: > > ISTM that we could do better with some fairly simple cooperation between > > index > > and table AM. It should be rather rare to look up a TID that was remove

Re: Crash: invalid DSA memory alloc request

2024-12-17 Thread Andres Freund
On 2024-12-17 15:32:06 -0600, Nathan Bossart wrote: > Committed. Thanks!

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Andres Freund
x27;ve seen enough bugs / corruption leading to indexes pointing to wrong and nonexisting tuples to make me think it's worth being a bit more proactive about raising errors for such cases. Of course what I described above can only detect index entries pointing to non-existing tuples, but that's still a good bit better than nothing. Greetings, Andres Freund

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-17 Thread Andres Freund
Hi, On 2024-12-17 19:57:13 +0200, Heikki Linnakangas wrote: > On 14/12/2024 01:44, Andres Freund wrote: > > The zero_damaged_pages path in bufmgr.c makes sense to me, but this one > > seems > > less sane to me. If you want to recover from a data corruption event and &g

Re: Adding NetBSD and OpenBSD to Postgres CI

2024-12-17 Thread Andres Freund
Hi, On 2024-12-17 14:25:01 -0500, Andres Freund wrote: > On 2024-12-17 13:09:49 -0500, Tom Lane wrote: > > I wrote: > > > Andres Freund writes: > > >> Was about to apply that, but then started to wonder if we don't need the > > >> same > >

Re: Adding NetBSD and OpenBSD to Postgres CI

2024-12-17 Thread Andres Freund
Hi, On 2024-12-17 13:09:49 -0500, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> Was about to apply that, but then started to wonder if we don't need the > >> same > >> for configure? And it sure looks like that has the same problem? > &

Re: Adding NetBSD and OpenBSD to Postgres CI

2024-12-17 Thread Andres Freund
Hi, On 2024-12-17 12:44:46 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2024-11-12 11:38:11 +0300, Nazir Bilal Yavuz wrote: > >> It was because initdb was failing on NetBSD when the LANG and LC_ALL > >> is not set to C. I rephrased the comment and moved

Re: Adding NetBSD and OpenBSD to Postgres CI

2024-12-17 Thread Andres Freund
.*\nci-os-only:[^\n]*netbsd.*' ... > +- name: OpenBSD - 7 - Meson > + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || > $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*openbsd.*' Think these probably should be added to src/tools/ci/README Greetings, Andres Freund

Re: Adding NetBSD and OpenBSD to Postgres CI

2024-12-17 Thread Andres Freund
Hi, On 2024-11-12 11:38:11 +0300, Nazir Bilal Yavuz wrote: > On Fri, 1 Nov 2024 at 21:44, Andres Freund wrote: > > > +CCACHE_DIR: /tmp/ccache_dir > > > + > > > +PATH: /usr/sbin:$PATH > > > + > > > +# Postgres interprets LANG as a &#x

Re: Crash: invalid DSA memory alloc request

2024-12-17 Thread Andres Freund
s alone, and if you enable e.g. function execution stats it can reasonably reach higher numbers more quickly. And use DSA_ALLOC_HUGE in that place feels like a rather low risk change? Greetings, Andres Freund

Re: Regression tests fail on OpenBSD due to low semmns value

2024-12-16 Thread Andres Freund
_senders alongside lower max_connections settings. It seems to make sense to have a lower max_wal_senders settings on machines that don't have enough resources to run with max_connections=100. Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-16 Thread Andres Freund
Hi, On 2024-12-16 18:05:59 +0100, Alvaro Herrera wrote: > On 2024-Dec-16, Robert Haas wrote: > > > On Mon, Dec 16, 2024 at 9:12 AM Andres Freund wrote: > > > Personally I don't like the obfuscation of "allocate" and "zero" vs just > > > n

Standardizing the file header?

2024-12-16 Thread Andres Freund
ike it'd be good to define one pattern as canonical so one doesn't have to guess what pattern one should follow when creating a new file. Thoughts? Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-16 Thread Andres Freund
Hi, On 2024-12-16 14:45:37 +0100, Jakub Wartak wrote: > On Thu, Dec 12, 2024 at 12:50 AM Andres Freund wrote: > An extended cycle of 80 backends copying into relations and occasionally > > truncating them (to simulate the partitions being dropped and new ones > > created). Fo

Re: FileFallocate misbehaving on XFS

2024-12-16 Thread Andres Freund
Hi, On 2024-12-14 09:29:12 +0100, Alvaro Herrera wrote: > On 2024-Dec-11, Andres Freund wrote: > > > One thing that I think we should definitely do is to include more detail in > > the error message. mdzeroextend()'s error messages don't include how many >

Re: pg_attribute_noreturn(), MSVC, C11

2024-12-15 Thread Andres Freund
Hi, On 2024-12-14 18:18:35 +0100, Peter Eisentraut wrote: > On 13.12.24 20:54, Andres Freund wrote: > > Another wrinkle: While __attribute__((noreturn)) works for function pointers > > (or function pointer typedefs) _Noreturn doesn't. Gah. We only use it that > > wa

Re: pg_attribute_noreturn(), MSVC, C11

2024-12-15 Thread Andres Freund
Hi, On 2024-12-14 18:15:13 +0100, Peter Eisentraut wrote: > On 13.12.24 20:10, Andres Freund wrote: > > (or alternatively include stdreturn if __STDC_VERSION__ indicates support > > and > > define a bare 'noreturn' if not) > > > > For msvc that mean

Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

2024-12-14 Thread Andres Freund
Hi, On 2024-12-14 12:08:57 +0700, John Naylor wrote: > On Thu, Jun 13, 2024 at 2:37 AM Andres Freund wrote: > > > > It's hard to understand, but a nonetheless helpful page is > > https://users.ece.cmu.edu/~koopman/crc/crc32.html which lists properties

Re: Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-13 Thread Andres Freund
Hi, On 2024-12-13 19:06:16 -0500, Tom Lane wrote: > Andres Freund writes: > > We have a fair number of special paths in md.c that are specific to > > recovery. E.g. in mdreadv() we do: > > ... > > As far as I can tell, nearly all - including the above - InRecovery path

Re: Hot standby queries see transient all-zeros pages

2024-12-13 Thread Andres Freund
Hi, On 2024-12-14 13:41:40 +1300, Thomas Munro wrote: > On Sat, Dec 14, 2024 at 11:41 AM Andres Freund wrote: > > On 2024-05-12 10:16:58 -0700, Noah Misch wrote: > > > The use of RBM_ZERO_AND_LOCK is incompatible with that. See a similar > > > argument at https://po

Re: Hot standby queries see transient all-zeros pages

2024-12-13 Thread Andres Freund
Hi, On 2024-12-13 16:38:05 -0800, Noah Misch wrote: > On Fri, Dec 13, 2024 at 05:41:15PM -0500, Andres Freund wrote: > > Afaics we didn't do anything about this issue? > > postgr.es/c/e656657 fixed this. I thought this was longstanding, but it > turned out to have sta

Exceptional md.c paths for recovery and zero_damaged_pages

2024-12-13 Thread Andres Freund
to use zero_damaged_pages, because that's almost a guarantee for worsening corruption in the future, because the now empty heap page will eventually be filled with new tuples, which now will be pointed to by index entries pointing that were created before the zeroing. So what's the point

Re: Hot standby queries see transient all-zeros pages

2024-12-13 Thread Andres Freund
CK_ZERO_ON_MISS? RBM_DWIM? > Like RBM_ZERO_AND_LOCK, it avoids wasting disk reads > on data we discard. Are there other strategies to consider? I guess we could just use RBM_ZERO_AND_CLEANUP_LOCK for this path. But that seems like it has too many obvious downsides. Greetings, Andres Freund

Re: checkpointer: PANIC: could not fsync file: No such file or directory

2024-12-13 Thread Andres Freund
Hi, On 2019-12-13 17:41:56 +1300, Thomas Munro wrote: > From 9609c9a153232fb2de169bf76158781d354c633b Mon Sep 17 00:00:00 2001 > From: Thomas Munro > Date: Fri, 13 Dec 2019 17:12:42 +1300 > Subject: [PATCH] Don't use _mdfd_getseg() in mdsyncfiletag(). > > _mdfd_getseg() opens all segments up to

Re: pg_attribute_noreturn(), MSVC, C11

2024-12-13 Thread Andres Freund
Hi, On 2024-12-13 14:10:13 -0500, Andres Freund wrote: > I just encountered another > warning C4715: 'XYZ: not all control paths return a value > > with msvc in CI in a case where it should be trivial for the compiler to > recognize that the function return isn't reac

pg_attribute_noreturn(), MSVC, C11

2024-12-13 Thread Andres Freund
it, but not as clearly as before. For an example of _Noreturn being used: https://godbolt.org/z/j15d35dao Greetings, Andres Freund [1] The msvc implementation of VA_ARGS_NARGS only works with the traditional preprocessor, but C11 support enables the new one. But we can detect tha

Re: FileFallocate misbehaving on XFS

2024-12-12 Thread Andres Freund
Hi, On 2024-12-12 14:14:20 +1100, Michael Harris wrote: > On Thu, 12 Dec 2024 at 10:50, Andres Freund wrote: > > Just to make sure - you're absolutely certain that you actually have space > > at > > the time of the errors? > > As sure as I can be. The RHEL

Re: TransactionXmin != MyProc->xmin

2024-12-12 Thread Andres Freund
t relevant rows being removed. > In any case, I propose that we set TransactionXmin in all of those cases as > well, so that TransactionXmin is always the equal to MyProc->xmin. Maybe > even rename it to MyProcXmin to make that more clear. I'm not sure it's really right to do that. If we don't hold a snapshot, what makes sure that we then call SnapshotResetXmin() or such to reset TransactionXmin? Greetings, Andres Freund

Re: Add Postgres module info

2024-12-12 Thread Andres Freund
o, maybe it makes sense to design this feature with re-fetching libraries, > supposing it is already implemented somehow and elements of the > DynamicFileList may be removed. I am quite certain we'll not support unloading libraries anytime soon. We used to support it and it caused problems... Changing anything about how exactly things are tracked in dfmgr.c will be the smallest part of supporting unloading libraries again. Greetings, Andres Freund

Re: Proposal for Updating CRC32C with AVX-512 Algorithm.

2024-12-12 Thread Andres Freund
Hi, On 2024-12-12 18:32:20 +0700, John Naylor wrote: > I went and looked at the Chromium source, and found the following > snippet that uses the same technique, but only requires 128-bit CLMUL > and has a minimum input size of 64 bytes, rather than 256. This seems > like it might be better suited

Re: FileFallocate misbehaving on XFS

2024-12-11 Thread Andres Freund
9 is the initial relfilenode for pg_attribute. There could also be some parallelism leading to bulk extension, due to the parallel restore. I don't remember which commands pg_restore actually executes in parallel. Greetings, Andres Freund

Re: Add Postgres module info

2024-12-11 Thread Andres Freund
e except #4, authors who want their source code to > support multiple PG major versions would be forced into using #if > tests on CATALOG_VERSION_NO to decide what to write. That's a > bit annoying but hardly unusual. #2 would be bit more annoying than #1, I'd say, because it'd affect every single extension, even ones not interested in any of this. Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-11 Thread Andres Freund
Hi, On 2024-12-10 16:33:06 -0500, Andres Freund wrote: > Maybe. I think we would have gotten a lot more reports if it were common. I > know of quite a few very busy installs using xfs. > > I think there must be some as-of-yet-unknown condition gating it. E.g. that > the file

Re: FileFallocate misbehaving on XFS

2024-12-10 Thread Andres Freund
Hi, On 2024-12-10 12:36:40 -0500, Robert Haas wrote: > On Mon, Dec 9, 2024 at 7:31 PM Andres Freund wrote: > > Pretty unexcited about all of these - XFS is fairly widely used for PG, but > > this problem doesn't seem very common. It seems to me that we're missing >

Re: FileFallocate misbehaving on XFS

2024-12-10 Thread Andres Freund
c/postgresql$ git grep CATALOG_VERSION_NO upstream/REL_16_STABLE src/include/catalog/catversion.h upstream/REL_16_STABLE:src/include/catalog/catversion.h:#define CATALOG_VERSION_NO 202307071 Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-10 Thread Andres Freund
On 2024-12-10 11:34:15 -0500, Andres Freund wrote: > On 2024-12-10 12:36:33 +0100, Jakub Wartak wrote: > > On Tue, Dec 10, 2024 at 7:34 AM Michael Harris wrote: > > 2. > > > > > # xfs_info /dev/mapper/ippvg-ipplv > > > meta-data=/dev/mapper/ippvg-i

Re: FileFallocate misbehaving on XFS

2024-12-10 Thread Andres Freund
Hi, On 2024-12-10 17:28:21 +1100, Michael Harris wrote: > On Tue, 10 Dec 2024 at 11:31, Andres Freund wrote: > > It'd be useful to get the xfs_info output that Jakub asked for. Perhaps also > > xfs_spaceman -c 'freesp -s' /mountpoint > > xfs_spaceman -c &#

Re: FileFallocate misbehaving on XFS

2024-12-09 Thread Andres Freund
us too much on the fact that > we first encounter it during the upgrade. I assume the file that actually errors out changes over time? It's always fallocate() that fails? Can you tell us anything about the workload / data? Lots of tiny tables, lots of big tables, write heavy, ...? Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-09 Thread Andres Freund
Hi, On 2024-12-10 09:34:08 +1100, Michael Harris wrote: > On Tue, 10 Dec 2024 at 03:31, Andres Freund wrote: > > I found some references for bugs that were fixed in 5.13. But I think at > > least > > some of this would persist if the filesystem ran into the issue with a

Re: Unmark gen_random_uuid() function leakproof

2024-12-09 Thread Andres Freund
e sense to add a test to opr_sanity.sql so we don't reintroduce such cases? Greetings, Andres Freund

Managing IO workers in postmaster's state machine

2024-12-09 Thread Andres Freund
ive_mode being PGC_POSTMASTER - but of course that's a different scale. I guess this could be argued to better be in a separate thread... Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-09 Thread Andres Freund
gt; https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1791323 Doubt it, we never do this as far as I am aware. Greetings, Andres Freund

Re: FileFallocate misbehaving on XFS

2024-12-09 Thread Andres Freund
[range] and compute how much free space you will need allocate and then do the a second pass through the file, because the file layout might have changed concurrently... This issue seems independent of the issue Michael is having though. Postgres, afaik, won't fallocate huge ranges with already allocated space. Greetings, Andres Freund

Re: meson missing test dependencies

2024-12-09 Thread Andres Freund
Hi, On 2024-12-09 16:06:18 +0100, Peter Eisentraut wrote: > On 07.12.24 21:45, Andres Freund wrote: > > If a test target does not have ay dependency 'meson test' treats it has > > having > > a dependency on the project default test. Which in turn currently inclu

Re: Do not scan index in right table if condition for left join evaluates to false using columns in left table

2024-12-08 Thread Andres Freund
On 2024-12-08 15:44:23 +0700, Andrei Lepikhov wrote: > On 8/12/2024 09:52, Andres Freund wrote: > > > I think avoiding touching a hash table and an index under MergeJoin can > > > also > > > be beneficial. > > > > How would you get significant w

Re: Do not scan index in right table if condition for left join evaluates to false using columns in left table

2024-12-07 Thread Andres Freund
gh both inner and outer anyway? Greetings, Andres Freund

  1   2   3   4   5   6   7   8   9   10   >