Hi,
On 2025-04-15 21:00:00 +0300, Alexander Lakhin wrote:
> Please take a look also at the simple reproducer for the crash inside
> pg_get_aios() I mentioned upthread:
> for i in {1..100}; do
> numjobs=12
> echo "iteration $i"
> date
> for ((j=1;j<=numjobs;j++)); do
> ( createdb db$j;
Hello Andres,
24.04.2025 03:40, Andres Freund wrote:
Hi,
On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
2025-04-08 01:41:54.670 UTC [4013251][client backend][0/2:0] DEBUG: waiting for
self with 0 pending
I'd like to extend this debug message with the number of in-flight IOs. I
assum
Hi,
On 2025-04-20 18:00:00 +0300, Alexander Lakhin wrote:
> 31.03.2025 02:46, Andres Freund wrote:
> > I've pushed most of these after some very light further editing. Yay.
> > Thanks
> > a lot for all the reviews!
> >
> > So far the buildfarm hasn't been complaining, but it's early days.
>
>
On 3/18/25 21:12, Andres Freund wrote:
> Hi,
>
> Attached is v2.10, with the following changes:
>
> ...
>
> - committed io_method=worker
>
There's an open item related to this commit (247ce06b883d), based on:
For now the default io_method is changed to "worker". We should re-
evaluat
Hello Andres,
31.03.2025 02:46, Andres Freund wrote:
I've pushed most of these after some very light further editing. Yay. Thanks
a lot for all the reviews!
So far the buildfarm hasn't been complaining, but it's early days.
I found one complaint against commit 12ce89fd0, expressed as:
https
Hello Andres,
14.04.2025 19:06, Andres Freund wrote:
Unfortunately I'm several hundred iterations in, without reproducing the
issue. I'm bad at statistics, but I think that makes it rather unlikely that I
will, without changing some aspect.
Was this an assert enabled build? What compiler and wh
Hi,
On 2025-04-13 09:00:01 +0300, Alexander Lakhin wrote:
> 07.04.2025 22:10, Alexander Lakhin wrote:
> > > I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor
> > > on
> > > tmpfs.
> >
> > Before sharing the script I tested it on two my machines, but I had
> > anticipated
Hello Andres,
07.04.2025 22:10, Alexander Lakhin wrote:
I ran it for a while in a VM, it hasn't triggered yet. Neither on xfs nor on
tmpfs.
Before sharing the script I tested it on two my machines, but I had
anticipated that the error can be hard to reproduce. Will try to reduce
the reproducer
Hi,
On 2025-04-06 23:00:00 +0300, Alexander Lakhin wrote:
> 02.04.2025 14:58, Andres Freund wrote:
> When running multiple installcheck's against a single server (please find
> the ready-to-use script attached (I use more sophisticated version with
> additional patches to make installcheck pass cl
Hi,
On 2025-04-05 06:43:52 -0700, Noah Misch wrote:
> Yeah. Maybe this (untested):
Something like that works. I adopted your formulation of this, mine was in
GetLocalVictimBuffer(), which seems slightly less future proof.
> > > If that's right, it would still be nice to reach the right
> > >
Hello Andres,
07.04.2025 19:20, Andres Freund wrote:
iteration 8: Sun Apr 6 19:22:39 UTC 2025
installchecks finished: Sun Apr 6 19:23:47 UTC 2025
2025-04-06 19:22:44.216 UTC [349525] LOG: could not read blocks 0..0 in file
"base/6179194/2606": Operation canceled
2025-04-06 19:22:44.216 UTC [
Hello Andres,
02.04.2025 14:58, Andres Freund wrote:
Hi,
I've pushed fixes for 1) and 2) and am working on 3).
When running multiple installcheck's against a single server (please find
the ready-to-use script attached (I use more sophisticated version with
additional patches to make installch
Hi,
On 2025-04-03 16:16:50 -0300, Ranier Vilela wrote:
> Em qui., 3 de abr. de 2025 às 15:35, Andres Freund
> escreveu:> > On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > > Em qua., 2 de abr. de 2025 às 08:58, Andres Freund
> > > escreveu:
> > >
> > > > Hi,
> > > >
> > > > I've pushed fix
Hi,
On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> There are three different types of failures in the test_aio test so far:
And a fourth, visible after I enabled liburing support for skink.
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2025-04-03%2007%3A06%3A19&stg
On Sun, Mar 23, 2025 at 11:57:48AM -0400, Andres Freund wrote:
> On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Attached v2.11
> >
> > > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
> > > --- a/src/backend/utils/
On Mon, Mar 24, 2025 at 11:43:47AM -0400, Andres Freund wrote:
> On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> > commit 247ce06b wrote:
> > > + pgaio_io_reopen(ioh);
> > > +
> > > + /*
> > > + * To be able to exercise the reopen-fails path, allow
Hi,
I've pushed fixes for 1) and 2) and am working on 3).
On 2025-04-01 17:13:24 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> > On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and
> > > CATCA
On Thu, Mar 20, 2025 at 02:54:14PM -0400, Andres Freund wrote:
> On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> > On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > I see this relies on md_readv_complete having converted "
Hi,
On 2025-03-24 11:43:47 -0400, Andres Freund wrote:
> > I didn't write a test to prove it, but I'm suspecting we'll reach the above
> > ERROR with this sequence:
> >
> > CREATE TEMP TABLE foo ...;
> > [some command that starts reading a block of foo into local buffers, then
> > ERROR with
Hi,
On 2025-03-19 13:20:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund wrote:
> >
> > Attached is v2.10,
>
> I noticed a few comments could be improved in 0011: bufmgr: Use AIO
> in StartReadBuffers()
> [...]
Yep.
> Above and in AsyncReadBuffers()
>
> * T
On Fri, Apr 04, 2025 at 11:53:13PM -0400, Andres Freund wrote:
> On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> > On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > > > - Make DEFINED in completor before verifying page. It might be
> > > > cleaner to
> > > > do this when
Hi,
On 2025-04-04 14:18:02 -0700, Noah Misch wrote:
> On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> > > - Make DEFINED in completor before verifying page. It might be cleaner
> > > to
> > > do this when the completor first retrieves a return value from
> > > io_uring,
>
On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote:
> On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> > On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> > In the general case, we could want client requests as follows:
> >
> > - If completor==definer and has not droppe
Hi,
Sorry for the slow work on this. The cycle times are humonguous due to
valgrind being so slow...
On 2025-04-03 12:40:23 -0700, Noah Misch wrote:
> On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> > The best fix for that one would, I think, be to have method_io_uring()
> > it
Hi,
On 2025-04-01 09:07:27 -0700, Noah Misch wrote:
> On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> > WRT the locking issues, I've been wondering whether we could make
> > LWLockWaitForVar() work that purpose, but I doubt it's the right approach.
> > Probably better to get rid o
On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote:
> 4b)
>
> That's not all though, after getting past this failure, I see uninitialized
> memory errors for reads into temporary buffers:
>
> ==3334031== VALGRINDERROR-BEGIN
> ==3334031== Conditional jump or move depends on uninitialise
Em qui., 3 de abr. de 2025 às 15:35, Andres Freund
escreveu:
> Hi,
>
> On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> > Em qua., 2 de abr. de 2025 às 08:58, Andres Freund
> > escreveu:
> >
> > > Hi,
> > >
> > > I've pushed fixes for 1) and 2) and am working on 3).
> > >
> > Coverity has on
Hi,
On 2025-04-03 13:46:39 -0300, Ranier Vilela wrote:
> Em qua., 2 de abr. de 2025 às 08:58, Andres Freund
> escreveu:
>
> > Hi,
> >
> > I've pushed fixes for 1) and 2) and am working on 3).
> >
> Coverity has one report about this.
>
> CID 1596092: (#1 of 1): Uninitialized scalar variable (UN
Hi.
Em qua., 2 de abr. de 2025 às 08:58, Andres Freund
escreveu:
> Hi,
>
> I've pushed fixes for 1) and 2) and am working on 3).
>
Coverity has one report about this.
CID 1596092: (#1 of 1): Uninitialized scalar variable (UNINIT)
13. uninit_use_in_call: Using uninitialized value result_one. Fie
On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote:
> On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE
> > are defined:
> >
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2
Hi,
On 2025-04-01 17:47:51 -0400, Andres Freund wrote:
> 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE
> are defined:
>
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2019%3A23%3A07
>
> # +++ tap check in src/test/modules/test_aio +++
Hi,
On 2025-04-01 11:55:20 -0400, Andres Freund wrote:
> I haven't yet pushed the changes, but will work on that in the afternoon.
There are three different types of failures in the test_aio test so far:
1) TEMP_CONFIG
See
https://postgr.es/m/zh5u22wbpcyfw2ddl3lsvmsxf4yvsrvgxqwwmfjddc4c2khsgp%
Hi Andres,
> > I didn't yet push
> >
> > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher level
> > > > design
I have several notes about 0003 / README.md:
1. I noticed that the use of "Postgres" and "postgres" is inconsistent.
2.
```
+pgaio_io_register_callbacks(ioh, PGA
On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote:
> On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> > On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> I haven't yet pushed the changes, but will work on that in the afternoon.
>
> I plan to afterwards close the CF ent
Hi,
On 2025-04-01 08:11:59 -0700, Noah Misch wrote:
> On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> > updated version
>
> All non-write patches (1-7) are ready for commit, though I have some cosmetic
> recommendations below. I've marked the commitfest entry Ready for Committer.
On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote:
> updated version
All non-write patches (1-7) are ready for commit, though I have some cosmetic
recommendations below. I've marked the commitfest entry Ready for Committer.
> + # Check a page validity error in another blo
Hi,
On 2025-04-01 14:56:17 +0300, Aleksander Alekseev wrote:
> Hi Andres,
>
> > > I didn't yet push
> > >
> > > > > Subject: [PATCH v2.14 13/29] aio: Add README.md explaining higher
> > > > > level design
>
> I have several notes about 0003 / README.md:
>
> 1. I noticed that the use of "Postgres"
Hi,
On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > Subject: [PATCH v2.14 11/29] Let caller of PageIsVerified() control
> > ignore_checksum_failure
> > Subject: [PATCH v2.14 12/29] bufmgr: Use AIO in StartReadBuffers()
> > Subject: [PATCH v2.14 14/29] aio: Basic read_stream adjustments for re
Hi,
On 2025-03-27 10:52:10 +1300, Thomas Munro wrote:
> On Thu, Mar 27, 2025 at 10:41 AM Andres Freund wrote:
> > > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> > >
> > > Consider also updating this comment to stop focusing on prefetch; I think
> > > changing that aligns
On Tue, Mar 25, 2025 at 11:58 AM Andres Freund wrote:
>
> > Another thought on complete_shared running on other backends: I wonder if we
> > should push an ErrorContextCallback that adds "CONTEXT: completing I/O of
> > other process" or similar, so people wonder less about how "SELECT FROM a"
> >
On Sat, Mar 29, 2025 at 08:39:54PM -0400, Andres Freund wrote:
> On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> > On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> > The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT.
> > If a client has decided to set cl
Hi,
On 2025-03-29 14:29:29 -0700, Noah Misch wrote:
> Flushing half-baked review comments before going offline for a few hours:
>
> On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> > Attached v2.13, with the following changes:
>
> > 5) The WARNING in the callback is now a LOG, as
Flushing half-baked review comments before going offline for a few hours:
On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote:
> Attached v2.13, with the following changes:
> 5) The WARNING in the callback is now a LOG, as it will be sent to the
> client as a WARNING explicitly w
Hi,
On 2025-03-29 10:48:10 -0400, Andres Freund wrote:
> Attached is v2.14:
FWIW, there was a last minute change in the test that fails in one task on CI,
due to reading across the smaller segment size configured for one of the
runs. Doesn't quite seem worth posting a new version for.
> - push
On Sat, Mar 29, 2025 at 2:25 PM Andres Freund wrote:
>
> I think I found an issue with this one - as it stands the view was viewable by
> everyone. While it doesn't provide a *lot* of insight, it still seems a bit
> too much for an unprivileged user to learn what part of a relation any other
> use
On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote:
> The number of combinations is annoyingly large. It's e.g. plausible to use
> ignore_checksum_failure=on and zero_damaged_pages=on at the same time for
> recovery.
That's intricate indeed.
> But I finally got to a point where the cod
Hi,
On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> > On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > > - We need to register a local callback for shared buffer reads, which
> > > don't
> > > need them today . That's a sm
Hi,
On 2025-03-28 08:54:42 -0400, Andres Freund wrote:
> One simplification that we could make is to only ever report one checksum
> failure for each IO, even if N buffers failed - after all that's what HEAD
> does (by virtue of throwing an error after the first). Then we'd not track the
> number
Hi,
On 2025-03-27 20:22:23 -0700, Noah Misch wrote:
> On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> > Don't report the error in the completion callback. The obvious place would
> > be
> > to do it where we we'll raise the warning/error in the issuing process. The
> > big disa
On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote:
> I now wrote some tests. And I both regret doing so (because it found problems,
> which would have been apparent long ago, if the feature had come with *any*
> tests, if I had gone the same way I could have just pushed stuff) and am gl
Hi,
On 2025-03-26 21:07:40 -0400, Andres Freund wrote:
> TODO
> ...
> - Add an explicit test for the checksum verification in the completion
> callback
>
> There is an existing test for testing an invalid page due to page header
> verification in test_aio, but not for checksum failures.
>
>
On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> On 2025-03-11 20:57:43 -0700, Noah Misch wrote:
> > - Like you say, "redefine max_files_per_process to be about the number of
> > files each *backend* will additionally open". It will become normal that
> > each backend's actual
On Thu, Mar 27, 2025 at 10:41 AM Andres Freund wrote:
> > > Subject: [PATCH v2.12 13/28] Enable IO concurrency on all systems
> >
> > Consider also updating this comment to stop focusing on prefetch; I think
> > changing that aligns with the patch's other changes:
> >
> > /*
> > * How many buffer
Hi,
On 2025-03-26 21:20:47 +, Thom Brown wrote:
> I took a quick gander through this just out of curiosity (yes, I know
> I'm late), and found these show-stoppers:
>
> v2.12-0015-aio-Add-pg_aios-view.patch:
>
> + ERROR mean the I/O failed with an error.
>
> s/mean/means/
>
>
> v2
Hi,
On 2025-03-26 11:31:02 -0700, Noah Misch wrote:
> I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
> Comment fixes", the last patch before write support.
Thanks!
> postgr.es/m/20250326001915.bc.nmi...@google.com covered patches 1-9, and this
> email covers patches
On Wed, Mar 26, 2025 at 04:33:49PM -0400, Andres Freund wrote:
> On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > Second, the aio_internal.h comment changes discussed in
> > postgr.es/m/20250325155808.f7.nmi...@google.com and ea
On Tue, 25 Mar 2025 at 01:18, Andres Freund wrote:
>
> Hi,
>
> Attached v2.12, with the following changes:
I took a quick gander through this just out of curiosity (yes, I know
I'm late), and found these show-stoppers:
v2.12-0015-aio-Add-pg_aios-view.patch:
+ ERROR mean the I/O failed
Hi,
On 2025-03-25 17:19:15 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> > @@ -296,7 +299,9 @@ pgaio_io_call_complete_local(PgAioHandle *ioh)
> >
> > /*
> > * Note that we don't save the result in ioh->distilled_result, the
> > local
> > -
I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr:
Comment fixes", the last patch before write support.
postgr.es/m/20250326001915.bc.nmi...@google.com covered patches 1-9, and this
email covers patches 10-17. All remaining review comments are minor, so I've
marked the com
Hi,
On 2025-03-22 19:09:55 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Attached v2.11
>
> > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
>
> Apart from some isolated cosmetic points, this is ready to commit:
>
> > + er
Hi,
On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > I don't know if that's an intentional or unintentional behavioral
> > difference.
> >
> > There are 2 1/2 ways around this:
> >
> > 1) Stop using IOSQE_ASYNC heuristic
> > 2a
Hi,
On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > FWIW, I prototyped this, it's not hard.
> >
> > But it can't replace the current WARNING with 100% fidelity: If we read 60
> > blocks in a single smgrreadv, we today would woul
On Tue, Mar 25, 2025 at 08:17:17PM -0400, Andres Freund wrote:
> On 2025-03-25 09:15:43 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> > > FWIW, I prototyped this, it's not hard.
> > >
> > > But it can't replace the current WARNING with 100% fidelity:
On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote:
> Attached v2.12, with the following changes:
> TODO:
> Wonder if it's worth adding some coverage for when checksums are disabled?
> Probably not necessary?
Probably not necessary, agreed. Orthogonal to AIO, it's likely worth a
On Tue, Mar 25, 2025 at 04:56:53PM -0400, Andres Freund wrote:
> The repeated-iteration approach taken in pgaio_closing_fd() isn't the
> prettiest, but it's hard to to imagine that ever being a noticeable.
Yep. I've reviewed the fixup code, and it looks all good.
> This survives a testrun where
Hi,
On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> While having nagging thoughts that we might be releasing FDs before io_uring
> gets them into kernel custody, I tried this hack to maximize FD turnover:
>
> static void
> ReleaseLruFiles(void)
> {
> #if 0
> while (nfile + numAllocatedDes
Hi,
On 2025-03-25 13:18:50 -0700, Noah Misch wrote:
> On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> > On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > > There are 2 1/2 ways around this:
> > > >
> > > > 1
On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote:
> On 2025-03-25 12:39:56 -0700, Noah Misch wrote:
> > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> > > There are 2 1/2 ways around this:
> > >
> > > 1) Stop using IOSQE_ASYNC heuristic
> > > 2a) Wait for all in-flig
On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote:
> On 2025-03-25 08:58:08 -0700, Noah Misch wrote:
> > While having nagging thoughts that we might be releasing FDs before io_uring
> > gets them into kernel custody, I tried this hack to maximize FD turnover:
> >
> > static void
> > Re
On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote:
> On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > > If we want to implement it, I think we could introduce PGAIO_RS_WARN,
> > > which
> > > then could tell the stag
Hi,
On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > (We may be due for a test mode that does smgrreleaseall() at every
> > > CHECK_FOR_INTERRUPTS()?)
> >
> > I suspect we are.
On Tue, Mar 25, 2025 at 11:26:14AM -0400, Andres Freund wrote:
> On 2025-03-25 06:33:21 -0700, Noah Misch wrote:
> > On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> > > On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > > > (We may be due for a test mode that does smgrreleaseall()
Hi,
On 2025-03-25 07:11:20 -0700, Noah Misch wrote:
> On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> > Is it actually sane to use WARNING here? At least for ZERO_ON_ERROR that
> > could
> > trigger a rather massive flood of messages to the client in a *normal*
> > situation. I'm
Hi,
On 2025-03-25 17:10:19 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 2:18 PM Andres Freund wrote:
> > Attached v2.12, with the following changes:
>
> Here's a tiny fixup to make io_concurrency=0 turn on
> READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME. Without this, AIO
> will still
On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote:
> On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > +static pg_attribute_always_inline PgAioResult
> > > +buffer_readv_complete_one(uint8 buf_off, Buffer buffer, uin
On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote:
> On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> > (We may be due for a test mode that does smgrreleaseall() at every
> > CHECK_FOR_INTERRUPTS()?)
>
> I suspect we are. I'm a bit afraid of even trying...
>
> ...
>
> It's extremely
On Tue, Mar 25, 2025 at 2:18 PM Andres Freund wrote:
> Attached v2.12, with the following changes:
Here's a tiny fixup to make io_concurrency=0 turn on
READ_BUFFERS_SYNCHRONOUSLY as mooted in a FIXME. Without this, AIO
will still run at level 1 even if you asked for 0. Feel free to
squash, or i
Hi,
On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> Unrelated to the above, another question about io_uring:
>
> commit da722699 wrote:
> > +/*
> > + * Need to submit staged but not yet submitted IOs using the fd, otherwise
> > +
Hi,
On 2025-03-24 19:20:37 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > static void
> > TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits,
> > - bool forget_owner)
> > +
Hi,
On 2025-03-24 17:45:37 -0700, Noah Misch wrote:
> (We may be due for a test mode that does smgrreleaseall() at every
> CHECK_FOR_INTERRUPTS()?)
I suspect we are. I'm a bit afraid of even trying...
...
It's extremely slow - but at least the main regression as well as the aio tests
pass!
>
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support
[I checked that v2.12 doesn't invalidate these review comments, but I didn't
technically rebase the review onto v2.12's line numbers.]
> static void
> TerminateBuffer
Hi,
On 2025-03-23 09:32:48 -0700, Noah Misch wrote:
> Another candidate description string:
>
> AIO_COMPLETED_SHARED "Waiting for another process to complete IO."
I liked that one and adopted it.
> > A more minimal change would be to narrow AIO_IO_URING_COMPLETION to
> > "execution" or someth
Hi,
On 2025-03-25 13:07:49 +1300, Thomas Munro wrote:
> On Tue, Mar 25, 2025 at 11:55 AM Andres Freund wrote:
> > #define READ_STREAM_USE_BATCHING 0x08
>
> +1
>
> I wonder if something more like READ_STREAM_CALLBACK_BATCHMODE_AWARE
> would be better, to highlight that you are making a declarati
On Tue, Mar 25, 2025 at 11:55 AM Andres Freund wrote:
> If a callback may sometimes need to block, it can still opt into
> READ_STREAM_USE_BATCHING, by submitting all staged IO before blocking.
>
> The hardest part is to explain the flag. Here's my current attempt:
>
> /* ---
> * Opt-in to using
Hi,
On 2025-03-23 17:29:39 -0700, Noah Misch wrote:
> commit 247ce06b wrote:
> > + pgaio_io_reopen(ioh);
> > +
> > + /*
> > +* To be able to exercise the reopen-fails path, allow
> > injection
> > +* points to trigger a f
On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote:
> On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > > Not sure yet how to best disable testing io_uring in this case. We can't
> > > just query EXEC_BACKEND from p
On Mon, Mar 24, 2025 at 5:59 AM Andres Freund wrote:
> On 2025-03-23 08:55:29 -0700, Noah Misch wrote:
> > An IO in PGAIO_HS_STAGED clearly blocks closing the IO's FD, and an IO in
> > PGAIO_HS_COMPLETED_IO clearly doesn't block that close. For
> > io_method=worker,
> > closing in PGAIO_HS_SUBMI
commit 247ce06b wrote:
> + pgaio_io_reopen(ioh);
> +
> + /*
> + * To be able to exercise the reopen-fails path, allow
> injection
> + * points to trigger a failure at this point.
> + */
> +
Hi,
On 2025-03-22 17:20:56 -0700, Noah Misch wrote:
> On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> > Not sure yet how to best disable testing io_uring in this case. We can't
> > just query EXEC_BACKEND from pg_config.h unfortunately. I guess making
> > the
> > initdb no
Hi,
On 2025-03-19 18:11:18 -0700, Noah Misch wrote:
> On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > Hm, we retry more frequently that that if there are new connections...
> > Maybe
> > just "try again next time"?
>
> Work
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11, with the following changes:
> - Added an error check for FileStartReadV() failing
>
> FileStartReadV() actually can fail, if the file can't be re-opened. I
> thought it'd be important for the error message to dif
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote:
> Attached v2.11
> Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring
Apart from some isolated cosmetic points, this is ready to commit:
> + ereport(ERROR,
> + errcode(err
On Thu, Mar 20, 2025 at 01:05:05PM -0400, Andres Freund wrote:
> On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> > On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > > +* marked as failed. In case of a partial read, some
> > > > buffers may be
> > > > +* ok.
Hi,
On 2025-03-19 18:17:37 -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > > + * marked as failed. In case of a partial read, some buffers
> > > may be
> > > + * ok.
> > > + */
> > > + failed =
> > > + prior_resu
On Tue, Mar 18, 2025 at 9:12 PM Andres Freund wrote:
>
> Hi,
>
> Attached is v2.10, with the following changes:
>
> - committed core AIO infrastructure patch
Hi, yay, It's happening.jpg ;)
Some thoughts about 2.10-0004:
What do you think about putting there into (io_uring patch) info
about the
On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote:
> On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> > commit 55b454d wrote:
> > > aio: Infrastructure for io_method=worker
> >
> > > + /* Try to launch one. */
> > > + child = StartChildProcess(B_IO_WORKER);
> > > +
Hi,
On 2025-03-19 14:25:30 -0700, Noah Misch wrote:
> On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote:
> > - Right now effective_io_concurrency cannot be set > 0 on Windows and other
> > platforms that lack posix_fadvise. But with AIO we can read ahead without
> > posix_fadvise()
On Tue, Mar 18, 2025 at 4:12 PM Andres Freund wrote:
>
> Attached is v2.10
This is a review of 0002: bufmgr: Improve stats when buffer is read
in concurrently
In the commit message, it might be worth distinguishing that
pg_stat_io and vacuum didn't double count reads, they under-counted
hits. p
On Tue, Mar 18, 2025 at 4:12 PM Andres Freund wrote:
>
> Attached is v2.10,
I noticed a few comments could be improved in 0011: bufmgr: Use AIO
in StartReadBuffers()
In WaitReadBuffers(), this comment is incomplete:
/*
-* Skip this block if someone else has already completed it
Hi,
On 2025-03-18 21:00:17 -0400, Melanie Plageman wrote:
> On Tue, Mar 18, 2025 at 4:12 PM Andres Freund wrote:
> > Attached is v2.10,
>
> This is a review of 0008: bufmgr: Implement AIO read support
>
> I'm afraid it is more of a cosmetic review than a sign-off on the
> patch's correctnes
1 - 100 of 121 matches
Mail list logo
