Re: Blocking I/O, async I/O and io_uring

2020-12-08 Thread Craig Ringer
On Tue, 8 Dec 2020 at 15:04, Andres Freund wrote: > Hi, > > On 2020-12-08 04:24:44 +, tsunakawa.ta...@fujitsu.com wrote: > > I'm looking forward to this from the async+direct I/O, since the > > throughput of some write-heavy workload decreased by half or more > > during checkpointing (due to

Re: Blocking I/O, async I/O and io_uring

2020-12-08 Thread Fujii Masao
On 2020/12/08 11:55, Craig Ringer wrote: Hi all A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. io_uring appears to offer a way to make system calls including reads, writes, fsync()s, and more in a non-b

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 04:24:44 +, tsunakawa.ta...@fujitsu.com wrote: > I'm looking forward to this from the async+direct I/O, since the > throughput of some write-heavy workload decreased by half or more > during checkpointing (due to fsync?) Depends on why that is. The most common, I think, caus

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 13:01:38 +0800, Craig Ringer wrote: > Have you done much bpf / systemtap / perf based work on measurement and > tracing of latencies etc? If not that's something I'd be keen to help with. > I've mostly been using systemtap so far but I'm trying to pivot over to > bpf. Not much -

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
On Tue, 8 Dec 2020 at 12:02, Andres Freund wrote: > Hi, > > On 2020-12-08 10:55:37 +0800, Craig Ringer wrote: > > A new kernel API called io_uring has recently come to my attention. I > > assume some of you (Andres?) have been following it for a while. > > Yea, I've spent a *lot* of time working

RE: Blocking I/O, async I/O and io_uring

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > Especially with direct IO > checkpointing can be a lot faster *and* less impactful on the "regular" > load. I'm looking forward to this from the async+direct I/O, since the throughput of some write-heavy workload decreased by half or more during checkpointing (due to fsync

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andres Freund
Hi, On 2020-12-08 10:55:37 +0800, Craig Ringer wrote: > A new kernel API called io_uring has recently come to my attention. I > assume some of you (Andres?) have been following it for a while. Yea, I've spent a *lot* of time working on AIO support, utilizing io_uring. Recently Thomas also joined

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Andreas Karlsson
On 12/8/20 3:55 AM, Craig Ringer wrote: A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. Andres did a talk on this at FOSDEM PGDay earlier this year. You can see his slides below, but since they are from Janu

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Thomas Munro
On Tue, Dec 8, 2020 at 3:56 PM Craig Ringer wrote: > I thought I'd start the discussion on this and see where we can go with it. > What incremental steps can be done to move us toward parallelisable I/O > without having to redesign everything? > > I'm thinking that redo is probably a good first

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
References to get things started: * https://lwn.net/Articles/810414/ * https://unixism.net/loti/what_is_io_uring.html * https://blogs.oracle.com/linux/an-introduction-to-the-io_uring-asynchronous-io-framework * https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/

Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
Hi all A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. io_uring appears to offer a way to make system calls including reads, writes, fsync()s, and more in a non-blocking, batched and pipelined manner, with or w