The new overhead for creating connections when io_method=io_uring is also a function of max_connections. I have been using the default (=100). But when I increase it to =1000 then the time to create a connection almost triples. That isn't a big surprise given the usage of TotalProcs here: https://github.com/postgres/postgres/blob/REL_18_BETA1/src/backend/storage/aio/method_io_uring.c#L129
On Tue, Jun 3, 2025 at 12:24 PM MARK CALLAGHAN <mdcal...@gmail.com> wrote: > When measuring the time to create a connection, it is ~2.3X longer with > io_method=io_uring then with io_method=sync (6.9ms vs 3ms), and the > postmaster process uses ~3.5X more CPU to create connections. > > The reproduction case so far is my usage of the Insert Benchmark on a > large server with 48 cores. I need to fix the benchmark client -- today it > creates ~1000 connections/s to run a monitoring query in between every 100 > queries and the extra latency from connection create makes results worse > for one of the benchmark steps. While I can fix the benchmark client to > avoid this, I am curious about the extra latency in connection create. > > I used "perf record -e cycles -F 333 -g -p $pidof_postmaster -- sleep 30" > but I have yet to find a big difference from the reports generated with > that for io_method=io_uring vs =sync. It shows that much time is spent in > the kernel dealing with the VM (page tables, etc). > > The server runs Ubuntu 22.04.4. I compiled the Postgres 18beta1 release > from source via: > ./configure --prefix=$pfx --enable-debug CFLAGS="-O2 > -fno-omit-frame-pointer" --with-lz4 --with-liburing > > Output from configure includes: > checking whether to build with liburing support... yes > checking for liburing... yes > > io_uring support was installed via: sudo apt install liburing-dev and I > have 2.1-2build1 > libc is Ubuntu GLIBC 2.35-0ubuntu3.10 > gcc is 11.4.0 > > More performance info is here: > https://mdcallag.github.io/reports/25_06_01.pg.all.mem.hetz/all.html#summary > > The config files I used only differ WRT io_method > * io_method=sync - > https://github.com/mdcallag/mytools/blob/master/bench/conf/arc/may25.hetzner/pg18b1git_o2nofp/conf.diff.cx10b_c32r128 > * io_method=workers - > https://github.com/mdcallag/mytools/blob/master/bench/conf/arc/may25.hetzner/pg18b1git_o2nofp/conf.diff.cx10cw4_c32r128 > * io_method=io_uring - > https://github.com/mdcallag/mytools/blob/master/bench/conf/arc/may25.hetzner/pg18b1git_o2nofp/conf.diff.cx10d_c32r128 > > The symptoms are: > * ~20% reduction in point queries/s with io_method=io_uring vs =sync, > =workers or Postgres 17.4, and the issue here is not that SELECT > performance has changed, it is that my benchmark client sometimes creates > connections in between running queries and the new latency from that for > io_method=io_uring hurts throughput > * CPU/query and context switches /query are similar, with io_uring the > CPU/query might be ~4% larger > > From sampled thread stacks of the postmaster when I use io_uring the > common stack is: > > arch_fork,__GI__Fork,__libc_fork,fork_process,postmaster_child_launch,BackendStartup,ServerLoop,PostmasterMain,main > > While the typical stack with io_method=sync is: > > epoll_wait,WaitEventSetWaitBlock,WaitEventSetWait,ServerLoop,PostmasterMain,main > > I run "ps" during each benchmark step and on example of what I see during > a point query benchmarks step (qp100.L2) with io_method=uring is below. The > benchmark step runs for 300 seconds. > ---> from the start of the step > mdcallag 3762684 0.9 1.5 103027276 2031612 ? Ss 03:12 0:14 > /home/mdcallag/d/pg18beta1_o2nofp/bin/postgres -D /data/m/pg > ---> from the end of the step > mdcallag 3762684 15.9 1.5 103027276 2031612 ? Rs 03:12 5:04 > /home/mdcallag/d/pg18beta1_o2nofp/bin/postgres -D /data/m/pg > > And from top I see: > ---> with =io_uring > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 3762684 mdcallag 20 0 98.3g 1.9g 1.9g R 99.4 1.5 3:04.87 > /home/mdcallag/d/pg18beta1_o2nofp/bin/postgres -D /data/m/pg > > --> with =sync > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 2913673 mdcallag 20 0 98.3g 1.9g 1.9g S 28.3 1.5 0:54.13 > /home/mdcallag/d/pg18beta1_o2nofp/bin/postgres -D /data/m/pg > > The postmaster had used 0:14 (14 seconds) of CPU time by the start of the > benchmark step and 5:04 (304 seconds) by the end. For the same step with > io_method=sync it was 0:05 at the start and 1:27 at the end. So the > postmaster used ~290 seconds of cpu with =io_uring vs ~82 with =sync, which > is ~3.5X more CPU on the postmaster per connection attempt. > > From vmstat what I see is that some of the rates (cs = context switches, > us = user CPU) are ~20% smaller with =io_uring, which is reasonable given > that the throughput is also ~20% smaller. But sy (system CPU) is not 20% > smaller because of the overhead from all of those calls to fork (or clone). > > Avg rates from vmstat > cs us sy us+sy > 492961 25.0 14.0 39.0 --> with =sync > 401233 20.1 14.0 34.1 ---> with =io_uring > > -- > Mark Callaghan > mdcal...@gmail.com > -- Mark Callaghan mdcal...@gmail.com
