1.7 GHz SPR, 'D' stepping
1 DSA device, 1 WQ (size=128, 1 engine, mode="dedicated", type="kernel",
name="dmaengine")
io_uring-bench with some mods:
1 thread, 1 io_uring context (sq_ring=64,cq_ring=128), random reads over one
1GB file on tmpfs
Without touching data in user-space:
SW copy (KIOPS) DMA copy (irq) (KIOPS) DMA copy
(poll) (KIOPS)
1KB 1488.8 583.9 1189.0
2KB 1300.7 582.0 1170.8
4KB 1045.5 580.1 1214.8
8KB 709.7 532.2 952.3
16KB 393.3 441.1 690.7
32KB 203.2 313.6 434.2
64KB 91.5 229.3 266.8
128KB 46.1 128.9 155.2
With touching data in user-space:
SW copy (KIOPS) DMA copy (irq) (KIOPS) DMA copy
(poll) (KIOPS)
1KB 1281.4 586.8 1157.0
2KB 1129.0 603.9 1058.1
4K 888.9 615.7 935.6
8KB 506.2 511.8 757.8
16KB 309.4 363.1 472.6
32KB 131.1 221.4 275.2
64KB 65.7 147.2 152.8
128KB 31.4 79.5 76.2
FS layer 1KB 4KB 8KB 16KB
32KB 128KB
-----------
read_iter (get_page + sw copy) 986 1367 2194 3994 8237
35441
read_iter (get_page) 442 459 652 1089 1852
5599
sw copy (deduced) 542 908 1542 2905 6385
29842
io_uring layer
-------------------
alloc dma_copy_vec 222 214 244 248 284
381
dma_map_sg 130 132 146 210 296
1034
cleanup dma_copy_vec 208 303 375 451 700
2108
(dma_unmap_sg, put_page, kfree)
dma driver layer
----------------------
dma_memcpy_sg_prep 345 344 329 479 713
2611
dma_txd_submit 108 109 108 109 109 111
dma_tx_status 327 349 431 534 797
2346
idxd_dma_tx_status (deduced) 119 41 56 83 97
238 ???
SW copy
=======
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -- /tmp/xz1
[ 152.636953] _io_ring_call_stat_print(read_iter):
nb_call=37514528,nb_cycle=37008954492,cycle/call=986;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -- /tmp/xz1
[ 70.028931] _io_ring_call_stat_print(read_iter):
nb_call=28191936,nb_cycle=38541193956,cycle/call=1367;
taskset -c 1 /bin/bash -x /mnt/share0/buildroot-prev1/build/images/io_uring -b
8192 -d 64 -- /tmp/xz1
[ 218.785124] _io_ring_call_stat_print(read_iter):
nb_call=19109312,nb_cycle=41928352830,cycle/call=2194;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -- /tmp/xz1
[ 337.191163] _io_ring_call_stat_print(read_iter):
nb_call=11452608,nb_cycle=45751699798,cycle/call=3994;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -- /tmp/xz1
[ 294.087806] _io_ring_call_stat_print(read_iter):
nb_call=5717280,nb_cycle=47097616396,cycle/call=8237;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 131072 -d 64 -- /tmp/xz1
[ 152.502022] _io_ring_call_stat_print(read_iter):
nb_call=1404480,nb_cycle=49777091724,cycle/call=35441;
DMA copy (ipoll)
==============
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[ 838.747911] _io_ring_call_stat_print(read_iter):
nb_call=24296523,nb_cycle=10747261984,cycle/call=442;
[ 838.749273] _io_ring_call_stat_print(dma_copy):
nb_call=24296523,nb_cycle=10502724540,cycle/call=432;
[ 838.750632] _io_ring_call_stat_print(dma_tx_status):
nb_call=24296523,nb_cycle=7600041778,cycle/call=312;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[ 1007.960910] _io_ring_call_stat_print(read_iter):
nb_call=23038754,nb_cycle=10597803630,cycle/call=459;
[ 1007.962291] _io_ring_call_stat_print(dma_copy):
nb_call=23038754,nb_cycle=10078534928,cycle/call=437;
[ 1007.963670] _io_ring_call_stat_print(dma_tx_status):
nb_call=23038754,nb_cycle=8797313170,cycle/call=381;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 1111.263944] _io_ring_call_stat_print(read_iter):
nb_call=19890966,nb_cycle=12980210200,cycle/call=652;
[ 1111.265306] _io_ring_call_stat_print(dma_copy):
nb_call=19890966,nb_cycle=10791559664,cycle/call=542;
[ 1111.266677] _io_ring_call_stat_print(dma_tx_status):
nb_call=19890966,nb_cycle=8551330946,cycle/call=429;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 --
/tmp/xz1
[ 1185.235939] _io_ring_call_stat_print(read_iter):
nb_call=14168913,nb_cycle=15437025706,cycle/call=1089;
[ 1185.237311] _io_ring_call_stat_print(dma_copy):
nb_call=14168913,nb_cycle=13240559576,cycle/call=934;
[ 1185.238681] _io_ring_call_stat_print(dma_tx_status):
nb_call=14168913,nb_cycle=7891900714,cycle/call=556;
extra stats enabled:
--------------------------
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[ 165.854097] _io_ring_call_stat_print(read_iter):
nb_call=20243146,nb_cycle=9360535452,cycle/call=462;
[ 165.855472] _io_ring_call_stat_print(dma_map_page):
nb_call=20243146,nb_cycle=2645942826,cycle/call=130;
[ 165.856866] _io_ring_call_stat_print(dma_map_prep):
nb_call=20243146,nb_cycle=6984741092,cycle/call=345;
[ 165.858238] _io_ring_call_stat_print(dma_txd_submit):
nb_call=20243146,nb_cycle=2204089220,cycle/call=108;
[ 165.859657] _io_ring_call_stat_print(dma_tx_status):
nb_call=20243146,nb_cycle=6626478424,cycle/call=327;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[ 275.031263] _io_ring_call_stat_print(read_iter):
nb_call=19959625,nb_cycle=9161716298,cycle/call=459;
[ 275.033058] _io_ring_call_stat_print(dma_map_page):
nb_call=19959625,nb_cycle=2635932496,cycle/call=132;
[ 275.034480] _io_ring_call_stat_print(dma_map_prep):
nb_call=19959625,nb_cycle=6870646042,cycle/call=344;
[ 275.035862] _io_ring_call_stat_print(dma_txd_submit):
nb_call=19959625,nb_cycle=2176350008,cycle/call=109;
[ 275.037256] _io_ring_call_stat_print(dma_tx_status):
nb_call=19959625,nb_cycle=6977679726,cycle/call=349;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 401.383119] _io_ring_call_stat_print(read_iter):
nb_call=18062188,nb_cycle=11587080420,cycle/call=641;
[ 401.384486] _io_ring_call_stat_print(dma_map_page):
nb_call=18062188,nb_cycle=2637550686,cycle/call=146;
[ 401.385889] _io_ring_call_stat_print(dma_map_prep):
nb_call=18062188,nb_cycle=5952760396,cycle/call=329;
[ 401.387271] _io_ring_call_stat_print(dma_txd_submit):
nb_call=18062188,nb_cycle=1965616416,cycle/call=108;
[ 401.388691] _io_ring_call_stat_print(dma_tx_status):
nb_call=18062188,nb_cycle=7790097454,cycle/call=431;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 --
/tmp/xz1
[ 627.571132] _io_ring_call_stat_print(read_iter):
nb_call=13945661,nb_cycle=14898802216,cycle/call=1068;
[ 627.572494] _io_ring_call_stat_print(dma_map_page):
nb_call=13945661,nb_cycle=2929960406,cycle/call=210;
[ 627.573883] _io_ring_call_stat_print(dma_map_prep):
nb_call=13945661,nb_cycle=6693123644,cycle/call=479;
[ 627.575250] _io_ring_call_stat_print(dma_txd_submit):
nb_call=13945661,nb_cycle=1520918608,cycle/call=109;
[ 627.576649] _io_ring_call_stat_print(dma_tx_status):
nb_call=13945661,nb_cycle=7452250324,cycle/call=534;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 --
/tmp/xz1
[ 778.798192] _io_ring_call_stat_print(read_iter):
nb_call=10047303,nb_cycle=17626174618,cycle/call=1754;
[ 778.799583] _io_ring_call_stat_print(dma_map_page):
nb_call=10047303,nb_cycle=2975384344,cycle/call=296;
[ 778.801000] _io_ring_call_stat_print(dma_map_prep):
nb_call=10047303,nb_cycle=7166354834,cycle/call=713;
[ 778.802384] _io_ring_call_stat_print(dma_txd_submit):
nb_call=10047303,nb_cycle=1098932882,cycle/call=109;
[ 778.803807] _io_ring_call_stat_print(dma_tx_status):
nb_call=10047303,nb_cycle=8010400466,cycle/call=797;
even more stats enabled:
----------------------------------
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[ 227.487680] _io_ring_call_stat_print(read_iter):
nb_call=18531697,nb_cycle=9091525924,cycle/call=490;
[ 227.489095] _io_ring_call_stat_print(dma_map_page):
nb_call=18531697,nb_cycle=2642274344,cycle/call=142;
[ 227.490470] _io_ring_call_stat_print(dma_map_prep):
nb_call=18531697,nb_cycle=6435793506,cycle/call=347;
[ 227.491841] _io_ring_call_stat_print(dma_txd_submit):
nb_call=18531697,nb_cycle=2023143092,cycle/call=109;
[ 227.493257] _io_ring_call_stat_print(dma_tx_status):
nb_call=18531697,nb_cycle=8965236398,cycle/call=483;
[ 227.494650] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=18531697,nb_cycle=3868579740,cycle/call=208;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[ 335.794733] _io_ring_call_stat_print(read_iter):
nb_call=18385330,nb_cycle=8541024350,cycle/call=464;
[ 335.796064] _io_ring_call_stat_print(dma_map_page):
nb_call=18385330,nb_cycle=2000742552,cycle/call=108;
[ 335.797471] _io_ring_call_stat_print(dma_map_prep):
nb_call=18385330,nb_cycle=6616304978,cycle/call=359;
[ 335.798869] _io_ring_call_stat_print(dma_txd_submit):
nb_call=18385330,nb_cycle=2006059222,cycle/call=109;
[ 335.800289] _io_ring_call_stat_print(dma_tx_status):
nb_call=18385330,nb_cycle=10568171772,cycle/call=574;
[ 335.801702] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=18385330,nb_cycle=5578989814,cycle/call=303;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 935.314721] _io_ring_call_stat_print(read_iter):
nb_call=16560375,nb_cycle=10774634416,cycle/call=650;
[ 935.316063] _io_ring_call_stat_print(dma_map_page):
nb_call=16560375,nb_cycle=2548383934,cycle/call=153;
[ 935.317423] _io_ring_call_stat_print(dma_map_prep):
nb_call=16560375,nb_cycle=5842722324,cycle/call=352;
[ 935.318786] _io_ring_call_stat_print(dma_txd_submit):
nb_call=16560375,nb_cycle=1802839382,cycle/call=108;
[ 935.320174] _io_ring_call_stat_print(dma_tx_status):
nb_call=16560375,nb_cycle=10670571864,cycle/call=644;
[ 935.321607] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=16560375,nb_cycle=6226614110,cycle/call=375;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 1051.589712] _io_ring_call_stat_print(read_iter):
nb_call=13569679,nb_cycle=13826817960,cycle/call=1018;
[ 1051.591105] _io_ring_call_stat_print(dma_map_page):
nb_call=13569679,nb_cycle=2828194164,cycle/call=208;
[ 1051.592483] _io_ring_call_stat_print(dma_map_prep):
nb_call=13569679,nb_cycle=6270020674,cycle/call=462;
[ 1051.593857] _io_ring_call_stat_print(dma_txd_submit):
nb_call=13569679,nb_cycle=1482618160,cycle/call=109;
[ 1051.595262] _io_ring_call_stat_print(dma_tx_status):
nb_call=13569679,nb_cycle=9804848154,cycle/call=722;
[ 1051.596671] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=13569679,nb_cycle=6125278794,cycle/call=451;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 --
/tmp/xz1
[ 1167.621755] _io_ring_call_stat_print(read_iter):
nb_call=9356189,nb_cycle=17348247116,cycle/call=1854;
[ 1167.623112] _io_ring_call_stat_print(dma_map_page):
nb_call=9356189,nb_cycle=3255963092,cycle/call=348;
[ 1167.624480] _io_ring_call_stat_print(dma_map_prep):
nb_call=9356189,nb_cycle=7065992002,cycle/call=755;
[ 1167.625861] _io_ring_call_stat_print(dma_txd_submit):
nb_call=9356189,nb_cycle=1023755914,cycle/call=109;
[ 1167.627276] _io_ring_call_stat_print(dma_tx_status):
nb_call=9356189,nb_cycle=9096623304,cycle/call=972;
[ 1167.628657] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=9356189,nb_cycle=6554425776,cycle/call=700;
and even more stats enabled:
----------------------------------------
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[ 413.308768] _io_ring_call_stat_print(read_iter):
nb_call=17436862,nb_cycle=8339876898,cycle/call=478;
[ 413.310115] _io_ring_call_stat_print(dma_map_page):
nb_call=17436862,nb_cycle=2397826664,cycle/call=137;
[ 413.311477] _io_ring_call_stat_print(dma_map_prep):
nb_call=17436862,nb_cycle=5871816456,cycle/call=336;
[ 413.312827] _io_ring_call_stat_print(dma_txd_submit):
nb_call=17436862,nb_cycle=1930111314,cycle/call=110;
[ 413.314212] _io_ring_call_stat_print(dma_tx_status):
nb_call=17436862,nb_cycle=9608856944,cycle/call=551;
[ 413.315580] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=17436862,nb_cycle=3873159708,cycle/call=222;
[ 413.317007] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=17436862,nb_cycle=4812252522,cycle/call=275;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[ 491.803957] _io_ring_call_stat_print(read_iter):
nb_call=17431464,nb_cycle=7976750478,cycle/call=457;
[ 491.805309] _io_ring_call_stat_print(dma_map_page):
nb_call=17431464,nb_cycle=2506301976,cycle/call=143;
[ 491.806670] _io_ring_call_stat_print(dma_map_prep):
nb_call=17431464,nb_cycle=6338791238,cycle/call=363;
[ 491.808039] _io_ring_call_stat_print(dma_txd_submit):
nb_call=17431464,nb_cycle=1944660584,cycle/call=111;
[ 491.809436] _io_ring_call_stat_print(dma_tx_status):
nb_call=17431464,nb_cycle=9289975568,cycle/call=532;
[ 491.810802] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=17431464,nb_cycle=3738193802,cycle/call=214;
[ 491.812247] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=17431464,nb_cycle=4546959358,cycle/call=260;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 596.820790] _io_ring_call_stat_print(read_iter):
nb_call=15143794,nb_cycle=10155218664,cycle/call=670;
[ 596.822172] _io_ring_call_stat_print(dma_map_page):
nb_call=15143794,nb_cycle=2554489340,cycle/call=168;
[ 596.823568] _io_ring_call_stat_print(dma_map_prep):
nb_call=15143794,nb_cycle=5858304540,cycle/call=386;
[ 596.824943] _io_ring_call_stat_print(dma_txd_submit):
nb_call=15143794,nb_cycle=1679767784,cycle/call=110;
[ 596.826358] _io_ring_call_stat_print(dma_tx_status):
nb_call=15143794,nb_cycle=10300601004,cycle/call=680;
[ 596.827758] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=15143794,nb_cycle=3702212752,cycle/call=244;
[ 596.829219] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=15143794,nb_cycle=6186738794,cycle/call=408;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 --
/tmp/xz1
[ 673.975774] _io_ring_call_stat_print(read_iter):
nb_call=12651137,nb_cycle=13348866092,cycle/call=1055;
[ 673.977163] _io_ring_call_stat_print(dma_map_page):
nb_call=12651137,nb_cycle=2657725706,cycle/call=210;
[ 673.978553] _io_ring_call_stat_print(dma_map_prep):
nb_call=12651137,nb_cycle=6720093690,cycle/call=531;
[ 673.979918] _io_ring_call_stat_print(dma_txd_submit):
nb_call=12651137,nb_cycle=1403316410,cycle/call=110;
[ 673.981316] _io_ring_call_stat_print(dma_tx_status):
nb_call=12651137,nb_cycle=9043971908,cycle/call=714;
[ 673.982718] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=12651137,nb_cycle=3138149536,cycle/call=248;
[ 673.984144] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=12651137,nb_cycle=5639966438,cycle/call=445;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 --
/tmp/xz1
[ 763.673818] _io_ring_call_stat_print(read_iter):
nb_call=8953781,nb_cycle=16585739778,cycle/call=1852;
[ 763.675148] _io_ring_call_stat_print(dma_map_page):
nb_call=8953781,nb_cycle=2772644866,cycle/call=309;
[ 763.676504] _io_ring_call_stat_print(dma_map_prep):
nb_call=8953781,nb_cycle=7655287758,cycle/call=854;
[ 763.677845] _io_ring_call_stat_print(dma_txd_submit):
nb_call=8953781,nb_cycle=985388208,cycle/call=110;
[ 763.679198] _io_ring_call_stat_print(dma_tx_status):
nb_call=8953781,nb_cycle=9035643372,cycle/call=1009;
[ 763.680572] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=8953781,nb_cycle=2548060728,cycle/call=284;
[ 763.681994] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=8953781,nb_cycle=6571959652,cycle/call=733;
taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30
/mnt/share0/buildroot-prev1/build/images/io_uring -b 131072 -d 64 -p 1 --
/tmp/xz1
[ 851.217379] _io_ring_call_stat_print(read_iter):
nb_call=3744307,nb_cycle=20966938344,cycle/call=5599;
[ 851.218765] _io_ring_call_stat_print(dma_map_page):
nb_call=3744307,nb_cycle=3872112822,cycle/call=1034;
[ 851.220061] _io_ring_call_stat_print(dma_map_prep):
nb_call=3744307,nb_cycle=9778748434,cycle/call=2611;
[ 851.221416] _io_ring_call_stat_print(dma_txd_submit):
nb_call=3744307,nb_cycle=417579118,cycle/call=111;
[ 851.222755] _io_ring_call_stat_print(dma_tx_status):
nb_call=11076628,nb_cycle=9711661782,cycle/call=876;
[ 851.224114] _io_ring_call_stat_print(dma_alloc_copy_vec):
nb_call=3744307,nb_cycle=1430044398,cycle/call=381;
[ 851.225592] _io_ring_call_stat_print(dma_cb_free_copy_vec):
nb_call=3744307,nb_cycle=7895268808,cycle/call=2108;
