I would suspect the checksum part of this (I do believe it's being
actively worked on) :
6533726 single-threaded checksum & raidz2 parity calculations limit
write bandwidth on thumper
-r
Robert Milkowski writes:
> Hi,
>
> snv_74, x4500, 48x 500GB, 16GB RAM, 2x dual core
>
> # zpool create test c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0 c0t7d0
> c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0 c4t0d0 c4t1d0 c4t2d0
> c4t3d0 c4t4d0 c4t5d0 c4t6d0 c4t7d0 c5t1d0 c5t2d0 c5t3d0 c5t5d0 c5t6d0 c5t7d0
> c6t0d0 c6t1d0 c6t2d0 c6t3d0 c6t4d0 c6t5d0 c6t6d0 c6t7d0 c7t0d0 c7t1d0 c7t2d0
> c7t3d0 c7t4d0 c7t5d0 c7t6d0 c7t7d0
> [46x 500GB]
>
> # ls -lh /test/q1
> -rw-r--r-- 1 root root 82G Oct 18 09:43 /test/q1
>
> # dd if=/test/q1 of=/dev/null bs=16384k &
> # zpool iostat 1
> capacity operations bandwidth
> pool used avail read write read write
> ---------- ----- ----- ----- ----- ----- -----
> test 213G 20.6T 645 120 80.1M 14.7M
> test 213G 20.6T 9.26K 0 1.16G 0
> test 213G 20.6T 9.66K 0 1.21G 0
> test 213G 20.6T 9.41K 0 1.18G 0
> test 213G 20.6T 9.41K 0 1.18G 0
> test 213G 20.6T 7.45K 0 953M 0
> test 213G 20.6T 7.59K 0 971M 0
> test 213G 20.6T 7.41K 0 948M 0
> test 213G 20.6T 8.25K 0 1.03G 0
> test 213G 20.6T 9.17K 0 1.15G 0
> test 213G 20.6T 9.54K 0 1.19G 0
> test 213G 20.6T 9.89K 0 1.24G 0
> test 213G 20.6T 9.41K 0 1.18G 0
> test 213G 20.6T 9.31K 0 1.16G 0
> test 213G 20.6T 9.80K 0 1.22G 0
> test 213G 20.6T 8.72K 0 1.09G 0
> test 213G 20.6T 7.86K 0 1006M 0
> test 213G 20.6T 7.21K 0 923M 0
> test 213G 20.6T 7.62K 0 975M 0
> test 213G 20.6T 8.68K 0 1.08G 0
> test 213G 20.6T 9.81K 0 1.23G 0
> test 213G 20.6T 9.57K 0 1.20G 0
>
> So it's around 1GB/s.
>
> # dd if=/dev/zero of=/test/q10 bs=128k &
> # zpool iostat 1
> capacity operations bandwidth
> pool used avail read write read write
> ---------- ----- ----- ----- ----- ----- -----
> test 223G 20.6T 656 170 81.5M 20.8M
> test 223G 20.6T 0 8.10K 0 1021M
> test 223G 20.6T 0 7.94K 0 1001M
> test 216G 20.6T 0 6.53K 0 812M
> test 216G 20.6T 0 7.19K 0 906M
> test 216G 20.6T 0 6.78K 0 854M
> test 216G 20.6T 0 7.88K 0 993M
> test 216G 20.6T 0 10.3K 0 1.27G
> test 222G 20.6T 0 8.61K 0 1.04G
> test 222G 20.6T 0 7.30K 0 919M
> test 222G 20.6T 0 8.16K 0 1.00G
> test 222G 20.6T 0 8.82K 0 1.09G
> test 225G 20.6T 0 4.19K 0 511M
> test 225G 20.6T 0 10.2K 0 1.26G
> test 225G 20.6T 0 9.15K 0 1.13G
> test 225G 20.6T 0 8.46K 0 1.04G
> test 225G 20.6T 0 8.48K 0 1.04G
> test 225G 20.6T 0 10.9K 0 1.33G
> test 231G 20.6T 0 3 0 3.96K
> test 231G 20.6T 0 0 0 0
> test 231G 20.6T 0 0 0 0
> test 231G 20.6T 0 9.02K 0 1.11G
> test 231G 20.6T 0 12.2K 0 1.50G
> test 231G 20.6T 0 9.14K 0 1.13G
> test 231G 20.6T 0 10.3K 0 1.27G
> test 231G 20.6T 0 9.08K 0 1.10G
> test 237G 20.6T 0 0 0 0
> test 237G 20.6T 0 0 0 0
> test 237G 20.6T 0 6.03K 0 760M
> test 237G 20.6T 0 9.18K 0 1.13G
> test 237G 20.6T 0 8.40K 0 1.03G
> test 237G 20.6T 0 8.45K 0 1.04G
> test 237G 20.6T 0 11.1K 0 1.36G
>
> Well, writing could be faster than reading here... there're gaps due to bug
> 6415647 I guess.
>
>
> # zpool destroy test
>
> # metainit d100 1 46 c0t0d0s0 c0t1d0s0 c0t2d0s0 c0t3d0s0 c0t4d0s0 c0t5d0s0
> c0t6d0s0 c0t7d0s0 c1t0d0s0 c1t1d0s0 c1t2d0s0 c1t3d0s0 c1t4d0s0 c1t5d0s0
> c1t6d0s0 c1t7d0s0 c4t0d0s0 c4t1d0s0 c4t2d0s0 c4t3d0s0 c4t4d0s0 c4t5d0s0
> c4t6d0s0 c4t7d0s0 c5t1d0s0 c5t2d0s0 c5t3d0s0 c5t5d0s0 c5t6d0s0 c5t7d0s0
> c6t0d0s0 c6t1d0s0 c6t2d0s0 c6t3d0s0 c6t4d0s0 c6t5d0s0 c6t6d0s0 c6t7d0s0
> c7t0d0s0 c7t1d0s0 c7t2d0s0 c7t3d0s0 c7t4d0s0 c7t5d0s0 c7t6d0s0 c7t7d0s0 -i
> 128k
> d100: Concat/Stripe is setup
> [46x 500GB]
>
> And I get not so good results - maximum 1GB of reading... hmmmm...
>
> maxphys is 56K - I thought it was increased some time ago on x86!
>
> Still no performance increase.
>
> # metainit d101 -r c0t0d0s0 c1t0d0s0 c4t0d0s0 c6t0d0s0 c7t0d0s0 -i 128k
> # metainit d102 -r c0t1d0s0 c1t1d0s0 c5t1d0s0 c6t1d0s0 c7t1d0s0 -i 128k
> # metainit d103 -r c0t2d0s0 c1t2d0s0 c5t2d0s0 c6t2d0s0 c7t2d0s0 -i 128k
> # metainit d104 -r c0t4d0s0 c1t4d0s0 c4t4d0s0 c6t4d0s0 c7t4d0s0 -i 128k
> # metainit d105 -r c0t3d0s0 c1t3d0s0 c4t3d0s0 c5t3d0s0 c6t3d0s0 c7t3d0s0 -i
> 128k
> # metainit d106 -r c0t5d0s0 c1t5d0s0 c4t5d0s0 c5t5d0s0 c6t5d0s0 c7t5d0s0 -i
> 128k
> # metainit d107 -r c0t6d0s0 c1t6d0s0 c4t6d0s0 c5t6d0s0 c6t6d0s0 c7t6d0s0 -i
> 128k
> # metainit d108 -r c0t7d0s0 c1t7d0s0 c4t7d0s0 c5t7d0s0 c6t7d0s0 c7t7d0s0 -i
> 128k
>
> # iostat -xnzCM 1 | egrep "device| c[0-7]$"
> [...]
> extended device statistics
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
> 0.0 362.0 0.0 362.0 0.0 7.0 0.0 19.3 0 698 c0
> 0.0 377.0 0.0 377.0 0.0 7.0 0.0 18.5 0 698 c1
> 0.0 320.0 0.0 320.0 0.0 6.0 0.0 18.7 0 598 c4
> 0.0 268.0 0.0 268.0 0.0 5.0 0.0 18.6 0 499 c5
> 0.0 372.0 0.0 372.0 0.0 7.0 0.0 18.8 0 698 c6
> 0.0 374.0 0.0 374.0 0.0 7.0 0.0 18.7 0 698 c7
>
> Sometimes I get even more - around 2.3GB/s
>
> The question is - why I can't get that kind of performance with single zfs
> pool (striping accross all te disks)? Concurrency problem or something else?
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
