Hello,
we have set up an Opensolaris 2009.06 based storage server. It provides NFS
shares and iSCSI targets primarily for Virtual Machines. From the beginning we
noticed extremely slow disk speed on the Linux virtualization host. After some
tests we discovered that the local disk performance is okay. Write speeds are
around 60 - 140 MiB / s. But all access via NFS or iSCSI drops after the first
10 MiB down to 250 KiB / s. The network throughput looks good. Netperf shows
920 - 940 Mbit. We first focused on writespeed over iSCSI. Reading and NFS
access are slow as well. The CPU is not very active.
Local writes to ZFS (partially cached):
$ dd if=/dev/zero of=out bs=64k count=16k
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 4.21787 s, 255 MB/s
$ for i in {1..6}; do echo "#${i}"; dd if=/dev/zero of=out bs=64k count=80; done
#1
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00345736 s, 1.5 GB/s
#2
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00324686 s, 1.6 GB/s
#3
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00326196 s, 1.6 GB/s
#4
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00332794 s, 1.6 GB/s
#5
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00326972 s, 1.6 GB/s
#6
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00341859 s, 1.5 GB/s
iSCSI writes from remote:
$ dd if=/dev/zero of=out bs=64k count=16k
More than an hour. Speed below 1 MiB / s.
$ for i in {1..6}; do echo "#${i}"; dd if=/dev/zero of=out bs=64k count=80; done
#1
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00337874 s, 1.6 GB/s
#2
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.00399618 s, 1.3 GB/s
#3
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 0.0390615 s, 134 MB/s
#4
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 8.75981 s, 599 kB/s
#5
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 7.97124 s, 658 kB/s
#6
80+0 records in
80+0 records out
5242880 bytes (5.2 MB) copied, 8.36449 s, 627 kB/s
The machine is a Intel Nehalem Xeon with 4 GiB RAM. Mainboard is an ASUS Z8NR
D12. The hard drives are Western Digital SATA drives. They are connected via a
hardware raid controller with 256 MB RAM building Raid 6 arrays. On top we
setup a default ZFS pool with several ZFS filesystems.
What could be the source of this?
Thanks.
--
This message posted from opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
