Hi all!! I'm setting up a Ceph (version 0.80.6) cluster and I'mbenchmarking the infrastructure and Ceph itself. I've got 3 rack servers (Dell R630) each with it's own disks in enclosures. The cluster network bandwidth is of 10Gbps, the bandwidth between the RAID controller (Dell H830) and enclosure (MD1400) is of 48 Gbps and negotiated speed with each disk (configured as JBOD) is of 6 Gbps. We use 4 TB SAS disks, 7200 RPM with sustained bandwidth of 175 MB/s (confirmed with fio).
I tried to simulate a Ceph pattern by writing/reading on a single disk with XFS, a great number of 4 MB files with on instance of fio. [writetest] > ioengine=libaio > directory=/var/local/xfs > filesize=4m > iodepth=256 > rw= write > direct=1 > numjobs=1 > loops=1 > nrfiles=18000 > bs=4096k For the other tests (sequential reads, random writes, random reads), only the "rw" changes. I got the following bandwidth: Sequential write speed: 137 MB/s Sequential read speed: 144.5 MB/s Radom write speed: 134 MB/s Radom read speed: 144 MB/s Ok, it has an overhead associated with writing a great amount of 4 MB files. Now, I move on to bechmark with an OSD on top of the disk, so I've got the cluster with only 1 OSD (separate partitions for journal and data) and run rados bench. 1 Thread: Writes: 27.7 MB/s Reads: 90 MB/s Random reads: 82 MB/s 16 Threads (default): Writes: 37 MB/s Reads: 79.2 MB/s Random reads: 71.4 MB/s As you can see, writes are awfully slow. What I have notice is very high latencies: Total writes made: 16868 > Write size: 4194304 > Bandwidth (MB/sec): 37.449 > Stddev Bandwidth: 24.606 > Max bandwidth (MB/sec): 120 > Min bandwidth (MB/sec): 0 > Average Latency: 1.70862 > Stddev Latency: 1.13098 > Max latency: 6.33929 > Min latency: 0.20767 Is this throughput to be expected from 7.2K RPM disks with 4 TB size? Or is anything in the Ceph configuration that might be changed to decrease the observed latency? Any sugestions? Appreciated, Pedro Miranda.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
