Barry, I have a similar setup and found that the 600GB 15K SAS drives work well. The 2TB 7200 disks did not work as well due to my not using SSD. Running the journal and the data on big slow drives will result in slow writes. All the big boys I've encountered are running SSDs.
Currently, I'm using two 515s with 5 600GB drives in a RAID0x1 configuration on each host. I do not currently have a cluster network setup, but my OSDs have 8 bonded 1gbe nics. Even with the fastest drives that you can get for the R515, I am considering trying to get an SSD sometime in the near future. This is due to performance issues I've run into trying to run an Oracle VM on Openstack Folsom. To not make this sound all doom and gloom. I am also running a website consisting of six vms that does not have as heavy random read write I/O and it runs fine. Here's a quick performance display with various block sizes on a host with 1 public 1Gbe link and 1 1Gbe link on the same vlan as the ceph cluster. I'm using RBD writeback caching on this VM. To accomplish this, I had to hack the libvirt volume.py file in Openstack, and enable it in the /etc/ceph/ceph.conf file on the host I was running it on. I know nothing of OpenNebula, so I can't speak to what it can or cannot do, and how to enable writeback caching in it. The rbd caching settings for ceph.conf can be found here. http://ceph.com/docs/master/rbd/rbd-config-ref/ Ex. ;Global Client Setting [client] rbd cache = true 4K: [root@optog3 temp]# dd if=/dev/zero of=here bs=4k count=50k oflag=direct 51200+0 records in 51200+0 records out 209715200 bytes (210 MB) copied, 7.09731 seconds, 29.5 MB/s [root@optog3 temp]# 8K [root@optog3 temp]# dd if=/dev/zero of=here bs=8192 count=50k oflag=direct 51200+0 records in 51200+0 records out 419430400 bytes (419 MB) copied, 7.36243 seconds, 57.0 MB/s [root@optog3 temp]# 4MB blocks. [root@optog3 temp]# dd if=/dev/zero of=here bs=4M count=500 oflag=direct 500+0 records in 500+0 records out 2097152000 bytes (2.1 GB) copied, 23.5803 seconds, 88.9 MB/s [root@optog3 temp]# 1GB blocks: [root@optog3 temp]# dd if=/dev/zero of=here bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.0053 seconds, 89.4 MB/s [root@optog3 temp]# This article by Hastexo, which I wish I would've seen before going to production may help you greatly with this decision. http://www.hastexo.com/resources/hints-and-kinks/solid-state-drives-and-ceph-osd-journals Dave Spano Optogenics Systems Administrator ----- Original Message ----- From: "Mark Nelson" <mark.nel...@inktank.com> To: ceph-users@lists.ceph.com Sent: Tuesday, May 7, 2013 9:17:24 AM Subject: Re: [ceph-users] Dell R515 performance and specification question On 05/07/2013 06:50 AM, Barry O'Rourke wrote: > Hi, > > I'm looking to purchase a production cluster of 3 Dell Poweredge R515's > which I intend to run in 3 x replication. I've opted for the following > configuration; > > 2 x 6 core processors > 32Gb RAM > H700 controller (1Gb cache) > 2 x SAS OS disks (in RAID1) > 2 x 1Gb ethernet (bonded for cluster network) > 2 x 1Gb ethernet (bonded for client network) > > and either 4 x 2Tb nearline SAS OSDs or 8 x 1Tb nearline SAS OSDs. Hi Barry, With so few disks and the inability to do 10GbE, you may want to consider doing something like 5-6 R410s or R415s and just using the on-board controller with a couple of SATA disks and 1 SSD for the journal. That should give you better aggregate performance since in your case you can't use 10GbE. It will also spread your OSDs across more hosts for better redundancy and may not cost that much more per GB since you won't need to use the H700 card if you are using an SSD for journals. It's not as dense as R515s or R720XDs can be when fully loaded, but for small clusters with few disks I think it's a good trade-off to get the added redundancy and avoid expander/controller complications. > > At the moment I'm undecided on the OSDs, although I'm swaying towards > the second option at the moment as it would give me more flexibility and > the option of using some of the disks as journals. > > I'm intending to use this cluster to host the images for ~100 virtual > machines, which will run on different hardware most likely be managed by > OpenNebula. > > I'd be interested to hear from anyone running a similar configuration > with a similar use case, especially people who have spent some time > benchmarking a similar configuration and still have a copy of the results. > > I'd also welcome any comments or critique on the above specification. > Purchases have to be made via Dell and 10Gb ethernet is out of the > question at the moment. > > Cheers, > > Barry > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
