Hi Sascha, What is your file system type, XFS or Btrfs ?
Thanks Wade On Wed, Feb 3, 2016 at 7:01 AM Sascha Vogt <sascha.v...@gmail.com> wrote: > Hi all, > > we recently tried adding a cache tier to our ceph cluster. We had 5 > spinning disks per hosts with a single journal NVMe disk, hosting the 5 > journals (1 OSD per spinning disk). We have 4 hosts up to now, so > overall 4 NVMes hosting 20 journals for 20 spinning disks. > > As we had some space left on the NVMes so we made two additional > partitions on each NVMe and created a 4 OSD cache tier. > > To our surprise the 4 OSD cache pool was able to deliver the same > performance then the previous 20 OSD pool while reducing the OPs on the > spinning disk to zero as long as the cache pool was sufficient to hold > all / most data (ceph is used for very short living KVM virtual machines > which do pretty heavy disk IO). > > As we don't need that much more storage right now we decided to extend > our cluster by adding 8 additional NVMe disks solely as a cache pool and > freeing the journal NVMes again. Now the question is: How to organize > the OSDs on the NVMe disks (2 per host)? > > As the NVMes peak around 5-7 concurrent sequential writes (tested with > fio) I thought about using 5 OSDs per NVMe. That would mean 10 > partitions (5 journals, 5 data). On the other hand the NVMes are only > 400GB large, so that would result in OSD disk sizes for <80 GB > (depending on the journal size). > > Would it make sense to skip the separate Journal partition and leave the > journal on the data disk itself and limitting it to a rather small > amount (lets say 1 GB or even less?) as SSDs typically don't like > sequential writes anyway? > > Or, if I leave journal and data on separate partitions should I reduce > the number of OSDs per disk to 3 as Ceph will most likly write to > journal and data in parallel and I therefore already get 6 parallel > "threads" of IO? > > Any feedback is highly appreciated :) > > Greetings > -Sascha- > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
