Hi all, thanks for your insights.
Eneko, > We tried to use a Samsung 840 Pro SSD as OSD some time ago and it was a > no-go; it wasn't that performance was bad, it > just didn't work for the kind of use of OSD. Any HDD was better than it (the > disk was healthy and have been used in a > software raid-1 for a pair of years). > > I suggest you check first that your Samsung 860 Pro disks work well for Ceph. > Also, how is your host's RAM? As already mentioned the hosts each have 64GB RAM. Each host has 3 SSDs for OSD usage. Each OSD is using about 1.3GB virtual memory / 400MB residual memory. Joachim, > I can only recommend the use of enterprise SSDs. We've tested many consumer > SSDs in the past, including your SSDs. Many > of them are not suitable for long-term use and some weard out within 6 months. Unfortunately I couldn't afford enterprise grade SSDs. But I suspect that my workload (about 20 VMs for our infrastructure, the most IO demanding is probably LDAP) is light enough that wearout won't be a problem. The issue I'm seeing then is probably related to direct IO if using bluestore. But with filestore, the file system cache probably hides the latency issues. Igor, > AFAIR Samsung 860 Pro isn't for enterprise market, you shouldn't use consumer > SSDs for Ceph. > > I had some experience with Samsung 960 Pro a while ago and it turned out that > it handled fsync-ed writes very slowly > (comparing to the original/advertised performance). Which one can probably > explain by the lack of power loss protection > for these drives. I suppose it's the same in your case. > > Here are a couple links on the topic: > > https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/ > > https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ Power loss protection wasn't a criteria for me as the cluster hosts are distributed in two buildings with separate battery backed UPSs. As mentioned above I suspect the main difference for my case between filestore and bluestore is file system cache vs. direct IO. Which means I will keep using filestore. Regards, Uwe _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
