David, You should move to Hammer to get all the benefits of performance. It's all added to Giant and migrated to the present hammer LTS release. FYI, focus was so far with read performance improvement and what we saw in our environment with 6Gb SAS SSDs so far that we are able to saturate drives BW wise with 64K onwards. But, with smaller block like 4K we are not able to saturate the SAS SSD drives yet. But, considering Ceph's scale out nature you can get some very good numbers out of a cluster. For example, with 8 SAS SSD drives (in a JBOF) and having 2 heads in front (So, a 2 node Ceph cluster) we are able to hit ~300K Random read iops while 8 SSD aggregated performance would be ~400K. Not too bad. At this point we are saturating host cpus. We have seen almost linear scaling if you add similar setups i.e adding say ~3 of the above setup, you could hit ~900K RR iops. So, I would say it is definitely there in terms read iops and more improvement are coming. But, write path is very awful compare to read and that's where the problem is. Because, in the mainstream, no workload is 100% RR (IMO). So, even if you have say 90-10 read/write the performance numbers would be ~6/7 X slower. So, it is very much dependent on your workload/application access pattern and obviously the cost you are willing to spend.
Thanks & Regards Somnath -----Original Message----- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson Sent: Wednesday, September 30, 2015 12:04 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Ceph, SSD, and NVMe On 09/30/2015 09:34 AM, J David wrote: > Because we have a good thing going, our Ceph clusters are still > running Firefly on all of our clusters including our largest, all-SSD > cluster. > > If I understand right, newer versions of Ceph make much better use of > SSDs and give overall much higher performance on the same equipment. > However, the impression I get of newer versions is that they are also > not as stable as Firefly and should only be used with caution. > > Given our storage consumers have an effectively unlimited appetite for > IOPs and throughput, more performance would be very welcome. But not > if it leads to cluster crashes and lost data. > > What really prompts this is that we are starting to see large-scale > NVMe equipment appearing in the channel ( e.g. > http://www.supermicro.com/products/system/1U/1028/SYS-1028U-TN10RT_.cf > m ). The cost is significantly higher with commensurately higher > theoretical perfomance. But if we're already not pushing our SSD's to > the max over SAS, the added benefit of NVMe would largely be lost. > > On the other hand, if we could safely upgrade to a more recent version > that is as stable and bulletproof as Firefly has been for us, but has > better performance with SSDs, that would not only benefit our current > setup, it would be a necessary first step for moving onto NVMe. > > So this raises three questions: > > 1) Have I correctly understood that one or more post-FireFly releases > exist that (c.p.) perform significantly better with all-SSD setups? > > 2) Is there any such release that (generally) is as rock-solid as > FireFly. Of course this is somewhat situationally dependent, so I > would settle for: is there any such release that doesn't have any > known minding-my-own-business-suddenly-lost-data bugs in a 100% RBD > use case? > > 3) Has anyone done anything with NVMe as storage (not just journals) > who would care to share what kind of performance they experienced? > > (Of course if we do upgrade we will do so carefully, do a test cluster > first, have backups standing by, etc. But if it's already known that > doing so will either not improve anything or is likely to blow up in > our faces, it would be better to leave well enough alone. The current > performance is by no means bad, we're just always greedy for more. :) > ) > > Thanks for any advice/suggestions! Hi David, The single biggest performance improvement we've seen for SSDs has resulted from the memory allocator investigation that Chaitanya Hulgol and Somnath Roy spearheaded at Sandisk and others including myself have followed up and tried to expand on since then. See: http://www.spinics.net/lists/ceph-devel/msg25823.html https://www.mail-archive.com/ceph-devel@vger.kernel.org/msg23100.html http://www.spinics.net/lists/ceph-devel/msg21582.html I haven't tested firefly, but there's a good chance that you may see a significant performance improvement simply by upgrading your systems to tcmalloc 2.4 and loading the OSDs with 128MB of thread cache or LD_PRELOAD jemalloc. This isn't something we officially support in RHCS yet, but we'll likely be moving toward it for future releases based on the very positive results we are seeing. The biggest thing to keep in mind is that this does increase per-OSD memory usage by several hundred MB, so 3-4X IOPS increase does come with a cost. On the plus side, it also reduces CPU usage, sometimes dramatically. You may be able to offset the increased memory usage somewhat by disabling transparent huge pages (especially with jemalloc). See: http://www.spinics.net/lists/ceph-devel/msg26483.html FWIW, between Sage's newstore work, and recent work by Somnath Roy to optimize the write path, we may see further improvement, but neither of those are ready for production yet. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
