Mark, thanks a lot for experimenting this for me. I’m gonna try master soon and will tell you how much I can get.
It’s interesting to see that using 2 SSDs brings up more performance, even both SSDs are under-utilized… They should be able to sustain both loads at the same time (journal and osd data). On 01 Sep 2014, at 09:51, Somnath Roy <somnath....@sandisk.com> wrote: > As I said, 107K with IOs serving from memory, not hitting the disk.. > > From: Jian Zhang [mailto:amberzhan...@gmail.com] > Sent: Sunday, August 31, 2014 8:54 PM > To: Somnath Roy > Cc: Haomai Wang; ceph-users@lists.ceph.com > Subject: Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K > IOPS > > Somnath, > on the small workload performance, 107k is higher than the theoretical IOPS > of 520, any idea why? > > > > >>Single client is ~14K iops, but scaling as number of clients increases. 10 > >>clients ~107K iops. ~25 cpu cores are used. > > > 2014-09-01 11:52 GMT+08:00 Jian Zhang <amberzhan...@gmail.com>: > Somnath, > on the small workload performance, > > > > 2014-08-29 14:37 GMT+08:00 Somnath Roy <somnath....@sandisk.com>: > > Thanks Haomai ! > > Here is some of the data from my setup. > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > Set up: > > -------- > > > > 32 core cpu with HT enabled, 128 GB RAM, one SSD (both journal and data) -> > one OSD. 5 client m/c with 12 core cpu and each running two instances of > ceph_smalliobench (10 clients total). Network is 10GbE. > > > > Workload: > > ------------- > > > > Small workload – 20K objects with 4K size and io_size is also 4K RR. The > intent is to serve the ios from memory so that it can uncover the performance > problems within single OSD. > > > > Results from Firefly: > > -------------------------- > > > > Single client throughput is ~14K iops, but as the number of client increases > the aggregated throughput is not increasing. 10 clients ~15K iops. ~9-10 cpu > cores are used. > > > > Result with latest master: > > ------------------------------ > > > > Single client is ~14K iops, but scaling as number of clients increases. 10 > clients ~107K iops. ~25 cpu cores are used. > > > > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > > > More realistic workload: > > ----------------------------- > > Let’s see how it is performing while > 90% of the ios are served from disks > > Setup: > > ------- > > 40 cpu core server as a cluster node (single node cluster) with 64 GB RAM. 8 > SSDs -> 8 OSDs. One similar node for monitor and rgw. Another node for client > running fio/vdbench. 4 rbds are configured with ‘noshare’ option. 40 GbE > network > > > > Workload: > > ------------ > > > > 8 SSDs are populated , so, 8 * 800GB = ~6.4 TB of data. Io_size = 4K RR. > > > > Results from Firefly: > > ------------------------ > > > > Aggregated output while 4 rbd clients stressing the cluster in parallel is > ~20-25K IOPS , cpu cores used ~8-10 cores (may be less can’t remember > precisely) > > > > Results from latest master: > > -------------------------------- > > > > Aggregated output while 4 rbd clients stressing the cluster in parallel is > ~120K IOPS , cpu is 7% idle i.e ~37-38 cpu cores. > > > > Hope this helps. > > > > Thanks & Regards > > Somnath > > > > -----Original Message----- > From: Haomai Wang [mailto:haomaiw...@gmail.com] > Sent: Thursday, August 28, 2014 8:01 PM > To: Somnath Roy > Cc: Andrey Korolyov; ceph-users@lists.ceph.com > Subject: Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K > IOPS > > > Hi Roy, > > > > I already scan your merged codes about "fdcache" and "optimizing for > lfn_find/lfn_open", could you give some performance improvement data about > it? I fully agree with your orientation, do you have any update about it? > > > > As for messenger level, I have some very early works on > it(https://github.com/yuyuyu101/ceph/tree/msg-event), it contains a new > messenger implementation which support different event mechanism. > > It looks like at least one more week to make it work. > > > > On Fri, Aug 29, 2014 at 5:48 AM, Somnath Roy <somnath....@sandisk.com> wrote: > > > Yes, what I saw the messenger level bottleneck is still huge ! > > > Hopefully RDMA messenger will resolve that and the performance gain will be > > significant for Read (on SSDs). For write we need to uncover the OSD > > bottlenecks first to take advantage of the improved upstream. > > > What I experienced that till you remove the very last bottleneck the > > performance improvement will not be visible and that could be confusing > > because you might think that the upstream improvement you did is not valid > > (which is not). > > > > > > Thanks & Regards > > > Somnath > > > -----Original Message----- > > > From: Andrey Korolyov [mailto:and...@xdel.ru] > > > Sent: Thursday, August 28, 2014 12:57 PM > > > To: Somnath Roy > > > Cc: David Moreau Simard; Mark Nelson; ceph-users@lists.ceph.com > > > Subject: Re: [ceph-users] [Single OSD performance on SSD] Can't go > > > over 3, 2K IOPS > > > > > > On Thu, Aug 28, 2014 at 10:48 PM, Somnath Roy <somnath....@sandisk.com> > > wrote: > > >> Nope, this will not be back ported to Firefly I guess. > > >> > > >> Thanks & Regards > > >> Somnath > > >> > > > > > > Thanks for sharing this, the first thing in thought when I looked at > > > this thread, was your patches :) > > > > > > If Giant will incorporate them, both the RDMA support and those should give > > a huge performance boost for RDMA-enabled Ceph backnets. > > > > > > ________________________________ > > > > > > PLEASE NOTE: The information contained in this electronic mail message is > > intended only for the use of the designated recipient(s) named above. If > > the reader of this message is not the intended recipient, you are hereby > > notified that you have received this message in error and that any review, > > dissemination, distribution, or copying of this message is strictly > > prohibited. If you have received this communication in error, please notify > > the sender by telephone or e-mail (as shown above) immediately and destroy > > any and all copies of this message in your possession (whether hard copies > > or electronically stored copies). > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Best Regards, > > > > Wheat > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Cheers. –––– Sébastien Han Cloud Architect "Always give 100%. Unless you're giving blood." Phone: +33 (0)1 49 70 99 72 Mail: sebastien....@enovance.com Address : 11 bis, rue Roquépine - 75008 Paris Web : www.enovance.com - Twitter : @enovance
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
