Anthony, I doubt the manufacturer reported 315MB/s for 4K block size. Most likely they've used 1M or 4M as the block size to achieve the 300MB/s+ speeds
Andrei ----- Original Message ----- > From: "Alexandre DERUMIER" <aderum...@odiso.com> > To: "Anthony Levesque" <aleves...@gtcomm.net> > Cc: "ceph-users" <ceph-users@lists.ceph.com> > Sent: Saturday, 25 April, 2015 5:32:30 PM > Subject: Re: [ceph-users] Possible improvements for a slow write > speed (excluding independent SSD journals) > I'm able to reach around 20000-25000iops with 4k block with s3500 > (with o_dsync) (so yes, around 80-100MB/S). > I'l bench new s3610 soon to compare. > ----- Mail original ----- > De: "Anthony Levesque" <aleves...@gtcomm.net> > À: "Christian Balzer" <ch...@gol.com> > Cc: "ceph-users" <ceph-users@lists.ceph.com> > Envoyé: Vendredi 24 Avril 2015 22:00:44 > Objet: Re: [ceph-users] Possible improvements for a slow write speed > (excluding independent SSD journals) > Hi Christian, > We tested some DC S3500 300GB using dd if=randfile of=/dev/sda bs=4k > count=100000 oflag=direct,dsync > we got 96 MB/s which is far from the 315 MB/s from the website. > Can I ask you or anyone on the mailing list how you are testing the > write speed for journals? > Thanks > --- > Anthony Lévesque > GloboTech Communications > Phone: 1-514-907-0050 x 208 > Toll Free: 1-(888)-GTCOMM1 x 208 > Phone Urgency: 1-(514) 907-0047 > 1-(866)-500-1555 > Fax: 1-(514)-907-0750 > aleves...@gtcomm.net > http://www.gtcomm.net > On Apr 23, 2015, at 9:05 PM, Christian Balzer < ch...@gol.com > > wrote: > Hello, > On Thu, 23 Apr 2015 18:40:38 -0400 Anthony Levesque wrote: > BQ_BEGIN > To update you on the current test in our lab: > 1.We tested the Samsung OSD in Recovery mode and the speed was able > to > maxout 2x 10GbE port(transferring data at 2200+ MB/s during > recovery). > So for normal write operation without O_DSYNC writes Samsung drives > seem > ok. > 2.We then tested a couple of different model of SSD we had in stock > with > the following command: > dd if=randfile of=/dev/sda bs=4k count=100000 oflag=direct,dsync > This was from a blog written by Sebastien Han and I think should be > able > to show how the drives would perform in O_DSYNC writes. For people > interested in some result of what we tested here they are: > Intel DC S3500 120GB = 114 MB/s > Samsung Pro 128GB = 2.4 MB/s > WD Black 1TB (HDD) = 409 KB/s > Intel 330 120GB = 105 MB/s > Intel 520 120GB = 9.4 MB/s > Intel 335 80GB = 9.4 MB/s > Samsung EVO 1TB = 2.5 MB/s > Intel 320 120GB = 78 MB/s > OCZ Revo Drive 240GB = 60.8 MB/s > 4x Samsung EVO 1TB LSI RAID0 HW + BBU = 28.4 MB/s > No real surprises here, but a nice summary nonetheless. > You _really_ want to avoid consumer SSDs for journals and have a good > idea > on how much data you'll write per day and how long you expect your > SSDs to > last (the TBW/$ ratio). > BQ_BEGIN > Please let us know if the command we ran was not optimal to test > O_DSYNC > writes > We order larger drive from Intel DC series to see if we could get > more > than 200 MB/s per SSD. We will keep you posted on tests if that > interested you guys. We dint test multiple parallel test yet (to > simulate multiple journal on one SSD). > BQ_END > You can totally trust the numbers on Intel's site: > http://ark.intel.com/products/family/83425/Data-Center-SSDs > The S3500s are by far the slowest and have the lowest endurance. > Again, depending on your expected write level the S3610 or S3700 > models > are going to be a better fit regarding price/performance. > Especially when you consider that loosing a journal SSD will result > in > several dead OSDs. > BQ_BEGIN > 3.We remove the Journal from all Samsung OSD and put 2x Intel 330 > 120GB > on all 6 Node to test. The overall speed we were getting from the > rados > bench went from 1000 MB/s(approx.) to 450 MB/s which might only be > because the intel cannot do too much in term of journaling (was > tested > at around 100 MB/s). It will be interesting to test with bigger Intel > DC S3500 drives(and more journals) per node to see if I can back up > to > 1000MB/s or even surpass it. > We also wanted to test if the CPU could be a huge bottle neck so we > swap > the Dual E5-2620v2 from node #6 and replace them with Dual > E5-2609v2(Which are much smaller in core and speed) and the 450 MB/s > we > got from he rados bench went even lower to 180 MB/s. > BQ_END > You really don't have to swap CPUs around, monitor things with atop > or > other tools to see where your bottlenecks are. > BQ_BEGIN > So Im wondering if the 1000MB/s we got when the Journal was shared on > the OSD SSD was not limited by the CPUs (even though the samsung are > not > good for journals on the long run) and not just by the fact Samsung > SSD > are bad in O_DSYNC writes(or maybe both). It is probable that 16 SSD > OSD per node in a full SSD cluster is too much and the major > bottleneck > will be from the CPU. > BQ_END > That's what I kept saying. ^.^ > BQ_BEGIN > 4.Im wondering if we find good SSD for the journal and keep the > samsung > for normal writes and read(We can saturate 20GbE easy with read > benchmark. We will test 40GbE soon) if the cluster will keep healthy > since Samsung seem to get burnt from O_DSYNC writes. > BQ_END > They will get burned, as in have their cells worn out by any and all > writes. > BQ_BEGIN > 5.In term of HBA controller, did you guys have made any test for a > full > SSD cluster or even just for SSD Journal. > BQ_END > If you have separate journals and OSDs, it often makes good sense to > have > them on separate controllers as well. > It all depends on density of your setup and capabilities of the > controllers. > LSI HBAs in IT mode are a known and working entity. > Christian > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Fusion Communications > http://www.gol.com/ > BQ_END > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
