Hi Christian, Yes we have purchased all our hardware, was very hard to convince management/finance to approve it, so some of the stuff we have is a bit cheap.
We have four storage nodes each with 6 x 6TB Western Digital Red SATA Drives (WD60EFRX-68M) and 6 x 1TB Samsung EVO 850s SSDs and 2x250GB Samsung EVO 850s (for OS raid). CPUs are Intel Atom C2750 @ 2.40GHz (8 Cores) with 32 GB of RAM. We have a 10Gig Network. The two options we are considering are: 1) Use two of the 1TB SSDs for the spinning disk journals (3 each) and then use the remaining 900+GB of each drive as an OSD to be part of the cache pool. 2) Put the spinning disk journals on the OS SSDs and use the 2 1TB SSDs for the cache pool. In both cases the other 4 1TB SSDs will be part of their own tier. Thanks a lot! Cameron Scrace Infrastructure Engineer Mobile +64 22 610 4629 Phone +64 4 462 5085 Email cameron.scr...@solnet.co.nz Solnet Solutions Limited Level 12, Solnet House 70 The Terrace, Wellington 6011 PO Box 397, Wellington 6140 www.solnet.co.nz From: Christian Balzer <ch...@gol.com> To: "ceph-us...@ceph.com" <ceph-us...@ceph.com> Cc: cameron.scr...@solnet.co.nz Date: 08/06/2015 12:18 p.m. Subject: Re: [ceph-users] Multiple journals and an OSD on one SSD doable? Hello, On Mon, 8 Jun 2015 09:55:56 +1200 cameron.scr...@solnet.co.nz wrote: > The other option we were considering was putting the journals on the OS > SSDs, they are only 250GB and the rest would be for the OS. Is that a > decent option? > You'll be getting a LOT better advice if you're telling us more details. For starters, have you bought the hardware yet? Tell us about your design, how many initial storage nodes, how many HDDs/SSDs per node, what CPUs/RAM/network? What SSDs are we talking about, exact models please. (Both the sizes you mentioned do not ring a bell for DC level SSDs I'm aware of) That said, I'm using Intel DC S3700s for mixed OS and journal use with good results. In your average Ceph storage node, normal OS (logging mostly) activity is a minute drop in the bucket for any decent SSD, so nearly all of it's resources are available to journals. You want to match the number of journals per SSD according to the capabilities of your SSD, HDDs and network. For example 8 HDD OSDs with 2 200GB DC S3700 and a 10Gb/s network is a decent match. The two SSDs at 900MB/s would appear to be the bottleneck, but in reality I'd expect the HDDs to be it. Never mind that you'd be more likely to be IOPS than bandwidth bound. Regards, Christian > Thanks! > > Cameron Scrace > Infrastructure Engineer > > Mobile +64 22 610 4629 > Phone +64 4 462 5085 > Email cameron.scr...@solnet.co.nz > Solnet Solutions Limited > Level 12, Solnet House > 70 The Terrace, Wellington 6011 > PO Box 397, Wellington 6140 > > www.solnet.co.nz > > > > From: Somnath Roy <somnath....@sandisk.com> > To: "cameron.scr...@solnet.co.nz" <cameron.scr...@solnet.co.nz>, > "ceph-us...@ceph.com" <ceph-us...@ceph.com> > Date: 08/06/2015 09:34 a.m. > Subject: RE: [ceph-users] Multiple journals and an OSD on one SSD > doable? > > > > Cameron, > Generally, it’s not a good idea. > You want to protect your SSDs used as journal.If any problem on that > disk, you will be losing all of your dependent OSDs. > I don’t think a bigger journal will gain you much performance , so, > default 5 GB journal size should be good enough. If you want to reduce > the fault domain and want to put 3 journals on a SSD , go for minimum > size and high endurance SSDs for that. > Now, if you want to use your rest of space of 1 TB ssd, creating just > OSDs will not gain you much (rather may get some burst performance). You > may want to consider the following. > > 1. If your spindle OSD size is much bigger than 900 GB , you don’t want > to make all OSDs of similar sizes, cache pool could be one of your > option. But, remember, cache pool can wear out your SSDs faster as > presently I guess it is not optimizing the extra writes. Sorry, I don’t > have exact data as I am yet to test that out. > > 2. If you want to make all the OSDs of similar sizes and you will be > able to create a substantial number of OSDs with your unused SSDs > (depends on how big the cluster is), you may want to put all of your > primary OSDs to SSD and gain significant performance boost for read. > Also, in this case, I don’t think you will be getting any burst > performance. > Thanks & Regards > Somnath > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > cameron.scr...@solnet.co.nz > Sent: Sunday, June 07, 2015 1:49 PM > To: ceph-us...@ceph.com > Subject: [ceph-users] Multiple journals and an OSD on one SSD doable? > > Setting up a Ceph cluster and we want the journals for our spinning > disks to be on SSDs but all of our SSDs are 1TB. We were planning on > putting 3 journals on each SSD, but that leaves 900+GB unused on the > drive, is it possible to use the leftover space as another OSD or will > it affect performance too much? > > Thanks, > > Cameron Scrace > Infrastructure Engineer > > Mobile +64 22 610 4629 > Phone +64 4 462 5085 > Email cameron.scr...@solnet.co.nz > Solnet Solutions Limited > Level 12, Solnet House > 70 The Terrace, Wellington 6011 > PO Box 397, Wellington 6140 > > www.solnet.co.nzAttention: This email may contain information intended > for the sole use of the original recipient. Please respect this when > sharing or disclosing this email's contents with any third party. If you > believe you have received this email in error, please delete it and > notify the sender or postmas...@solnetsolutions.co.nz as soon as > possible. The content of this email does not necessarily reflect the > views of Solnet Solutions Ltd. > > > PLEASE NOTE: The information contained in this electronic mail message > is intended only for the use of the designated recipient(s) named above. > If the reader of this message is not the intended recipient, you are > hereby notified that you have received this message in error and that > any review, dissemination, distribution, or copying of this message is > strictly prohibited. If you have received this communication in error, > please notify the sender by telephone or e-mail (as shown above) > immediately and destroy any and all copies of this message in your > possession (whether hard copies or electronically stored copies). > > > > Attention: > This email may contain information intended for the sole use of > the original recipient. Please respect this when sharing or > disclosing this email's contents with any third party. If you > believe you have received this email in error, please delete it > and notify the sender or postmas...@solnetsolutions.co.nz as > soon as possible. The content of this email does not necessarily > reflect the views of Solnet Solutions Ltd. > -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ Attention: This email may contain information intended for the sole use of the original recipient. Please respect this when sharing or disclosing this email's contents with any third party. If you believe you have received this email in error, please delete it and notify the sender or postmas...@solnetsolutions.co.nz as soon as possible. The content of this email does not necessarily reflect the views of Solnet Solutions Ltd.
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
