[ceph-users] Re: SSD recommendations for RBD and VM's
I would recommend Intel S4510 series, which has power loss protection (PLP). If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 should also be OK (with separate DB/WAL on enterprise SSD with PLP) Samuel huxia...@horebdata.cn From: by morphin Date: 2021-05-30 02:48 To: Anthony D'Atri CC: Ceph Users Subject: [ceph-users] Re: SSD recommendations for RBD and VM's Hello Anthony. I use Qemu and I don't need size. I've 1000 vm and usually they're clones from the same rbd image. The image is 30GB. Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read intensive. Usage is stable and does not grow. So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB will be enough. Actually 120PCS 256GB will be better but the price goes up. I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've 40PCS 100GB. I'm gonna make them OSD as well. 7 years and DC S3700 still rocks. Not even one of them is dead. The SSD must be Low price & High TBW life span. Rest is not important. Anthony D'Atri , 30 May 2021 Paz, 02:26 tarihinde şunu yazdı: > > The choice depends on scale, your choice of chassis / form factor, budget, > workload and needs. > > The sizes you list seem awfully small. Tell us more about your use-case. > OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > —aad > > > > On May 29, 2021, at 2:10 PM, by morphin wrote: > > > > Hello. > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > What are the best Performance / Price SSDs in the market right now? > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > Is there a SSD recommendation list for ceph? > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Fwd: Re: Ceph osd will not start.
I've actually managed to get a little further with my problem. As I've said before these servers are slightly distorted in config. 63 drives and only 48g if memory. Once I create about 15-20 osds it continues to format the disks but won't actually create the containers or start any service. Worse than that on reboot the disks disappear, not stop working but not detected by Linux, which makes me think I'm hitting some kernel limit. At this point I'm going to cut my loses and give up and use the small slightly more powerful 30x drive systems I have (with 256g memory), maybe transplanting the larger disks if I need more capacity. Peter On Sat, 29 May 2021, 23:19 Marco Pizzolo, wrote: > Thanks David > We will investigate the bugs as per your suggestion, and then will look to > test with the custom image. > > Appreciate it. > > On Sat, May 29, 2021, 4:11 PM David Orman wrote: > >> You may be running into the same issue we ran into (make sure to read >> the first issue, there's a few mingled in there), for which we >> submitted a patch: >> >> https://tracker.ceph.com/issues/50526 >> https://github.com/alfredodeza/remoto/issues/62 >> >> If you're brave (YMMV, test first non-prod), we pushed an image with >> the issue we encountered fixed as per above here: >> https://hub.docker.com/repository/docker/ormandj/ceph/tags?page=1 . We >> 'upgraded' to this when we encountered the mgr hanging on us after >> updating ceph to v16 and experiencing this issue using: "ceph orch >> upgrade start --image docker.io/ormandj/ceph:v16.2.3-mgrfix". I've not >> tried to boostrap a new cluster with a custom image, and I don't know >> when 16.2.4 will be released with this change (hopefully) integrated >> as remoto accepted the patch upstream. >> >> I'm not sure if this is your exact issue, see the bug reports and see >> if you see the lock/the behavior matches, if so - then it may help you >> out. The only change in that image is that patch to remoto being >> overlaid on the default 16.2.3 image. >> >> On Fri, May 28, 2021 at 1:15 PM Marco Pizzolo >> wrote: >> > >> > Peter, >> > >> > We're seeing the same issues as you are. We have 2 new hosts Intel(R) >> > Xeon(R) Gold 6248R CPU @ 3.00GHz w/ 48 cores, 384GB RAM, and 60x 10TB >> SED >> > drives and we have tried both 15.2.13 and 16.2.4 >> > >> > Cephadm does NOT properly deploy and activate OSDs on Ubuntu 20.04.2 >> with >> > Docker. >> > >> > Seems to be a bug in Cephadm and a product regression, as we have 4 near >> > identical nodes on Centos running Nautilus (240 x 10TB SED drives) and >> had >> > no problems. >> > >> > FWIW we had no luck yet with one-by-one OSD daemon additions through >> ceph >> > orch either. We also reproduced the issue easily in a virtual lab using >> > small virtual disks on a single ceph VM with 1 mon. >> > >> > We are now looking into whether we can get past this with a manual >> buildout. >> > >> > If you, or anyone, has hit the same stumbling block and gotten past it, >> I >> > would really appreciate some guidance. >> > >> > Thanks, >> > Marco >> > >> > On Thu, May 27, 2021 at 2:23 PM Peter Childs wrote: >> > >> > > In the end it looks like I might be able to get the node up to about >> 30 >> > > odds before it stops creating any more. >> > > >> > > Or more it formats the disks but freezes up starting the daemons. >> > > >> > > I suspect I'm missing somthing I can tune to get it working better. >> > > >> > > If I could see any error messages that might help, but I'm yet to spit >> > > anything. >> > > >> > > Peter. >> > > >> > > On Wed, 26 May 2021, 10:57 Eugen Block, wrote: >> > > >> > > > > If I add the osd daemons one at a time with >> > > > > >> > > > > ceph orch daemon add osd drywood12:/dev/sda >> > > > > >> > > > > It does actually work, >> > > > >> > > > Great! >> > > > >> > > > > I suspect what's happening is when my rule for creating osds run >> and >> > > > > creates them all-at-once it ties the orch it overloads cephadm >> and it >> > > > can't >> > > > > cope. >> > > > >> > > > It's possible, I guess. >> > > > >> > > > > I suspect what I might need to do at least to work around the >> issue is >> > > > set >> > > > > "limit:" and bring it up until it stops working. >> > > > >> > > > It's worth a try, yes, although the docs state you should try to >> avoid >> > > > it, it's possible that it doesn't work properly, in that case >> create a >> > > > bug report. ;-) >> > > > >> > > > > I did work out how to get ceph-volume to nearly work manually. >> > > > > >> > > > > cephadm shell >> > > > > ceph auth get client.bootstrap-osd -o >> > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring >> > > > > ceph-volume lvm create --data /dev/sda --dmcrypt >> > > > > >> > > > > but given I've now got "add osd" to work, I suspect I just need >> to fine >> > > > > tune my osd creation rules, so it does not try and create too >> many osds >> > > > on >> > > > > the same node at the same time. >> > > > >> > > > I agree, no need to do it manually if
[ceph-users] Re: [External Email] Re: XFS on RBD on EC painfully slow
Reed, I'd like to add to Sebastian's comments - the problem is probably rsync. I inherited a smaller setup than you when I assumed my current responsibilities - an XFS file system on a RAID and exported over NFS. The backup process is based on RSnapshot, which is based on rsync over SSH, but the target is another XFS on hardware RAID. The file system contains a couple thousand user home directories for Computer Science students, so wide and deep and lots of small files. I was tormented by a daily backup process that was taking 3 days to run - copying the entire file system in a single rsync. What I ultimately concluded is that rsync runs exponentially slower as the size of the file tree to be copied increases. To get around this, if you play some games with find and GNU parallel, you can break your file system into many small sub-trees and run many rsyncs in parallel. Copying this way, you will see amazing throughput. From the point of view of a programmer, I think that rsync must build a representation of the source and destination file trees in memory and then must traverse and re-traverse them to make sure everything got copies and that nothing changed in the source tree. I've never read the code, but I've see articles that confirm my theory. In my case, because of inflexibility in RSnapshot I have ended up with 26 consecutive rsyncs - a*, b*, c*, etc. and I still go about twice as fast as I would with one large rsync. However, when I transferred this file system to a new NFS server and new storage I was able to directly rsync each user in parallel. I filled up a 10GB pipe and copied the whole FS in an hour. Typing in a hurry. If my explanation is confusing, please don't hesitate to ask me to explain better. -Dave -- Dave Hall Binghamton University kdh...@binghamton.edu On Fri, May 28, 2021 at 11:12 AM Sebastian Knust < skn...@physik.uni-bielefeld.de> wrote: > Hi Reed, > > To add to this command by Weiwen: > > On 28.05.21 13:03, 胡 玮文 wrote: > > Have you tried just start multiple rsync process simultaneously to > transfer different directories? Distributed system like ceph often benefits > from more parallelism. > > When I migrated from XFS on iSCSI (legacy system, no Ceph) to CephFS a > few months ago, I used msrsync [1] and was quite happy with the speed. > For your use case, I would start with -p 12 but might experiment with up > to -p 24 (as you only have 6C/12T in your CPU). With many small files, > you also might want to increase -s from the default 1000. > > Note that msrsync does not work with the --delete rsync flag. As I was > syncing a live system, I ended up with this workflow: > > - Initial sync with msrsync (something like ./msrsync -p 12 --progress > --stats --rsync "-aS --numeric-ids" ...) > - Second sync with msrsync (to sync changes during the first sync) > - Take old storage off-line for users / read-only > - Final rsync with --delete (i.e. rsync -aS --numeric-ids --delete ...) > - Mount cephfs at location of old storage, adjust /etc/exports with fsid > entries where necessary, turn system back on-line / read-write > > Cheers > Sebastian > > [1] https://github.com/jbd/msrsync > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: SSD recommendations for RBD and VM's
Hello Samuel. Thanks for the answer. Yes the Intel S4510 series is a good choice but it's expensive. I have 21 server and data distribution is quite well. At power loss I don't think I'll lose data. All the VM's using same image and the rest is cookie. In this case I'm not sure I should spend extra money on PLP. Actually I like Samsung 870 EVO. It's cheap and I think 300TBW will be enough for 5-10years. Do you know any better ssd with the same price range as 870 EVO? Samsung 870 evo (500GB) = 5 Years or 300 TBW - $64.99 Samsung 860 pro (512GB) = 5 Years or 600 TBW - $99 > > I would recommend Intel S4510 series, which has power loss protection (PLP). > > If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 > should also be OK (with separate DB/WAL on enterprise SSD with PLP) > > Samuel > > > huxia...@horebdata.cn > > > From: by morphin > Date: 2021-05-30 02:48 > To: Anthony D'Atri > CC: Ceph Users > Subject: [ceph-users] Re: SSD recommendations for RBD and VM's > Hello Anthony. > > I use Qemu and I don't need size. > I've 1000 vm and usually they're clones from the same rbd image. The > image is 30GB. > Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read > intensive. Usage is stable and does not grow. > So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. > I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB > will be enough. Actually 120PCS 256GB will be better but the price > goes up. > I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've > 40PCS 100GB. I'm gonna make them OSD as well. > 7 years and DC S3700 still rocks. Not even one of them is dead. > The SSD must be Low price & High TBW life span. Rest is not important. > > > Anthony D'Atri , 30 May 2021 Paz, 02:26 > tarihinde şunu yazdı: > > > > The choice depends on scale, your choice of chassis / form factor, budget, > > workload and needs. > > > > The sizes you list seem awfully small. Tell us more about your use-case. > > OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > > —aad > > > > > > > On May 29, 2021, at 2:10 PM, by morphin wrote: > > > > > > Hello. > > > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > > What are the best Performance / Price SSDs in the market right now? > > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > > > Is there a SSD recommendation list for ceph? > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: SSD recommendations for RBD and VM's
Pls check Crucial MX500 2TB drive, i think it is a bit cheaper than Samsung 970 EVO, and it is reliable as well. samuel From: mhnx Date: 2021-05-30 20:45 To: huxia...@horebdata.cn CC: Anthony D'Atri; ceph-users Subject: Re: [ceph-users] Re: SSD recommendations for RBD and VM's Hello Samuel. Thanks for the answer. Yes the Intel S4510 series is a good choice but it's expensive. I have 21 server and data distribution is quite well. At power loss I don't think I'll lose data. All the VM's using same image and the rest is cookie. In this case I'm not sure I should spend extra money on PLP. Actually I like Samsung 870 EVO. It's cheap and I think 300TBW will be enough for 5-10years. Do you know any better ssd with the same price range as 870 EVO? Samsung 870 evo (500GB) = 5 Years or 300 TBW - $64.99 Samsung 860 pro (512GB) = 5 Years or 600 TBW - $99 > > I would recommend Intel S4510 series, which has power loss protection (PLP). > > If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 > should also be OK (with separate DB/WAL on enterprise SSD with PLP) > > Samuel > > > huxia...@horebdata.cn > > > From: by morphin > Date: 2021-05-30 02:48 > To: Anthony D'Atri > CC: Ceph Users > Subject: [ceph-users] Re: SSD recommendations for RBD and VM's > Hello Anthony. > > I use Qemu and I don't need size. > I've 1000 vm and usually they're clones from the same rbd image. The > image is 30GB. > Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read > intensive. Usage is stable and does not grow. > So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. > I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB > will be enough. Actually 120PCS 256GB will be better but the price > goes up. > I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've > 40PCS 100GB. I'm gonna make them OSD as well. > 7 years and DC S3700 still rocks. Not even one of them is dead. > The SSD must be Low price & High TBW life span. Rest is not important. > > > Anthony D'Atri , 30 May 2021 Paz, 02:26 > tarihinde şunu yazdı: > > > > The choice depends on scale, your choice of chassis / form factor, budget, > > workload and needs. > > > > The sizes you list seem awfully small. Tell us more about your use-case. > > OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > > —aad > > > > > > > On May 29, 2021, at 2:10 PM, by morphin wrote: > > > > > > Hello. > > > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > > What are the best Performance / Price SSDs in the market right now? > > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > > > Is there a SSD recommendation list for ceph? > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Cephadm/docker or install from packages
Hi all, I want to ask your opinion which Ceph deploy version is better: using cephadm(docker) or installing from packages? Cephadm brings lots of convenience: easy upgrade (which sometimes stucks but still), easy add new OSDs, ability to make placement policies etc. In contrary I seem to lose some transparency on how Ceph works as well as ability to apply fine-grained control, namely setting up daemon configs myself. Even running ceph-volume lvm list to see which osd uses a particular block devices requires entering a docker container. So which option is better to use in production cluster? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
