Re: [ceph-users] The OSD can be “down” but still “in”.
Hi, If the OSD represents the primary one for a PG, then all IO will be stopped..which may lead to application failure.. no, that's not how it works. You have an acting set of OSDs for a PG, typically 3 OSDs in a replicated pool. If the primary OSD goes down, the secondary becomes the primary immediately and serves client requests. I recommend to read the docs [1] to get a better understanding of the workflow, or set up a practice environment to test failure scenarios and watch what happens if an OSD/host/rack etc. fails. Regards, Eugen [1] http://docs.ceph.com/docs/master/architecture/#peering-and-sets Zitat von M Ranga Swami Reddy : Thanks for reply. If the OSD represents the primary one for a PG, then all IO will be stopped..which may lead to application failure.. On Tue, Jan 22, 2019 at 5:32 PM Matthew Vernon wrote: Hi, On 22/01/2019 10:02, M Ranga Swami Reddy wrote: > Hello - If an OSD shown as down and but its still "in" state..what > will happen with write/read operations on this down OSD? It depends ;-) In a typical 3-way replicated setup with min_size 2, writes to placement groups on that OSD will still go ahead - when 2 replicas are written OK, then the write will complete. Once the OSD comes back up, these writes will then be replicated to that OSD. If it stays down for long enough to be marked out, then pgs on that OSD will be replicated elsewhere. If you had min_size 3 as well, then writes would block until the OSD was back up (or marked out and the pgs replicated to another OSD). Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] osd bad crc cause whole cluster halt
Hi list, I have encounter this problem both on jewel cluster and luminous cluster. The symptom is some request will be blocked forever and the whole cluster won't able to receive any data anymore. Further investigating shows the blocked request happened on 2 osds(the pool size is 2 so I guess it will effect all osd in an pg acting set).The osd log have many repeated error message like below error message in jewel cluster first effected osd 2019-01-21 10:08:18.836864 941f9b10 0 bad crc in data 1155321283 != exp 1909584237 2019-01-21 10:08:18.837013 941f9b10 0 -- 10.0.2.39:6800/23795 >> 10.0.2.40:6804/28471 pipe(0x7c15c000 sd=149 :53242 s=2 pgs=5 cs=1 l=0 c=0x7a697e60).fault, initiating reconnect 2019-01-21 10:08:18.839328 7cbb4b10 0 -- 10.0.2.39:6800/23795 >> 10.0.2.40:6804/28471 pipe(0x6b782b00 sd=235 :6800 s=0 pgs=0 cs=0 l=0 c=0x8b2f5440).accept connect_seq 2 vs existing 2 state connecting 2019-01-21 10:08:18.850772 7cbb4b10 0 bad crc in data 1155321283 != exp 1909584237 2019-01-21 10:08:18.850910 7cbb4b10 0 -- 10.0.2.39:6800/23795 >> 10.0.2.40:6804/28471 pipe(0x6b782b00 sd=235 :6800 s=2 pgs=58 cs=3 l=0 c=0x7a697e60).fault with nothing to send, going to st andby second effected osd 2019-01-21 10:06:12.282115 9513cb10 0 bad crc in data 1035875608 != exp 3787091679 2019-01-21 10:06:12.290395 abdcdb10 0 -- 10.0.1.40:6804/28471 submit_message osd_op_reply(1031289084 rbd_data.28ae2238e1f29.00a7df16 [set-alloc-hint object_size 4194304 write_size 4194304,write 65536~524288] v5503'1224666 uv1224666 ondisk = 0) v7 remote, 10.0.1.121:0/3226500701, failed lossy con, dropping message 0x70df6800 2019-01-21 10:06:12.297356 9eb3cb10 0 -- 10.0.1.40:6804/28471 submit_message osd_op_reply(1031289067 rbd_data.28ae2238e1f29.00a7defd [set-alloc-hint object_size 4194304 write_size 4194304,write 3211264~524288] v5503'1236405 uv1236405 ondisk = 0) v7 remote, 10.0.1.121:0/3226500701, failed lossy con, dropping message 0x716a0e00 2019-01-21 10:06:12.303597 abdcdb10 0 -- 10.0.1.40:6804/28471 submit_message osd_op_reply(1031289085 rbd_data.28ae2238e1f29.00a7df16 [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~524288] v5503'1224667 uv1224667 ondisk = 0) v7 remote, 10.0.1.121:0/3226500701, failed lossy con, dropping message 0x6e537000 2019-01-21 10:06:12.310642 9c33cb10 0 -- 10.0.1.40:6804/28471 submit_message osd_op_reply(1031289069 rbd_data.28ae2238e1f29.00a7defd [set-alloc-hint object_size 4194304 write_size 4194304,write 3735552~458752] v5503'1236406 uv1236406 ondisk = 0) v7 remote, 10.0.1.121:0/3226500701, failed lossy con, dropping message 0x71655000 2019-01-21 10:08:18.837438 94b3cb10 0 -- 10.0.2.40:6804/28471 >> 10.0.2.39:6800/23795 pipe(0x888acd00 sd=129 :6804 s=2 pgs=3916 cs=1 l=0 c=0x9202a7e0).fault, initiating reconnect 2019-01-21 10:08:18.839301 9323cb10 0 -- 10.0.2.40:6804/28471 >> 10.0.2.39:6800/23795 pipe(0x702a6000 sd=80 :6804 s=0 pgs=0 cs=0 l=0 c=0x8d086480).accept connect_seq 2 vs existing 2 state connecting 2019-01-21 10:08:18.851839 94b3cb10 0 -- 10.0.2.40:6804/28471 >> 10.0.2.39:6800/23795 pipe(0x888acd00 sd=129 :42636 s=2 pgs=3930 cs=3 l=0 c=0x9202a7e0).fault, initiating reconnect 2019-01-21 10:08:18.860245 7eaf5b10 0 -- 10.0.2.40:6804/28471 >> 10.0.2.39:6800/23795 pipe(0x888acd00 sd=129 :42636 s=1 pgs=3930 cs=4 l=0 c=0x9202a7e0).fault 2019-01-21 10:08:18.877537 94b3cb10 0 -- 10.0.2.40:6804/28471 >> 10.0.2.39:6800/23795 pipe(0x888acd00 sd=80 :42638 s=2 pgs=3931 cs=5 l=0 c=0x9202a7e0).fault, initiating reconnect error message in luminous cluster first effected osd 2018-12-11 23:14:43.034926 b560c8e0 0 -- 10.0.2.2:6802/15865 >> 10.0.2.37:6800/13016 conn(0x7648e00 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=8870304 cs=8856031 l=0).handle_connect_msg: challenging authorizer 2018-12-11 23:14:43.042915 b560c8e0 0 bad crc in front 1566330326 != exp 3283985696 2018-12-11 23:14:43.044587 b4e0c8e0 0 -- 10.0.2.2:6802/15865 >> 10.0.2.37:6800/13016 conn(0x919e700 :6802 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer 2018-12-11 23:14:43.045153 b4e0c8e0 0 -- 10.0.2.2:6802/15865 >> 10.0.2.37:6800/13016 conn(0x919e700 :6802 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 8856034 vs existing csq=8856033 existing_state=STATE_STANDBY second effected osd 2018-12-11 23:15:23.693508 b56158e0 0 -- 10.0.2.37:6800/13016 >> 10.0.2.2:6802/15865 conn(0x2f984e00 :6800 s=STATE_OPEN pgs=4450977 cs=8863341 l=0).fault initiating reconnect 2018-12-11 23:15:23.704284 b56158e0 0 -- 10.0.2.37:6800/13016 >> 10.0.2.2:6802/15865 conn(0x2f984e00 :6800 s=STATE_OPEN pgs=4450978 cs=8863343 l=0).fault initiating reconnect 2018-12-11 23:15:23.714925 b56158e0 0 -- 10.0.2.37:6800/13016 >> 10.0.2.2:6802/15865 conn(0x2f984e00 :6800 s=STATE_OPEN pgs=4450979 cs=8863345 l=0).fault initiating reconnect 2018-12-11 23:15:23.725507 b56158e0 0 -- 10.0.2.37:6800/13016 >> 10.0.2.2:6802/15865 conn(0x
Re: [ceph-users] read-only mounts of RBD images on multiple nodes for parallel reads
On Tue, Jan 22, 2019 at 01:26:29PM -0800, Void Star Nill wrote: > Regarding Mykola's suggestion to use Read-Only snapshots, what is the > overhead of creating these snapshots? I assume these are copy-on-write > snapshots, so there's no extra space consumed except for the metadata? Yes. -- Mykola Golub ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
Hi, thats a bad news. round about 5000 OSDs are affected from this issue. It's not realy a solution to redeploy this OSDs. Is it possible to migrate the local keys to the monitors? I see that the OSDs with the "lockbox feature" has only one key for data and journal partition and the older OSDs have individual keys for journal and data. Might this be a problem? And a other question. Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one host? So I could only migrate newer OSDs to ceph-volume and deploy new ones (after disk replacements) with ceph-volume until hopefuly there is a solution. Regards Manuel On Tue, 22 Jan 2019 07:44:02 -0500 Alfredo Deza wrote: > This is one case we didn't anticipate :/ We supported the wonky > lockbox setup and thought we wouldn't need to go further back, > although we did add support for both > plain and luks keys. > > Looking through the code, it is very tightly couple to > storing/retrieving keys from the monitors, and I don't know what > workarounds might be possible here other than throwing away the OSD > and deploying a new one (I take it this is not an option for you at > all) > > Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Migrating to a dedicated cluster network
Hello, Ceph users, is it possible to migrate already deployed Ceph cluster, which uses public network only, to a split public/dedicated networks? If so, can this be done without service disruption? I have now got a new hardware which makes this possible, but I am not sure how to do it. Another question is whether the cluster network can be done solely on top of IPv6 link-local addresses without any public address prefix. When deploying this cluster (Ceph Firefly, IIRC), I had problems with mixed IPv4/IPv6 addressing, and ended up with ms_bind_ipv6 = false in my Ceph conf. Thanks, -Yenya -- | Jan "Yenya" Kasprzak | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | This is the world we live in: the way to deal with computers is to google the symptoms, and hope that you don't have to watch a video. --P. Zaitcev ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating to a dedicated cluster network
Hi Yenya, Can I ask how your cluster looks and why you want to do the network splitting? We used to set up 9-12 OSD nodes (12-16 HDDs each) clusters using 2x10Gb for access and 2x10Gb for cluster network, however, I don't see the reasons to not use just one network for next cluster setup. Thanks Jakub śr., 23 sty 2019, 10:40: Jan Kasprzak napisał(a): > Hello, Ceph users, > > is it possible to migrate already deployed Ceph cluster, which uses > public network only, to a split public/dedicated networks? If so, > can this be done without service disruption? I have now got a new > hardware which makes this possible, but I am not sure how to do it. > > Another question is whether the cluster network can be done > solely on top of IPv6 link-local addresses without any public address > prefix. > > When deploying this cluster (Ceph Firefly, IIRC), I had problems > with mixed IPv4/IPv6 addressing, and ended up with ms_bind_ipv6 = false > in my Ceph conf. > > Thanks, > > -Yenya > > -- > | Jan "Yenya" Kasprzak > | > | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 > | > This is the world we live in: the way to deal with computers is to google > the symptoms, and hope that you don't have to watch a video. --P. Zaitcev > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Process stuck in D+ on cephfs mount
Yes sort of. I do have an inconsistent pg for a while, but it is on a different pool. But I take it this is related to a networking issue I currently have with rsync and broken pipe. Where exactly does it go wrong? The cephfs kernel clients is sending a request to the osd, but the osd never replies? > > >> >> >> I got one again >> >> [] wait_on_page_bit_killable+0x83/0xa0 >> [] __lock_page_or_retry+0xb2/0xc0 >> [] filemap_fault+0x3b7/0x410 >> [] ceph_filemap_fault+0x13c/0x310 [ceph] >> [] __do_fault+0x4c/0xc0 >> [] do_read_fault.isra.42+0x43/0x130 >> [] handle_mm_fault+0x6b1/0x1040 >> [] __do_page_fault+0x154/0x450 >> [] do_page_fault+0x35/0x90 >> [] page_fault+0x28/0x30 >> [] 0x >> >> > >This is likely caused by hang osd request, was you cluster health? > > >> >check /proc//stack to find where it is stuck >> > >> >> >> >> >> >> I have a process stuck in D+ writing to cephfs kernel mount. >> Anything >> >> can be done about this? (without rebooting) >> >> >> >> >> >> CentOS Linux release 7.5.1804 (Core) >> >> Linux 3.10.0-514.21.2.el7.x86_64 >> >> >> >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Using Ceph central backup storage - Best practice creating pools
Hi, due to performance issues RGW is not an option. This statement may be wrong, but there's the following aspect to consider. If I write a backup that is typically a large file, this is normally a single IO stream. This causes massive performance issues on Ceph because this single IO stream is sequentially written in small pieces on OSDs. To overcome this issue multi IO stream should be used when writing large files, and this means the application writing the backup must support multi IO stream. Considering this the following question comes up: If I write a backup into a RBD (that could be considered as a network share), will Ceph use single IO stream or multi IO stream on storage side? THX Am Di., 22. Jan. 2019 um 23:20 Uhr schrieb Christian Wuerdig : > > If you use librados directly it's up to you to ensure you can identify your > objects. Generally RADOS stores objects and not files so when you provide > your object ids you need to come up with a convention so you can correctly > identify them. If you need to provide meta data (i.e. a list of all existing > backups, when they were taken etc.) then again you need to manage that > yourself (probably in dedicated meta-data objects). Using RADOS namespaces > (like one per database) is probably a good idea. > Also keep in mind that for example Bluestore has a maximum object size of 4GB > so mapping files 1:1 to object is probably not a wise approach and you should > breakup your files into smaller chunks when storing them. There is > libradosstriper which handles the striping of large objects transparently but > not sure if that has support for RADOS namespaces. > > Using RGW instead might be an easier route to go down > > On Wed, 23 Jan 2019 at 10:10, cmonty14 <74cmo...@gmail.com> wrote: >> >> My backup client is using librados. >> I understand that defining a pool for the same application is recommended. >> >> However this would not answer my other questions: >> How can I identify a backup created by client A that I want to restore >> on another client Z? >> I mean typically client A would write a backup file identified by the >> filename. >> Would it be possible on client Z to identify this backup file by >> filename? If yes, how? >> >> Am Di., 22. Jan. 2019 um 15:07 Uhr schrieb : >> > >> > Hi, >> > >> > Ceph's pool are meant to let you define specific engineering rules >> > and/or application (rbd, cephfs, rgw) >> > They are not designed to be created in a massive fashion (see pgs etc) >> > So, create a pool for each engineering ruleset, and store your data in them >> > For what is left of your project, I believe you have to implement that >> > on top of Ceph >> > >> > For instance, let say you simply create a pool, with a rbd volume in it >> > You then create a filesystem on that, and map it on some server >> > Finally, you can push your files on that mountpoint, using various >> > Linux's user, acl or whatever : beyond that point, there is nothing more >> > specific to Ceph, it is "just" a mounted filesystem >> > >> > Regards, >> > >> > On 01/22/2019 02:16 PM, cmonty14 wrote: >> > > Hi, >> > > >> > > my use case for Ceph is providing a central backup storage. >> > > This means I will backup multiple databases in Ceph storage cluster. >> > > >> > > This is my question: >> > > What is the best practice for creating pools & images? >> > > Should I create multiple pools, means one pool per database? >> > > Or should I create a single pool "backup" and use namespace when writing >> > > data in the pool? >> > > >> > > This is the security demand that should be considered: >> > > DB-owner A can only modify the files that belong to A; other files >> > > (owned by B, C or D) are accessible for A. >> > > >> > > And there's another issue: >> > > How can I identify a backup created by client A that I want to restore >> > > on another client Z? >> > > I mean typically client A would write a backup file identified by the >> > > filename. >> > > Would it be possible on client Z to identify this backup file by >> > > filename? If yes, how? >> > > >> > > >> > > THX >> > > ___ >> > > ceph-users mailing list >> > > ceph-users@lists.ceph.com >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >> > ___ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] crush location hook with mimic
Hi,
we are having issues with the crush location hooks on Mimic:
we deployed the same script we have been using since Hammer (and has
been working fine also in Jewel) that returns:
root=fresh-install host=$(hostname -s)-fresh
however it seems the output of the script is completely disregarded.
Upon OSD restart the MON logs:
mon-l19-31: 2019-01-23 11:23:42.247 7f1d533f8700 0
mon.mon-l19-31@0(leader) e3 handle_command mon_command({"prefix": "osd
crush create-or-move", "id": 16, "weight":1.4555, "args":
["host=vhp-l6-37", "root=default"]} v 0) v1
mon-l19-31: 2019-01-23 11:23:42.247 7f1d533f8700 0 log_channel(audit)
log [INF] : from='osd.16 10.129.48.32:6806/905786' entity='osd.16'
cmd=[{"prefix": "osd crush create-or-move", "id": 16, "weight":1.4555,
"args": ["host=vhp-l6-37", "root=default"]}]: dispatch
Anyone has any pointer on how to solve this?
(Someone was having a similar issue in the past but that seemed to be
related to an extra intermediate field like datacenter)
Kind regards,
Mattia
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating to a dedicated cluster network
Jakub Jaszewski wrote: : Hi Yenya, : : Can I ask how your cluster looks and why you want to do the network : splitting? Jakub, we have deployed the Ceph cluster originally as a proof of concept for a private cloud. We run OpenNebula and Ceph on about 30 old servers with old HDDs (2 OSDs per host), all connected via 1 Gbit ethernet with 10Gbit backbone. Since then our private cloud got pretty popular among our users, so we are planning to upgrade it to a smaller amount of modern servers. The new servers have two 10GbE interfaces, so the primary reasoning behind it is "why not use them both when we already have them". Of course, interface teaming/bonding is another option. Currently I see the network being saturated only when doing a live migration of a VM between the physical hosts, and then during a Ceph cluster rebalance. So, I don't think moving to a dedicated cluster network is a necessity for us. Anyway, does anybody use the cluster network with larger MTU (jumbo frames)? : We used to set up 9-12 OSD nodes (12-16 HDDs each) clusters using 2x10Gb : for access and 2x10Gb for cluster network, however, I don't see the reasons : to not use just one network for next cluster setup. -Yenya : śr., 23 sty 2019, 10:40: Jan Kasprzak napisał(a): : : > Hello, Ceph users, : > : > is it possible to migrate already deployed Ceph cluster, which uses : > public network only, to a split public/dedicated networks? If so, : > can this be done without service disruption? I have now got a new : > hardware which makes this possible, but I am not sure how to do it. : > : > Another question is whether the cluster network can be done : > solely on top of IPv6 link-local addresses without any public address : > prefix. : > : > When deploying this cluster (Ceph Firefly, IIRC), I had problems : > with mixed IPv4/IPv6 addressing, and ended up with ms_bind_ipv6 = false : > in my Ceph conf. : > : > Thanks, : > : > -Yenya : > : > -- : > | Jan "Yenya" Kasprzak : > | : > | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 : > | : > This is the world we live in: the way to deal with computers is to google : > the symptoms, and hope that you don't have to watch a video. --P. Zaitcev : > ___ : > ceph-users mailing list : > ceph-users@lists.ceph.com : > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com : > -- | Jan "Yenya" Kasprzak | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | This is the world we live in: the way to deal with computers is to google the symptoms, and hope that you don't have to watch a video. --P. Zaitcev ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Cephfs snapshot create date
How can I get the snapshot create date on cephfs. When I do an ls on .snap dir it will give me the date of the snapshot source date. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Spec for Ceph Mon+Mgr?
jes...@krogh.cc wrote: : Hi. : : We're currently co-locating our mons with the head node of our Hadoop : installation. That may be giving us some problems, we dont know yet, but : thus I'm speculation about moving them to dedicated hardware. : : It is hard to get specifications "small" engough .. the specs for the : mon is where we usually virtualize our way out of if .. which seems very : wrong here. : : Are other people just co-locating it with something random or what are : others typically using in a small ceph cluster (< 100 OSDs .. 7 OSD hosts) Jesper, FWIW, we colocate our mons/mgrs with OpenNebula master node and minor OpenNebula host nodes. As an example, one of them is AMD Opteron 6134 (8 cores, 2.3 GHz), 16 GB RAM, 1 Gbit ethernet. We have three mons. I want to keep this setup also in the future, but I may move the OpenNebula virtualization out of mon hosts - not because the hosts are overloaded, but they are getting too old/slow/small for the VMs themselves :-). We have three mons with a similar configuration. -Yenya -- | Jan "Yenya" Kasprzak | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | This is the world we live in: the way to deal with computers is to google the symptoms, and hope that you don't have to watch a video. --P. Zaitcev ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, Jan 23, 2019 at 4:01 AM Manuel Lausch wrote:
>
> Hi,
>
> thats a bad news.
>
> round about 5000 OSDs are affected from this issue. It's not realy a
> solution to redeploy this OSDs.
>
> Is it possible to migrate the local keys to the monitors?
> I see that the OSDs with the "lockbox feature" has only one key for
> data and journal partition and the older OSDs have individual keys for
> journal and data. Might this be a problem?
I don't know how that would look like, but I think it is worth a try
if re-deploying OSDs is not feasible for you.
The key api for encryption is *very* odd and a lot of its quirks are
undocumented. For example, ceph-volume is stuck supporting naming
files and keys 'lockbox'
(for backwards compatibility) but there is no real lockbox anymore.
Another quirk is that when storing the secret in the monitor, it is
done using the following convention:
dm-crypt/osd/{OSD FSID}/luks
The 'luks' part there doesn't indicate anything about the type of
encryption (!!) so regardless of the type of encryption (luks or
plain) the key would still go there.
If you manage to get the keys into the monitors you still wouldn't be
able to scan OSDs to produce the JSON files, but you would be able to
create the JSON file with the
metadata that ceph-volume needs to run the OSD.
The contents are documented here:
http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents
>
> And a other question.
> Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one
> host?
I don't think it is a problem, but we don't test it so I can't say
with certainty.
> So I could only migrate newer OSDs to ceph-volume and deploy new
> ones (after disk replacements) with ceph-volume until hopefuly there is
> a solution.
I would strongly suggest implementing some automation to get all those
OSDs into 100% ceph-volume. The ceph-volume tooling for handling
ceph-disk OSDs is very
very robust and works very well, but it shouldn't be a long term
solution for OSDs that have been deployed with ceph-disk.
>
> Regards
> Manuel
>
>
> On Tue, 22 Jan 2019 07:44:02 -0500
> Alfredo Deza wrote:
>
>
> > This is one case we didn't anticipate :/ We supported the wonky
> > lockbox setup and thought we wouldn't need to go further back,
> > although we did add support for both
> > plain and luks keys.
> >
> > Looking through the code, it is very tightly couple to
> > storing/retrieving keys from the monitors, and I don't know what
> > workarounds might be possible here other than throwing away the OSD
> > and deploying a new one (I take it this is not an option for you at
> > all)
> >
> >
> Manuel Lausch
>
> Systemadministrator
> Storage Services
>
> 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 |
> 76135 Karlsruhe | Germany Phone: +49 721 91374-1847
> E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de
>
> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452
>
> Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer
>
>
> Member of United Internet
>
> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte
> Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat
> sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie
> bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem
> bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern,
> weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu
> verwenden.
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient of this e-mail, you are hereby
> notified that saving, distribution or use of the content of this e-mail
> in any way is prohibited. If you have received this e-mail in error,
> please notify the sender and delete the e-mail.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, Jan 23, 2019 at 10:01:05AM +0100, Manuel Lausch wrote: Hi, thats a bad news. round about 5000 OSDs are affected from this issue. It's not realy a solution to redeploy this OSDs. Is it possible to migrate the local keys to the monitors? I see that the OSDs with the "lockbox feature" has only one key for data and journal partition and the older OSDs have individual keys for journal and data. Might this be a problem? And a other question. Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one host? So I could only migrate newer OSDs to ceph-volume and deploy new ones (after disk replacements) with ceph-volume until hopefuly there is a solution. I might be wrong on this, since its been a while since I played with that. But iirc you can't migrate a subset of ceph-disk OSDs to ceph-volume on one host. Once you run ceph-volume simple activate, the ceph-disk systemd units and udev profiles will be disabled. While the remaining ceph-disk OSDs will continue to run, they won't come up after a reboot. I'm sure there's a way to get them running again, but I imagine you'd rather not manually deal with that. Regards Manuel On Tue, 22 Jan 2019 07:44:02 -0500 Alfredo Deza wrote: This is one case we didn't anticipate :/ We supported the wonky lockbox setup and thought we wouldn't need to go further back, although we did add support for both plain and luks keys. Looking through the code, it is very tightly couple to storing/retrieving keys from the monitors, and I don't know what workarounds might be possible here other than throwing away the OSD and deploying a new one (I take it this is not an option for you at all) Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jan Fajerski Engineer Enterprise Storage SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, Jan 23, 2019 at 8:25 AM Jan Fajerski wrote: > > On Wed, Jan 23, 2019 at 10:01:05AM +0100, Manuel Lausch wrote: > >Hi, > > > >thats a bad news. > > > >round about 5000 OSDs are affected from this issue. It's not realy a > >solution to redeploy this OSDs. > > > >Is it possible to migrate the local keys to the monitors? > >I see that the OSDs with the "lockbox feature" has only one key for > >data and journal partition and the older OSDs have individual keys for > >journal and data. Might this be a problem? > > > >And a other question. > >Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one > >host? > >So I could only migrate newer OSDs to ceph-volume and deploy new > >ones (after disk replacements) with ceph-volume until hopefuly there is > >a solution. > I might be wrong on this, since its been a while since I played with that. But > iirc you can't migrate a subset of ceph-disk OSDs to ceph-volume on one host. > Once you run ceph-volume simple activate, the ceph-disk systemd units and udev > profiles will be disabled. While the remaining ceph-disk OSDs will continue to > run, they won't come up after a reboot. This is correct, once you "activate" ceph-disk OSDs via ceph-volume you are disabling all udev/systemd triggers for those OSDs, so you must migrate all. I was assuming the question was more of a way to keep existing ceph-disk OSDs and create new ceph-volume OSDs, which you can, as long as this is not Nautilus or newer where ceph-disk doesn't exist > I'm sure there's a way to get them running again, but I imagine you'd rather > not > manually deal with that. > > > >Regards > >Manuel > > > > > >On Tue, 22 Jan 2019 07:44:02 -0500 > >Alfredo Deza wrote: > > > > > >> This is one case we didn't anticipate :/ We supported the wonky > >> lockbox setup and thought we wouldn't need to go further back, > >> although we did add support for both > >> plain and luks keys. > >> > >> Looking through the code, it is very tightly couple to > >> storing/retrieving keys from the monitors, and I don't know what > >> workarounds might be possible here other than throwing away the OSD > >> and deploying a new one (I take it this is not an option for you at > >> all) > >> > >> > >Manuel Lausch > > > >Systemadministrator > >Storage Services > > > >1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | > >76135 Karlsruhe | Germany Phone: +49 721 91374-1847 > >E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de > > > >Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 > > > >Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer > > > > > >Member of United Internet > > > >Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte > >Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat > >sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie > >bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem > >bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, > >weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu > >verwenden. > > > >This e-mail may contain confidential and/or privileged information. If > >you are not the intended recipient of this e-mail, you are hereby > >notified that saving, distribution or use of the content of this e-mail > >in any way is prohibited. If you have received this e-mail in error, > >please notify the sender and delete the e-mail. > >___ > >ceph-users mailing list > >ceph-users@lists.ceph.com > >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- > Jan Fajerski > Engineer Enterprise Storage > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > HRB 21284 (AG Nürnberg) > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Process stuck in D+ on cephfs mount
On Wed, Jan 23, 2019 at 6:07 PM Marc Roos wrote: > > Yes sort of. I do have an inconsistent pg for a while, but it is on a > different pool. But I take it this is related to a networking issue I > currently have with rsync and broken pipe. > > Where exactly does it go wrong? The cephfs kernel clients is sending a > request to the osd, but the osd never replies? > yes, please check if there are hang requests in /sys/kernel/debug/ceph/xxx/osdc > > > > > >> > >> > >> I got one again > >> > >> [] wait_on_page_bit_killable+0x83/0xa0 > >> [] __lock_page_or_retry+0xb2/0xc0 > >> [] filemap_fault+0x3b7/0x410 > >> [] ceph_filemap_fault+0x13c/0x310 [ceph] > >> [] __do_fault+0x4c/0xc0 > >> [] do_read_fault.isra.42+0x43/0x130 > >> [] handle_mm_fault+0x6b1/0x1040 > >> [] __do_page_fault+0x154/0x450 > >> [] do_page_fault+0x35/0x90 > >> [] page_fault+0x28/0x30 > >> [] 0x > >> > >> > > > >This is likely caused by hang osd request, was you cluster health? > > > > > >> >check /proc//stack to find where it is stuck > >> > > >> >> > >> >> > >> >> I have a process stuck in D+ writing to cephfs kernel mount. > >> Anything > >> >> can be done about this? (without rebooting) > >> >> > >> >> > >> >> CentOS Linux release 7.5.1804 (Core) > >> >> Linux 3.10.0-514.21.2.el7.x86_64 > >> >> > >> > >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, 23 Jan 2019 14:25:00 +0100 Jan Fajerski wrote: > I might be wrong on this, since its been a while since I played with > that. But iirc you can't migrate a subset of ceph-disk OSDs to > ceph-volume on one host. Once you run ceph-volume simple activate, > the ceph-disk systemd units and udev profiles will be disabled. While > the remaining ceph-disk OSDs will continue to run, they won't come up > after a reboot. I'm sure there's a way to get them running again, but > I imagine you'd rather not manually deal with that. yes you are right. The activate disables system wide the ceph-disk. This is done by symlinking /etc/systemd/system/ceph-disk@.service to /dev/null. After deleting this symlink my OSDs started again after reboot. The startup processes from ceph-volume and ceph-disk might conflicts each other but on a QA system this did work. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, 23 Jan 2019 08:11:31 -0500
Alfredo Deza wrote:
> I don't know how that would look like, but I think it is worth a try
> if re-deploying OSDs is not feasible for you.
yes, is there a working way to migrate this I will have a try it.
>
> The key api for encryption is *very* odd and a lot of its quirks are
> undocumented. For example, ceph-volume is stuck supporting naming
> files and keys 'lockbox'
> (for backwards compatibility) but there is no real lockbox anymore.
> Another quirk is that when storing the secret in the monitor, it is
> done using the following convention:
>
> dm-crypt/osd/{OSD FSID}/luks
>
> The 'luks' part there doesn't indicate anything about the type of
> encryption (!!) so regardless of the type of encryption (luks or
> plain) the key would still go there.
>
> If you manage to get the keys into the monitors you still wouldn't be
> able to scan OSDs to produce the JSON files, but you would be able to
> create the JSON file with the
> metadata that ceph-volume needs to run the OSD.
I think it is not that problem to create the json files by myself.
Moving the Keys to the monitors and creating appropriate auth-keys
should be more or less easy as well.
The problem I see is, that there are individual keys for the journal
and data partition while the new process useses only one key for both
partitions.
maybe I can recreate the journal partition with the other key. But is
this possible? Are there important data ramaining on the journal after
clean stopping the OSD which I cannot throw away without trashing the
whole OSD?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, Jan 23, 2019 at 4:15 PM Manuel Lausch wrote: > yes you are right. The activate disables system wide the ceph-disk. > This is done by symlinking /etc/systemd/system/ceph-disk@.service > to /dev/null. > After deleting this symlink my OSDs started again after reboot. > The startup processes from ceph-volume and ceph-disk might conflicts > each other but on a QA system this did work. Yeah, we are also running some clusters with mixed ceph-volume and unmigrated ceph-disk OSDs, the startup works fine. Paul > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On 1/23/19 3:05 PM, Alfredo Deza wrote: > On Wed, Jan 23, 2019 at 8:25 AM Jan Fajerski wrote: >> >> On Wed, Jan 23, 2019 at 10:01:05AM +0100, Manuel Lausch wrote: >>> Hi, >>> >>> thats a bad news. >>> >>> round about 5000 OSDs are affected from this issue. It's not realy a >>> solution to redeploy this OSDs. >>> >>> Is it possible to migrate the local keys to the monitors? >>> I see that the OSDs with the "lockbox feature" has only one key for >>> data and journal partition and the older OSDs have individual keys for >>> journal and data. Might this be a problem? >>> >>> And a other question. >>> Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one >>> host? >>> So I could only migrate newer OSDs to ceph-volume and deploy new >>> ones (after disk replacements) with ceph-volume until hopefuly there is >>> a solution. >> I might be wrong on this, since its been a while since I played with that. >> But >> iirc you can't migrate a subset of ceph-disk OSDs to ceph-volume on one host. >> Once you run ceph-volume simple activate, the ceph-disk systemd units and >> udev >> profiles will be disabled. While the remaining ceph-disk OSDs will continue >> to >> run, they won't come up after a reboot. > > This is correct, once you "activate" ceph-disk OSDs via ceph-volume > you are disabling all udev/systemd triggers for > those OSDs, so you must migrate all. > > I was assuming the question was more of a way to keep existing > ceph-disk OSDs and create new ceph-volume OSDs, which you can, as long > as this is not Nautilus or newer where ceph-disk doesn't exist > Will there be any plans to implement a command in ceph-volume that allows to create simple volumes like the ones that are migrated from ceph-disk using the scan and activate commands from ceph-volume? >> I'm sure there's a way to get them running again, but I imagine you'd rather >> not >> manually deal with that. >>> >>> Regards >>> Manuel >>> >>> >>> On Tue, 22 Jan 2019 07:44:02 -0500 >>> Alfredo Deza wrote: >>> >>> This is one case we didn't anticipate :/ We supported the wonky lockbox setup and thought we wouldn't need to go further back, although we did add support for both plain and luks keys. Looking through the code, it is very tightly couple to storing/retrieving keys from the monitors, and I don't know what workarounds might be possible here other than throwing away the OSD and deploying a new one (I take it this is not an option for you at all) >>> Manuel Lausch >>> >>> Systemadministrator >>> Storage Services >>> >>> 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | >>> 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 >>> E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de >>> >>> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 >>> >>> Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer >>> >>> >>> Member of United Internet >>> >>> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte >>> Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat >>> sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie >>> bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem >>> bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, >>> weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu >>> verwenden. >>> >>> This e-mail may contain confidential and/or privileged information. If >>> you are not the intended recipient of this e-mail, you are hereby >>> notified that saving, distribution or use of the content of this e-mail >>> in any way is prohibited. If you have received this e-mail in error, >>> please notify the sender and delete the e-mail. >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> -- >> Jan Fajerski >> Engineer Enterprise Storage >> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, >> HRB 21284 (AG Nürnberg) >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- _ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Division for Bioinformatics Innrain 80, 6020 Innsbruck Email: dietmar.rie...@i-med.ac.at Web: http://www.icbi.at signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] migrate ceph-disk to ceph-volume fails with dmcrypt
On Wed, Jan 23, 2019 at 11:03 AM Dietmar Rieder wrote: > > On 1/23/19 3:05 PM, Alfredo Deza wrote: > > On Wed, Jan 23, 2019 at 8:25 AM Jan Fajerski wrote: > >> > >> On Wed, Jan 23, 2019 at 10:01:05AM +0100, Manuel Lausch wrote: > >>> Hi, > >>> > >>> thats a bad news. > >>> > >>> round about 5000 OSDs are affected from this issue. It's not realy a > >>> solution to redeploy this OSDs. > >>> > >>> Is it possible to migrate the local keys to the monitors? > >>> I see that the OSDs with the "lockbox feature" has only one key for > >>> data and journal partition and the older OSDs have individual keys for > >>> journal and data. Might this be a problem? > >>> > >>> And a other question. > >>> Is it a good idea to mix ceph-disk and ceph-volume managed OSDSs on one > >>> host? > >>> So I could only migrate newer OSDs to ceph-volume and deploy new > >>> ones (after disk replacements) with ceph-volume until hopefuly there is > >>> a solution. > >> I might be wrong on this, since its been a while since I played with that. > >> But > >> iirc you can't migrate a subset of ceph-disk OSDs to ceph-volume on one > >> host. > >> Once you run ceph-volume simple activate, the ceph-disk systemd units and > >> udev > >> profiles will be disabled. While the remaining ceph-disk OSDs will > >> continue to > >> run, they won't come up after a reboot. > > > > This is correct, once you "activate" ceph-disk OSDs via ceph-volume > > you are disabling all udev/systemd triggers for > > those OSDs, so you must migrate all. > > > > I was assuming the question was more of a way to keep existing > > ceph-disk OSDs and create new ceph-volume OSDs, which you can, as long > > as this is not Nautilus or newer where ceph-disk doesn't exist > > > > Will there be any plans to implement a command in ceph-volume that > allows to create simple volumes like the ones that are migrated from > ceph-disk using the scan and activate commands from ceph-volume? The idea is that this is open-ended: you can create your OSDs in whatever way you want, and add the information to the JSON files to get them up and running. Implementing a 'create' for it (if I'm understanding this correctly) would imply having some sort of opinion on the creation process, which would go against what the intention was for the command. > > > >> I'm sure there's a way to get them running again, but I imagine you'd > >> rather not > >> manually deal with that. > >>> > >>> Regards > >>> Manuel > >>> > >>> > >>> On Tue, 22 Jan 2019 07:44:02 -0500 > >>> Alfredo Deza wrote: > >>> > >>> > This is one case we didn't anticipate :/ We supported the wonky > lockbox setup and thought we wouldn't need to go further back, > although we did add support for both > plain and luks keys. > > Looking through the code, it is very tightly couple to > storing/retrieving keys from the monitors, and I don't know what > workarounds might be possible here other than throwing away the OSD > and deploying a new one (I take it this is not an option for you at > all) > > > >>> Manuel Lausch > >>> > >>> Systemadministrator > >>> Storage Services > >>> > >>> 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | > >>> 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 > >>> E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de > >>> > >>> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 > >>> > >>> Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer > >>> > >>> > >>> Member of United Internet > >>> > >>> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte > >>> Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat > >>> sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie > >>> bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem > >>> bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, > >>> weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu > >>> verwenden. > >>> > >>> This e-mail may contain confidential and/or privileged information. If > >>> you are not the intended recipient of this e-mail, you are hereby > >>> notified that saving, distribution or use of the content of this e-mail > >>> in any way is prohibited. If you have received this e-mail in error, > >>> please notify the sender and delete the e-mail. > >>> ___ > >>> ceph-users mailing list > >>> ceph-users@lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > >> -- > >> Jan Fajerski > >> Engineer Enterprise Storage > >> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > >> HRB 21284 (AG Nürnberg) > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http:
Re: [ceph-users] Process stuck in D+ on cephfs mount
Are there any others I need to grab? So can I do all at once. I do not like to have to restart this one so often. > > Yes sort of. I do have an inconsistent pg for a while, but it is on a > different pool. But I take it this is related to a networking issue I > currently have with rsync and broken pipe. > > Where exactly does it go wrong? The cephfs kernel clients is sending a > request to the osd, but the osd never replies? > yes, please check if there are hang requests in /sys/kernel/debug/ceph/xxx/osdc > > > > > >> > >> > >> I got one again > >> > >> [] wait_on_page_bit_killable+0x83/0xa0 > >> [] __lock_page_or_retry+0xb2/0xc0 >> > [] filemap_fault+0x3b7/0x410 >> > [] ceph_filemap_fault+0x13c/0x310 [ceph] >> > [] __do_fault+0x4c/0xc0 >> [] > do_read_fault.isra.42+0x43/0x130 >> [] > handle_mm_fault+0x6b1/0x1040 >> [] > __do_page_fault+0x154/0x450 >> [] > do_page_fault+0x35/0x90 >> [] page_fault+0x28/0x30 > >> [] 0x >> >> > >This is likely > caused by hang osd request, was you cluster health? > > > > > >> >check /proc//stack to find where it is stuck >> > > >> >> >> >> >> >> I have a process stuck in D+ writing to > cephfs kernel mount. > >> Anything > >> >> can be done about this? (without rebooting) >> >> >> >> > >> >> CentOS Linux release 7.5.1804 (Core) >> >> Linux > 3.10.0-514.21.2.el7.x86_64 >> >> >> >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Playbook Deployment - [TASK ceph-mon : test if rbd exists ]
Hi Ceph Community, I am using ansible 2.2 and ceph branch stable-2.2, on centos7, to deploy the playbook. But the deployment get hangs in this step "TASK [ceph-mon : test if rbd exists]". it gets hangs there and doesnot move. I have all the three ceph nodes ceph-admin, ceph-mon, ceph-osd I appreciate any help! Here I am providing log: ---Log -- TASK [ceph-mon : test if rbd exists] *** task path: /root/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml:60 Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r ceph2mon '/bin/sh -c '"'"'echo ~ && sleep 0'"'"'' ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r ceph2mon '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896 `" && echo ansible-tmp-1547740115.56-213823795856896="` echo /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896 `" ) && sleep 0'"'"'' PUT /tmp/tmpG7u1eN TO /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896/command.py SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r '[ceph2mon]' ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r ceph2mon '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896/ /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896/command.py && sleep 0'"'"'' ESTABLISH SSH CONNECTION FOR USER: None SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r -tt ceph2mon '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-iefqzergptqzfhqmxouabfjfvdvbadku; /usr/bin/python /root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896/command.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1547740115.56-213823795856896/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"'' - Thanks, Meysam Kamali Mobile: (416) 890-8804 *P** Please consider the environment before printing this email* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Commercial support
Hi, How is the commercial support for Ceph? More specifically, I was recently pointed in the direction of the very interesting combination of CephFS, Samba and ctdb. Is anyone familiar with companies that provide commercial support for in-house solutions like this? Regards, Ketil ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Commercial support
On Wed, Jan 23, 2019 at 5:29 PM Ketil Froyn wrote: > > Hi, > > How is the commercial support for Ceph? More specifically, I was recently > pointed in the direction of the very interesting combination of CephFS, Samba > and ctdb. Is anyone familiar with companies that provide commercial support > for in-house solutions like this? > > Regards, Ketil Hi Ketil, We provide a commercial solution based on Ceph, which is geared toward a business consumer with 10s or perhaps 100s, not 1000s of machines. Full hardware support, monitoring, integration, etc. http://storcium.com is the web site and VMWare certification is here: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=41781&vcl=true Red Hat, of course, provides commercial support for Ceph as RHES (https://redhatstorage.redhat.com/category/enterprise-storage/) Proxmox supports Ceph integrated with their clusters (we are liking that technology as well, more and more due to very good thought-through design and quality). If you provide more information on the specific use cases, it would be helpful. -- Alex Gorbachev Storcium > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Commercial support
Suse as well https://www.suse.com/products/suse-enterprise-storage/ On Wed, Jan 23, 2019, 6:01 PM Alex Gorbachev On Wed, Jan 23, 2019 at 5:29 PM Ketil Froyn wrote: > > > > Hi, > > > > How is the commercial support for Ceph? More specifically, I was > recently pointed in the direction of the very interesting combination of > CephFS, Samba and ctdb. Is anyone familiar with companies that provide > commercial support for in-house solutions like this? > > > > Regards, Ketil > > Hi Ketil, > > We provide a commercial solution based on Ceph, which is geared toward > a business consumer with 10s or perhaps 100s, not 1000s of machines. > Full hardware support, monitoring, integration, etc. > http://storcium.com is the web site and VMWare certification is here: > > https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=41781&vcl=true > > Red Hat, of course, provides commercial support for Ceph as RHES > (https://redhatstorage.redhat.com/category/enterprise-storage/) > > Proxmox supports Ceph integrated with their clusters (we are liking > that technology as well, more and more due to very good > thought-through design and quality). > > If you provide more information on the specific use cases, it would be > helpful. > -- > Alex Gorbachev > Storcium > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
