On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch <peter.ei...@virginpulse.com>
wrote:
> Hi,
>
>
>
> I appreciate the insistency that the directions be followed. I wholly
> agree. The only liberty I took was to do a ‘yum update’ instead of just
> ‘yum update ceph-osd’ and then reboot. (Also my MDS runs on the MON hosts,
> so it got update a step early.)
>
>
>
> As for the logs:
>
>
>
> [2019-07-24 15:07:22,713][ceph_volume.main][INFO ] Running command:
> ceph-volume simple scan
>
> [2019-07-24 15:07:22,714][ceph_volume.process][INFO ] Running command:
> /bin/systemctl show --no-pager --property=Id --state=running ceph-osd@*
>
> [2019-07-24 15:07:27,574][ceph_volume.main][INFO ] Running command:
> ceph-volume simple activate --all
>
> [2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO ]
> activating OSD specified in
> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
>
> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
> Required devices (block and data) not present for bluestore
>
> [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
> bluestore devices found: [u'data']
>
> [2019-07-24 15:07:27,576][ceph_volume][ERROR ] exception caught by
> decorator
>
> Traceback (most recent call last):
>
> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
> 59, in newfunc
>
> return f(*a, **kw)
>
> File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148,
> in main
>
> terminal.dispatch(self.mapper, subcommand_args)
>
> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> 182, in dispatch
>
> instance.main()
>
> File
> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line
> 33, in main
>
> terminal.dispatch(self.mapper, self.argv)
>
> File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> 182, in dispatch
>
> instance.main()
>
> File
> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> line 272, in main
>
> self.activate(args)
>
> File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line
> 16, in is_root
>
> return func(*a, **kw)
>
> File
> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> line 131, in activate
>
> self.validate_devices(osd_metadata)
>
> File
> "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> line 62, in validate_devices
>
> raise RuntimeError('Unable to activate bluestore OSD due to missing
> devices')
>
> RuntimeError: Unable to activate bluestore OSD due to missing devices
>
>
>
> (this is repeated for each of the 16 drives)
>
>
>
> Any other thoughts? (I’ll delete/create the OSDs with ceph-deply
> otherwise.)
>
Try using `ceph-volume simple scan --stdout` so that it doesn't persist
data onto /etc/ceph/osd/ and inspect that the JSON produced is capturing
all the necessary details for OSDs.
Alternatively, I would look into the JSON files already produced in
/etc/ceph/osd/ and check if the details are correct. The `scan` sub-command
does a tremendous effort to cover all cases where ceph-disk
created an OSD (filestore, bluestore, dmcrypt, etc...) but it is possible
that it may be hitting a problem. This is why the tool made these JSON
files available, so that they could be inspected and corrected if anything.
The details of the scan sub-command can be found at
http://docs.ceph.com/docs/master/ceph-volume/simple/scan/ and the JSON
structure is described in detail below at
http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents
In this particular case the tool is refusing to activate what seems to be a
bluestore OSD. Is it really a bluestore OSD? if so, then it can't find
where is the data partition. What does that partition look like (for any of
the failing OSDs) ? Does it use dmcrypt, how was it created? (hopefully
with ceph-disk!)
If you know the data partition for a given OSD, try and pass it onto
'scan'. For example if it is /dev/sda1 you could do `ceph-volume simple
scan /dev/sda1` and check its output.
>
> peter
>
>
>
>
> Peter Eisch
> Senior Site Reliability Engineer
> T *1.612.659.3228* <1.612.659.3228>
> [image: Facebook] <https://www.facebook.com/VirginPulse>
> [image: LinkedIn] <https://www.linkedin.com/company/virgin-pulse>
> [image: Twitter] <https://twitter.com/virginpulse>
> *virginpulse.com* <https://www.virginpulse.com/>
> | *virginpulse.com/global-challenge*
> <https://www.virginpulse.com/en-gb/global-challenge/>
>
> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
> Switzerland | United Kingdom | USA
> Confidentiality Notice: The information contained in this e-mail,
> including any attachment(s), is intended solely for use by the designated
> recipient(s). Unauthorized use, dissemination, distribution, or
> reproduction of this message by anyone other than the intended
> recipient(s), or a person designated as responsible for delivering such
> messages to the intended recipient, is strictly prohibited and may be
> unlawful. This e-mail may contain proprietary, confidential or privileged
> information. Any views or opinions expressed are solely those of the author
> and do not necessarily represent those of Virgin Pulse, Inc. If you have
> received this message in error, or are not the named recipient(s), please
> immediately notify the sender and delete this e-mail message.
> v2.59
>
> *From: *Alfredo Deza <ad...@redhat.com>
> *Date: *Wednesday, July 24, 2019 at 3:02 PM
> *To: *Peter Eisch <peter.ei...@virginpulse.com>
> *Cc: *Paul Emmerich <paul.emmer...@croit.io>, "ceph-users@lists.ceph.com"
> <ceph-users@lists.ceph.com>
> *Subject: *Re: [ceph-users] Upgrading and lost OSDs
>
>
>
>
>
> On Wed, Jul 24, 2019 at 3:49 PM Peter Eisch <peter.ei...@virginpulse.com>
> wrote:
>
>
>
> I’m at step 6. I updated/rebooted the host to complete “installing the
> new packages and restarting the ceph-osd daemon” on the first OSD host.
> All the systemctl definitions to start the OSDs were deleted, all the
> properties in /var/lib/ceph/osd/ceph-* directories were deleted. All the
> files in /var/lib/ceph/osd-lockbox, for comparison, were untouched and
> still present.
>
>
>
> Peeking into step 7 I can run ceph-volume:
>
>
>
> # ceph-volume simple scan /dev/sda1
>
> Running command: /usr/sbin/cryptsetup status /dev/sda1
>
> Running command: /usr/sbin/cryptsetup status
> 93fb5f2f-0273-4c87-a718-886d7e6db983
>
> Running command: /bin/mount -v /dev/sda5 /tmp/tmpF5F8t2
>
> stdout: mount: /dev/sda5 mounted on /tmp/tmpF5F8t2.
>
> Running command: /usr/sbin/cryptsetup status /dev/sda5
>
> Running command: /bin/ceph --cluster ceph --name
> client.osd-lockbox.93fb5f2f-0273-4c87-a718-886d7e6db983 --keyring
> /tmp/tmpF5F8t2/keyring config-key get
> dm-crypt/osd/93fb5f2f-0273-4c87-a718-886d7e6db983/luks
>
> Running command: /bin/umount -v /tmp/tmpF5F8t2
>
> stderr: umount: /tmp/tmpF5F8t2 (/dev/sda5) unmounted
>
> Running command: /usr/sbin/cryptsetup --key-file - --allow-discards
> luksOpen /dev/sda1 93fb5f2f-0273-4c87-a718-886d7e6db983
>
> Running command: /bin/mount -v
> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 /tmp/tmpYK0WEV
>
> stdout: mount: /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 mounted on
> /tmp/tmpYK0WEV.
>
> --> broken symlink found /tmp/tmpYK0WEV/block ->
> /dev/mapper/a05b447c-c901-4690-a249-cc1a2d62a110
>
> Running command: /usr/sbin/cryptsetup status /tmp/tmpYK0WEV/block_dmcrypt
>
> Running command: /usr/sbin/cryptsetup status
> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
>
> Running command: /bin/umount -v /tmp/tmpYK0WEV
>
> stderr: umount: /tmp/tmpYK0WEV
> (/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983) unmounted
>
> Running command: /usr/sbin/cryptsetup remove
> /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
>
> --> OSD 0 got scanned and metadata persisted to file:
> /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
>
> --> To take over management of this scanned OSD, and disable ceph-disk and
> udev, run:
>
> --> ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983
>
> #
>
> #
>
> # ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983
>
> --> Required devices (block and data) not present for bluestore
>
> --> bluestore devices found: [u'data']
>
> --> RuntimeError: Unable to activate bluestore OSD due to missing devices
>
> #
>
>
>
> The tool detected bluestore, or rather, it failed to find a journal
> associated with /dev/sda1. Scanning a single partition can cause that.
> There is a flag to spit out the findings to STDOUT instead of persisting
> them in /etc/ceph/osd/
>
>
>
> Since this is a "whole system" upgrade, then the upgrade documentation
> instructions need to be followed:
>
>
>
> ceph-volume simple scan
> ceph-volume simple activate --all
>
>
>
> If the `scan` command doesn't display any information (not even with the
> --stdout flag) then the logs at /var/log/ceph/ceph-volume.log need to be
> inspected. It would be useful to check any findings in there
>
>
>
>
> Okay, this created /etc/ceph/osd/*.json. This is cool. Is there a
> command or option which will read these files and mount the devices?
>
>
>
> peter
>
>
>
>
>
>
>
> *Peter Eisch*
>
> Senior Site Reliability Engineer
>
> *T*
>
> 1.612.659.3228
>
> [image: Facebook]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227707952&sdata=B2JiNp12z7gsfF2i5T2l%2FSjfg6Fhg8E85OpdyGpEMHg%3D&reserved=0>
>
> [image: LinkedIn]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=z3Ii%2BGgPKe7fCOhNGXw%2BlD9j28YCY4gH81is%2BJoiSJU%3D&reserved=0>
>
> [image: Twitter]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=zUaGW2Fm16sdyJdHUPtDN6CzaMXtxMOHvmNDi9VshCw%3D&reserved=0>
>
> virginpulse.com
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=n3m9S%2Bt8fYzGY%2BqPw2wT433TQhf2oPXp9wAum9s9%2BUk%3D&reserved=0>
>
> |
>
> virginpulse.com/global-challenge
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=5wn%2F%2FkY1IL0d4BxXNpqCJUHG09gUFRTr2S9KWv1mVG4%3D&reserved=0>
>
>
> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
> Switzerland | United Kingdom | USA
>
> Confidentiality Notice: The information contained in this e-mail,
> including any attachment(s), is intended solely for use by the designated
> recipient(s). Unauthorized use, dissemination, distribution, or
> reproduction of this message by anyone other than the intended
> recipient(s), or a person designated as responsible for delivering such
> messages to the intended recipient, is strictly prohibited and may be
> unlawful. This e-mail may contain proprietary, confidential or privileged
> information. Any views or opinions expressed are solely those of the author
> and do not necessarily represent those of Virgin Pulse, Inc. If you have
> received this message in error, or are not the named recipient(s), please
> immediately notify the sender and delete this e-mail message.
>
> v2.59
>
> *From: *Alfredo Deza <ad...@redhat.com>
> *Date: *Wednesday, July 24, 2019 at 2:20 PM
> *To: *Peter Eisch <peter.ei...@virginpulse.com>
> *Cc: *Paul Emmerich <paul.emmer...@croit.io>, "ceph-users@lists.ceph.com"
> <ceph-users@lists.ceph.com>
> *Subject: *Re: [ceph-users] Upgrading and lost OSDs
>
>
>
> On Wed, Jul 24, 2019 at 2:56 PM Peter Eisch <peter.ei...@virginpulse.com>
> wrote:
>
> Hi Paul,
>
> To do better to answer you question, I'm following:
> http://docs.ceph.com/docs/nautilus/releases/nautilus/
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fnautilus%2Freleases%2Fnautilus%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227727937&sdata=D1Q0Hrg9mq2tTtfTt1kp4Ts3vBc7mertKvYy8vBWNF8%3D&reserved=0>
>
> At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum
> update' for patching the host and rebooted to pick up the current centos
> kernel.
>
>
>
> If you are at Step 6 then it is *crucial* to understand that the tooling
> used to create the OSDs is no longer available and Step 7 *is absolutely
> required*.
>
>
>
> ceph-volume has to scan the system and give you the output of all OSDs
> found so that it can persist them in /etc/ceph/osd/*.json files and then
> can later be
>
> "activated".
>
>
>
>
> I didn't do anything to specific commands for just updating the ceph RPMs
> in this process.
>
>
>
> It is not clear if you are at Step 6 and wondering why OSDs are not up, or
> you are past that and ceph-volume wasn't able to detect anything.
>
>
>
> peter
>
> *Peter Eisch*
>
> Senior Site Reliability Engineer
>
> *T*
>
> 1.612.659.3228
>
> [image: Facebook]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=uncHzIOSXt25%2F0NydXSJaLAf6E3Ad05N%2BJLBKYYJQ%2Fw%3D&reserved=0>
>
> [image: LinkedIn]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=NxQ3BmhgWo93uoQfJ1W7lLcDdSUQHgoXu1I49vzibwE%3D&reserved=0>
>
> [image: Twitter]
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=sYUapxFqHq0LVyxO4I7kkwN1y9PG5ZLHd83gseRIxvM%3D&reserved=0>
>
> virginpulse.com
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=aoXH94QOjdngJXkAPcz9kmJAK5BA6c9rR5BP01lX0bw%3D&reserved=0>
>
> |
>
> virginpulse.com/global-challenge
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpulse.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=aUbx8uIlqldr5JtKT1PL05nbzcNPHONpouVkj1qXRdM%3D&reserved=0>
>
>
> Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore |
> Switzerland | United Kingdom | USA
>
> Confidentiality Notice: The information contained in this e-mail,
> including any attachment(s), is intended solely for use by the designated
> recipient(s). Unauthorized use, dissemination, distribution, or
> reproduction of this message by anyone other than the intended
> recipient(s), or a person designated as responsible for delivering such
> messages to the intended recipient, is strictly prohibited and may be
> unlawful. This e-mail may contain proprietary, confidential or privileged
> information. Any views or opinions expressed are solely those of the author
> and do not necessarily represent those of Virgin Pulse, Inc. If you have
> received this message in error, or are not the named recipient(s), please
> immediately notify the sender and delete this e-mail message.
>
> v2.59
>
>
> From: Paul Emmerich <paul.emmer...@croit.io>
> Date: Wednesday, July 24, 2019 at 1:39 PM
> To: Peter Eisch <peter.ei...@virginpulse.com>
> Cc: Xavier Trilla <xavier.tri...@clouding.io>, "ceph-users@lists.ceph.com"
> <ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] Upgrading and lost OSDs
>
> On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto:
> peter.ei...@virginpulse.com> wrote:
> # lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 1.7T 0 disk
> ├─sda1 8:1 0 100M 0 part
> ├─sda2 8:2 0 1.7T 0 part
> └─sda5 8:5 0 10M 0 part
> sdb 8:16 0 1.7T 0 disk
> ├─sdb1 8:17 0 100M 0 part
> ├─sdb2 8:18 0 1.7T 0 part
> └─sdb5 8:21 0 10M 0 part
> sdc 8:32 0 1.7T 0 disk
> ├─sdc1 8:33 0 100M 0 part
>
> That's ceph-disk which was removed, run "ceph-volume simple scan"
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at
> https://nam02.safelinks.protection.outlook.com/?url=https://croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=gAFSElTqBRhu1zDIQYvWQ2WZtqqHoW%2FLa3stBfqXXHQ%3D&reserved=0>
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
>
> https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.croit.io&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=LWJKCZ4VxuwoHMJGSWiEWAcawFPw7pDGC48%2B6bnXk6A%3D&reserved=0>
> Tel: +49 89 1896585 90
>
>
> ...
> I'm thinking the OSD would start (I can recreate the .service definitions
> in systemctl) if the above were mounted in a way like they are on another
> of my hosts:
> # lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 1.7T 0 disk
> ├─sda1 8:1 0 100M 0 part
> │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt
> /var/lib/ceph/osd/ceph-16
> ├─sda2 8:2 0 1.7T 0 part
> │ └─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt
> └─sda5 8:5 0 10M 0 part
> /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5
> sdb 8:16 0 1.7T 0 disk
> ├─sdb1 8:17 0 100M 0 part
> │ └─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt
> /var/lib/ceph/osd/ceph-17
> ├─sdb2 8:18 0 1.7T 0 part
> │ └─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt
> └─sdb5 8:21 0 10M 0 part
> /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2
> sdc 8:32 0 1.7T 0 disk
> ├─sdc1 8:33 0 100M 0 part
> │ └─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt
> /var/lib/ceph/osd/ceph-18
> ├─sdc2 8:34 0 1.7T 0 part
> │ └─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt
> ...
>
> Thank you for your time on this,
>
> peter
>
> From: Xavier Trilla <mailto:xavier.tri...@clouding.io>
> Date: Wednesday, July 24, 2019 at 1:25 PM
> To: Peter Eisch <mailto:peter.ei...@virginpulse.com>
> Cc: "mailto:ceph-users@lists.ceph.com"; <mailto:ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] Upgrading and lost OSDs
>
> Hi Peter,
>
> Im not sure but maybe after some changes the OSDs are not being
> recongnized by ceph scripts.
>
> Ceph used to use udev to detect the OSDs and then moved to lvm, which kind
> of OSDs are you running? Blustore or filestore? Which version did you use
> to create them?
>
> Cheers!
>
> El 24 jul 2019, a les 20:04, Peter Eisch <mailto:mailto:
> peter.ei...@virginpulse.com> va escriure:
> Hi,
>
> I’m working through updating from 12.2.12/luminious to 14.2.2/nautilus on
> centos 7.6. The managers are updated alright:
>
> # ceph -s
> cluster:
> id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec
> health: HEALTH_WARN
> Degraded data redundancy: 24177/9555955 objects degraded
> (0.253%), 7 pgs degraded, 1285 pgs undersized
> 3 monitors have not enabled msgr2
> ...
>
> I updated ceph on a OSD host with 'yum update' and then rebooted to grab
> the current kernel. Along the way, the contents of all the directories in
> /var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this.
> I can manage the undersized but I'd like to get these drives working again
> without deleting each OSD and recreating them.
>
> So far I've pulled the respective cephx key into the 'keyring' file and
> populated 'bluestore' into the 'type' files but I'm unsure how to get the
> lockboxes mounted to where I can get the OSDs running. The osd-lockbox
> directory is otherwise untouched from when the OSDs were deployed.
>
> Is there a way to run ceph-deploy or some other tool to rebuild the mounts
> for the drives?
>
> peter
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com%2Flistinfo.cgi%2Fceph-users-ceph.com&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=jeAj%2FfzN%2BOG1NPFeZwqYnQiB4mgbgLGqq85tz99xBz8%3D&reserved=0>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
