On Tue, Jan 22, 2019 at 6:45 AM Manuel Lausch <manuel.lau...@1und1.de> wrote: > > Hi, > > we want upgrade our ceph clusters from jewel to luminous. And also want > to migrate the osds to ceph-volume described in > http://docs.ceph.com/docs/luminous/ceph-volume/simple/scan/#ceph-volume-simple-scan > > The clusters are running since dumpling and are setup with dmcrypt. > Since dumpling there are until now three different types of dmcrypt > > plain dmcrypt with keys local > luks with keys local > luks with keys on the ceph monitors > > Now it seems only the last type can be migrated to ceph-volume. > > ceph-volume simple scan trys to mount a lockbox which does not exists > on the older OSDs. Are those OSDs not supported with ceph-volume?
This is one case we didn't anticipate :/ We supported the wonky lockbox setup and thought we wouldn't need to go further back, although we did add support for both plain and luks keys. Looking through the code, it is very tightly couple to storing/retrieving keys from the monitors, and I don't know what workarounds might be possible here other than throwing away the OSD and deploying a new one (I take it this is not an option for you at all) > > This are the errors: > > # ceph-volume simple scan /var/lib/ceph/osd/ceph-183 > stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device > stderr: lsblk: /var/lib/ceph/osd/ceph-183: not a block device > Running command: /usr/sbin/cryptsetup status > /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 > Running command: /usr/sbin/cryptsetup status > 21ad7722-002f-464c-b460-a8976a7b4872 > Running command: mount -v /tmp/tmp3t1WRC > stderr: mount: is write-protected, mounting read-only > stderr: mount: unknown filesystem type '(null)' > --> RuntimeError: command returned non-zero exit status: 32 > > > and this is in the ceph-volume.log > > [2019-01-22 12:39:31,456][ceph_volume.process][INFO ] Running command: > /usr/sbin/blkid -p /dev/mapper/9b68b7e9-854e-498a-8381-4eef128a9d7a > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] detecting > if argument is a device or a directory: /var/lib/ceph/osd/ceph-183 > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan > directly, path is a directory > [2019-01-22 12:39:31,533][ceph_volume.devices.simple.scan][INFO ] will scan > encrypted OSD directory at path: /var/lib/ceph/osd/ceph-183 > [2019-01-22 12:39:31,534][ceph_volume.process][INFO ] Running command: > /usr/sbin/blkid -s PARTUUID -o value /dev/sdv1 > [2019-01-22 12:39:31,539][ceph_volume.process][INFO ] stdout > 21ad7722-002f-464c-b460-a8976a7b4872 > [2019-01-22 12:39:31,540][ceph_volume.process][INFO ] Running command: > /usr/sbin/cryptsetup status 21ad7722-002f-464c-b460-a8976a7b4872 > [2019-01-22 12:39:31,546][ceph_volume.process][INFO ] stdout > /dev/mapper/21ad7722-002f-464c-b460-a8976a7b4872 is active and is in use. > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout type: PLAIN > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout cipher: > aes-cbc-essiv:sha256 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout keysize: 256 > bits > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout key location: > dm-crypt > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout device: > /dev/sdv1 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout sector size: > 512 > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout offset: 0 > sectors > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout size: > 7805646479 sectors > [2019-01-22 12:39:31,547][ceph_volume.process][INFO ] stdout mode: > read/write > [2019-01-22 12:39:31,548][ceph_volume.process][INFO ] Running command: mount > -v /tmp/tmp3t1WRC > [2019-01-22 12:39:31,597][ceph_volume.process][INFO ] stderr mount: is > write-protected, mounting read-only > [2019-01-22 12:39:31,622][ceph_volume.process][INFO ] stderr mount: unknown > filesystem type '(null)' > [2019-01-22 12:39:31,622][ceph_volume][ERROR ] exception caught by decorator > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, > in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in > main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", > line 33, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", > line 353, in main > self.scan(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, > in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", > line 244, in scan > osd_metadata = self.scan_encrypted(osd_path) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/scan.py", > line 169, in scan_encrypted > with system.tmp_mount(lockbox) as lockbox_path: > File "/usr/lib/python2.7/site-packages/ceph_volume/util/system.py", line > 145, in __enter__ > self.path > File "/usr/lib/python2.7/site-packages/ceph_volume/process.py", line 153, > in run > raise RuntimeError(msg) > RuntimeError: command returned non-zero exit status: 32 > > > > ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) > luminous (stable) > > > Regards > Manuel > > > > > -- > Manuel Lausch > > Systemadministrator > Storage Services > > 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 > Karlsruhe | Germany > Phone: +49 721 91374-1847 > E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de > > Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 > > Geschäftsführer: Thomas Ludwig, Jan Oetjen, Sascha Vollmer > > > Member of United Internet > > Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen > enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese > E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und > vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten > ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt > auf welche Weise auch immer zu verwenden. > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient of this e-mail, you are hereby notified that > saving, distribution or use of the content of this e-mail in any way is > prohibited. If you have received this e-mail in error, please notify the > sender and delete the e-mail. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
