On Tue, Jan 28, 2020 at 08:03:35PM +0100, bauen1 wrote:
>Hi,
>
>I've run into the same issue while testing:
>
>ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
>nautilus (stable)
>
>debian bullseye
>
>Ceph was installed using ceph-ansible on a vm from the repo
>http://download.ceph.com/debian-nautilus
>
>The output of `sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume
>--cluster test lvm batch --bluestore /dev/vdb'` has been attached.
Thx, I opened https://tracker.ceph.com/issues/43868.
This looks like a bluestore/osd issue to me, though it might end up being
ceph-volumes fault.
>
>Also worth noting might be that '/var/lib/ceph/osd/test-0/fsid' is
>empty (but I don't know too much about the internals)
>
>- bauen1
>
>On 1/28/20 4:54 PM, Dave Hall wrote:
>>Jan,
>>
>>Unfortunately I'm under immense pressure right now to get some form
>>of Ceph into production, so it's going to be Luminous for now, or
>>maybe a live upgrade to Nautilus without recreating the OSDs (if
>>that's possible).
>>
>>The good news is that in the next couple months I expect to add more
>>hardware that should be nearly identical. I will gladly give it a
>>go at that time and see if I can recreate. (Or, if I manage to
>>thoroughly crash my current fledgling cluster, I'll give it another
>>go on one node while I'm up all night recovering.)
>>
>>If you could tell me where to look I'd gladly read some code and see
>>if I can find anything that way. Or if there's any sort of design
>>document describing the deep internals I'd be glad to scan it to see
>>if I've hit a corner case of some sort. Actually, I'd be interested
>>in reading those documents anyway if I could.
>>
>>Thanks.
>>
>>-Dave
>>
>>Dave Hall
>>
>>On 1/28/2020 3:05 AM, Jan Fajerski wrote:
>>>On Mon, Jan 27, 2020 at 03:23:55PM -0500, Dave Hall wrote:
>>>>All,
>>>>
>>>>I've just spent a significant amount of time unsuccessfully chasing
>>>>the _read_fsid unparsable uuid error on Debian 10 / Natilus 14.2.6.
>>>>Since this is a brand new cluster, last night I gave up and moved back
>>>>to Debian 9 / Luminous 12.2.11. In both cases I'm using the packages
>>>>from Debian Backports with ceph-ansible as my deployment tool.
>>>>Note that above I said 'the _read_fsid unparsable uuid' error. I've
>>>>searched around a bit and found some previously reported issues, but I
>>>>did not see any conclusive resolutions.
>>>>
>>>>I would like to get to Nautilus as quickly as possible, so I'd gladly
>>>>provide additional information to help track down the cause of this
>>>>symptom. I can confirm that, looking at the ceph-volume.log on the
>>>>OSD host I see no difference between the ceph-volume lvm batch command
>>>>generated by the ceph-ansible versions associated with these two Ceph
>>>>releases:
>>>>
>>>> ceph-volume --cluster ceph lvm batch --bluestore --yes
>>>> --block-db-size 133358734540 /dev/sdc /dev/sdd /dev/sde /dev/sdf
>>>> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/nvme0n1
>>>>
>>>>Note that I'm using --block-db-size to divide my NVMe into 12 segments
>>>>as I have 4 empty drive bays on my OSD servers that I may eventually
>>>>be able to fill.
>>>>
>>>>My OSD hardware is:
>>>>
>>>> Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 3125627568 sectors
>>>> Disk /dev/sdc: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdd: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sde: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdf: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdg: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdh: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdi: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>> Disk /dev/sdj: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
>>>>
>>>>I'd send the output of ceph-volume inventory on Luminous, but I'm
>>>>getting -->: KeyError: 'human_readable_size'.
>>>>
>>>>Please let me know if I can provide any further information.
>>>Mind re-running you ceph-volume command with debug output
>>>enabled:
>>>CEPH_VOLUME_DEBUG=true ceph-volume --cluster ceph lvm batch
>>>--bluestore ...
>>>
>>>Ideally you could also openen a bug report here
>>>https://tracker.ceph.com/projects/ceph-volume/issues/new
>>>
>>>Thanks!
>>>>Thanks.
>>>>
>>>>-Dave
>>>>
>>>>--
>>>>Dave Hall
>>>>Binghamton University
>>>>
>>>>_______________________________________________
>>>>ceph-users mailing list -- ceph-users@ceph.io
>>>>To unsubscribe send an email to ceph-users-le...@ceph.io
>>_______________________________________________
>>ceph-users mailing list -- ceph-users@ceph.io
>>To unsubscribe send an email to ceph-users-le...@ceph.io
>sysadmin@ceph-test:~$ sudo setenforce 0
>sysadmin@ceph-test:~$ sudo sh -c 'CEPH_VOLUME_DEBUG=true ceph-volume --cluster
>test lvm batch --bluestore /dev/vdb'
>
>Total OSDs: 1
>
> Type Path LV
> Size % of device
>----------------------------------------------------------------------------------------------------
> [data] /dev/vdb
> 63.00 GB 100.0%
>--> The above OSDs would be created if the operation continues
>--> do you want to proceed? (yes/no) yes
>Running command: /usr/sbin/vgcreate -s 1G --force --yes
>ceph-1cc81d7c-a153-462a-8080-ec3d217c7180 /dev/vdb
> stdout: Physical volume "/dev/vdb" successfully created.
> stdout: Volume group "ceph-1cc81d7c-a153-462a-8080-ec3d217c7180" successfully
> created
>Running command: /usr/sbin/lvcreate --yes -l 63 -n
>osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e
>ceph-1cc81d7c-a153-462a-8080-ec3d217c7180
> stdout: Wiping ceph_bluestore signature on
> /dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e.
> stdout: Logical volume "osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e"
> created.
>Running command: /bin/ceph-authtool --gen-print-key
>Running command: /bin/ceph --cluster test --name client.bootstrap-osd
>--keyring /var/lib/ceph/bootstrap-osd/test.keyring -i - osd new
>e3ebb6e0-82c8-4088-a6bd-abd729a575bb
>Running command: /bin/ceph-authtool --gen-print-key
>Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/test-0
>Running command: /usr/sbin/restorecon /var/lib/ceph/osd/test-0
>Running command: /bin/chown -h ceph:ceph
>/dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e
>Running command: /bin/chown -R ceph:ceph /dev/dm-1
>Running command: /bin/ln -s
>/dev/ceph-1cc81d7c-a153-462a-8080-ec3d217c7180/osd-data-bbd7752f-fad9-41d5-bbbe-e6fd512bcf8e
> /var/lib/ceph/osd/test-0/block
>Running command: /bin/ceph --cluster test --name client.bootstrap-osd
>--keyring /var/lib/ceph/bootstrap-osd/test.keyring mon getmap -o
>/var/lib/ceph/osd/test-0/activate.monmap
> stderr: got monmap epoch 1
>Running command: /bin/ceph-authtool /var/lib/ceph/osd/test-0/keyring
>--create-keyring --name osd.0 --add-key
>AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==
> stdout: creating /var/lib/ceph/osd/test-0/keyring
> stdout: added entity osd.0 auth(key=AQAcgzBeTlc5BxAApXJgwyoRAHtrL9kk1tbs9w==)
>Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/keyring
>Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/test-0/
>Running command: /bin/ceph-osd --cluster test --osd-objectstore bluestore
>--mkfs -i 0 --monmap /var/lib/ceph/osd/test-0/activate.monmap --keyfile -
>--osd-data /var/lib/ceph/osd/test-0/ --osd-uuid
>e3ebb6e0-82c8-4088-a6bd-abd729a575bb --setuser ceph --setgroup ceph
> stderr: 2020-01-28 18:53:20.438 7f17de7b3c00 -1
> bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid
> stderr: terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get>
> >'
> stderr: what(): boost::bad_get: failed value get using boost::get
> stderr: *** Caught signal (Aborted) **
> stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
> stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
> nautilus (stable)
> stderr: 1: (()+0x13520) [0x7f17dee75520]
> stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
> stderr: 3: (abort()+0x121) [0x7f17de926535]
> stderr: 4: (()+0x9a643) [0x7f17decba643]
> stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
> stderr: 6: (()+0xa6041) [0x7f17decc6041]
> stderr: 7: (()+0xa6295) [0x7f17decc6295]
> stderr: 8: (()+0x49a92c) [0x56027edc792c]
> stderr: 9: (Option::size_t const
> md_config_t::get_val<Option::size_t>(ConfigValues const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&) const+0x51) [0x56027eedeea1]
> stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
> stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
> stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
> stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3)
> [0x56027eef9b23]
> stderr: 14: (main()+0x1821) [0x56027eea68d1]
> stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
> stderr: 16: (_start()+0x2a) [0x56027eed903a]
> stderr: 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal (Aborted) **
> stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
> stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
> nautilus (stable)
> stderr: 1: (()+0x13520) [0x7f17dee75520]
> stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
> stderr: 3: (abort()+0x121) [0x7f17de926535]
> stderr: 4: (()+0x9a643) [0x7f17decba643]
> stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
> stderr: 6: (()+0xa6041) [0x7f17decc6041]
> stderr: 7: (()+0xa6295) [0x7f17decc6295]
> stderr: 8: (()+0x49a92c) [0x56027edc792c]
> stderr: 9: (Option::size_t const
> md_config_t::get_val<Option::size_t>(ConfigValues const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&) const+0x51) [0x56027eedeea1]
> stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
> stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
> stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
> stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3)
> [0x56027eef9b23]
> stderr: 14: (main()+0x1821) [0x56027eea68d1]
> stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
> stderr: 16: (_start()+0x2a) [0x56027eed903a]
> stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
> stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1
> bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid
> stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal
> (Aborted) **
> stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
> stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
> nautilus (stable)
> stderr: 1: (()+0x13520) [0x7f17dee75520]
> stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
> stderr: 3: (abort()+0x121) [0x7f17de926535]
> stderr: 4: (()+0x9a643) [0x7f17decba643]
> stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
> stderr: 6: (()+0xa6041) [0x7f17decc6041]
> stderr: 7: (()+0xa6295) [0x7f17decc6295]
> stderr: 8: (()+0x49a92c) [0x56027edc792c]
> stderr: 9: (Option::size_t const
> md_config_t::get_val<Option::size_t>(ConfigValues const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&) const+0x51) [0x56027eedeea1]
> stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
> stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
> stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
> stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3)
> [0x56027eef9b23]
> stderr: 14: (main()+0x1821) [0x56027eea68d1]
> stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
> stderr: 16: (_start()+0x2a) [0x56027eed903a]
> stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
> stderr: -5> 2020-01-28 18:53:20.438 7f17de7b3c00 -1
> bluestore(/var/lib/ceph/osd/test-0/) _read_fsid unparsable uuid
> stderr: 0> 2020-01-28 18:53:20.486 7f17de7b3c00 -1 *** Caught signal
> (Aborted) **
> stderr: in thread 7f17de7b3c00 thread_name:ceph-osd
> stderr: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9)
> nautilus (stable)
> stderr: 1: (()+0x13520) [0x7f17dee75520]
> stderr: 2: (gsignal()+0x141) [0x7f17de93b081]
> stderr: 3: (abort()+0x121) [0x7f17de926535]
> stderr: 4: (()+0x9a643) [0x7f17decba643]
> stderr: 5: (()+0xa5fd6) [0x7f17decc5fd6]
> stderr: 6: (()+0xa6041) [0x7f17decc6041]
> stderr: 7: (()+0xa6295) [0x7f17decc6295]
> stderr: 8: (()+0x49a92c) [0x56027edc792c]
> stderr: 9: (Option::size_t const
> md_config_t::get_val<Option::size_t>(ConfigValues const&,
> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&) const+0x51) [0x56027eedeea1]
> stderr: 10: (BlueStore::_set_cache_sizes()+0x174) [0x56027f3fba44]
> stderr: 11: (BlueStore::_open_bdev(bool)+0x1c5) [0x56027f3fe845]
> stderr: 12: (BlueStore::mkfs()+0x6e0) [0x56027f484620]
> stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1b3)
> [0x56027eef9b23]
> stderr: 14: (main()+0x1821) [0x56027eea68d1]
> stderr: 15: (__libc_start_main()+0xeb) [0x7f17de927bbb]
> stderr: 16: (_start()+0x2a) [0x56027eed903a]
> stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>--> Was unable to complete a new OSD, will rollback changes
>Running command: /bin/ceph --cluster test --name client.bootstrap-osd
>--keyring /var/lib/ceph/bootstrap-osd/test.keyring osd purge-new osd.0
>--yes-i-really-mean-it
> stderr: purged osd.0
>Traceback (most recent call last):
> File "/usr/sbin/ceph-volume", line 11, in <module>
> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
> File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 38, in
> __init__
> self.main(self.argv)
> File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in
> newfunc
> return f(*a, **kw)
> File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 149, in main
> terminal.dispatch(self.mapper, subcommand_args)
> File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in
> dispatch
> instance.main()
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line
> 40, in main
> terminal.dispatch(self.mapper, self.argv)
> File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in
> dispatch
> instance.main()
> File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
> is_root
> return func(*a, **kw)
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line
> 325, in main
> self.execute()
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/batch.py", line
> 288, in execute
> self.strategy.execute()
> File
> "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/strategies/bluestore.py",
> line 124, in execute
> Create(command).main()
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py",
> line 69, in main
> self.create(args)
> File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
> is_root
> return func(*a, **kw)
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py",
> line 26, in create
> prepare_step.safe_prepare(args)
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py",
> line 219, in safe_prepare
> self.prepare()
> File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in
> is_root
> return func(*a, **kw)
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py",
> line 320, in prepare
> osd_fsid,
> File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py",
> line 119, in prepare_bluestore
> db=db
> File "/usr/lib/python3/dist-packages/ceph_volume/util/prepare.py", line 430,
> in osd_mkfs_bluestore
> raise RuntimeError('Command failed with exit code %s: %s' % (returncode, '
> '.join(command)))
>RuntimeError: Command failed with exit code 250: /bin/ceph-osd --cluster test
>--osd-objectstore bluestore --mkfs -i 0 --monmap
>/var/lib/ceph/osd/test-0/activate.monmap --keyfile - --osd-data
>/var/lib/ceph/osd/test-0/ --osd-uuid e3ebb6e0-82c8-4088-a6bd-abd729a575bb
>--setuser ceph --setgroup ceph
>sysadmin@ceph-test:~$ sudo setenforce 1
>sysadmin@ceph-test:~$
>
>_______________________________________________
>ceph-users mailing list -- ceph-users@ceph.io
>To unsubscribe send an email to ceph-users-le...@ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
