Le 23/08/2018 à 16:13, Alfredo Deza a écrit :
What you mean is that, at this stage, I must directly declare the UUID paths
in value of --block.db (i.e. replace /dev/nvme0n1p1 with its PARTUUID), that
is ?
No, this all looks correct. How does the ceph-volume.log and
ceph-volume-systemd.log look when you are booting up for the OSDs that
aren't coming up?
Anything useful in there?
ceph-volume.log (extract)
[2018-08-20 11:26:29,430][ceph_volume.process][INFO ] Running command:
systemctl start ceph-osd@1
[2018-08-20 11:26:32,268][ceph_volume.main][INFO ] Running command:
ceph-volume lvm trigger 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4
[2018-08-20 11:26:32,269][ceph_volume.process][INFO ] Running command:
/sbin/lvs --noheadings --readonly --separator=";" -o
lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-425a36ca-8b60-471d-80ed-19726714ea3a/osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037,ceph.block_uuid=kkykJn-EmUa-EvZj-Ffr7-dk06-VUG0-nhnUvB,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p4,ceph.db_uuid=ec4cf738-1627-4ddf-a7ef-91810fb8aaab,ceph.encrypted=0,ceph.osd_fsid=9380cd27-c0fe-4ede-9ed3-d09eff545037,ceph.osd_id=3,ceph.type=block,ceph.vdo=0";"/dev/ceph-425a36ca-8b60-471d-80ed-19726714ea3a/osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037";"osd-block-9380cd27-c0fe-4ede-9ed3-d09eff545037";"ceph-425a36ca-8b60-471d-80ed-19726714ea3a";"kkykJn-EmUa-EvZj-Ffr7-dk06-VUG0-nhnUvB
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421/osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8,ceph.block_uuid=WdXALj-0bCs-Ndfy-xGIN-q1rF-hcmg-utIiNB,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p8,ceph.db_uuid=9642a688-0991-4a19-b49a-994a22edaf60,ceph.encrypted=0,ceph.osd_fsid=5d4af2fc-388c-4795-9d1a-53ad8aba56d8,ceph.osd_id=7,ceph.type=block,ceph.vdo=0";"/dev/ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421/osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8";"osd-block-5d4af2fc-388c-4795-9d1a-53ad8aba56d8";"ceph-597bbc7c-33ba-4020-a1c6-fe4297ad0421";"WdXALj-0bCs-Ndfy-xGIN-q1rF-hcmg-utIiNB
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb/osd-block-b8e82f22-e993-4458-984b-90232b8b3d55,ceph.block_uuid=Q1Aeu5-bm6V-m9yB-O3vX-SFvX-8oHo-SNkrfe,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p3,ceph.db_uuid=c7bb1f33-65f7-4116-ae27-0d73dde82208,ceph.encrypted=0,ceph.osd_fsid=b8e82f22-e993-4458-984b-90232b8b3d55,ceph.osd_id=2,ceph.type=block,ceph.vdo=0";"/dev/ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb/osd-block-b8e82f22-e993-4458-984b-90232b8b3d55";"osd-block-b8e82f22-e993-4458-984b-90232b8b3d55";"ceph-5dbc3f3a-d781-4d22-9e9c-8c7a74ce39fb";"Q1Aeu5-bm6V-m9yB-O3vX-SFvX-8oHo-SNkrfe
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-70e7b5f7-408d-4810-9d92-d6679d621db0/osd-block-02540fff-5478-4a67-bf5c-679c72150e8d,ceph.block_uuid=jp7YJ5-vNcj-93at-49LK-bSWT-ke4m-Y0Brzy,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p5,ceph.db_uuid=b5ea7d3a-8c90-4163-a2af-a4dfa045ced6,ceph.encrypted=0,ceph.osd_fsid=02540fff-5478-4a67-bf5c-679c72150e8d,ceph.osd_id=4,ceph.type=block,ceph.vdo=0";"/dev/ceph-70e7b5f7-408d-4810-9d92-d6679d621db0/osd-block-02540fff-5478-4a67-bf5c-679c72150e8d";"osd-block-02540fff-5478-4a67-bf5c-679c72150e8d";"ceph-70e7b5f7-408d-4810-9d92-d6679d621db0";"jp7YJ5-vNcj-93at-49LK-bSWT-ke4m-Y0Brzy
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d/osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe,ceph.block_uuid=enriOd-To2e-lLWi-Dc91-keUr-1tf7-4Hvrj2,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p1,ceph.db_uuid=99870da1-b7d1-479d-ad63-14407c593278,ceph.encrypted=0,ceph.osd_fsid=98bfb597-009b-4e88-bc5e-dd22587d21fe,ceph.osd_id=0,ceph.type=block,ceph.vdo=0";"/dev/ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d/osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe";"osd-block-98bfb597-009b-4e88-bc5e-dd22587d21fe";"ceph-766bd78c-ed1a-4e27-8b4d-7adc4c4f2f0d";"enriOd-To2e-lLWi-Dc91-keUr-1tf7-4Hvrj2
[2018-08-20 11:26:32,347][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-8df436c9-1302-4f1f-b185-c060f5e177d5/osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8,ceph.block_uuid=nqgDsS-dz7R-Ddlf-j31m-Q98Y-Qjua-Lmi5vq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p10,ceph.db_uuid=d7ac0b16-a18c-4e8a-973f-829970e5a5ec,ceph.encrypted=0,ceph.osd_fsid=2e5b3463-5904-4aee-9ae1-7d31d8576dc8,ceph.osd_id=9,ceph.type=block,ceph.vdo=0";"/dev/ceph-8df436c9-1302-4f1f-b185-c060f5e177d5/osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8";"osd-block-2e5b3463-5904-4aee-9ae1-7d31d8576dc8";"ceph-8df436c9-1302-4f1f-b185-c060f5e177d5";"nqgDsS-dz7R-Ddlf-j31m-Q98Y-Qjua-Lmi5vq
[2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389/osd-block-b7100200-9eef-4c85-b855-b5a0a435354c,ceph.block_uuid=xa3cq8-cGm1-1aKI-RC5M-mcqw-zadE-tdcHhK,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p6,ceph.db_uuid=fd88634e-7381-40e4-9aec-1df115ef64cf,ceph.encrypted=0,ceph.osd_fsid=b7100200-9eef-4c85-b855-b5a0a435354c,ceph.osd_id=5,ceph.type=block,ceph.vdo=0";"/dev/ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389/osd-block-b7100200-9eef-4c85-b855-b5a0a435354c";"osd-block-b7100200-9eef-4c85-b855-b5a0a435354c";"ceph-a23b8ade-25ae-43b2-ae42-8990bae6f389";"xa3cq8-cGm1-1aKI-RC5M-mcqw-zadE-tdcHhK
[2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c/osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd,ceph.block_uuid=S51Kgl-r9QD-P7Dn-PYzp-6mXi-TYfD-RGxefs,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p2,ceph.db_uuid=3ee5298a-2cbd-424b-b26d-e9eb04fdbc84,ceph.encrypted=0,ceph.osd_fsid=bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd,ceph.osd_id=1,ceph.type=block,ceph.vdo=0";"/dev/ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c/osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd";"osd-block-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd";"ceph-cb9bd619-dfd8-4a09-9b60-ecfb50b2f13c";"S51Kgl-r9QD-P7Dn-PYzp-6mXi-TYfD-RGxefs
[2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb/osd-block-ba351d69-5c48-418e-a377-4034f503af93,ceph.block_uuid=kpvBCz-Ikuu-52r0-0FQ2-NGR1-hlGP-bHpMi4,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p7,ceph.db_uuid=61bc909c-0930-49b7-90ad-c5e2d27eed9e,ceph.encrypted=0,ceph.osd_fsid=ba351d69-5c48-418e-a377-4034f503af93,ceph.osd_id=6,ceph.type=block,ceph.vdo=0";"/dev/ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb/osd-block-ba351d69-5c48-418e-a377-4034f503af93";"osd-block-ba351d69-5c48-418e-a377-4034f503af93";"ceph-db6fc227-11fb-4510-8a8b-29c16c7d9bfb";"kpvBCz-Ikuu-52r0-0FQ2-NGR1-hlGP-bHpMi4
[2018-08-20 11:26:32,348][ceph_volume.process][INFO ] stdout
ceph.block_device=/dev/ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a/osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593,ceph.block_uuid=ZctH0P-9zjl-zvTf-xHcd-bIIV-bMGa-vS4wOP,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=838506b7-e0c6-4022-9e17-2d1cf9458be6,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/nvme0n1p9,ceph.db_uuid=70e9e629-2c82-4aaa-bd73-abf22ebc9736,ceph.encrypted=0,ceph.osd_fsid=913e65e3-62d9-48f8-a0ef-45315cf64593,ceph.osd_id=8,ceph.type=block,ceph.vdo=0";"/dev/ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a/osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593";"osd-block-913e65e3-62d9-48f8-a0ef-45315cf64593";"ceph-ee343712-359b-4972-bbdf-c66ae5d4bf4a";"ZctH0P-9zjl-zvTf-xHcd-bIIV-bMGa-vS4wOP
[2018-08-20 11:26:32,349][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py",
line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line
153, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line
182, in dispatch
instance.main()
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/main.py", line
38, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line
182, in dispatch
instance.main()
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py",
line 16, in is_root
return func(*a, **kw)
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/trigger.py",
line 70, in main
Activate(['--auto-detect-objectstore', osd_id, osd_uuid]).main()
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/activate.py",
line 320, in main
self.activate(args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py",
line 16, in is_root
return func(*a, **kw)
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/activate.py",
line 230, in activate
raise RuntimeError('could not find osd.%s with fsid %s' % (osd_id,
osd_fsid))
RuntimeError: could not find osd.1 with fsid
4a9954ce-0a0f-432b-a91d-eaacb45287d4
ceph-volume-systemd.log (extract)
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-6-ba351d69-5c48-418e-a377-4034f503af93
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-3-9380cd27-c0fe-4ede-9ed3-d09eff545037
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-4-02540fff-5478-4a67-bf5c-679c72150e8d
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-0-98bfb597-009b-4e88-bc5e-dd22587d21fe
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-8-913e65e3-62d9-48f8-a0ef-45315cf64593
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-5-b7100200-9eef-4c85-b855-b5a0a435354c
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8
[2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm,
extra data: 6-ba351d69-5c48-418e-a377-4034f503af93
[2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm,
extra data: 3-9380cd27-c0fe-4ede-9ed3-d09eff545037
[2018-08-20 11:26:26,386][systemd][INFO ] parsed sub-command: lvm,
extra data: 1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 4-02540fff-5478-4a67-bf5c-679c72150e8d
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 0-98bfb597-009b-4e88-bc5e-dd22587d21fe
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 8-913e65e3-62d9-48f8-a0ef-45315cf64593
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 5-b7100200-9eef-4c85-b855-b5a0a435354c
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-2-b8e82f22-e993-4458-984b-90232b8b3d55
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8
[2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received:
lvm-1-4a9954ce-0a0f-432b-a91d-eaacb45287d4
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8
[2018-08-20 11:26:26,387][systemd][INFO ] parsed sub-command: lvm,
extra data: 2-b8e82f22-e993-4458-984b-90232b8b3d55
[2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 3-9380cd27-c0fe-4ede-9ed3-d09eff545037
[2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 2-b8e82f22-e993-4458-984b-90232b8b3d55
[2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 5-b7100200-9eef-4c85-b855-b5a0a435354c
[2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 6-ba351d69-5c48-418e-a377-4034f503af93
[2018-08-20 11:26:26,458][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 4-02540fff-5478-4a67-bf5c-679c72150e8d
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 8-913e65e3-62d9-48f8-a0ef-45315cf64593
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 0-98bfb597-009b-4e88-bc5e-dd22587d21fe
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 7-5d4af2fc-388c-4795-9d1a-53ad8aba56d8
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 1-4a9954ce-0a0f-432b-a91d-eaacb45287d4
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 9-2e5b3463-5904-4aee-9ae1-7d31d8576dc8
[2018-08-20 11:26:26,459][ceph_volume.process][INFO ] Running command:
ceph-volume lvm trigger 1-bcb9d7e6-44ea-449b-ad97-1aa5f880dfdd
[2018-08-20 11:26:27,068][ceph_volume.process][INFO ] stderr -->
RuntimeError: could not find osd.1 with fsid
4a9954ce-0a0f-432b-a91d-eaacb45287d4
[2018-08-20 11:26:27,086][systemd][WARNING] command returned non-zero
exit status: 1
[2018-08-20 11:26:27,086][systemd][WARNING] failed activating OSD,
retries left: 30
This sounds very odd. Could you clarify where block and block.db are?
Also useful here would be to take a look at
/var/log/ceph/ceph-volume-systemd.log and ceph-volume.log to
see how ceph-volume is trying to get this OSD up and running.
Also useful would be to check `ceph-volume lvm list` to verify that
regardless of the name change, it recognizes the correct partition
mapped to the OSD
Oops !
# ceph-volume lvm list
--> KeyError: 'devices'
Can you re-run this like:
CEPH_VOLUME_DEBUG=1 ceph-volume lvm list
And paste the output? I think this has been fixed since, but want to
double check
# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
(stable)
# CEPH_VOLUME_DEBUG=1 ceph-volume lvm list
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 6, in <module>
main.Volume()
File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line 37,
in __init__
self.main(self.argv)
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py",
line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/dist-packages/ceph_volume/main.py", line
153, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line
182, in dispatch
instance.main()
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/main.py", line
38, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/dist-packages/ceph_volume/terminal.py", line
182, in dispatch
instance.main()
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py",
line 300, in main
self.list(args)
File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py",
line 16, in is_root
return func(*a, **kw)
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py",
line 120, in list
pretty_report(report)
File
"/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/listing.py",
line 53, in pretty_report
device_metadata_item_template.format(tag_name='devices',
value=','.join(device['devices'])))
KeyError: 'devices'
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
