Public bug reported: [Impact] * ceph-iscsi on Focal talking to a Pacific or later Ceph cluster
* rbd-target-api service fails to start if there is a blocklist entry for the unit. * When the rbd-target-api service starts it checks if any of the ip addresses on the machine it is running on are listed as blocked. If there are entries it tries to remove them. When it issues the block removal command it checks stdout from the removal command looking for the string `un-blacklisting`. However from Pacific onward a successful unblocking returns `un-blocklisting` instead (https://github.com/ceph/ceph/commit/dfd01d765304ed8783cef613930e65980d9aee23) [Test Plan] If an existing ceph-iscsi deployment is available then skip to step 3. 1) Deploy the bundle below (tested with OpenStack provider). series: focal applications: ceph-iscsi: charm: cs:ceph-iscsi num_units: 2 ceph-osd: charm: ch:ceph-osd num_units: 3 storage: osd-devices: 'cinder,10G' options: osd-devices: '/dev/test-non-existent' source: yoga channel: latest/edge ceph-mon: charm: ch:ceph-mon num_units: 3 options: monitor-count: '3' source: yoga channel: latest/edge relations: - - 'ceph-mon:client' - 'ceph-iscsi:ceph-client' - - 'ceph-osd:mon' - 'ceph-mon:osd' 2) Connect to ceph-iscsi unit: juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0 3) Stop rbd-target-api via systemd to make test case clearer: sudo systemctl stop rbd-target-api 4) Add 2 blocklist entries for this unit (due to another issue the ordering of the output from `osd blacklist ls` matters which can lead to the reproduction of this bug being intermittent. To avoid this add two entries which ensures there is always an entry for this node in the list of blocklist entries to be removed). sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1 sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2 sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls listed 2 entries 172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000 172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000 5) Attempt to start service: sudo /usr/bin/python3 /usr/bin/rbd-target-api At this point the process should be running in the foreground but instead it will die. The log from the service will have an entry like: 2022-04-21 12:35:21,695 CRITICAL [gateway.py:51:ceph_rm_blacklist()] - blacklist removal failed. Run 'ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist rm 172.20.0.156:0/1' [Where problems could occur] * Problems could occur with the service starting as this blocklist check is done at startup. * Blocklist entries could fail to be removed. This issue is very similar to Bug #1883112 ** Affects: ceph-iscsi (Ubuntu) Importance: Undecided Status: New ** Description changed: [Impact] - * ceph-iscsi on Focal talking to a Pacific or later Ceph cluster + * ceph-iscsi on Focal talking to a Pacific or later Ceph cluster - * rbd-target-api service fails to start if there is a blocklist - entry for the unit. + * rbd-target-api service fails to start if there is a blocklist + entry for the unit. - * When the rbd-target-api service starts it checks if any of the - ip addresses on the machine it is running on are listed as - blocked. If there are entries it tries to remove them. When it - issues the block removal command it checks stdout from the - removal command looking for the string `un-blacklisting`. - However from Pacific onward a successful unblocking returns - `un-blocklisting` instead (https://github.com/ceph/ceph/commit/dfd01d765304ed8783cef613930e65980d9aee23) - + * When the rbd-target-api service starts it checks if any of the + ip addresses on the machine it is running on are listed as + blocked. If there are entries it tries to remove them. When it + issues the block removal command it checks stdout from the + removal command looking for the string `un-blacklisting`. + However from Pacific onward a successful unblocking returns + `un-blocklisting` instead (https://github.com/ceph/ceph/commit/dfd01d765304ed8783cef613930e65980d9aee23) [Test Plan] - If an existing ceph-iscsi deployment is available then skip to - step 3. + If an existing ceph-iscsi deployment is available then skip to + step 3. - 1) Deploy the bundle below (tested with OpenStack provider). + 1) Deploy the bundle below (tested with OpenStack provider). series: focal applications: - ceph-iscsi: - charm: cs:ceph-iscsi - num_units: 2 - ceph-osd: - charm: ch:ceph-osd - num_units: 3 - storage: - osd-devices: 'cinder,10G' - options: - osd-devices: '/dev/test-non-existent' - source: yoga - channel: latest/edge - ceph-mon: - charm: ch:ceph-mon - num_units: 3 - options: - monitor-count: '3' - source: yoga - channel: latest/edge + ceph-iscsi: + charm: cs:ceph-iscsi + num_units: 2 + ceph-osd: + charm: ch:ceph-osd + num_units: 3 + storage: + osd-devices: 'cinder,10G' + options: + osd-devices: '/dev/test-non-existent' + source: yoga + channel: latest/edge + ceph-mon: + charm: ch:ceph-mon + num_units: 3 + options: + monitor-count: '3' + source: yoga + channel: latest/edge relations: - - - 'ceph-mon:client' - - 'ceph-iscsi:ceph-client' - - - 'ceph-osd:mon' - - 'ceph-mon:osd' + - - 'ceph-mon:client' + - 'ceph-iscsi:ceph-client' + - - 'ceph-osd:mon' + - 'ceph-mon:osd' - 2) Connect to ceph-iscsi unit: + 2) Connect to ceph-iscsi unit: juju ssh -m zaza-a1d88053ab85 ceph-iscsi/0 - 3) Stop rbd-target-api via systemd to make test case clearer: + 3) Stop rbd-target-api via systemd to make test case clearer: sudo systemctl stop rbd-target-api - 4) Add 2 blocklist entries for this unit (due to another issue the + 4) Add 2 blocklist entries for this unit (due to another issue the ordering of the output from `osd blacklist ls` matters which can lead to the reproduction of this bug being intermittent. To avoid this add two entries which ensures there is always an entry for this node in the list of blocklist entries to be removed). sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/1 sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist add $(hostname --all-ip-addresses | awk '{print $1}'):0/2 sudo ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist ls - listed 2 entries - 172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000 - 172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000 + listed 2 entries + 172.20.0.135:0/2 2022-02-23T11:14:54.850352+0000 + 172.20.0.135:0/1 2022-02-23T11:14:52.502592+0000 - 5) Attempt to start service: + 5) Attempt to start service: sudo /usr/bin/python3 /usr/bin/rbd-target-api At this point the process should be running in the foreground but instead it will die. The log from the service will have an entry like: 2022-04-21 12:35:21,695 CRITICAL [gateway.py:51:ceph_rm_blacklist()] - blacklist removal failed. Run 'ceph -n client.ceph-iscsi --conf /etc/ceph/iscsi/ceph.conf osd blacklist rm 172.20.0.156:0/1' - [Where problems could occur] - * Problems could occur with the service starting as this blocklist + * Problems could occur with the service starting as this blocklist check is done at startup. - * Blocklist entries could fail to be removed. + * Blocklist entries could fail to be removed. + + + This issue is very similar to Bug #1883112 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1969775 Title: rbd-target-api crashes with `blacklist removal failed` To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1969775/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
