Package: open-iscsi
Version: 2.1.10-2
Severity: important
Hi,
I'm seeing an issue where automatic login at boot doesn't work on trixie, and
even after manual login, automatic logout at shutdown doesn't work on some
setups (see below for details).
I did see #932799 which sounds related but since I only see this on trixie, not
on bookworm, it most likely isn't the same issue. Marked as important because
#932799 was also marked as important.
Steps to reproduce:
- edit /etc/iscsi/iscsid.conf to set `node.startup = automatic`
- perform discovery and login:
# iscsiadm -m discovery --portal 10.1.1.128 --type sendtargets
10.1.1.128:3260,1
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209
# iscsiadm -m node --login
Logging in to [iface: default, target:
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209, portal:
10.1.1.128,3260]
Login to [iface: default, target:
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209, portal:
10.1.1.128,3260] successful.
# iscsiadm -m session
tcp: [1] 10.1.1.128:3260,1
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209 (non-flash)
- reboot
- after boot, no iSCSI session is established even though `node.startup` was
set to automatic:
# iscsiadm -m session
iscsiadm: No active sessions.
# journalctl -b | grep open-iscsi
Dec 18 10:42:48 debian-testing systemd[1]: open-iscsi.service - Login to
default iSCSI targets was skipped because no trigger condition checks were met.
# LANG=C systemctl status open-iscsi
* open-iscsi.service - Login to default iSCSI targets
Loaded: loaded (/usr/lib/systemd/system/open-iscsi.service; enabled;
preset: enabled)
Active: inactive (dead)
Condition: start condition unmet at Wed 2024-12-18 10:42:48 CET; 30s ago
|- ConditionDirectoryNotEmpty=|/etc/iscsi/nodes was not met
`- ConditionDirectoryNotEmpty=|/sys/class/iscsi_session was not met
Docs: man:iscsiadm(8)
man:iscsid(8)
Dec 18 10:42:48 debian-testing systemd[1]: open-iscsi.service - Login to
default iSCSI targets was skipped because no trigger condition checks were met.
I believe the reason is that upstream open-iscsi 2.1.9 changed [1] the location
of the persistent discovery database from /etc/iscsi/nodes to
/var/lib/iscsi/nodes:
# ls -al /etc/iscsi/nodes
ls: cannot access '/etc/iscsi/nodes': No such file or directory
# ls -al /var/lib/iscsi/nodes
total 12
drwx------ 3 root root 4096 Dec 18 09:22 .
drwx------ 4 root root 4096 Dec 18 09:22 ..
drwx------ 3 root root 4096 Dec 18 09:22
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209
But open-iscsi.service [2] checks for existence of /etc/iscsi/nodes (or
/sys/class/iscsi_session, which doesn't exist if there is no iSCSI session):
# systemctl cat open-iscsi.service | grep Condition
ConditionDirectoryNotEmpty=|/etc/iscsi/nodes
ConditionDirectoryNotEmpty=|/sys/class/iscsi_session
And hence doesn't start on boot.
Another side-effect is even after manual login, it looks like in some
situations automatic logout at shutdown doesn't work and shutdown hangs. I can
reproduce this if an iSCSI LUN is an LVM PV/VG with an active LV:
# iscsiadm -m node --login
Logging in to [iface: default, target:
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209, portal:
10.1.1.128,3260]
Login to [iface: default, target:
iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209, portal:
10.1.1.128,3260] successful.
# ls -al
/dev/disk/by-path/ip-10.1.1.128\:3260-iscsi-iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664\:sn.a65a5f75b209-lun-0
lrwxrwxrwx 1 root root 9 Dec 18 09:52
/dev/disk/by-path/ip-10.1.1.128:3260-iscsi-iqn.2003-01.org.linux-iscsi.iscsi-target-1path.x8664:sn.a65a5f75b209-lun-0
-> ../../sdb
# pvs
PV VG Fmt Attr PSize PFree
/dev/sdb myvg lvm2 a-- <3.97g 3.48g
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
mylv myvg -wi-a----- 500.00m
# poweroff
Shutdown hangs and I get the following warnings in the console:
Dec 18 09:53:43 debian-testing kernel: connection1:0: ping timeout of 5 secs
expired, recv timeout 5, last rx 4294911518, last ping 4294912832, now
4294914112
If I manually override ConditionDirectoryNotEmpty, both automatic login and
logout seem to work again:
# cat /etc/systemd/system/open-iscsi.service.d/override.conf
[Unit]
ConditionDirectoryNotEmpty=|/var/lib/iscsi/nodes
There is likely an additional problem on upgrades: On setups using automatic
login on bookworm, automatic login won't work after upgrading to trixie. These
setups have records at /etc/iscsi/nodes, so current open-iscsi.service will
start, but iscsiadm will not find them and thus won't log in.
As for how to fix this, I guess there are at least two options, but none seems
optimal:
- set dbroot back to /etc/iscsi, as hinted in [1].
But this might break setups already using the new location that don't rely on
automatic login.
- change open-iscsi.service to use the new location, i.e.,
ConditionDirectoryNotEmpty=|/var/lib/iscsi/nodes.
But this would still break automatic login on setups upgrading from bookworm
that have records in /etc/iscsi/nodes, as post-upgrade open-iscsi.service
won't start on boot then. Even if it started, iscsiadm wouldn't find the
discovery records under the old location and wouldn't login (see above).
Happy to provide more information if needed, just let me know.
[1]
https://github.com/open-iscsi/open-iscsi/commit/2eda28874196ed02c0d7b3696ccf81142c60409c
[2]
https://salsa.debian.org/linux-blocks-team/open-iscsi/-/blob/83da404e431b3020af7a122170b00a5bb79c69d1/debian/open-iscsi.service
-- System Information:
Debian Release: trixie/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 6.11.10-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages open-iscsi depends on:
ii debconf [debconf-2.0] 1.5.87
ii init-system-helpers 1.67
ii libc6 2.40-4
ii libisns0t64 0.101-1+b1
ii libkmod2 33+20240816-2
ii libmount1 2.40.2-12
ii libopeniscsiusr 2.1.10-2
ii libssl3t64 3.3.2-2
ii libsystemd0 257-2
ii udev 257-2
Versions of packages open-iscsi recommends:
ii busybox 1:1.37.0-4
pn finalrd <none>
open-iscsi suggests no packages.
-- Configuration Files:
/etc/iscsi/iscsid.conf changed:
iscsid.startup = /bin/systemctl start iscsid.socket
node.startup = automatic
node.leading_login = No
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.initial_login_retry_max = 8
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.session.nr_sessions = 1
node.session.reopen_max = 0
node.session.iscsi.FastAbort = Yes
node.session.scan = auto
-- debconf information:
open-iscsi/remove_even_with_active_sessions:
open-iscsi/upgrade_recovery_error:
open-iscsi/upgrade_even_with_failed_sessions:
open-iscsi/downgrade_and_break_system:
