I verified the fix using openssh-server 1:9.0p1-1ubuntu7.2 from kinetic-
proposed:
Test 1:
root@kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu7.2
Candidate: 1:9.0p1-1ubuntu7.2
Version table:
*** 1:9.0p1-1ubuntu7.2 500
500 http://archive.ubuntu.com/ubuntu kinetic-proposed/main amd64
Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu7.1 500
500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
1:9.0p1-1ubuntu7 500
500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages
root@kinetic:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:DduZSXZNbtS3h9D91h0NQfRK7wnuxpWrj3f8/0J4ajc.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? no
Host key verification failed.
root@kinetic:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 14:00:03 UTC; 11s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1013 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1014 (sshd)
Tasks: 1 (limit: 18854)
Memory: 1.3M
CPU: 40ms
CGroup: /system.slice/ssh.service
└─1014 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 15 14:00:03 kinetic systemd[1]: Starting OpenBSD Secure Shell server...
May 15 14:00:03 kinetic sshd[1014]: Server listening on :: port 22.
May 15 14:00:03 kinetic systemd[1]: Started OpenBSD Secure Shell server.
May 15 14:00:04 kinetic sshd[1015]: Connection closed by ::1 port 43046
[preauth]
root@kinetic:~# systemctl reload ssh.service
root@kinetic:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 14:00:03 UTC; 22s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1013 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1021 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1022 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1014 (sshd)
Tasks: 1 (limit: 18854)
Memory: 1.4M
CPU: 97ms
CGroup: /system.slice/ssh.service
└─1014 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 15 14:00:03 kinetic systemd[1]: Starting OpenBSD Secure Shell server...
May 15 14:00:03 kinetic sshd[1014]: Server listening on :: port 22.
May 15 14:00:03 kinetic systemd[1]: Started OpenBSD Secure Shell server.
May 15 14:00:04 kinetic sshd[1015]: Connection closed by ::1 port 43046
[preauth]
May 15 14:00:23 kinetic systemd[1]: Reloading OpenBSD Secure Shell server...
May 15 14:00:23 kinetic sshd[1014]: Received SIGHUP; restarting.
May 15 14:00:23 kinetic systemd[1]: Reloaded OpenBSD Secure Shell server.
May 15 14:00:23 kinetic sshd[1014]: Server listening on :: port 22.
Test 2:
root@kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu7.2
Candidate: 1:9.0p1-1ubuntu7.2
Version table:
*** 1:9.0p1-1ubuntu7.2 500
500 http://archive.ubuntu.com/ubuntu kinetic-proposed/main amd64
Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu7.1 500
500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
1:9.0p1-1ubuntu7 500
500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages
root@kinetic:~# vi /etc/default/ssh
root@kinetic:~# cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
root@kinetic:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:DduZSXZNbtS3h9D91h0NQfRK7wnuxpWrj3f8/0J4ajc.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.10 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: Copy environment: XDG_SESSION_ID=15
debug3: Copy environment: XDG_RUNTIME_DIR=/run/user/0
debug3: Copy environment: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
debug3: Copy environment: XDG_SESSION_TYPE=tty
debug3: Copy environment: XDG_SESSION_CLASS=user
debug3: Copy environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
debug3: Copy environment: LANG=en_US.UTF-8
Environment:
LANG=en_US.UTF-8
USER=root
LOGNAME=root
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL=/bin/bash
TERM=xterm-256color
XDG_SESSION_ID=15
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=::1 36376 22
SSH_CONNECTION=::1 36376 ::1 22
SSH_TTY=/dev/pts/2
root@kinetic:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Mon 2023-05-15 14:19:05 UTC; 11s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 150 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 151 (sshd)
Tasks: 0 (limit: 18854)
Memory: 2.2M
CPU: 91ms
CGroup: /system.slice/ssh.service
‣ 151 "sshd: root@pts/2"
May 15 14:19:06 kinetic sshd[151]: debug1: server_input_channel_req: channel 0
request shell reply 1
May 15 14:19:06 kinetic sshd[151]: debug1: session_by_channel: session 0
channel 0
May 15 14:19:06 kinetic sshd[151]: debug1: session_input_channel_req: session 0
req shell
May 15 14:19:06 kinetic sshd[151]: Starting session: shell on pts/2 for root
from ::1 port 36376 id 0
May 15 14:19:06 kinetic sshd[151]: debug2: fd 5 setting TCP_NODELAY
May 15 14:19:06 kinetic sshd[151]: debug3: set_sock_tos: set socket 5
IPV6_TCLASS 0x10
May 15 14:19:06 kinetic sshd[151]: debug2: channel 0: rfd 11 isatty
May 15 14:19:06 kinetic sshd[151]: debug2: fd 11 setting O_NONBLOCK
May 15 14:19:06 kinetic sshd[151]: debug3: fd 8 is O_NONBLOCK
May 15 14:19:06 kinetic sshd[151]: debug3: send packet: type 99
** Tags removed: verification-needed verification-needed-kinetic
verification-needed-lunar
** Tags added: verification-done verification-done-kinetic
verification-done-lunar
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2011458
Title:
ssh fails to rebind when it is killed with -HUP
Status in openssh package in Ubuntu:
Fix Committed
Status in openssh source package in Kinetic:
Fix Committed
Status in openssh source package in Lunar:
Fix Committed
Bug description:
[Impact]
The sshd re-execution logic is generally broken with systemd socket
activation, which means that (1) sshd fails when it is told to re-exec
via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in
debug mode.
[Test Case]
(1) Test systemctl reload ssh:
* On a machine with openssh-server installed, make a connection to
localhost to activate ssh.service (the connection does not need to be
complete, so you can just say "no" at the host key verification
stage):
$ ssh localhost
[...]
* Send SIGHUP to sshd by calling systemctl reload ssh:
$ systemctl reload ssh
* Check the service state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s
ago
Duration: 2min 44.132s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1112 (code=exited, status=255/EXCEPTION)
CPU: 79ms
Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666
[preauth]
Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure
Shell server...
Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting.
Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address.
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
* On an affected machine, the service will fail as shown above.
(2) Test debug mode:
* On a machine with openssh-server installed, edit /etc/default/ssh to
configure debug mode for sshd:
$ cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
* Attempt to make a connection to localhost:
$ ssh localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
* On an affected machine, the attempt will fail as shown above, and
the service will be in a failed state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC;
2min 27s ago
Duration: 5ms
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited,
status=0/SUCCESS)
Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Main PID: 1167 (code=exited, status=255/EXCEPTION)
CPU: 40ms
Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22.
Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK
Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running
in debugging mode.
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7
config len 3456
Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done
Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4
pipe -1 sock 7
Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
[Where problems could occur]
The fix expands Ubuntu's patch for systemd socket activation to try
and make sure that any fds passed from systemd are not closed across
re-executions of sshd. If we saw a problem, it would most likely be an
attempt to operate on a closed fd, or the wrong fd, as a result of an
edge case in one of the re-execution paths.
[Original Description]
In kinetic and lunar gce images we are facing an issue when ssh is being
killed with -HUP
SSH is failing to rebind port 22. It is not failing in other previous systems.
It can be reproduced by running
# pkill -o -HUP sshd || true
# journalctl -n 20
Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on
0.0.0.0 failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on ::
failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address.
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process
exited, code=exited, status=255/EXCEPTION
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result
'exit-code'.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
