(I am subscribed now, no need to cc me anymore.)
On Fri, Jan 13, 2017 at 13:03:23 -0700, Bob Proulx wrote:
> Hmm... This feels like something where the smallest of details can
> cause a lot of ... I usually say problems but in this case I will say
> noise. And there are a lot of different ways to configure every
> possible part of the system including ssh. It just isn't possible to
> avoid tripping over one snag or another.
Agree!
> 208.118.235.201 is the IP address of the new server. 208.118.235.72
> is the IP address of the old server. Using crytographic keys to
> verify host systems is much stronger than using IP addresses.
> Therefore personally I don't track IP addresses and I personally
> configure ssh to track host keys only. I know that I am in the
> minority since the default is to do both. But there is no way to help
> it as the IP address must change. All we can do is try to minimize
> the amount of thrash by limiting the changes as much as possible.
>
> When I test switching from the previous IP address to the new IP
> address using default ssh options I see behavior like this:
>
> Perform action using ssh adds this known_hosts entry:
>
> hg.sv.gnu.org,208.118.235.72 ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
>
> Switch IP address of hg.sv.gnu.org to new server 208.118.235.201.
> Perform another ssh action.
>
> Warning: Permanently added the RSA host key for IP address
> '208.118.235.201' to the list of known hosts.
>
> Now the known_hosts entries looks like this (intentionally using
> HashKnownHosts=no
> so as to show the entries, otherwise we would just have the hash):
>
> hg.sv.gnu.org,208.118.235.72 ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
> 208.118.235.201 ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
>
> Switching the IP address back to the previous 208.118.235.72 causes no
> key warnings as the hg.sv.gnu.org,208.118.235.72 is a consistent state
> for that action.
Indeed, that's my experience with other server migrations where the keys
are preserved. Before I saw the thread about the old keys being cloned
to the new server, this looked to me like a case where it was a new
server with completely new keys.
> Well... Just removing the old entry and trying again seems to defeat
> the anti-MITM checks here. And once the entry has been deleted if it
> is a MITM attack then you have lost the original key to verify the
> previously known real host. Let's say I am on a laptop at an airport
> and I really am being attacked by having a MITM try to spoof things.
> Let's say I stop doing anything at that point and get on the airplane
> and travel. I maintain my known_hosts with the previously verified
> keys. I travel elsewhere. I connect again. If I have avoided the
> attack then the verified connectivity now works without changing
> anything.
Of course. In this case I was on my regular network and a change was
fully expected. I could have kept the previous entry until I got
definitive confirmation of the new server keys. Either way, I did not
accept the host key until I got independent confirmation over a separate
(preferably signed in some way) channel, which is how I would have added
a new server in the first place.
> Since your behavior differs from my testing with the default options
> (and with my CheckHostIP=no case) I can only assume that once again
> the smallest of details is causing a lot of, in this case, noise. I
> don't know why your case is different. Perhaps you can debug your
> case to root cause? It needs your access to your ssh version, config
> and known_hosts state.
Yes, I can reproduce this on a separate machine from a clean slate (with
CheckHostIP=yes).
First, assume I am a user who has only ever logged in to the old server:
$ rm -rf ~/.ssh/known_hosts
$ ssh -V
OpenSSH_7.4p1 Debian-5, OpenSSL 1.0.2j 26 Sep 2016
## first connect to hg.savannah.gnu.org bound to the old server
$ ssh -o FingerprintHash=md5 u...@hg.savannah.gnu.org true
The authenticity of host 'hg.savannah.gnu.org (208.118.235.72)' can't be
established.
RSA key fingerprint is MD5:80:5a:b0:0c:ec:93:66:29:49:7e:04:2b:fd:ba:2c:d5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hg.savannah.gnu.org,208.118.235.72' (RSA) to
the list of known hosts.
You tried to execute: true
Sorry, you are not allowed to execute that command.
## I'll cut these lines from subsequent examples, assume success
## ok, time passes, dns now points at the new server
$ ssh -o FingerprintHash=md5 u...@hg.savannah.gnu.org true
Warning: Permanently added the RSA host key for IP address
'208.118.235.201' to the list of known hosts.
...
Ok, this works as you described it should.
Now pretend that I have used the old server, but about a month ago I was
asked to test the new server under a different name:
$ rm -rf ~/.ssh/known_hosts
$ ssh -V
OpenSSH_7.4p1 Debian-5, OpenSSL 1.0.2j 26 Sep 2016
## first connect to hg.savannah.gnu.org bound to the old server
$ ssh -o FingerprintHash=md5 u...@hg.savannah.gnu.org true
The authenticity of host 'hg.savannah.gnu.org (208.118.235.72)' can't be
established.
RSA key fingerprint is MD5:80:5a:b0:0c:ec:93:66:29:49:7e:04:2b:fd:ba:2c:d5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hg.savannah.gnu.org,208.118.235.72' (RSA) to
the list of known hosts.
...
## now connect to the new server, but not as the same name
$ ssh -o FingerprintHash=md5 u...@hg0.savannah.gnu.org true
The authenticity of host 'hg0.savannah.gnu.org (208.118.235.201)' can't be
established.
ECDSA key fingerprint is
MD5:65:b8:1c:2f:82:7c:0e:39:e1:4a:63:f2:13:10:e8:9c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hg0.savannah.gnu.org,208.118.235.201' (ECDSA)
to the list of known hosts.
...
## now with the old name pointing to the new server
$ ssh -o FingerprintHash=md5 u...@hg.savannah.gnu.org true
Warning: the RSA host key for 'hg.savannah.gnu.org' differs from the key
for the IP address '208.118.235.201'
Offending key for IP in /home/user/.ssh/known_hosts:4
Matching host key in /home/user/.ssh/known_hosts:1
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.
A combination of (the default) CheckHostIP=yes, the fact that the new
server has keys that my client prefers over the old keys, and that I
must have connected to the new server before is what appears to be
triggering this. By the way, answering "yes" to this question does not
change known_hosts at all, the user will be asked the same question on
every connection from this point on, until one or more entries are
manually removed from known_hosts.
I can confirm that CheckHostIP=no eliminates this warning and prompt,
and connects to the new server using the original RSA host key.
Alternatively, if I start over and always connect with
HostKeyAlgorithms=ssh-rsa, the stronger key is ignored and there is also
no warning and no prompt.
I also observe that, in the case of a seamless migration where only the
warning about the new IP address is shown but no questions are asked,
~/.ssh/known_hosts only ever contains the original ssh-rsa key, until
the entry is completely removed from the file.
For my part, I have what I need, we don't need to investigate this any
further, but maybe this is helpful to you, let me know if I can test
anything else.
--
mike
