Public bug reported: Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g. AuthorizedKeysFile /etc/ssh/authorized_keys We observed that the file and parent folder permissions had been modified, to: /etc/ssh - 0700 /etc/ssh/authorized_key - 0600 These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file. We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c- 0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble. https://github.com/canonical/cloud- init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option: https://github.com/canonical/cloud- init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write. There are similarities to https://bugs.launchpad.net/cloud- init/+bug/1839061, but this is a different issue. ** Affects: cloud-init Importance: Undecided Status: New ** Description changed: Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds. We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g. - AuthorizedKeysFile /etc/ssh/authorized_keys + AuthorizedKeysFile /etc/ssh/authorized_keys We observed that the file and parent folder permissions had been modified, to: - /etc/ssh - 0700 - /etc/ssh/authorized_key - 0600 + /etc/ssh - 0700 + /etc/ssh/authorized_key - 0600 These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file. We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c- 0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble. https://github.com/canonical/cloud- init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option: https://github.com/canonical/cloud- init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write. + + There are similarities to https://bugs.launchpad.net/cloud- + init/+bug/1839061, but this is a different issue. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1911680 Title: Wrong access permissions of authorized keys file and parent directory when using absolute AuthorizedKeysFile Status in cloud-init: New Bug description: Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds. We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g. AuthorizedKeysFile /etc/ssh/authorized_keys We observed that the file and parent folder permissions had been modified, to: /etc/ssh - 0700 /etc/ssh/authorized_key - 0600 These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file. We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c- 0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble. https://github.com/canonical/cloud- init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option: https://github.com/canonical/cloud- init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write. There are similarities to https://bugs.launchpad.net/cloud- init/+bug/1839061, but this is a different issue. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1911680/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
