Public bug reported: On Focal EC2 cloud, Cloud-init v.20.1-5-g67c8e53c-0ubuntu1 recently redacted header values from logs for security, but the redact operation was operating on the source headers dictionary. Ec2 API token requests are mangled with a 'REDACTED' value as a result and cloud-init can no longer obtain a valid IMDSv2 token and falls back to IMDSv2 after 2 minutes of failed retries.
Commit introducing the regression: https://github.com/canonical/cloud- init/commit/87cd040ed8fe7195cbb357ed3bbf53cd2a81436c#diff- a779470bb47168497ada0a33f7990b01R284 Error logs of interest 2020-03-03 13:27:47,839 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (400, 341b) after 1 attempts 2020-03-03 13:27:47,839 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/1s]: bad status code [400] 2020-03-03 13:27:47,847 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again 2020-03-03 13:27:48,848 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token 2020-03-03 13:27:48,848 - url_helper.py[DEBUG]: [0/1] open 'http://None/latest/api/token' with {'url': 'http://None/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration 2020-03-03 13:27:48,849 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc18d46cb10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) 2020-03-03 13:27:48,850 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1'}} configuration 2020-03-03 13:27:48,869 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 19b) after 1 attempts 2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254' 2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token ** Affects: cloud-init Importance: High Assignee: Chad Smith (chad.smith) Status: Triaged ** Changed in: cloud-init Importance: Undecided => High ** Changed in: cloud-init Assignee: (unassigned) => Chad Smith (chad.smith) ** Changed in: cloud-init Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1865882 Title: EC2: token requests 400 due to redacted headers Status in cloud-init: Triaged Bug description: On Focal EC2 cloud, Cloud-init v.20.1-5-g67c8e53c-0ubuntu1 recently redacted header values from logs for security, but the redact operation was operating on the source headers dictionary. Ec2 API token requests are mangled with a 'REDACTED' value as a result and cloud-init can no longer obtain a valid IMDSv2 token and falls back to IMDSv2 after 2 minutes of failed retries. Commit introducing the regression: https://github.com/canonical/cloud- init/commit/87cd040ed8fe7195cbb357ed3bbf53cd2a81436c#diff- a779470bb47168497ada0a33f7990b01R284 Error logs of interest 2020-03-03 13:27:47,839 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (400, 341b) after 1 attempts 2020-03-03 13:27:47,839 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/1s]: bad status code [400] 2020-03-03 13:27:47,847 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again 2020-03-03 13:27:48,848 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token 2020-03-03 13:27:48,848 - url_helper.py[DEBUG]: [0/1] open 'http://None/latest/api/token' with {'url': 'http://None/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration 2020-03-03 13:27:48,849 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc18d46cb10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) 2020-03-03 13:27:48,850 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1'}} configuration 2020-03-03 13:27:48,869 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 19b) after 1 attempts 2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254' 2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1865882/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
