Seems like there's some consensus for this being a security hardening
opportunity which we can safely work through in public, and possibly
accompany with some written guidance to administrators if anyone wants
to draft that.
** Description changed:
- This issue is being treated as a potential security risk under
- embargo. Please do not make any public mention of embargoed
- (private) security vulnerabilities before their coordinated
- publication by the OpenStack Vulnerability Management Team in the
- form of an official OpenStack Security Advisory. This includes
- discussion of the bug or associated fixes in public forums such as
- mailing lists, code review systems and bug trackers. Please also
- avoid private disclosure to other individuals not already approved
- for access to this information, and provide this same reminder to
- those who are made aware of the issue prior to publication. All
- discussion should remain confined to this private bug report, and
- any proposed fixes should be added to the bug as attachments. This
- embargo shall not extend past 2025-01-07 and will be made
- public by or on that date even if no fix is identified.
-
This was reported to me by a G-Research engineer, Andrew Ellard, as
tested against Nova+Ironic unmaintained/yoga branch. I am providing
information from him here, but will validate it in devstack and against
other branches in the next couple of days.
Add the following to nova.conf on a nova-compute instance running Ironic
driver (I'm unsure if this also impacts other drivers). Topic names,
urls, and paths redacted (I have not validated if this happens with
other oslo.notification drivers):
[DEFAULT]
notification_level = info
versioned_notifications_topics = oslo.notifications
[oslo_messaging_notifications]
driver = messagingv2
transport_url = $kafka_url
topics = oslo.notifications.$env.$region
[oslo_messaging_kafka]
security_protocol = SASL_SSL
sasl_mechanism = GSSAPI
ssl_ca_file = /path/to/tls-ca-bundle.pem
Rebuild a machine, and make that rebuild fail.
Observe that the payload in the rebuild_instance notification contains an
auth_token:
"priority"=>"ERROR",
"publisher_id"=>"compute.hostname-ironic",
"timestamp"=>"2024-10-08 14:51:23.891320",
"payload"=>{
"exception"=>"{'kwargs': {'code': 400}, 'message': 'Failed to provision
instance 91aa8063-cf88-4dbb-8644-5787ee7d0581: None', 'class':
'InstanceDeployFailure'}",
"args"=>{...
}
},
"message_id"=>"cdaaf8b9-1bc3-4d31-be58-315dd8acb9cd",
"_context"=>{...
},
"event_type"=>"rebuild_instance"
Auth tokens are found in:
- payload.args.instance.ec2_ids_context.auth_token
- payload.args.instance_info__cache_context.auth_token
- payload.args.request_spec_context.auth_token
- payload.args.request.spec_obj_image_context.auth_token
- payload.args.request_spec_obj_limits_context.auth_token
- payload.args.request_spec_obj_security_groups_context.auth_token
All of the auth_tokens appear to be valid.
It's notable that this notification appears different than most
notifications from nova (e.g. compute.instance.XXX).
I know VMT does not manage unmaintained branches, but I'd request we
keep this treated as an embargoed bug until we determine what the scope
is.
** Changed in: ossa
Status: Incomplete => Won't Fix
** Information type changed from Private Security to Public Security
** Information type changed from Public Security to Public
** Tags added: security
** Also affects: ossn
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2084081
Title:
Payload of "rebuild_instance" notification contains an auth_token
Status in OpenStack Compute (nova):
New
Status in OpenStack Security Advisory:
Won't Fix
Status in OpenStack Security Notes:
New
Bug description:
This was reported to me by a G-Research engineer, Andrew Ellard, as
tested against Nova+Ironic unmaintained/yoga branch. I am providing
information from him here, but will validate it in devstack and
against other branches in the next couple of days.
Add the following to nova.conf on a nova-compute instance running
Ironic driver (I'm unsure if this also impacts other drivers). Topic
names, urls, and paths redacted (I have not validated if this happens
with other oslo.notification drivers):
[DEFAULT]
notification_level = info
versioned_notifications_topics = oslo.notifications
[oslo_messaging_notifications]
driver = messagingv2
transport_url = $kafka_url
topics = oslo.notifications.$env.$region
[oslo_messaging_kafka]
security_protocol = SASL_SSL
sasl_mechanism = GSSAPI
ssl_ca_file = /path/to/tls-ca-bundle.pem
Rebuild a machine, and make that rebuild fail.
Observe that the payload in the rebuild_instance notification contains an
auth_token:
"priority"=>"ERROR",
"publisher_id"=>"compute.hostname-ironic",
"timestamp"=>"2024-10-08 14:51:23.891320",
"payload"=>{
"exception"=>"{'kwargs': {'code': 400}, 'message': 'Failed to provision
instance 91aa8063-cf88-4dbb-8644-5787ee7d0581: None', 'class':
'InstanceDeployFailure'}",
"args"=>{...
}
},
"message_id"=>"cdaaf8b9-1bc3-4d31-be58-315dd8acb9cd",
"_context"=>{...
},
"event_type"=>"rebuild_instance"
Auth tokens are found in:
- payload.args.instance.ec2_ids_context.auth_token
- payload.args.instance_info__cache_context.auth_token
- payload.args.request_spec_context.auth_token
- payload.args.request.spec_obj_image_context.auth_token
- payload.args.request_spec_obj_limits_context.auth_token
- payload.args.request_spec_obj_security_groups_context.auth_token
All of the auth_tokens appear to be valid.
It's notable that this notification appears different than most
notifications from nova (e.g. compute.instance.XXX).
I know VMT does not manage unmaintained branches, but I'd request we
keep this treated as an embargoed bug until we determine what the
scope is.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2084081/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
