Op 25-08-2026 om 02:27 schreef Mitchell Drage:
Thanks for the feedback Wido.
It depends on how you look at this. Inside Ceph's RGW it's normal that
there is a user who has multiple buckets [...] The same applies to a
CloudStack account [...] If you want to separate, you can create
multiple accounts in CloudStack, correct?
I think there's a distinction to make between accounts owning resources that
are within Cloudstack, vs Access/Secret keys for S3 buckets where the intention
of those keys is to use them externally to the platform - i.e. in a backup
tool, a CI/CD tool, a third-party or SaaS app. These things are typically
outside of the span of control of Cloudstack or the administrator that's
interacting with Cloudstack itself, so to share these credentials across
buckets is the concern for me. Each bucket is typically for a different tool
with different supplier/supply chain risks, therefore its own security boundary.
I can't disagree with you there. I would think that it does make sense,
but then you also want to be able to rotate the keys of buckets, that is
a very valid use-case as well.
Yes, using separate accounts in Cloudstack to get different secrets is an
option, but I feel it's quite a heavy and inflexible requirement and conflates
tenancy with storage-specific requirements.
Ideally, I would see this being some form of IAM that could handle this, but I
don't see that being a nearby feature.
Agree with you there.
Wouldn't this overload the backend? You keep creating users and I don't
know how that scales and if both Minio and Ceph were designed for this.
Fair question, and I think you'd know the RGW internals far better than I
would. My understanding is that RGW users are stored in metadata, and
production deployments run into the thousands of users without issue. Buckets
and users are stored in the metadata pool and just consume space rather than
memory, and don't materially impact performance.
True, I don't think Ceph will have a real problem in handling this.
If proliferation is a concern, perhaps one option would be to make per-bucket
identity opt-in per bucket rather than the default, so an identity is only
created when an administrator actually wants to isolate or delegate a bucket.
The default stays account-level and nothing changes for the common case.
The last part to cover - we don't have any ability to rotate keys right now, so
at a minimum I feel there's a need for a second key and the ability to
revoke/rotate regardless of where we land on identity.
I know for Ceph at least, it's just a matter of adding a second key to the
existing RGW user and having Cloudstack handle these changes.
True. The longer I think about it, the more it makes sense to have
unique keys per bucket. How we achieve this technically and per backend,
that's the next step, but for now this seems reasonable and logic to do.
And key rotation should be there as well :-)
Wido
Mitch
-----Original Message-----
From: Wido den Hollander <[email protected]>
Sent: Monday, 24 August 2026 10:33 PM
To: [email protected]; Mitchell Drage <[email protected]>
Subject: Re: [DISCUSS] Per-bucket credentials and key rotation for object
storage
This email comes from outside the organization.
Do not click links or open attachments unless it is an email you expected to
receive.
Op 24-08-2026 om 06:17 schreef Mitchell Drage:
Hi all,
I'd like to propose an improvement to the object storage feature and get
feedback before I start to think about developing something.
Thanks for this! Discussions and feedback a great.
Current behaviour
Object storage credentials are provisioned per CloudStack account, not per
bucket. The first bucket creates a backing user (e.g. a radosgw-admin user on
the RGW provider) and its access/secret key is stored against the account.
Every subsequent bucket in that account reuses the same key pair. Deleting and
recreating a bucket returns the same keys, and there's no API to rotate or
regenerate them.
Problem
- A leaked key exposes every bucket the account owns, not just one.
- You can't delegate access to a single bucket without handing over access to
all of them.
- There's no safe way (or any way at all, from what I can see) to rotate a
credential, since there's only one and revoking it breaks every client at once.
It depends on how you look at this. Inside Ceph's RGW it's normal that there is
a user who has multiple buckets, with the Access and Secret key of that user
you can access the buckets.
This is very similar to other S3 services. The same applies to a CloudStack
account, if you gain access to that account, you gain access to all the
Instances of that user.
If you want to separate, you can create multiple accounts in CloudStack,
correct?
Proposal
1. Per-bucket credentials: each bucket gets its own backing identity rather
than sharing the account's. This contains the impact of a leaked key to a
single bucket.
Wouldn't this overload the backend? You keep creating users and I don't know
how that scales and if both Minio and Ceph were designed for this.
I'm not convinced yet, but I do appreciate feedback on features!
Wido
2. Two key pairs per bucket, with rotate and revoke: two independent slots so
an operator can rotate one while clients continue on the other, then cut over
and revoke the old one. This mirrors Azure's two-key model and AWS's
two-access-keys-per-user.
I'd keep the backing primitive as plain users (RGW/MinIO/Cloudian all support
multiple keys per user), rather than the newer IAM-account constructs, to keep
the change contained and provider-agnostic.
Compatibility / migration
- Existing buckets are untouched on upgrade. The bucket gains a nullable
credential association; when null, it falls back to the account key exactly as
today.
- New buckets get their own credentials by default.
- Operators opt in to migrating existing buckets via an explicit action (single
and bulk), so they choose when any disruption happens rather than the upgrade
forcing it.
- Every supported S3 backend would need code changes which would be larger than
I can take on practically, but also because I only have access to Ceph. I could
develop for Ceph, and the others would need to be a roadmap item.
Alternatively, other people familiar with those systems could develop code and
test. Ideally everything would move together, but I'm not confident that we
could have someone contribute for the MinIO and Cloudian backends, and I also
see that there's an open PR for Dell ECS support
(https://github.com/apache/cloudstack/pull/12124), so that would need updating
as well.
Future direction (I'm proposing it be out of scope here, for now)
The longer-term destination might be an IAM / service-account style model where
identities are decoupled from buckets and bound via policy, which is where AWS,
Azure, GCP and now Ceph (accounts in Squid) all sit.
That requires a significant amount of change in CloudStack and the introduction
of IAM and granular policy, so I'm deliberately not tackling it now. This
proposal is designed as a step towards it rather than against it: once
credentials are their own entity, a many-to-many with permissions is an
extension rather than a rewrite.
Rough scope is a moderate amount of change touching the object store driver
interface, the ceph provider plugin (ideally all the three provider plugins),
the bucket schema, a new API command + UI tab, and an upgrade path.
Does this seem like a reasonable direction?
Thanks,
Mitch
