Thanks for the feedback Wido. > It depends on how you look at this. Inside Ceph's RGW it's normal that > there is a user who has multiple buckets [...] The same applies to a > CloudStack account [...] If you want to separate, you can create > multiple accounts in CloudStack, correct?
I think there's a distinction to make between accounts owning resources that are within Cloudstack, vs Access/Secret keys for S3 buckets where the intention of those keys is to use them externally to the platform - i.e. in a backup tool, a CI/CD tool, a third-party or SaaS app. These things are typically outside of the span of control of Cloudstack or the administrator that's interacting with Cloudstack itself, so to share these credentials across buckets is the concern for me. Each bucket is typically for a different tool with different supplier/supply chain risks, therefore its own security boundary. Yes, using separate accounts in Cloudstack to get different secrets is an option, but I feel it's quite a heavy and inflexible requirement and conflates tenancy with storage-specific requirements. Ideally, I would see this being some form of IAM that could handle this, but I don't see that being a nearby feature. > Wouldn't this overload the backend? You keep creating users and I don't > know how that scales and if both Minio and Ceph were designed for this. Fair question, and I think you'd know the RGW internals far better than I would. My understanding is that RGW users are stored in metadata, and production deployments run into the thousands of users without issue. Buckets and users are stored in the metadata pool and just consume space rather than memory, and don't materially impact performance. If proliferation is a concern, perhaps one option would be to make per-bucket identity opt-in per bucket rather than the default, so an identity is only created when an administrator actually wants to isolate or delegate a bucket. The default stays account-level and nothing changes for the common case. The last part to cover - we don't have any ability to rotate keys right now, so at a minimum I feel there's a need for a second key and the ability to revoke/rotate regardless of where we land on identity. I know for Ceph at least, it's just a matter of adding a second key to the existing RGW user and having Cloudstack handle these changes. Mitch -----Original Message----- From: Wido den Hollander <[email protected]> Sent: Monday, 24 August 2026 10:33 PM To: [email protected]; Mitchell Drage <[email protected]> Subject: Re: [DISCUSS] Per-bucket credentials and key rotation for object storage This email comes from outside the organization. Do not click links or open attachments unless it is an email you expected to receive. Op 24-08-2026 om 06:17 schreef Mitchell Drage: > Hi all, > > I'd like to propose an improvement to the object storage feature and get > feedback before I start to think about developing something. > Thanks for this! Discussions and feedback a great. > > Current behaviour > > Object storage credentials are provisioned per CloudStack account, not per > bucket. The first bucket creates a backing user (e.g. a radosgw-admin user on > the RGW provider) and its access/secret key is stored against the account. > Every subsequent bucket in that account reuses the same key pair. Deleting > and recreating a bucket returns the same keys, and there's no API to rotate > or regenerate them. > > > Problem > > - A leaked key exposes every bucket the account owns, not just one. > - You can't delegate access to a single bucket without handing over access to > all of them. > - There's no safe way (or any way at all, from what I can see) to rotate a > credential, since there's only one and revoking it breaks every client at > once. > It depends on how you look at this. Inside Ceph's RGW it's normal that there is a user who has multiple buckets, with the Access and Secret key of that user you can access the buckets. This is very similar to other S3 services. The same applies to a CloudStack account, if you gain access to that account, you gain access to all the Instances of that user. If you want to separate, you can create multiple accounts in CloudStack, correct? > > Proposal > > 1. Per-bucket credentials: each bucket gets its own backing identity rather > than sharing the account's. This contains the impact of a leaked key to a > single bucket. Wouldn't this overload the backend? You keep creating users and I don't know how that scales and if both Minio and Ceph were designed for this. I'm not convinced yet, but I do appreciate feedback on features! Wido > 2. Two key pairs per bucket, with rotate and revoke: two independent slots so > an operator can rotate one while clients continue on the other, then cut over > and revoke the old one. This mirrors Azure's two-key model and AWS's > two-access-keys-per-user. > > I'd keep the backing primitive as plain users (RGW/MinIO/Cloudian all support > multiple keys per user), rather than the newer IAM-account constructs, to > keep the change contained and provider-agnostic. > > > Compatibility / migration > > - Existing buckets are untouched on upgrade. The bucket gains a nullable > credential association; when null, it falls back to the account key exactly > as today. > - New buckets get their own credentials by default. > - Operators opt in to migrating existing buckets via an explicit action > (single and bulk), so they choose when any disruption happens rather than the > upgrade forcing it. > - Every supported S3 backend would need code changes which would be larger > than I can take on practically, but also because I only have access to Ceph. > I could develop for Ceph, and the others would need to be a roadmap item. > Alternatively, other people familiar with those systems could develop code > and test. Ideally everything would move together, but I'm not confident that > we could have someone contribute for the MinIO and Cloudian backends, and I > also see that there's an open PR for Dell ECS support > (https://github.com/apache/cloudstack/pull/12124), so that would need > updating as well. > > > Future direction (I'm proposing it be out of scope here, for now) > > The longer-term destination might be an IAM / service-account style model > where identities are decoupled from buckets and bound via policy, which is > where AWS, Azure, GCP and now Ceph (accounts in Squid) all sit. > That requires a significant amount of change in CloudStack and the > introduction of IAM and granular policy, so I'm deliberately not tackling it > now. This proposal is designed as a step towards it rather than against it: > once credentials are their own entity, a many-to-many with permissions is an > extension rather than a rewrite. > > Rough scope is a moderate amount of change touching the object store driver > interface, the ceph provider plugin (ideally all the three provider plugins), > the bucket schema, a new API command + UI tab, and an upgrade path. > > Does this seem like a reasonable direction? > > > Thanks, > Mitch
