Hi all, I’ve been quietly working on implementing a Cloudian (an S3 Object Storage Vendor) plugin for the new Object Storage Framework in CloudStack. I’m getting there but I have some questions. If you are reading this and don’t have all the answers but know some of them or have an opinion on some of it, let me know.
Firstly, I’m using the following mapping currently. CloudStack Domain is mapped to a HyperStore Group. CloudStack Account is mapped to a HyperStore User (in the domain appropriate group) CloudStack Projects will probably be automatically mapped to HyperStore Users also (haven’t tried it yet). We have 3 server endpoints to manage (for this plugin) 1. The Admin API endpoint 2. An S3 service endpoint 3. An IAM service endpoint I plan to look at changing the “Add new object storage” UI to accept those 3 different endpoints if Cloudian is choosen as the provider. I haven’t looked into that yet. The Object Storage “url” will be the Admin API endpoint. I’ll probably add the other endpoint urls to the “details” Map similar to how the accesskey and secretkey are set already. I think this should work. I’ll probably get to this part next week. I am thinking that the main idea for this plugin is to have all bucket management (creation and deletion in particular) done from the CloudStack UI. I say this because, when you look at the “Details” UI element for a bucket, you can find the Access Key and Secret Key for that bucket there. If you add those credentials to a 3rd party s3 application etc, then it is possible to create other buckets that CloudStack will not know about or manage under that user account. As this seems wrong to me, I am planning on creating an IAM user with a policy that grants bucket and object actions except for createBucket and deleteBucket. The IAM user credentials will then be displayed instead of the Root User credentials. The Root user credentials will be used by the plugin itself to manage the bucket. In this manner, Users created by CloudStack can only create and delete buckets via the CloudStack UI. I don’t see anywhere to update the credentials associated with the ObjectStorage providers in case they change or get rotated? Right now I have the following credentials: ADMIN API - user/password combination Each HyperStore Account - Root User Credentials (hidden from user but used by plugin) Each HyperStore Account - IAM User with Restricted Credentials -> visible as part of Bucket “Details" I noticed that if there are 2 or more object storages configured and one of them has an exception when collecting bucket usage stats, then the other one can be skipped. I have a fix for this coming also. One other thing that I have done differently is the “public” bucket policy option. Minio and Ceph currently enable public bucket listing (index of objects in the bucket). I have only enabled GetObject on the bucket’s objects. I could enable the same as Minio and Ceph but I think it is over permissive. I think the developer of Ceph just copied what Minio was doing so I figured I’d ask. Another difference that I noticed is that Minio has CORS enabled by default on it's buckets. This seems to be required for the S3 Javascript browser built into CloudStack. By default, Cloudian does not enable CORS (similar to AWS). I plan to enable it as part of the createBucket API so that the CloudStack browser functionality also works but I am wondering how to set the Allowed Origin. Mino uses “*” allowing any Origin. I have not checked Ceph but there is nothing explicity done in the code. Do you think using “*” is ok? If not, I’m currently stuck for where to get the correct setting for Origin given that there could be many management servers. Also there is no place in the UI to manage this right now should the origins change. So maybe “*” is the right direction here. Where should documentation regarding this go? I think an administrator would like to know how for the different providers types the Accounts are mapped to the Storage etc. Things I am not going to fix in my first PR (but should be fixed): 1. INFO level logging currently dumps S3 credentials in the log file. This happens in a few places. Just look for “secretkey”. This should be addressed so as not to log clear text “passwords". 2. The secret key is visible in the Bucket “Details” page. This should possibly be hidden with a click to “copy” type browser action in place. Thanks, Tom. > On Sep 11, 2024, at 17:17, Tom O'Dowd <tpod...@cloudian.com> wrote: > > Sorry. answering my own question again. I found I could delete from the > object_store table in the cloud database directly so that works for me right > now rather than dropping everything. > > One thing that I have noticed about the UI is that the secret key is > “displayed” and “remembered” by the browser when configuring a new object > storage. This should probably behave more like a password field where you > cannot see the value normally and the browser doesn’t remember previously > used values. > > <Screenshot 2024-09-11 at 17.07.15.png> > > Tom. > >> On Sep 11, 2024, at 16:56, Tom O'Dowd <tpod...@cloudian.com> wrote: >> >> Hi guys, >> >> I’ve started working on the Object Storage plugin. One quick question which >> I can’t find yet in the docs… >> >> Once I have added an object store as the admin user, how do I remove it? ie, >> I just want to blow away the configuration for now so I can try re-add it >> repeatedly as I fix things? Maybe the opposite of deploydb will do? >> >> Thanks, >> >> Tom. >> >>> On Sep 6, 2024, at 18:55, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: >>> >>> Hi Tom, >>> >>> In addition to what has been answered and what you've found, here are some >>> useful links for reference: >>> >>> https://github.com/shapeblue/hackerbook/tree/main (self-learning dev >>> material) >>> https://github.com/shapeblue/hackerbook/blob/main/2-dev.md#contributing-to-cloudstack >>> https://github.com/shapeblue/mbx (local CI/dev-test setup) >>> https://github.com/apache/cloudstack/commit/c3f0d14d31583fdf2570de2633db5c87a0f08fa4 >>> (learn from new and existing plugins as reference implementation) >>> https://github.com/apache/cloudstack/tree/main/plugins/storage/object/minio >>> (original minio plugin for reference) >>> >>> >>> Regards. >>> >>> >>> >>> >>> ________________________________ >>> From: Tom O'Dowd <tpod...@cloudian.com.INVALID> >>> Sent: Friday, September 6, 2024 14:37 >>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> >>> Subject: Re: Object Storage Framework integration for Cloudian HyperStore >>> >>> Sorry for the noise. I answered my own question (or rather I found a page >>> that did): https://cloudstack.apache.org/developers/ >>> >>> Thanks, >>> >>> Tom. >>> >>>> On Sep 6, 2024, at 17:06, Tom O'Dowd <tpod...@cloudian.com> wrote: >>>> >>>> Hi Daan, >>>> >>>> I have made some progress. I have got CloudStack running on Ubuntu 24.04 >>>> now. I have installed Minio and configurated it out as the Object Storage >>>> Provider to understand how it being used from an Admin and Regular user’s >>>> perspective. Additionally, I have also enabled the existing Cloudian >>>> Plugin and tried that out to ensure that it's all working also. >>>> >>>> There is one small item that I would like to fix in the existing plugin >>>> relating to a path change in the default HyperStore URL that the SSO uses. >>>> That’s just a string change. Additionally, we have moved away from Puppet >>>> in recent HyperStore versions for configuration management and I’d like to >>>> update the documentation specific to that point. My question is which >>>> branch I should use as a base for my PRs for both cloudstack and >>>> cloudstack-documentation. >>>> >>>> Thanks, >>>> >>>> Tom. >>>> >>>>> On Sep 3, 2024, at 17:38, Daan Hoogland <daan.hoogl...@gmail.com> wrote: >>>>> >>>>> Tom, >>>>> You are welcome to submit a PR and we'll happily review it and help you >>>>> tackle problems. The framework is there to use and adjust to your needs. >>>>> As >>>>> for the API, what level of stability do you require? I expect some changes >>>>> on behalf of you if you are serious about implementing it. The minio is a >>>>> PoC implementation and at least a ceph one is underway. >>>>> >>>>> On Tue, Sep 3, 2024 at 9:59 AM Tom O'Dowd <tpod...@cloudian.com.invalid> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We (Cloudian) have been getting some inquiries requesting a new Cloudian >>>>>> HyperStore plugin to work with CloudStacks new Object Storage Framework. >>>>>> We >>>>>> currently have a different CloudStack plugin which adds integration >>>>>> between >>>>>> the CloudStack UI and our UI that works for both admin and users as well >>>>>> as >>>>>> providing secondary storage. I’ve been trying to find out more >>>>>> information >>>>>> regarding the new Object Storage Framework and see there is a Minio >>>>>> plugin >>>>>> and a simulator plugin. I’ve also just checked out the git repo and have >>>>>> found the relevant code. Is the API stable at this point and are you >>>>>> accepting new plugin contributions? Can I just implement the same base >>>>>> objects that the Minio plugin is implementing in order to get our own >>>>>> plugin working? >>>>>> >>>>>> I notice in the “Add Object Storage” UI, there are fields: >>>>>> >>>>>> Name >>>>>> Provider >>>>>> URL >>>>>> Access key >>>>>> Secret key >>>>>> >>>>>> I think that we would use an https URL, but some installations would >>>>>> probably have a self-signed cert so they might need the option to skip >>>>>> validation. Didn’t check how this was working with Minio yet. Currently, >>>>>> we >>>>>> do not use an access/secret key to connect to the administration API but >>>>>> rather a username and password. Once connected, this should be able to >>>>>> allocate users to match CloudStack accounts. I’d need to look at what >>>>>> mappings to use there. The users should have Access/Secret keys so I >>>>>> would >>>>>> expect all of the S3 bucket browser stuff will just work once its all >>>>>> connected. >>>>>> >>>>>> Development wise, I have followed the “Developers Guide” under >>>>>> https://docs.cloudstack.apache.org/en/latest/developersguide/developer_guide.html#prerequisites >>>>>> and got as far as connecting to the management server without the >>>>>> simulator. I’m currently running on Rocky8. I’m wondering if I’m going to >>>>>> run into trouble on this OS and if I should switch to Ubuntu 24.04 which >>>>>> it >>>>>> seems the instructions were for? >>>>>> >>>>>> I guess my first goal is to seem it working with Minio and then try to >>>>>> get >>>>>> it working with HyperStore. >>>>>> >>>>>> Any advice appreciated, >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Tom. >>>>> >>>>> >>>>> >>>>> -- >>>>> Daan >>>> >>> >> >
