Hi all, I have finished working on the plugin and have submitted a PR which is currently waiting on workflow approval.
https://github.com/apache/cloudstack/pull/9748 One part of my changes that I am a little unsure of relates to maven edits to pom.xml. Please let me know if something looks amiss in that area. Thanks, Tom. > On Sep 21, 2024, at 17:21, Tom O'Dowd <tpod...@cloudian.com> wrote: > > Hi all, > > I’ve been quietly working on implementing a Cloudian (an S3 Object Storage > Vendor) plugin for the new Object Storage Framework in CloudStack. I’m > getting there but I have some questions. If you are reading this and don’t > have all the answers but know some of them or have an opinion on some of it, > let me know. > > Firstly, I’m using the following mapping currently. > > CloudStack Domain is mapped to a HyperStore Group. > CloudStack Account is mapped to a HyperStore User (in the domain appropriate > group) > CloudStack Projects will probably be automatically mapped to HyperStore Users > also (haven’t tried it yet). > > We have 3 server endpoints to manage (for this plugin) > 1. The Admin API endpoint > 2. An S3 service endpoint > 3. An IAM service endpoint > > I plan to look at changing the “Add new object storage” UI to accept those 3 > different endpoints if Cloudian is choosen as the provider. I haven’t looked > into that yet. > > The Object Storage “url” will be the Admin API endpoint. I’ll probably add > the other endpoint urls to the “details” Map similar to how the accesskey > and secretkey are set already. I think this should work. I’ll probably get to > this part next week. > > I am thinking that the main idea for this plugin is to have all bucket > management (creation and deletion in particular) done from the CloudStack UI. > I say this because, when you look at the “Details” UI element for a bucket, > you can find the Access Key and Secret Key for that bucket there. If you add > those credentials to a 3rd party s3 application etc, then it is possible to > create other buckets that CloudStack will not know about or manage under that > user account. As this seems wrong to me, I am planning on creating an IAM > user with a policy that grants bucket and object actions except for > createBucket and deleteBucket. The IAM user credentials will then be > displayed instead of the Root User credentials. The Root user credentials > will be used by the plugin itself to manage the bucket. In this manner, Users > created by CloudStack can only create and delete buckets via the CloudStack > UI. > > I don’t see anywhere to update the credentials associated with the > ObjectStorage providers in case they change or get rotated? > > Right now I have the following credentials: > ADMIN API - user/password combination > Each HyperStore Account - Root User Credentials (hidden from user but used by > plugin) > Each HyperStore Account - IAM User with Restricted Credentials -> visible as > part of Bucket “Details" > > I noticed that if there are 2 or more object storages configured and one of > them has an exception when collecting bucket usage stats, then the other one > can be skipped. I have a fix for this coming also. > > One other thing that I have done differently is the “public” bucket policy > option. Minio and Ceph currently enable public bucket listing (index of > objects in the bucket). I have only enabled GetObject on the bucket’s > objects. I could enable the same as Minio and Ceph but I think it is over > permissive. I think the developer of Ceph just copied what Minio was doing so > I figured I’d ask. > > Another difference that I noticed is that Minio has CORS enabled by default > on it's buckets. This seems to be required for the S3 Javascript browser > built into CloudStack. By default, Cloudian does not enable CORS (similar to > AWS). I plan to enable it as part of the createBucket API so that the > CloudStack browser functionality also works but I am wondering how to set the > Allowed Origin. Mino uses “*” allowing any Origin. I have not checked Ceph > but there is nothing explicity done in the code. Do you think using “*” is > ok? If not, I’m currently stuck for where to get the correct setting for > Origin given that there could be many management servers. Also there is no > place in the UI to manage this right now should the origins change. So maybe > “*” is the right direction here. > > Where should documentation regarding this go? I think an administrator would > like to know how for the different providers types the Accounts are mapped to > the Storage etc. > > Things I am not going to fix in my first PR (but should be fixed): > > 1. INFO level logging currently dumps S3 credentials in the log file. This > happens in a few places. Just look for “secretkey”. This should be addressed > so as not to log clear text “passwords". > > 2. The secret key is visible in the Bucket “Details” page. This should > possibly be hidden with a click to “copy” type browser action in place. > > Thanks, > > Tom. > >> On Sep 11, 2024, at 17:17, Tom O'Dowd <tpod...@cloudian.com> wrote: >> >> Sorry. answering my own question again. I found I could delete from the >> object_store table in the cloud database directly so that works for me right >> now rather than dropping everything. >> >> One thing that I have noticed about the UI is that the secret key is >> “displayed” and “remembered” by the browser when configuring a new object >> storage. This should probably behave more like a password field where you >> cannot see the value normally and the browser doesn’t remember previously >> used values. >> >> <Screenshot 2024-09-11 at 17.07.15.png> >> >> Tom. >> >>> On Sep 11, 2024, at 16:56, Tom O'Dowd <tpod...@cloudian.com> wrote: >>> >>> Hi guys, >>> >>> I’ve started working on the Object Storage plugin. One quick question which >>> I can’t find yet in the docs… >>> >>> Once I have added an object store as the admin user, how do I remove it? >>> ie, I just want to blow away the configuration for now so I can try re-add >>> it repeatedly as I fix things? Maybe the opposite of deploydb will do? >>> >>> Thanks, >>> >>> Tom. >>> >>>> On Sep 6, 2024, at 18:55, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: >>>> >>>> Hi Tom, >>>> >>>> In addition to what has been answered and what you've found, here are some >>>> useful links for reference: >>>> >>>> https://github.com/shapeblue/hackerbook/tree/main (self-learning dev >>>> material) >>>> https://github.com/shapeblue/hackerbook/blob/main/2-dev.md#contributing-to-cloudstack >>>> https://github.com/shapeblue/mbx (local CI/dev-test setup) >>>> https://github.com/apache/cloudstack/commit/c3f0d14d31583fdf2570de2633db5c87a0f08fa4 >>>> (learn from new and existing plugins as reference implementation) >>>> https://github.com/apache/cloudstack/tree/main/plugins/storage/object/minio >>>> (original minio plugin for reference) >>>> >>>> >>>> Regards. >>>> >>>> >>>> >>>> >>>> ________________________________ >>>> From: Tom O'Dowd <tpod...@cloudian.com.INVALID> >>>> Sent: Friday, September 6, 2024 14:37 >>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> >>>> Subject: Re: Object Storage Framework integration for Cloudian HyperStore >>>> >>>> Sorry for the noise. I answered my own question (or rather I found a page >>>> that did): https://cloudstack.apache.org/developers/ >>>> >>>> Thanks, >>>> >>>> Tom. >>>> >>>>> On Sep 6, 2024, at 17:06, Tom O'Dowd <tpod...@cloudian.com> wrote: >>>>> >>>>> Hi Daan, >>>>> >>>>> I have made some progress. I have got CloudStack running on Ubuntu 24.04 >>>>> now. I have installed Minio and configurated it out as the Object Storage >>>>> Provider to understand how it being used from an Admin and Regular user’s >>>>> perspective. Additionally, I have also enabled the existing Cloudian >>>>> Plugin and tried that out to ensure that it's all working also. >>>>> >>>>> There is one small item that I would like to fix in the existing plugin >>>>> relating to a path change in the default HyperStore URL that the SSO >>>>> uses. That’s just a string change. Additionally, we have moved away from >>>>> Puppet in recent HyperStore versions for configuration management and I’d >>>>> like to update the documentation specific to that point. My question is >>>>> which branch I should use as a base for my PRs for both cloudstack and >>>>> cloudstack-documentation. >>>>> >>>>> Thanks, >>>>> >>>>> Tom. >>>>> >>>>>> On Sep 3, 2024, at 17:38, Daan Hoogland <daan.hoogl...@gmail.com> wrote: >>>>>> >>>>>> Tom, >>>>>> You are welcome to submit a PR and we'll happily review it and help you >>>>>> tackle problems. The framework is there to use and adjust to your needs. >>>>>> As >>>>>> for the API, what level of stability do you require? I expect some >>>>>> changes >>>>>> on behalf of you if you are serious about implementing it. The minio is a >>>>>> PoC implementation and at least a ceph one is underway. >>>>>> >>>>>> On Tue, Sep 3, 2024 at 9:59 AM Tom O'Dowd <tpod...@cloudian.com.invalid> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We (Cloudian) have been getting some inquiries requesting a new Cloudian >>>>>>> HyperStore plugin to work with CloudStacks new Object Storage >>>>>>> Framework. We >>>>>>> currently have a different CloudStack plugin which adds integration >>>>>>> between >>>>>>> the CloudStack UI and our UI that works for both admin and users as >>>>>>> well as >>>>>>> providing secondary storage. I’ve been trying to find out more >>>>>>> information >>>>>>> regarding the new Object Storage Framework and see there is a Minio >>>>>>> plugin >>>>>>> and a simulator plugin. I’ve also just checked out the git repo and have >>>>>>> found the relevant code. Is the API stable at this point and are you >>>>>>> accepting new plugin contributions? Can I just implement the same base >>>>>>> objects that the Minio plugin is implementing in order to get our own >>>>>>> plugin working? >>>>>>> >>>>>>> I notice in the “Add Object Storage” UI, there are fields: >>>>>>> >>>>>>> Name >>>>>>> Provider >>>>>>> URL >>>>>>> Access key >>>>>>> Secret key >>>>>>> >>>>>>> I think that we would use an https URL, but some installations would >>>>>>> probably have a self-signed cert so they might need the option to skip >>>>>>> validation. Didn’t check how this was working with Minio yet. >>>>>>> Currently, we >>>>>>> do not use an access/secret key to connect to the administration API but >>>>>>> rather a username and password. Once connected, this should be able to >>>>>>> allocate users to match CloudStack accounts. I’d need to look at what >>>>>>> mappings to use there. The users should have Access/Secret keys so I >>>>>>> would >>>>>>> expect all of the S3 bucket browser stuff will just work once its all >>>>>>> connected. >>>>>>> >>>>>>> Development wise, I have followed the “Developers Guide” under >>>>>>> https://docs.cloudstack.apache.org/en/latest/developersguide/developer_guide.html#prerequisites >>>>>>> and got as far as connecting to the management server without the >>>>>>> simulator. I’m currently running on Rocky8. I’m wondering if I’m going >>>>>>> to >>>>>>> run into trouble on this OS and if I should switch to Ubuntu 24.04 >>>>>>> which it >>>>>>> seems the instructions were for? >>>>>>> >>>>>>> I guess my first goal is to seem it working with Minio and then try to >>>>>>> get >>>>>>> it working with HyperStore. >>>>>>> >>>>>>> Any advice appreciated, >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Tom. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Daan >>>>> >>>> >>> >> >
