Re: [DISCUSSION] Management of the database's "VIEW"s

[Wei ZHOU]

I totally agree with Rohit.

Anyway, it is a very nice idea. It will be very helpful when we make SQL
changes on VIEWs on both a fork and main branch.

Can we have all views in a separated file, for
example schema-create-views.sql, and run the SQL as the last step when all
other SQL changes are done in upgrade ?
This could reduce the complexity of coding , porting/backporting etc.


-Wei







On Wed, 8 Feb 2023 at 08:01, Rohit Yadav  wrote:

> That's a good idea but would create two ways of applying SQL changes
> during upgrade. Nicolas's point where removal is needed may be handled by
> just a drop statement in the views sql file.
>
> The other issue I see is lack of some kind of enforcement, validation, or
> check (it may be possible to do those via a Github Actions validation
> check, or simply changing how we declare the views/schema programmatically
> using something like jooq [1] that gives type-checking and API and can work
> with GenericDaoBase).
>
> The bigger and general issue of error-prone SQL upgrade paths remains
> unanswered. We may need to explore migrating to something like flyway [2]
> or similar. For example, when working with something like Django or
> Ruby-on-Rails, DB migration is something very fun and automatic - that sort
> of automation and developer experience in CloudStack would be great.
>
> [1]
> https://www.jooq.org/doc/latest/manual/sql-building/ddl-statements/create-statement/create-view-statement/
> [2] https://flywaydb.org/
>
>
> Regards.
>
> 
> From: Daan Hoogland 
> Sent: Tuesday, February 7, 2023 19:12
> To: dev@cloudstack.apache.org 
> Subject: Re: [DISCUSSION] Management of the database's "VIEW"s
>
> nice guys, i think we should go with it. Less error prone than our current
> MO
>
> On Tue, Feb 7, 2023 at 2:13 PM Daniel Salvador 
> wrote:
>
> > Nicolas,
> >
> > I had not thought about this case. I think your suggestion is nice; we
> can
> > use this approach.
> >
> > Best regards,
> > Daniel Salvador (gutoveronezi)
> >
> > On Tue, Feb 7, 2023 at 9:45 AM Nicolas Vazquez <
> > nicolas.vazq...@shapeblue.com> wrote:
> >
> > > Thanks Daniel, that approach looks nice to me.
> > >
> > > How would it work in case a view needs to be removed? I would think we
> > can
> > > remove the file from the views directory and add the drop view SQL on
> the
> > > schema file.
> > >
> > > Regards,
> > > Nicolas Vazquez
> > > 
> > > From: Daniel Augusto Veronezi Salvador 
> > > Sent: Monday, February 6, 2023 9:43 PM
> > > To: dev@cloudstack.apache.org 
> > > Subject: [DISCUSSION] Management of the database's "VIEW"s
> > >
> > > Hello guys,
> > >
> > > I would like to open a discussion on our current management of the
> > > database's "VIEW"s.
> > >
> > > Currently, we have to look at the changes in the previous "schema"
> files
> > > and replicate the whole "CREATE VIEW" command in the current "schema"
> > file,
> > > modifying what we need (adding/removing columns, and so on). This
> process
> > > makes the changes in a "VIEW" to be distributed through several files,
> > > increasing the number of lines modified for simple changes (e.g.: for
> > > adding a single field to the "VIEW" result we need to replicate the
> whole
> > > command); thus, making it difficult to maintain and track.
> > >
> > > With that in mind, I had some ideas of how we can improve this process.
> > > The proposal is: instead of adding the changes to the current "schema"
> > > file, we create unique files for each "VIEW" and manage them; more
> > detailed:
> > > 1. under the directory "db", where the "schema" files are, we would
> > create
> > > a sub-directory called "views";
> > > 2. in the sub-directory "views", we would create a file for each
> "VIEW",
> > > named with the "VIEW" name (for instance,
> > > "cloud.network_offering_view.sql");
> > > 3. in the "VIEW" file, we would put the "DROP VIEW" command, followed
> by
> > > the "CREATE VIEW" command, just as we do in the "schema" file; for
> > > instance, the content of file "cloud.network_offering_view.sql" would
> be:
> > >
> > > ```
> > > DROP VIEW IF EXISTS `cloud`.`network_offering_view`;
> > >
> > > CREATE VIEW `cloud`.`network_offering_view` AS
> > >  SELECT
> > >  `network_offerings`.`id` AS `id`,
> > >  `network_offerings`.`uuid` AS `uuid`,
> > >  `network_offerings`.`name` AS `name`,
> > >  
> > > ```
> > >
> > > 4. then, after each version upgrade, in Java we execute all the files
> in
> > > the sub-directory "views"; this way, if a "VIEW" changed, it would be
> > > recreated with the new changes; otherwise, it would be only recreated
> as
> > is;
> > >
> > > That would allow us to easily track "VIEW" modifications, as we would
> > just
> > > change the "VIEW" declaration in the same file, instead of re-declaring
> > the
> > > whole "VIEW" in a different file; and we would have a better history of
> > the
> > > changes. Also, we would not need to migra

Re: [DISCUSSION] Management of the database's "VIEW"s

[Daan Hoogland]

I like your ideas as well @Rohit Yadav  and @Wei
ZHOU  , but that is a step further than what Daniel
is proposing and his idea is a good step forward.
As for the schema-create-views.sql, that would be possible and maybe even
neater to split in a schema-drop-views.sql to be run before upgrades and
the create script for afterwards.

Any more beautiful implementation like jooq or flyway is good but will
require a lot of zeal and resources. For one thing we must deal with the
GenericDao<> in a backwards compatible way. There are other technical debt
issues that would take precedence when it comes down to me. (e.g. gson
upgrade and a lot of other older dependencies).

this ^ is no minus one on any idea, just caution.


On Wed, Feb 8, 2023 at 9:02 AM Wei ZHOU  wrote:

> I totally agree with Rohit.
>
> Anyway, it is a very nice idea. It will be very helpful when we make SQL
> changes on VIEWs on both a fork and main branch.
>
> Can we have all views in a separated file, for
> example schema-create-views.sql, and run the SQL as the last step when all
> other SQL changes are done in upgrade ?
> This could reduce the complexity of coding , porting/backporting etc.
>
>
> -Wei
>
>
>
>
>
>
>
> On Wed, 8 Feb 2023 at 08:01, Rohit Yadav 
> wrote:
>
> > That's a good idea but would create two ways of applying SQL changes
> > during upgrade. Nicolas's point where removal is needed may be handled by
> > just a drop statement in the views sql file.
> >
> > The other issue I see is lack of some kind of enforcement, validation, or
> > check (it may be possible to do those via a Github Actions validation
> > check, or simply changing how we declare the views/schema
> programmatically
> > using something like jooq [1] that gives type-checking and API and can
> work
> > with GenericDaoBase).
> >
> > The bigger and general issue of error-prone SQL upgrade paths remains
> > unanswered. We may need to explore migrating to something like flyway [2]
> > or similar. For example, when working with something like Django or
> > Ruby-on-Rails, DB migration is something very fun and automatic - that
> sort
> > of automation and developer experience in CloudStack would be great.
> >
> > [1]
> >
> https://www.jooq.org/doc/latest/manual/sql-building/ddl-statements/create-statement/create-view-statement/
> > [2] https://flywaydb.org/
> >
> >
> > Regards.
> >
> > 
> > From: Daan Hoogland 
> > Sent: Tuesday, February 7, 2023 19:12
> > To: dev@cloudstack.apache.org 
> > Subject: Re: [DISCUSSION] Management of the database's "VIEW"s
> >
> > nice guys, i think we should go with it. Less error prone than our
> current
> > MO
> >
> > On Tue, Feb 7, 2023 at 2:13 PM Daniel Salvador 
> > wrote:
> >
> > > Nicolas,
> > >
> > > I had not thought about this case. I think your suggestion is nice; we
> > can
> > > use this approach.
> > >
> > > Best regards,
> > > Daniel Salvador (gutoveronezi)
> > >
> > > On Tue, Feb 7, 2023 at 9:45 AM Nicolas Vazquez <
> > > nicolas.vazq...@shapeblue.com> wrote:
> > >
> > > > Thanks Daniel, that approach looks nice to me.
> > > >
> > > > How would it work in case a view needs to be removed? I would think
> we
> > > can
> > > > remove the file from the views directory and add the drop view SQL on
> > the
> > > > schema file.
> > > >
> > > > Regards,
> > > > Nicolas Vazquez
> > > > 
> > > > From: Daniel Augusto Veronezi Salvador 
> > > > Sent: Monday, February 6, 2023 9:43 PM
> > > > To: dev@cloudstack.apache.org 
> > > > Subject: [DISCUSSION] Management of the database's "VIEW"s
> > > >
> > > > Hello guys,
> > > >
> > > > I would like to open a discussion on our current management of the
> > > > database's "VIEW"s.
> > > >
> > > > Currently, we have to look at the changes in the previous "schema"
> > files
> > > > and replicate the whole "CREATE VIEW" command in the current "schema"
> > > file,
> > > > modifying what we need (adding/removing columns, and so on). This
> > process
> > > > makes the changes in a "VIEW" to be distributed through several
> files,
> > > > increasing the number of lines modified for simple changes (e.g.: for
> > > > adding a single field to the "VIEW" result we need to replicate the
> > whole
> > > > command); thus, making it difficult to maintain and track.
> > > >
> > > > With that in mind, I had some ideas of how we can improve this
> process.
> > > > The proposal is: instead of adding the changes to the current
> "schema"
> > > > file, we create unique files for each "VIEW" and manage them; more
> > > detailed:
> > > > 1. under the directory "db", where the "schema" files are, we would
> > > create
> > > > a sub-directory called "views";
> > > > 2. in the sub-directory "views", we would create a file for each
> > "VIEW",
> > > > named with the "VIEW" name (for instance,
> > > > "cloud.network_offering_view.sql");
> > > > 3. in the "VIEW" file, we would put the "DROP VIEW" command, followed
> > by
> > > > the "CREATE VIEW" command, ju

Re: [DISCUSS] CloudStack Website build and modernisation

[Rohit Yadav]

All,

The ASF infra has advised recently that they're deprecating and removing Roller 
support, our project blog is based on Roller.

Given this hasn't received any objections on this thread, I've spent some time 
in the last two months exploring options that solve most of our website and 
blog requirements. I explored several other top-level ASF project websites and 
found they had built their website/blog/article using something like Pelican, 
Hugo, Jekyll and recently Docusaurus. Among these Jekyll is the oldest and most 
stable platform that Github supports out of the box. Most of these TLP websites 
use a CI/CD based staging->publishing pipeline, using Github Actions, Gitpod, 
Netlify etc.

Among these various options, Docusaurus seems to satisfy many of the 
requirements and integrates with Netlify CMS (to support our marketing 
contributors who aren't necessarily git-experts; 
https://www.netlifycms.org/docs/docusaurus/).

If there aren't any objections I would like to set up a staging/beta website 
(as per 
https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-WebsitedeploymentserviceforGitrepositories
 this can be configured, using default it would be on 
cloudstack.staged.apach.org).

Example TLP website source repos:

Docusaurus:
https://github.com/apache/apisix-website
https://github.com/apache/shenyu-website
https://github.com/apache/incubator-kvrocks-website

Jekyll:
https://github.com/apache/openwhisk-website
https://github.com/apache/nuttx-website

Hugo:
https://github.com/apache/kyuubi-website
https://github.com/apache/hop-website
https://github.com/apache/dubbo-website


Regards.


From: Rohit Yadav 
Sent: Monday, December 19, 2022 14:57
To: dev@cloudstack.apache.org 
Subject: [DISCUSS] CloudStack Website build and modernisation

All,

For 4.17.2.0 release publication on the website, I had to manually edit the 
website [0] build as I couldn't get middleman to work on either Ubuntu 20.04, 
22.04 (after spending a good hour to fix the ruby/gems build system). Any 
pointers on getting middleman/gems to work?

As our website is dated, and in the past ASF infra has advised against setting 
up a dynamic CMS such as Wordpres for us I'm looking into alternatives that are 
ASF infra approved [1] and looking into Pelican and Jekyll. I'm also exploring 
what other ASF TLP websites are using for inspiration. I also found several 
apache projects such as apisix [2] 
having complex website build systems that use Netlify or other means of 
publishing websites.

My high-level proposal is that we;
(a) explore/investigate and migrate to a build system that is compliant with 
ASF infra policies and easy for anybody (esp PMCs and release managers) to 
setup/iterate locally,
(b) integrate PR builds using Github actions or community CI/QA server 
(https://qa.cloudstack.cloud/) and,
(c) we update the project website to use a modern library (something like Ant 
Design which we use with the ACS UI, or bootstrap, bulma, etc).

Any suggestions, comments, or advice on this?

[0] https://github.com/apache/cloudstack-www
[1] 
https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Staticwebsitecontentgeneration
[2] https://github.com/apache/apisix-website


Regards.




 

[GitHub] [cloudstack-kubernetes-provider] DaanHoogland closed pull request #38: deployment.yaml: add tolerations

[via GitHub]

DaanHoogland closed pull request #38: deployment.yaml: add tolerations
URL: https://github.com/apache/cloudstack-kubernetes-provider/pull/38


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov commented on pull request #112: Enable passing of profile information via commandline

[via GitHub]

borisstoyanov commented on PR #112:
URL: 
https://github.com/apache/cloudstack-cloudmonkey/pull/112#issuecomment-1422612881

   Hi @Pearl1594 I've tried using the parameters but could not, could you 
please tell me what I'm doing wrong? 
   
   ```
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk -h
   usage: cmk [flags] [commands|apis] [-h]
   
   CloudMonkey (cmk) 🐵 is a command line interface for Apache CloudStack.
   
   Allowed flags:
 -hShow this help message or API doc when specified after an API
 -vPrint version
 -oAPI response output format: json, text, table, column, csv
 -pServer profile
 -dEnable debug mode
 -cDifferent config file path
 -u CloudStack's API endpoint URL
 -s CloudStack user's secret Key
 -a CloudStack user's API Key
   
   Default commands:
 exit  Exits
 help  Help
 set   Configures options for cmk
 sync  Discovers and updates APIs
 version   Version info
   
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk set -s 
secretkey -a apikey
   Invalid option provided: -s
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk set 
-s=secretkey -a apikey
   Invalid option provided: -s=secretkey
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk -s secretkey -a 
apikey
   Apache CloudStack 🐵 CloudMonkey 6.3.0
   Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues
   
   (admin) 🐱 > exit
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk set -d  -s 
secretkey -a apikey
   Invalid option provided: -d
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk -v
   Apache CloudStack 🐵 CloudMonkey 6.3.0 (build: 6d45f0c, 
2023-02-08T15:27:35+0200)
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk set profile -s 
secretkey -a apikey
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % cmk
   zsh: command not found: cmk
   bstoyanov@MacBookProlocal cloudstack-cloudmonkey % ./bin/cmk
   Apache CloudStack 🐵 CloudMonkey 6.3.0
   Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues
   
   (-s secretkey -a apikey) 🐱 >
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov merged pull request #111: Support for auto-completion of storage pool related APIs

[via GitHub]

borisstoyanov merged PR #111:
URL: https://github.com/apache/cloudstack-cloudmonkey/pull/111


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov closed issue #98: Auto-complete not working for 'id' param in update storagecapabilities

[via GitHub]

borisstoyanov closed issue #98: Auto-complete not working for 'id' param in 
update storagecapabilities
URL: https://github.com/apache/cloudstack-cloudmonkey/issues/98


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov commented on pull request #84: Follow standard project layout

[via GitHub]

borisstoyanov commented on PR #84:
URL: 
https://github.com/apache/cloudstack-cloudmonkey/pull/84#issuecomment-1422622686

   @davidjumani could you please address the conflicts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov commented on pull request #52: Add new filter based on jmespath for 6.0 version

[via GitHub]

borisstoyanov commented on PR #52:
URL: 
https://github.com/apache/cloudstack-cloudmonkey/pull/52#issuecomment-1423721674

   @snifbr could you please resolve the conflicts 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov commented on issue #117: filter is ignored on e.g. deploy virtual machine

[via GitHub]

borisstoyanov commented on issue #117:
URL: 
https://github.com/apache/cloudstack-cloudmonkey/issues/117#issuecomment-1423726653

   @DaanHoogland I'm working on the milestone for 6.3.0 and I'm wondering 
should we add this improvement. Do you have any idea how much effort it is 
going to take, are you willing to work on it? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-cloudmonkey] borisstoyanov commented on issue #94: add human readable output

[via GitHub]

borisstoyanov commented on issue #94:
URL: 
https://github.com/apache/cloudstack-cloudmonkey/issues/94#issuecomment-1423735115

   @Pearl1594 are you still looking to work on this issue? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org