Overprovisioning consideration in metrics API response

[Abhishek Kumar]

Hi devs,

There have been recurring issues and changes for API responses not considering 
the over-provisioning factor while reporting metrics for hosts, clusters, etc.
https://github.com/apache/cloudstack/issues/4778
https://github.com/apache/cloudstack/pull/4850
https://github.com/apache/cloudstack/pull/4499

While some of the metric parameters doesn't consider overprovisioning at all, 
some give value in the format- "memorytotalgb": "6.78 GB (x 1.0)".​
So, to address this should we consider a code/API-wide change?
And while fixing it should we introduce new parameters such as - 
cputotalwithoverprovisioning, memorytotalwithoverprovisioning, etc or should we 
apply the overprovisioning factors to the existing response parameters?
Please share your thoughts.

Regards,
Abhishek

abhishek.ku...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

[GitHub] [cloudstack-documentation] sureshanaparti commented on a change in pull request #205: VMware vSphere datastore cluster synchronisation

[GitBox]

sureshanaparti commented on a change in pull request #205:
URL: 
https://github.com/apache/cloudstack-documentation/pull/205#discussion_r604711626



##
File path: source/adminguide/storage.rst
##
@@ -121,6 +121,14 @@ OCFS2. In this case the CloudStack does not attempt to 
mount or unmount
 the storage as is done with NFS. The CloudStack requires that the
 administrator insure that the storage is available
 
+VMware vSphere supports VMFS5, VMFS6, vSAN, vVols, datastorecluster storage 
types.

Review comment:
   thanks @harikrishna-patnala 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Overprovisioning consideration in metrics API response

[Rohit Yadav]

Thanks for starting this thread Abhishek. I think all 'allocated' API response 
keys (irrespective of type such as CPU, RAM, storage/disk etc) across all 
list/metrics APIs should consider overprovisioning factor.

For example, if the total resource value/limit is 100 and overprovisioning 
factor is 1.5 that means CloudStack can effectively allocate 1.5*100=150 of 
that resource, which in actual or physical value is (allocated value / 
over-provisioning factor). Let me add user@ ML to hear if users agree with my 
interpretation of allocated values/metrics.


Regards.


From: Abhishek Kumar 
Sent: Wednesday, March 31, 2021 13:31
To: dev@cloudstack.apache.org 
Subject: Overprovisioning consideration in metrics API response

Hi devs,

There have been recurring issues and changes for API responses not considering 
the over-provisioning factor while reporting metrics for hosts, clusters, etc.
https://github.com/apache/cloudstack/issues/4778
https://github.com/apache/cloudstack/pull/4850
https://github.com/apache/cloudstack/pull/4499

While some of the metric parameters doesn't consider overprovisioning at all, 
some give value in the format- "memorytotalgb": "6.78 GB (x 1.0)".​
So, to address this should we consider a code/API-wide change?
And while fixing it should we introduce new parameters such as - 
cputotalwithoverprovisioning, memorytotalwithoverprovisioning, etc or should we 
apply the overprovisioning factors to the existing response parameters?
Please share your thoughts.

Regards,
Abhishek

abhishek.ku...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

RE: [DISCUSS] Marvin tests interaction

[Paul Angus]

I agree with Rohit, if people want to improve mavin ux that’s fine, but keep it 
a Marvin thing.  

- Marvin outputs a file that says what passed and what failed.
- A GUI is going to need constant ongoing maintenance for it to be able to 
understand tests to give meaningful additional information on tests or add any 
appreciable value.
- It's never going to be able to actually diagnose failures - so you're going 
to have to read logs anyway.

So in summary, the cons seem to heavily outweigh the pros to me.
Maybe a very simple TUI which generates the string which is ultimately run 
might help people.

A way to generate the file that describes the configuration of zone(s) that 
Marvin will be run against might be more helpful.


-Original Message-
From: Wei ZHOU  
Sent: 30 March 2021 15:51
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Marvin tests interaction

Can it be a standalone tool with ui ?

-Wei

On Tue, 30 Mar 2021 at 16:45, Nicolas Vazquez 
wrote:

> The idea is not to rewrite the tests in Java but to improve how we 
> interact with the tests and display results. The new service/agent 
> should not be responsible for testing but to invoke the execution of 
> the requested tests. This new service, lets say for example 'marvin 
> agent' will simply receive which tests are needed to run and invoke 
> them the same way as before. Then, it could use the information in the 
> logs for individual tests to retrieve back the results to the 
> management server. I imagine it could work similarly to the VR healths 
> checks functionality
>
>
> Regards,
>
> Nicolas Vazquez
>
> 
> From: Rohit Yadav 
> Sent: Tuesday, March 30, 2021 6:17 AM
> To: dev@cloudstack.apache.org 
> Subject: Re: [DISCUSS] Marvin tests interaction
>
> Sorry all, I don't agree - the thing being tested shouldn't be 
> responsible for testing (unless it's a well-defined limited set of 
> self-testing features, for example a set of sanity/intergrity checks).
>
> Unless we're saying the service/feature is used to run tests for other
> (nested?) CloudStack environments. The other issue is (and unless the 
> idea is to re-write tests in Java) most tests are written in Python, 
> so the CloudStack plugin/service would still fork and need to run 
> Python tests somehow (directly or via a runner such as nose) or 
> explore use of Jython or Python on JVM (which may make it complex).
>
>
> Regards.
>
> 
> From: Suresh Anaparti 
> Sent: Tuesday, March 30, 2021 11:23
> To: dev@cloudstack.apache.org 
> Subject: Re: [DISCUSS] Marvin tests interaction
>
> +1, added my thoughts in the ticket. This can ease/reduce the Dev & QA
> testing efforts. Also, it's good to see the testing results in the UI 
> itself.
>
> Regards,
> Suresh
>
> On 30/03/21, 9:41 AM, "David Jumani"  wrote:
>
> +1 on the idea and on Rakesh's suggestions!
> 
> From: Rakesh v  http://www.rakeshv@gmail.com http://www.rakeshv@gmail.com>>>
> Sent: Monday, March 29, 2021 11:50 PM
> To: dev@cloudstack.apache.org 
> Subject: Re: [DISCUSS] Marvin tests interaction
>
> I have added my thoughts in the issue link. Hope that's useful to you.
>
> Sent from my iPhone
>
>
> david.jum...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>
> suresh.anapa...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK 
> @shapeblue
>
>
>
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK 
> @shapeblue
>
>
>
>
> nicolas.vazq...@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK 
> @shapeblue
>
>
>
> > On Mar 29, 2021, at 4:51 AM, Nicolas Vazquez <
> nicolas.vazq...@shapeblue.com> wrote:
> >
> > Hi,
> >
> > I would like to propose an idea to improve the interaction with 
> the marvin tests through the management server. This could be useful 
> for development and test environments in which tests could be easily 
> started, configured and their results monitored through the UI.
> >
> > This could be achieved by creating a new service in charge of 
> the execution of the tests and sending results back to the management 
> server, so it can display them. A more detailed description:
> https://github.com/apache/cloudstack/issues/4799
> >
> > I would like to hear your thoughts and ideas about it. Would you 
> find this useful?
> >
> >
> > Regards,
> >
> > Nicolas Vazquez
> >
> > nicolas.vazq...@shapeblue.com
> > www.shapeblue.com
> > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>   

AW: [DISCUSS] Marvin tests interaction

[peter.muryshkin]

Hi there Nicolas,

why is actually interaction important in this context? I read that you want to 
enable developers to quickly run and check the results.
For my own scenario I had a requirement to have a set of functional tests only 
seeing ACS as a blackbox to have a "sanity check" of the service, where Marvin 
is then an overkill as it runs integration tests looking deeply into the 
database etc.
So it was indeed important to have results fast. My appoach has been to write a 
set of Python tests using the exoscale/cs library for most common API usage 
scenarios,
and then I run them in our GitLab instance using the dynamic parallel CI/CD 
jobs feature, one separate job for each test, that is the test runtime 
complexity is O(1) and not O(n).

The "UI" I can then interact with is GitLab CI/CD pipeline dashboard which 
allows to quickly identify failed tests.

Here is the link to the codebase.
https://gitlab.cc-asp.fraunhofer.de/fcs-public/apache-cs-extensions

kind regards
Peter

Von: Nicolas Vazquez 
Gesendet: Montag, 29. März 2021 04:50:44
An: dev@cloudstack.apache.org
Betreff: [DISCUSS] Marvin tests interaction

Hi,

I would like to propose an idea to improve the interaction with the marvin 
tests through the management server. This could be useful for development and 
test environments in which tests could be easily started, configured and their 
results monitored through the UI.

This could be achieved by creating a new service in charge of the execution of 
the tests and sending results back to the management server, so it can display 
them. A more detailed description: 
https://github.com/apache/cloudstack/issues/4799

I would like to hear your thoughts and ideas about it. Would you find this 
useful?


Regards,

Nicolas Vazquez

nicolas.vazq...@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue

Re: Hostbill struggles

[Hean Seng]

You need the hostname , or can choose the generated hostname

On Wed, Mar 31, 2021 at 8:51 PM  wrote:

> Hi Lucian,
>
>   I see that you're using cloudstack2,  just "cloudstack and select
> "CloudHosting" from "provisioning type, in case you want to provision
> accounts "flexible accounts" where a customer can use the resources to
> build custom VMs.  If not, you can go with Single VPS, where the resources
> are specified for each package.
>
>   Regarding hostname, you must pay attention to order page and order type
> template, for compatibility with cloudstack version module. See the img :
> https://imgur.com/daCKTYC
>
>
> Regards,
> Cristian
>
>
> -Original Message-
> From: n...@li.nux.ro 
> Sent: Tuesday, March 30, 2021 5:49 PM
> To: dev@cloudstack.apache.org
> Cc: us...@cloudstack.apache.org
> Subject: Hostbill struggles
>
> Hi,
>
> In the hope I am not the only poor soul trying to use Hostbill with
> Cloudstack, has anyone managed to get it working? Need some help.
>
> I have successfully created products linked to Cloudstack2 module, tried
> my best at configuring it, however several hours later I am yet to produce
> an actual VM launch.
>
> The first issue is on the "new server" page, as a client, I get a "Please
> select VM size" that seems to be unrelated to any field that is displayed
> (the product IS linked to a valid service offering!).
>
> Another random issue is when ordering the product it requires completing a
> "hostname", any random string seem to do it, but I do wonder what the heck
> is this fields purpose as it just seems to get in the way more than
> anything.
>
> Relevant screenshots: https://imgur.com/a/f3KsAWf
>
> Any thoughts?
>
> Regards,
> Lucian
>
>

-- 
Regards,
Hean Seng

RE: Hostbill struggles

[cristian.c]

Hi Lucian,

  I see that you're using cloudstack2,  just "cloudstack and select 
"CloudHosting" from "provisioning type, in case you want to provision accounts 
"flexible accounts" where a customer can use the resources to build custom VMs. 
 If not, you can go with Single VPS, where the resources are specified for each 
package.

  Regarding hostname, you must pay attention to order page and order type 
template, for compatibility with cloudstack version module. See the img : 
https://imgur.com/daCKTYC


Regards,
Cristian


-Original Message-
From: n...@li.nux.ro  
Sent: Tuesday, March 30, 2021 5:49 PM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Hostbill struggles

Hi,

In the hope I am not the only poor soul trying to use Hostbill with Cloudstack, 
has anyone managed to get it working? Need some help.

I have successfully created products linked to Cloudstack2 module, tried my 
best at configuring it, however several hours later I am yet to produce an 
actual VM launch.

The first issue is on the "new server" page, as a client, I get a "Please 
select VM size" that seems to be unrelated to any field that is displayed (the 
product IS linked to a valid service offering!).

Another random issue is when ordering the product it requires completing a 
"hostname", any random string seem to do it, but I do wonder what the heck is 
this fields purpose as it just seems to get in the way more than anything.

Relevant screenshots: https://imgur.com/a/f3KsAWf

Any thoughts?

Regards,
Lucian