RE: [ACS4.2.1] CLOUDSTACK-2804

[Hugo Trippaers]

Thanks for the reminder, I've closed the issue

From: Abhinandan Prateek [mailto:abhinandan.prat...@citrix.com]
Sent: Thursday, October 31, 2013 6:27 AM
To: CloudStack Dev; Hugo Trippaers
Subject: [ACS4.2.1] CLOUDSTACK-2804

Hi Hugo,

   There are some commits in 
CLOUDSTACK-2804, will 
you know if this is fixed ?

-abhi

Re: Review Request 15108: Grab MacAddress in a more deterministic fashion

[Laszlo Hornyak]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15108/#review27875
---


I haven't had a chance to test it yet, but I prefer this solution over my own 
patchset, since I was only trying to fix some broken functionality without 
radically changing the behavior.
One minor problem I see in there is that still lots of code in static 
initializer block, it is hard to write unit tests for it. I think we can solve 
that later.

- Laszlo Hornyak


On Oct. 31, 2013, 12:18 a.m., Darren Shepherd wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15108/
> ---
> 
> (Updated Oct. 31, 2013, 12:18 a.m.)
> 
> 
> Review request for cloudstack, Alex Huang and Laszlo Hornyak.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> The MAC address is determined in the following order
> 
> 1. Look for MAC that matches interface of cluster.node.IP
> 2. Look for first interface alphabetically that has a non-linklocal address
> 3. Look for first interface alphabetically
> 4. Randomly generate a mac address
> 
> 
> Diffs
> -
> 
>   utils/src/com/cloud/utils/net/MacAddress.java 15350c8 
> 
> Diff: https://reviews.apache.org/r/15108/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Darren Shepherd
> 
>

Review Request 15115: CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

[Girish Shilamkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15115/
---

Review request for cloudstack, Santhosh Edukulla and SrikanteswaraRao Talluri.


Bugs: CLOUDSTACK-4335
https://issues.apache.org/jira/browse/CLOUDSTACK-4335


Repository: cloudstack-git


Description
---

test_deployVmSharedNetworkWithoutIpRange creates a shared network
without start and end ip. This is no longer permitted and creation
of shared network will fail with error insufficient capacity. So
remove this test which is invalid.


Diffs
-

  test/integration/component/test_shared_network_offering.py 85f0892 

Diff: https://reviews.apache.org/r/15115/diff/


Testing
---


Thanks,

Girish Shilamkar

Re: Review Request 14426: Tests for Netscaler support as external LB Provider in VPC

[Santhosh Edukulla]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14426/#review27877
---

Ship it!


Ship It!

- Santhosh Edukulla


On Oct. 28, 2013, 7:03 a.m., Sowmya Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14426/
> ---
> 
> (Updated Oct. 28, 2013, 7:03 a.m.)
> 
> 
> Review request for cloudstack, Rajesh Battala, Santhosh Edukulla, and Sowmya 
> Krishnan.
> 
> 
> Bugs: CLOUDSTACK-4776
> https://issues.apache.org/jira/browse/CLOUDSTACK-4776
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Created tests for Netscaler as external LB provider in VPC
> Used ddt to achieve this without adding new tests but modifying the existing 
> tests
> Created new network_offering_vpcNS 
> Handled addition on NS as optional in setup - if NS addition fails, the 
> non-NS tests still work and NS tests alone will be skipped
> Removed the creation of vpc Offering for each test, instead, using Default 
> offering
> test_03_create_network_netscaler is no more valid - removed it. I am adding 
> new tests for NS as external LB provider. So this is not needed.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_vpc_network.py 1d0b2d2 
> 
> Diff: https://reviews.apache.org/r/14426/diff/
> 
> 
> Testing
> ---
> 
> Tested script locally. Long running script... Latest run looks good so far. 
> 
> output so far:
> Test create network in VPC ... ok
> Test create network in VPC ... ok
> Test create network in VPC mismatched services (Should fail) ... ok
> Test create network in VPC mismatched services (Should fail) ... ok
> Test create multiple networks with LB service (Should fail) ... ok
> Test create multiple networks with LB service (Should fail) ... ok
> Test create network with external LB devices ... ok
> Test create network with redundant router capability ... SKIP: skipped - RvR 
> didn't support VPC currently
> Test create network services not supported by VPC (Should fail) ... ok
> Test create network without sourceNAT service in VPC (should fail) ... ok
> Test create network with shared network offering ... ok
> Test create network with shared network offering ... ok
> Test create network with conserve mode ON ... ok
> Test create network with conserve mode ON ... ok
> Test network gc after shutdown of vms in the network ... FAIL
> Test network rules after starting a VpcVr that was shutdown after network.gc 
> ... ok
> Test Stop all the Vms that are part of the a Network ... ok
> Test create network outside cidr range of VPC ... ok
> Test create network outside cidr range of VPC ... ok
> Test create network outside cidr range of VPC ... ok
> Test create network inside cidr range of VPC ... ok
> Test create network inside cidr range of VPC ... ok
> Test create network overlapping cidr range of VPC ...
> 
> 
> Thanks,
> 
> Sowmya Krishnan
> 
>

Re: Review Request 14426: Tests for Netscaler support as external LB Provider in VPC

[Santhosh Edukulla]

> On Oct. 16, 2013, 4:43 p.m., Santhosh Edukulla wrote:
> > Sowmya,
> > 
> > 
> > Is it ok if we do a code walkthrough? Let me know i can create a gtm. 
> > Also,please add the path to test plan for this feature along with feature 
> > owner information, i will invite him\her as well.
> > 
> > santhosh
> 
> Sowmya Krishnan wrote:
> Sure. Test plan is here: https://cwiki.apache.org/confluence/x/xIzlAQ
> I had done the testing for the feature and dev was done by Rajesh(already 
> added as reviewer for this patch)
> 
> Prasanna Santhanam wrote:
> Any meeting notes or corrections from this review meeting?
> 
> Santhosh Edukulla wrote:
> There were few review comments. We are doing the review again tomorrow.
> 
> Sowmya Krishnan wrote:
> Did few more changes to the patch:
> 1. Edited Comments/Description for many tests. It was just 
> wrong/copy-paste of first test. 
> 2. Removed commented code
> 
> Following based on review comments:
> 1. Replaced global for Netscaler configuration in setUpClass() with class 
> attribute: ns_configured. Another option to this was to add an attribute to 
> services class. But that wouldn't work with the current design of tests since 
> we re-initialize services() class attributes in setUp() which runs before 
> every test. So the value assigned in setUpClass will be lost. Hence followed 
> this approach.
> 2. Removed non-functioning if statements
> 3. Removed VPC creation from test_10_create_network_with_conserve_mode
> 4. Few naming convention changes
> 
> Apart from these, few generic discussions came up, mostly around handling 
> exceptions for API calls which we presently don't religiously follow across 
> all tests.
> Filed couple of tickets to track those issues (something to be fixed 
> across all tests): 
> CLOUDSTACK-4972, CLOUDSTACK-4974
>


1. If you are done with it , may be you can close the review.
https://reviews.apache.org/r/14426/

2. Also, change the summary of the review to reflect ddt usage. netscalar as 
external lb still requires more test cases for automation i believe.


- Santhosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14426/#review27068
---


On Oct. 28, 2013, 7:03 a.m., Sowmya Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14426/
> ---
> 
> (Updated Oct. 28, 2013, 7:03 a.m.)
> 
> 
> Review request for cloudstack, Rajesh Battala, Santhosh Edukulla, and Sowmya 
> Krishnan.
> 
> 
> Bugs: CLOUDSTACK-4776
> https://issues.apache.org/jira/browse/CLOUDSTACK-4776
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Created tests for Netscaler as external LB provider in VPC
> Used ddt to achieve this without adding new tests but modifying the existing 
> tests
> Created new network_offering_vpcNS 
> Handled addition on NS as optional in setup - if NS addition fails, the 
> non-NS tests still work and NS tests alone will be skipped
> Removed the creation of vpc Offering for each test, instead, using Default 
> offering
> test_03_create_network_netscaler is no more valid - removed it. I am adding 
> new tests for NS as external LB provider. So this is not needed.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_vpc_network.py 1d0b2d2 
> 
> Diff: https://reviews.apache.org/r/14426/diff/
> 
> 
> Testing
> ---
> 
> Tested script locally. Long running script... Latest run looks good so far. 
> 
> output so far:
> Test create network in VPC ... ok
> Test create network in VPC ... ok
> Test create network in VPC mismatched services (Should fail) ... ok
> Test create network in VPC mismatched services (Should fail) ... ok
> Test create multiple networks with LB service (Should fail) ... ok
> Test create multiple networks with LB service (Should fail) ... ok
> Test create network with external LB devices ... ok
> Test create network with redundant router capability ... SKIP: skipped - RvR 
> didn't support VPC currently
> Test create network services not supported by VPC (Should fail) ... ok
> Test create network without sourceNAT service in VPC (should fail) ... ok
> Test create network with shared network offering ... ok
> Test create network with shared network offering ... ok
> Test create network with conserve mode ON ... ok
> Test create network with conserve mode ON ... ok
> Test network gc after shutdown of vms in the network ... FAIL
> Test network rules after starting a VpcVr that was shutdown after network.gc 
> ... ok
> Test Stop all the Vms that are part of the a Network ... ok
> Test create network outside cidr range of VPC ... ok
> Test create network outside cidr range of VPC ... ok
> Test create network 

Re: haPlanners error in fresh 4.2 checkout

[sebgoa]

On Oct 29, 2013, at 7:56 PM, Nitin Mehta  wrote:

> Yeah, that should have cleaned the workspace. Did you see what Prachi
> recommended below ?

Yes and it's there. I do a fresh 4.2 checkout

it compiles fine with:

mvn -Pdeveloper -Dsimulator -DskipTests clean install

But when I run with:

mvn -pl client jetty:run

I get this:

2013-10-31 10:01:31.403:INFO::Started SelectChannelConnector@0.0.0.0:8080
[INFO] Started Jetty Server
Exception in thread "Timer-2" 
org.springframework.beans.factory.BeanExpressionException: Expression parsing 
failed; nested exception is 
org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): 
Field or property 'haPlanners' cannot be found on object of type 
'org.springframework.beans.factory.config.BeanExpressionContext'
at 
org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:142)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1299)
at 
org.springframework.beans.factory.support.BeanDefinitionValueResolver.evaluate(BeanDefinitionValueResolver.java:210)
at 
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:182)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.configureBean(AbstractAutowireCapableBeanFactory.java:314)
at 
com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:71)
at 
com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: org.springframework.expression.spel.SpelEvaluationException: 
EL1008E:(pos 0): Field or property 'haPlanners' cannot be found on object of 
type 'org.springframework.beans.factory.config.BeanExpressionContext'
at 
org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:208)
at 
org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:72)
at 
org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:52)
at 
org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at 
org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:88)
at 
org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:139)
... 10 more
ERROR [cloud.api.ApiServlet] (770998563@qtp-534473248-6:) unknown exception 
writing api response
java.lang.NullPointerException
at com.cloud.api.ApiServer.loginUser(ApiServer.java:770)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:211)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:71)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
ERROR [cloud.api.ApiServlet] (770998563@qtp-534473248-6:) unknown exception 
writ

Re: Review Request 14459: CLOUDSTACK-2243: Add automation tests for VMs base image update faclity

[Santhosh Edukulla]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14459/#review27880
---

Ship it!


Ship It!

- Santhosh Edukulla


On Oct. 31, 2013, 6:39 a.m., Ashutosh Kelkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14459/
> ---
> 
> (Updated Oct. 31, 2013, 6:39 a.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar and Santhosh Edukulla.
> 
> 
> Bugs: CLOUDSTACK-2243
> https://issues.apache.org/jira/browse/CLOUDSTACK-2243
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Added new test class to test Base Image Updation Facility.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_base_image_updation.py PRE-CREATION 
>   tools/marvin/marvin/codes.py 3da90d6 
>   tools/marvin/marvin/integration/lib/base.py 4f15137 
> 
> Diff: https://reviews.apache.org/r/14459/diff/
> 
> 
> Testing
> ---
> 
> Tested locally on KVM advanced setup:
> 
> Log:
> 
> ==> result.log <==
> test_01_deploy_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test deploy an instance with service offerings with IsVolatile set. ... ok
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test rebooting instances created with isVolatile service offerings ...
> ==> client.log <==
> 2013-10-09 20:22:30,659 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=True
> 2013-10-09 20:22:30,695 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=False
> 
> ==> result.log <==
> ok
> test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate)
> Test restoring a vm with different template than the one it was created with 
> ...
> ==> client.log <==
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Registered a template of fo
> ith ID: 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Waiting for download of tem
> : 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:38:25,367 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=True
> 2013-10-09 20:38:25,386 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=False
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Creating recurring snapshot 
> po
>  disk on vm created with IsVolatile=True
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Snapshot Policy - Type : 
> HOURL
> inute : 53
> 2013-10-09 20:38:30,982 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Sleeping for 25 minutes till 
> t
> snapshoted
> 
> ==> result.log <==
> ok
> test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) ...
> ==> client.log <==
> 2013-10-09 21:11:51,780 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether root disk of
> atile=True was destroyed
> 2013-10-09 21:11:51,827 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether snapshot 
> rule
> isVolatile=True was destroyed
> 
> ==> result.log <==
> ok
> 
> --
> Ran 4 tests in 3534.241s
> 
> OK
> 
> 
> Thanks,
> 
> Ashutosh Kelkar
> 
>

Re: Review Request 14459: CLOUDSTACK-2243: Add automation tests for VMs base image update faclity

[ASF Subversion and Git Services]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14459/#review27881
---


Commit c53d4e9e141d5048a4ed2ea040fbef545c7ae9d1 in branch refs/heads/master 
from Ashutosh Kelkar
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=c53d4e9 ]

CLOUDSTACK-2243: Add automation tests for VMs base image
 update faclity

Signed-off-by: Girish Shilamkar 


- ASF Subversion and Git Services


On Oct. 31, 2013, 6:39 a.m., Ashutosh Kelkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14459/
> ---
> 
> (Updated Oct. 31, 2013, 6:39 a.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar and Santhosh Edukulla.
> 
> 
> Bugs: CLOUDSTACK-2243
> https://issues.apache.org/jira/browse/CLOUDSTACK-2243
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Added new test class to test Base Image Updation Facility.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_base_image_updation.py PRE-CREATION 
>   tools/marvin/marvin/codes.py 3da90d6 
>   tools/marvin/marvin/integration/lib/base.py 4f15137 
> 
> Diff: https://reviews.apache.org/r/14459/diff/
> 
> 
> Testing
> ---
> 
> Tested locally on KVM advanced setup:
> 
> Log:
> 
> ==> result.log <==
> test_01_deploy_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test deploy an instance with service offerings with IsVolatile set. ... ok
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test rebooting instances created with isVolatile service offerings ...
> ==> client.log <==
> 2013-10-09 20:22:30,659 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=True
> 2013-10-09 20:22:30,695 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=False
> 
> ==> result.log <==
> ok
> test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate)
> Test restoring a vm with different template than the one it was created with 
> ...
> ==> client.log <==
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Registered a template of fo
> ith ID: 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Waiting for download of tem
> : 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:38:25,367 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=True
> 2013-10-09 20:38:25,386 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=False
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Creating recurring snapshot 
> po
>  disk on vm created with IsVolatile=True
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Snapshot Policy - Type : 
> HOURL
> inute : 53
> 2013-10-09 20:38:30,982 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Sleeping for 25 minutes till 
> t
> snapshoted
> 
> ==> result.log <==
> ok
> test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) ...
> ==> client.log <==
> 2013-10-09 21:11:51,780 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether root disk of
> atile=True was destroyed
> 2013-10-09 21:11:51,827 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether snapshot 
> rule
> isVolatile=True was destroyed
> 
> ==> result.log <==
> ok
> 
> --
> Ran 4 tests in 3534.241s
> 
> OK
> 
> 
> Thanks,
> 
> Ashutosh Kelkar
> 
>

Re: Review Request 14459: CLOUDSTACK-2243: Add automation tests for VMs base image update faclity

[ASF Subversion and Git Services]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14459/#review27882
---


Commit e47dfb673cc71a0014fc320a0a14cb8942145e8f in branch refs/heads/4.2 from 
Ashutosh Kelkar
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=e47dfb6 ]

CLOUDSTACK-2243: Add automation tests for VMs base image update faclity

Signed-off-by: Girish Shilamkar 


- ASF Subversion and Git Services


On Oct. 31, 2013, 6:39 a.m., Ashutosh Kelkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14459/
> ---
> 
> (Updated Oct. 31, 2013, 6:39 a.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar and Santhosh Edukulla.
> 
> 
> Bugs: CLOUDSTACK-2243
> https://issues.apache.org/jira/browse/CLOUDSTACK-2243
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Added new test class to test Base Image Updation Facility.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_base_image_updation.py PRE-CREATION 
>   tools/marvin/marvin/codes.py 3da90d6 
>   tools/marvin/marvin/integration/lib/base.py 4f15137 
> 
> Diff: https://reviews.apache.org/r/14459/diff/
> 
> 
> Testing
> ---
> 
> Tested locally on KVM advanced setup:
> 
> Log:
> 
> ==> result.log <==
> test_01_deploy_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test deploy an instance with service offerings with IsVolatile set. ... ok
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate)
> Test rebooting instances created with isVolatile service offerings ...
> ==> client.log <==
> 2013-10-09 20:22:30,659 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=True
> 2013-10-09 20:22:30,695 - DEBUG - 
> test_02_reboot_instance_with_is_volatile_offering 
> (test_base_image_updation.TestBaseImageUpdate) - Checking root
> th isVolatile=False
> 
> ==> result.log <==
> ok
> test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate)
> Test restoring a vm with different template than the one it was created with 
> ...
> ==> client.log <==
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Registered a template of fo
> ith ID: 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:22:32,373 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Waiting for download of tem
> : 5a9190a9-8a59-487f-b90f-4a76a0f509a0
> 2013-10-09 20:38:25,367 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=True
> 2013-10-09 20:38:25,386 - DEBUG - test_03_restore_vm_with_new_template 
> (test_base_image_updation.TestBaseImageUpdate) - Checking template id of VM
> le=False
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Creating recurring snapshot 
> po
>  disk on vm created with IsVolatile=True
> 2013-10-09 20:38:30,541 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Snapshot Policy - Type : 
> HOURL
> inute : 53
> 2013-10-09 20:38:30,982 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Sleeping for 25 minutes till 
> t
> snapshoted
> 
> ==> result.log <==
> ok
> test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) ...
> ==> client.log <==
> 2013-10-09 21:11:51,780 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether root disk of
> atile=True was destroyed
> 2013-10-09 21:11:51,827 - DEBUG - test_04_reoccuring_snapshot_rules 
> (test_base_image_updation.TestBaseImageUpdate) - Checking whether snapshot 
> rule
> isVolatile=True was destroyed
> 
> ==> result.log <==
> ok
> 
> --
> Ran 4 tests in 3534.241s
> 
> OK
> 
> 
> Thanks,
> 
> Ashutosh Kelkar
> 
>

Re: [MERGE] network-guru-orchestration into master

[Hugo Trippaers]

On 31 okt. 2013, at 04:34, Marcus Sorensen  wrote:

> Here's a scenario... Lets say we currently have a zone deployed with Vlan
> isolation. There's nothing stopping us from running a vxlan isolation on
> the same hosts, even on the same physical interface. This is actually done
> in devcloud-kvm where public and mgmt traffic are selected by traffic label
> (which are tagged vlans), and guest traffic is handled by vxlan.
> 
> If one wanted to deprecate Vlan in favor of vxlan without a big zone
> shufflr it would be simple if isolation method were a property of network
> offering. I imagine other isolation offerings could also be deployed in
> tandem, even if it required multiple physical interfaces in the host.

Currently the system we have in place is indeed capable of running different 
isolation types on a host, it uses the physical network concept in cloudstack 
to determine which isolation type to use. This is mainly because the physical 
network also defines which tag or network interface label to use on the 
hypervisors.

I understand the case that it might be possible to combine different isolation 
types on physical networks, like the case you describe with vlan and vxlan. 
However in other cases it should not happen, like when nicira nvp is used which 
takes exclusive control of a traffic label or bridge on the hypervisor. As far 
as i understood the contrail implementation it actually takes control of the 
entire networking of a hypervisor, so its exclusive with any other isolation 
provider.

Within the database and (most of) the code it is already possible to set 
multiple isolation types on a physical network. That gives the admin the 
ability to link a certain physical interface/traffic label or bridge to a set 
of isolation types. For example cloudbr0 with NONE, VLAN and VXLAN and cloudbr1 
with STT. The offering system can be used to indicate which isolation provider 
to use for which particular virtual network.

Cheers,

Hugo


> On Wed, Oct 30, 2013 at 11:36 AM, Pedro Roque Marques
>  wrote:
>> 
>> Why ? That seems to me the most obvious way to do it.
>> There are different networking solutions: e.g. VLANs and overlays such as
> OpenContrail that assume an L3 switching topology.  For a deployment one
> would tend to choose a solution associated with the physical network.
> 
> Sure that would be simple.  The problem is that Zone also implies
> other things.  Secondary/Primary storage for example.  Zone puts other
> limitations on things.  So by tying networking to a zone you can get
> into the situation where somebody has a CloudStack installation, and
> they want to add OpenContrail.  If we tie it to the zone, that may
> mean that in order for them to use OpenContrail they may need to get
> another NFS secondary storage, primary storage, compute nodes, etc and
> then copy all the templates from one zone to another, etc, etc.
> 
> So currently you can't have Basic and Advanced networking in the same
> zone.  I think it should be possible.  Imagine I had a smaller
> dev/test cloud.  Today if I want basic and advance networking (because
> those are two distinict work loads), I have to have servers dedicated
> to each type.  So 10 for basic, 10 or advanced.  Now the networking
> under the hood for basic and advance can mix quite easily but just
> because we decided to implement it as it is today, I'm suddenly forced
> to managed two pools of physical resources, when it would have been
> must easier and more cost effective to have one pool.
> 
> I think this is general thing that should change with CloudStack over
> time.  Zone and Pod should not be so closely tied to network concepts.
> CloudStack should be capable of being looser for people who want more
> hybrid work loads.  We too easily say that a service provider is just
> going to choose one or the other so why support mixed configurations.
> But there are more people that could be using CloudStack but are
> turned off by the strict models and concepts that CloudStack enforces.
> I have to say I was one of those people.  CloudStack forced me to
> manage my infrastructure in a certain way that I didn't particularly
> care for.
> 
> Darren

Re: [MERGE] network-guru-orchestration into master

[Hugo Trippaers]

On 31 okt. 2013, at 03:48, Darren Shepherd  wrote:

> Its important to remember that the definition of a Network in
> CloudStack is quite purposefully very loose.  A network in the
> simplest terms is just a group of VMs.  A guru creates and manages
> that network.  A network is not strictly defined as a union of
> traffictype/isolationtype/networktype/guesttype.  So, for example,
> there need not be a 1-to-1 mapping between Guru and IsolationType.
> You should be able to have multiple gurus per any context.  That's the
> theory and vision.

I think that vision was put down somewhere before we entered the era of 
software defined networking. At the moment it is simply not true as long as the 
guru is responsible for creating the overlay networks there is a 1 on 1 
relation with the guru and the supported isolation types in several cases.

> 
> Now the reality is that the current selection logic of the guru in the
> orchestrator doesn't fully back up that vision because the selection
> logic too easily allows two Guru's to implement the same network.
> 
> So I think the current behavior of setupNetwork is flawed.  This
> change, I feel, muddies to a certain degree the definition of a
> network in CloudStack and attempts to say a network is strict union of
> traffic/isolation/network/guest-type.  If we merge this change, does
> it hurt anything?  No, its more just conceptual disagreement.  When I
> get to the point that I feel setupNetwork is limiting what I want to
> do, I'll propose a change.  Right now, I just accept that its not
> right in my mind.

I’ve fixed the definition of setupNetwork so the function returns a single 
network when called. This was already the effective behavior, but now its put 
down in code as well. I’ll push that change to the branch today.


> 
> So I'd have to say I'm neutral on this change.  I don't care too much
> either way as I think down the line this all needs to change.

Let’s find a moment to sit down at CCCEU and discuss this further. I have some 
ideas of where i want to go with the networking bit as well and it would be 
nice if we are more or less on the same page before we start working on it. 
Saying that you don’t care because you think its going to change it anyway 
isn’t really the best way to approach this.

> 
> Darren
> 
> On Wed, Oct 30, 2013 at 11:36 AM, Pedro Roque Marques
>  wrote:
>> 
>> On Oct 30, 2013, at 11:08 AM, Darren Shepherd  
>> wrote:
>> 
>>> I definitely don't want the setup to be done at a zone level.
>> 
>> Why ? That seems to me the most obvious way to do it.
>> There are different networking solutions: e.g. VLANs and overlays such as 
>> OpenContrail that assume an L3 switching topology.  For a deployment one 
>> would tend to choose a solution associated with the physical network.
>> 
>>> Specifically, I'd like to get rid of the notion of basic and advanced
>>> zones which would mean that you could be mixing gurus in the same
>>> zone.  If anything, I'd say the simplest would be to just specify the
>>> Guru in the network offering.
>> 
>> Basic and advanced zones is rather unhelpful, i agree. But you still can't 
>> Mix Guru's unless they can interoperate... The way for them to interop (i.e. 
>> minimum common denominator) is to connect at L3 in an router between zones.
>> 
>>> 
>>> I like the work that Hugo has done, but the implications of that
>>> change are 2 things.  First, we only want one guru to match.  If that
>>> is the case, then lets change the NetworkOrchestrator.setupNetwork()
>>> to just return one network.  I have a hard time understanding why you
>>> would want to return two or more.  Even if the guru created two
>>> networks internally, only one should be return to the caller.  There's
>>> not a good way to invoke setupNetwork() and handle cases where more
>>> than one network comes back.
>>> 
>>> Second, we are saying the only way to choose a guru is based on
>>> networkType/trafficType/guestType/isolationmethods.  I think the core
>>> logic should be more flexible than that.  We can first check those 4
>>> items, and then if we find more than one guru that matches, use the
>>> best one (before what I said about adding canHandle to the interface).
>>> But the check shouldn't be only those four criteria.  Hugo's change
>>> does not strictly enforce that, but it is essentially implied.
>>> 
>>> After saying all that, if we just put the Guru in the network
>>> offering.  Would that work for everything?  That would be a really
>>> simple and explicit way to configure this all.
>> 
>> My preference would be for a NetworkManager per zone.
>> 
>>> 
>>> Darren
>>> 
>>> 
>>> On Wed, Oct 30, 2013 at 10:31 AM, Alex Huang  wrote:
 I agree with Hugo that the current way of ask NetworkGuru to decide itself 
 is not desirable.  It makes absolute sense to change that but I wonder why 
 do it through capabilities.  Why not just ask the system admin to choose 
 when they setup the zone?   When I originally designed Netw

RE: [Merge] hyperv branch to master

[Devdeep Singh]

Thanks Darren. I'll start working on getting it merged into the master. I hope 
to finish it by Friday.

Regards,
Devdeep

-Original Message-
From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] 
Sent: Tuesday, October 29, 2013 10:45 PM
To: dev@cloudstack.apache.org
Cc: Donal Lafferty; Rajesh Battala; Anshul Gangwar
Subject: Re: [Merge] hyperv branch to master

+1

When you merge you will get a conflict in the spring configuration.
componentContext.xml and others are gone now in master.  You will need to 
update to the new style.  The easiest will be to refer to XenServer spring 
config for an example of how to do it.  You can largely copy and paste the 
contents of plugins/hypervisors/xen/resources/META-INF/cloudstack and replace 
xenserver with hyperv.  You can also refer to 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions
for more in depth explaination of the changes.

Darren

On Tue, Oct 29, 2013 at 9:35 AM, David Nalley  wrote:
> On Mon, Oct 28, 2013 at 7:41 AM, Devdeep Singh  
> wrote:
>> Hi,
>>
>> I would like to merge the support for Hyperv to the master branch. 
>> Development for this has been done by Donal, Rajesh, Anshul and I on branch 
>> [1]. The feature was proposed for merge earlier [3] but unit tests for 
>> hyperv agent code were requested [4].
>>
>
> I know this has been discussed several times, but what was the 
> decision about making the agent code live in its own repo?
> It's C#, might need to be signed to actually run easily on Windows, etc.
>
> --David

Re: Review Request 15018: CLOUDSTACK-4908: CPU socket count of host

[Harikrishna Patnala]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15018/
---

(Updated Oct. 31, 2013, 10:14 a.m.)


Review request for cloudstack and Kishan Kavala.


Changes
---

updated patch after resolving conflicts with latest master.


Bugs: CLOUDSTACK-4908
https://issues.apache.org/jira/browse/CLOUDSTACK-4908


Repository: cloudstack-git


Description
---

CLOUDSTACK-4908: CPU socket count of host 
Adding cpu socket count of the host in hostresponse


Diffs (updated)
-

  api/src/com/cloud/host/Host.java 17b0ba8 
  api/src/org/apache/cloudstack/api/command/admin/host/ListHostsCmd.java 
95b9ac3 
  api/src/org/apache/cloudstack/api/response/HostResponse.java 687687d 
  core/src/com/cloud/agent/api/StartupRoutingCommand.java d52666b 
  core/test/org/apache/cloudstack/api/agent/test/CheckOnHostCommandTest.java 
1b68276 
  
engine/orchestration/src/org/apache/cloudstack/engine/datacenter/entity/api/db/EngineHostVO.java
 249c2db 
  engine/schema/src/com/cloud/host/HostVO.java c814b94 
  
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
 8d3a0e9 
  
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
 423abe6 
  
plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/CitrixResourceBase.java
 3323a15 
  server/src/com/cloud/api/query/QueryManagerImpl.java 51ee249 
  server/src/com/cloud/api/query/dao/HostJoinDaoImpl.java 97f8bf9 
  server/src/com/cloud/api/query/vo/HostJoinVO.java fbc59cd 
  server/src/com/cloud/resource/ResourceManagerImpl.java e9d5193 
  setup/db/db/schema-421to430.sql 803f1d9 
  vmware-base/src/com/cloud/hypervisor/vmware/mo/HostMO.java 108a3ec 
  vmware-base/src/com/cloud/hypervisor/vmware/mo/VmwareHypervisorHost.java 
f7288f1 
  
vmware-base/src/com/cloud/hypervisor/vmware/mo/VmwareHypervisorHostResourceSummary.java
 fa11dc3 

Diff: https://reviews.apache.org/r/15018/diff/


Testing
---


Thanks,

Harikrishna Patnala

Re: Review Request 15021: Fixed Bug: 4899 : Added Configuration Support to Marvin.

[Santhosh Edukulla]

> On Oct. 30, 2013, 10:22 a.m., Santhosh Edukulla wrote:
> > tools/marvin/marvin/configGenerator.py, line 337
> > 
> >
> > Nopes, its right, user by default no need to mention any path while 
> > creating the object instead provides it under getConfig. User by default 
> > dont need to worry about the configuration provided by default, we are 
> > providing him the parsed dictionary by default by placing the configuration 
> > at a default location. Only if he wanted to change this configuration to a 
> > separate one he can still change by using getConfig method.
> 
> Prasanna Santhanam wrote:
> so self.__init__(self, filepath='defaultpath'): should work. The reason 
> for adding the filepath argument is simply to aid call to the command line 
> switch from nose. By losing the command line switch from nose to specify the 
> marvin-config path, do you not feel we are restricting?
> 
> also, i guess i'm having a hard time seeing how the said user will use 
> this getConfig utility. can you perhaps re-use the ldap test to show how the 
> hardcoded external resource is simplified using ConfigManager? that would 
> make it easier to understand and this patch can be absorbed with that test.
>

Added a simple usage of how this facility can be used as an example. 
'''
1. cloudstackTestClient will now have configuration object exposed
and available through its method getConfigParser()
2. The object will now have all options exposed through ConfigManager
3. cls.configObj will now provide ConfigManager interface
   User can use this to parse and get the dictionary represenation
   of his configuaration  
'''
@classmethod
def setUpClass(cls):
cls.api_client = super(
   TestAddMultipleNetScaler,
   cls
   ).getClsTestClient().getApiClient()
cls.configObj =  super(
   TestAddMultipleNetScaler,
   cls
   ).getConfigParser()
if cls.configObj is not None:
'''
Here, cls.configDict will now have all the configuration parsed and 
available as Dictionary
This Dictionary is available to total test feature code accessible 
wherever applicable
'''
cls.configDict = cls.configObj.getConfig(self, 
"/setup/config/myfeature.cfg", None )


- Santhosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15021/#review27765
---


On Oct. 30, 2013, 10:12 a.m., Santhosh Edukulla wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15021/
> ---
> 
> (Updated Oct. 30, 2013, 10:12 a.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
>  Added Configuration Support to Marvin.
> 
> 1. It provides the basic configuration facilities to marvin.
> 
> 2. User can just add configuration files for his tests, deployment
>   etc, under one config folder before running their tests.
>   cs/tools/marvin/marvin/config.
>   They can remove all hard coded values from code and separate
>   it out as config at this location.
>   Either add this to the existing setup.cfg as separate 
> section
>   or add new configuration.
> 3. This will thus removes hard coded tests and separate
>   data from tests.
> 
> 4. This API is provided as an additional facility under
>   cloudstackTestClient and users can get the
>   configuration object as similar to apiclient,dbconnection
>   etc to drive their test.
> 
> 5. They just add their configuration for a test,
>   setup etc,at one single place under configuration dir
>   and use "getConfigParser" API of cloudstackTestClient
>   It will give them "configObj".They can either pass their own
>   config file for parsing to "getConfig" or it will use
>   default config file @ config/setup.cfg.
> 6. They will then get the dictionary of parsed
>   configuration and can use it further to drive their tests or
>   config drive
> 7. Test features, can  drive their setups thus removing hard coded
>   values. Configuration default file will be under config and as
>   setup.cfg.
> 8. Users can use their own configuration file passed to
>   "getConfig" API,once configObj is returned.
> 

Re: Review Request 15021: Fixed Bug: 4899 : Added Configuration Support to Marvin.

[Santhosh Edukulla]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15021/
---

(Updated Oct. 31, 2013, 11:57 a.m.)


Review request for cloudstack, Girish Shilamkar and Prasanna Santhanam.


Changes
---

1. Removed few unwanted imports
2. Added default config dir path
3. Added config directory to setup.py 
4. Added config file path to be pickedup dynamically.


Repository: cloudstack-git


Description
---

 Added Configuration Support to Marvin.

1. It provides the basic configuration facilities to marvin.

2. User can just add configuration files for his tests, deployment
  etc, under one config folder before running their tests.
  cs/tools/marvin/marvin/config.
  They can remove all hard coded values from code and separate
  it out as config at this location.
  Either add this to the existing setup.cfg as separate section
  or add new configuration.
3. This will thus removes hard coded tests and separate
  data from tests.

4. This API is provided as an additional facility under
  cloudstackTestClient and users can get the
  configuration object as similar to apiclient,dbconnection
  etc to drive their test.

5. They just add their configuration for a test,
  setup etc,at one single place under configuration dir
  and use "getConfigParser" API of cloudstackTestClient
  It will give them "configObj".They can either pass their own
  config file for parsing to "getConfig" or it will use
  default config file @ config/setup.cfg.
6. They will then get the dictionary of parsed
  configuration and can use it further to drive their tests or
  config drive
7. Test features, can  drive their setups thus removing hard coded
  values. Configuration default file will be under config and as
  setup.cfg.
8. Users can use their own configuration file passed to
  "getConfig" API,once configObj is returned.

Another such case where we are using sed or bash script is  in between a build 
run for replacing hard coded ldap ip for region\setup specific. We can now 
change all parameters before run under configuration, the test features will 
use configuration object and thus values, rather hard coded strings which 
avoids replacement through shell script.

 Also, did few naming convention changes. Its better to follow uniform naming 
conventions. Currently, wherever iam seeing a specific notation not followed. 
We are incorporating those changes.

ToDo:
clean up of current config at places, making configuration required for 
marvin\tests unified and available at one place and clean up of files\code 
related to it. 


Diffs
-

  tools/marvin/marvin/cloudstackTestClient.py be93f35 
  tools/marvin/marvin/config/setup.cfg PRE-CREATION 
  tools/marvin/marvin/configGenerator.py 0cfad30 
  tools/marvin/marvin/integration/lib/utils.py 7d662af 

Diff: https://reviews.apache.org/r/15021/diff/


Testing
---

Basic check to retrieve configuration values is done.


File Attachments (updated)


Added changes as discussed
  
https://reviews.apache.org/media/uploaded/files/2013/10/31/0001-Fixed-Bug-4899_2.patch


Thanks,

Santhosh Edukulla

Re: ui styling

[Frankie Onuonga]

Hi guys,

I trust all is well.

I was actually referring to getting to assist you guys with the UI work you
have.
I was hoping to offer a helping hand in this.
I will still make a branch of my own.

Kind Regards,


Thanks for the response Brain.


On Wed, Oct 30, 2013 at 8:07 PM, Brian Federle wrote:

> Hi Frankie, unfortunately that would be hard to do, since they were just
> incremental changes applied on master. I would have to cherry pick all the
> commits I did over time, which would take a while :). What you can try is
> copying the updated CSS files (/css), images (/images) and all scripts
> under ui/widgets to your local build (in the same respective folders). Note
> that this would contain some work-in-progress UI functionality for 4.3 that
> may or may not be working.
>
> -Brian
>
> -Original Message-
> From: Frankie Onuonga [mailto:frankie.onuo...@gmail.com]
> Sent: Tuesday, October 29, 2013 10:11 PM
> To: dev@cloudstack.apache.org
> Subject: Re: ui styling
>
> I also wish this was the case.
> I am still trying to get in touch with the team to assist them with this.
> Hope someone gets back to me.
>
> I think I can commit a week to get this done.
>
> thanks
>
>
> On Wed, Oct 30, 2013 at 4:35 AM, Marcus Sorensen  >wrote:
>
> > I'm digging the UI styling changes!  It makes me wish they were all
> > merged in from a branch as a squashed commit so I could pull them out
> > and apply them to our local 4.2 build.
> >
>
>
>
> --
> Skype: Frankie Onuonga
> twitter: Frankie.onuonga
> irc #freenode: Frankie.onuonga
>



-- 
Skype: Frankie.Onuonga
twitter: Frankieonuonga
irc #freenode: Frankie.onuonga

Re: Review Request 15021: Fixed Bug: 4899 : Added Configuration Support to Marvin.

[Prasanna Santhanam]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15021/#review27887
---

Ship it!


0b617a1

- Prasanna Santhanam


On Oct. 31, 2013, 11:57 a.m., Santhosh Edukulla wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15021/
> ---
> 
> (Updated Oct. 31, 2013, 11:57 a.m.)
> 
> 
> Review request for cloudstack, Girish Shilamkar and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
>  Added Configuration Support to Marvin.
> 
> 1. It provides the basic configuration facilities to marvin.
> 
> 2. User can just add configuration files for his tests, deployment
>   etc, under one config folder before running their tests.
>   cs/tools/marvin/marvin/config.
>   They can remove all hard coded values from code and separate
>   it out as config at this location.
>   Either add this to the existing setup.cfg as separate 
> section
>   or add new configuration.
> 3. This will thus removes hard coded tests and separate
>   data from tests.
> 
> 4. This API is provided as an additional facility under
>   cloudstackTestClient and users can get the
>   configuration object as similar to apiclient,dbconnection
>   etc to drive their test.
> 
> 5. They just add their configuration for a test,
>   setup etc,at one single place under configuration dir
>   and use "getConfigParser" API of cloudstackTestClient
>   It will give them "configObj".They can either pass their own
>   config file for parsing to "getConfig" or it will use
>   default config file @ config/setup.cfg.
> 6. They will then get the dictionary of parsed
>   configuration and can use it further to drive their tests or
>   config drive
> 7. Test features, can  drive their setups thus removing hard coded
>   values. Configuration default file will be under config and as
>   setup.cfg.
> 8. Users can use their own configuration file passed to
>   "getConfig" API,once configObj is returned.
> 
> Another such case where we are using sed or bash script is  in between a 
> build run for replacing hard coded ldap ip for region\setup specific. We can 
> now change all parameters before run under configuration, the test features 
> will use configuration object and thus values, rather hard coded strings 
> which avoids replacement through shell script.
> 
>  Also, did few naming convention changes. Its better to follow uniform naming 
> conventions. Currently, wherever iam seeing a specific notation not followed. 
> We are incorporating those changes.
> 
> ToDo:
> clean up of current config at places, making configuration required for 
> marvin\tests unified and available at one place and clean up of files\code 
> related to it. 
> 
> 
> Diffs
> -
> 
>   tools/marvin/marvin/cloudstackTestClient.py be93f35 
>   tools/marvin/marvin/config/setup.cfg PRE-CREATION 
>   tools/marvin/marvin/configGenerator.py 0cfad30 
>   tools/marvin/marvin/integration/lib/utils.py 7d662af 
> 
> Diff: https://reviews.apache.org/r/15021/diff/
> 
> 
> Testing
> ---
> 
> Basic check to retrieve configuration values is done.
> 
> 
> File Attachments
> 
> 
> Added changes as discussed
>   
> https://reviews.apache.org/media/uploaded/files/2013/10/31/0001-Fixed-Bug-4899_2.patch
> 
> 
> Thanks,
> 
> Santhosh Edukulla
> 
>

Re: Review Request 15115: CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

[Santhosh Edukulla]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15115/#review27888
---

Ship it!


Ship It!

- Santhosh Edukulla


On Oct. 31, 2013, 8:32 a.m., Girish Shilamkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15115/
> ---
> 
> (Updated Oct. 31, 2013, 8:32 a.m.)
> 
> 
> Review request for cloudstack, Santhosh Edukulla and SrikanteswaraRao Talluri.
> 
> 
> Bugs: CLOUDSTACK-4335
> https://issues.apache.org/jira/browse/CLOUDSTACK-4335
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> test_deployVmSharedNetworkWithoutIpRange creates a shared network
> without start and end ip. This is no longer permitted and creation
> of shared network will fail with error insufficient capacity. So
> remove this test which is invalid.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_shared_network_offering.py 85f0892 
> 
> Diff: https://reviews.apache.org/r/15115/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Girish Shilamkar
> 
>

Review Request 15127: CLOUDSTACK-4495: Fixed TestResetSSHKeyUserRights.test_01_reset_keypair_normal_user

[Gaurav Aradhye]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15127/
---

Review request for cloudstack and Girish Shilamkar.


Repository: cloudstack-git


Description
---

Fixed test script issue which was checking the VM state by listing the VMs in a 
particular account and checking the state of first VM in the list.
It was assuming that there is only one VM in the account, in some cases earlier 
deleted VM was found in expunged/destroying state in the list.

Now the VMs are listed by VM id and state is checked.

Also did some code cleanup.


Diffs
-

  test/integration/component/test_reset_ssh_keypair.py 4d0c45e 
  tools/marvin/marvin/codes.py 39ef950 

Diff: https://reviews.apache.org/r/15127/diff/


Testing
---

Tested locally on KVM setup:

test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights)
Verify API resetSSHKeyForVirtualMachine for non admin non root ...
==> client.log <==CS_TEST]# tail -f client.log result.log
2013-10-31 01:27:19,814 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Check if the VM is 
properly deployed or
 not?
2013-10-31 01:27:19,834 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Stopping the virtual 
machine
2013-10-31 01:29:25,329 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Creating a new SSH keypair 
for account:
 test-5XE7P6
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Created a new keypair with 
name: W44C6E
.pem
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Writing the private key to 
local file
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - File path: /tmp/W44C6E.pem
2013-10-31 01:29:25,410 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Resetting the SSH key pair 
for instance
: 452be90f-d0e9-4b28-afcd-251eea1c23c3
2013-10-31 01:29:55,601 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Starting the virtual 
machine after rese
tting the keypair
2013-10-31 01:30:05,725 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - SSHing with new keypair

==> result.log <==
ok


Thanks,

Gaurav Aradhye

Re: Review Request 15115: CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

[ASF Subversion and Git Services]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15115/#review27891
---


Commit 58179d28840a37e6b098d731d4fe6e6e5ea8ecf9 in branch refs/heads/master 
from Girish Shilamkar
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=58179d2 ]

CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

test_deployVmSharedNetworkWithoutIpRange creates a shared network
without start and end ip. This is no longer permitted and creation
of shared network will fail with error "insufficient capacity". So
remove this test which is invalid.

Signed-off-by: Girish Shilamkar 


- ASF Subversion and Git Services


On Oct. 31, 2013, 8:32 a.m., Girish Shilamkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15115/
> ---
> 
> (Updated Oct. 31, 2013, 8:32 a.m.)
> 
> 
> Review request for cloudstack, Santhosh Edukulla and SrikanteswaraRao Talluri.
> 
> 
> Bugs: CLOUDSTACK-4335
> https://issues.apache.org/jira/browse/CLOUDSTACK-4335
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> test_deployVmSharedNetworkWithoutIpRange creates a shared network
> without start and end ip. This is no longer permitted and creation
> of shared network will fail with error insufficient capacity. So
> remove this test which is invalid.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_shared_network_offering.py 85f0892 
> 
> Diff: https://reviews.apache.org/r/15115/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Girish Shilamkar
> 
>

Re: Review Request 15115: CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

[ASF Subversion and Git Services]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15115/#review27893
---


Commit 733c355eaa364e5402876b1347b206454b3bec66 in branch refs/heads/4.2 from 
Girish Shilamkar
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=733c355 ]

CLOUDSTACK-4335: Delete test_deployVmSharedNetworkWithoutIpRange

test_deployVmSharedNetworkWithoutIpRange creates a shared network
without start and end ip. This is no longer permitted and creation
of shared network will fail with error "insufficient capacity". So
remove this test which is invalid.

Signed-off-by: Girish Shilamkar 


- ASF Subversion and Git Services


On Oct. 31, 2013, 8:32 a.m., Girish Shilamkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15115/
> ---
> 
> (Updated Oct. 31, 2013, 8:32 a.m.)
> 
> 
> Review request for cloudstack, Santhosh Edukulla and SrikanteswaraRao Talluri.
> 
> 
> Bugs: CLOUDSTACK-4335
> https://issues.apache.org/jira/browse/CLOUDSTACK-4335
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> test_deployVmSharedNetworkWithoutIpRange creates a shared network
> without start and end ip. This is no longer permitted and creation
> of shared network will fail with error insufficient capacity. So
> remove this test which is invalid.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_shared_network_offering.py 85f0892 
> 
> Diff: https://reviews.apache.org/r/15115/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Girish Shilamkar
> 
>

Re: Review Request 15127: CLOUDSTACK-4495: Fixed TestResetSSHKeyUserRights.test_01_reset_keypair_normal_user

[Girish Shilamkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15127/#review27894
---



test/integration/component/test_reset_ssh_keypair.py


If we are specifying vmid then no need to specify listall=True. Rest looks 
good.


- Girish Shilamkar


On Oct. 31, 2013, 12:41 p.m., Gaurav Aradhye wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15127/
> ---
> 
> (Updated Oct. 31, 2013, 12:41 p.m.)
> 
> 
> Review request for cloudstack and Girish Shilamkar.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Fixed test script issue which was checking the VM state by listing the VMs in 
> a particular account and checking the state of first VM in the list.
> It was assuming that there is only one VM in the account, in some cases 
> earlier deleted VM was found in expunged/destroying state in the list.
> 
> Now the VMs are listed by VM id and state is checked.
> 
> Also did some code cleanup.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_reset_ssh_keypair.py 4d0c45e 
>   tools/marvin/marvin/codes.py 39ef950 
> 
> Diff: https://reviews.apache.org/r/15127/diff/
> 
> 
> Testing
> ---
> 
> Tested locally on KVM setup:
> 
> test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights)
> Verify API resetSSHKeyForVirtualMachine for non admin non root ...
> ==> client.log <==CS_TEST]# tail -f client.log result.log
> 2013-10-31 01:27:19,814 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Check if the VM is 
> properly deployed or
>  not?
> 2013-10-31 01:27:19,834 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Stopping the virtual 
> machine
> 2013-10-31 01:29:25,329 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Creating a new SSH 
> keypair for account:
>  test-5XE7P6
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Created a new keypair 
> with name: W44C6E
> .pem
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Writing the private key 
> to local file
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - File path: 
> /tmp/W44C6E.pem
> 2013-10-31 01:29:25,410 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Resetting the SSH key 
> pair for instance
> : 452be90f-d0e9-4b28-afcd-251eea1c23c3
> 2013-10-31 01:29:55,601 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Starting the virtual 
> machine after rese
> tting the keypair
> 2013-10-31 01:30:05,725 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - SSHing with new keypair
> 
> ==> result.log <==
> ok
> 
> 
> Thanks,
> 
> Gaurav Aradhye
> 
>

Review Request 15128: added group and domain params to importLdapUsers api call

[Rajani Karuturi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15128/
---

Review request for cloudstack, Abhinandan Prateek and Ian Duffy.


Bugs: CLOUDSTACK-4866
https://issues.apache.org/jira/browse/CLOUDSTACK-4866


Repository: cloudstack-git


Description
---

added group and domain params to importLdapUsers api call.

If a group param is present, importUsers will import ldap users only from that 
group.
If a domain param is present, it will create the users in that domain. it will 
fallback to group or OU of the ldap user or the ROOT domain in the order


Diffs
-

  
plugins/user-authenticators/ldap/src/org/apache/cloudstack/api/command/LdapImportUsersCmd.java
 f872247 
  
plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapConfiguration.java
 a08dccb 
  
plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapManager.java
 683822d 
  
plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapManagerImpl.java
 90a79b3 
  
plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapUserManager.java
 47697c9 
  
plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapConfigurationSpec.groovy
 66b4673 
  
plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapCreateAccountCmdSpec.groovy
 cc849de 
  
plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapImportUsersCmdSpec.groovy
 d04b094 
  
plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapManagerImplSpec.groovy
 321e1af 
  
plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapUserManagerSpec.groovy
 339923e 
  server/src/com/cloud/configuration/Config.java 3fdc343 
  setup/db/db/schema-421to430.sql 0de9dfd 

Diff: https://reviews.apache.org/r/15128/diff/


Testing
---

Yes. Added unit tests.


Thanks,

Rajani Karuturi

Re: Review Request 15127: CLOUDSTACK-4495: Fixed TestResetSSHKeyUserRights.test_01_reset_keypair_normal_user

[Gaurav Aradhye]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15127/
---

(Updated Oct. 31, 2013, 7:18 p.m.)


Review request for cloudstack and Girish Shilamkar.


Changes
---

Removed unnecessary listall parameter.


Repository: cloudstack-git


Description
---

Fixed test script issue which was checking the VM state by listing the VMs in a 
particular account and checking the state of first VM in the list.
It was assuming that there is only one VM in the account, in some cases earlier 
deleted VM was found in expunged/destroying state in the list.

Now the VMs are listed by VM id and state is checked.

Also did some code cleanup.


Diffs (updated)
-

  test/integration/component/test_reset_ssh_keypair.py 4d0c45e 
  tools/marvin/marvin/codes.py 39ef950 

Diff: https://reviews.apache.org/r/15127/diff/


Testing
---

Tested locally on KVM setup:

test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights)
Verify API resetSSHKeyForVirtualMachine for non admin non root ...
==> client.log <==CS_TEST]# tail -f client.log result.log
2013-10-31 01:27:19,814 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Check if the VM is 
properly deployed or
 not?
2013-10-31 01:27:19,834 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Stopping the virtual 
machine
2013-10-31 01:29:25,329 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Creating a new SSH keypair 
for account:
 test-5XE7P6
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Created a new keypair with 
name: W44C6E
.pem
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Writing the private key to 
local file
2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - File path: /tmp/W44C6E.pem
2013-10-31 01:29:25,410 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Resetting the SSH key pair 
for instance
: 452be90f-d0e9-4b28-afcd-251eea1c23c3
2013-10-31 01:29:55,601 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Starting the virtual 
machine after rese
tting the keypair
2013-10-31 01:30:05,725 - DEBUG - test_01_reset_keypair_normal_user 
(test_reset_ssh_keypair.TestResetSSHKeyUserRights) - SSHing with new keypair

==> result.log <==
ok


Thanks,

Gaurav Aradhye

note to devcloud users on db.properties.override

[Hugo Trippaers]

Heya,

With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
loading of the db.properties file, however this also affected the 
DatabaseCreator used by the deploydb procedure to refresh the database. 
Effectively the db.properies.override is ignored at the moment and the 
db.proeprties file in utils/conf/ will be used instead. We need to come up with 
a nice solution for this in the near future, but in the meantime make any 
custom setting there. Just don’t commit them ;-)

Cheers,

Hugo

Re: [Question] CloudStack support tool: cloud-bugtool

[sebgoa]

On Oct 25, 2013, at 7:57 AM, Gavin Lee  wrote:

> Unlike the CS4.2 release notes say, I could not find cloud-bugtool in
> source tree or installed/upgrade CS4.2 platform.
> This script can only be found in CloudPlatform 4.2.
> 
> I think the support tool like this is very helpful, does anyone know the
> detailed info?
> Thanks!
> 

Gavin, I don't know anything about this tool, first time I hear about it.
But Google knows about it:
http://support.citrix.com/article/CTX136093

And it seems that anyone can download this script. As far as I know this is not 
part of the CloudStack source code.
Having a quick look at it, I don't think it will work for cloudstack 4.2+ 
because the package renaming…but it should be an easy fix.

That said, you are right that this is mentioned in the release notes.
It was not in 4.1, so I suspect this was introduced in 4.2.
since we moved the docs over to a separate repo, the 'git blame' shows David 
Nalley has the author, so I cannot track who introduced this to ask for help.

Can you enter an issue in JIRA ?
We need either to clean the release notes or see how to get this cloud-bugtool 
in the code...

-sebastien

> -- 
> Gavin

Re: Migrate from existing xenserver pool into cloudstack

[Daan Hoogland]

not that i know, the workaround involves dumping your machines to
templates and installing them after assigning the cluster to
cloudstack. not something you want to do with a big cluster. So it
will be nteresting to see other solutions.

regards,

On Mon, Oct 28, 2013 at 4:52 AM, 王健  wrote:
> Is there a tool for migrating from existing xenserver pool into cloudstack?
> ---
> Confidentiality Notice: The information contained in this e-mail and any 
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be 
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
> this communication is
> not the intended recipient, unauthorized use, forwarding, printing,  storing, 
> disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this 
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original 
> message and all copies from
> your system. Thank you.
> ---

Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

Edison, Min,

When creating a Zone and there is no SSVM so it sends to
LocalHostEndPoint.  I always get the below error.  Is there some fix
for this?

ERROR [o.a.c.s.r.NfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) Unable to create local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
ERROR [o.a.c.s.r.LocalNfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) GetRootDir for
nfs://192.168.3.134:/exports/secondary1 failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.ensureLocalRootPathExists(NfsSecondaryStorageResource.java:2456)
at 
org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.mount(LocalNfsSecondaryStorageResource.java:96)
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mountUri(NfsSecondaryStorageResource.java:2293)
at 
org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.getRootDir(LocalNfsSecondaryStorageResource.java:85)
at 
org.apache.cloudstack.storage.template.DownloadManagerImpl.handleDownloadCommand(DownloadManagerImpl.java:695)

Darren

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

I'll fix that.  Gimme ten mintues.  I specifically looked at that code
and thought I didn't change the behavior, but I guess I screwed it up.

Just a general comment.  There's too many ways to do the same thing in
CloudStack.  Especially the database.  The way databases are setup for
developers shouldn't be so different from production.  The way we
manage the DB at the moment is so problematic in my mind.

Darren


On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
> Heya,
>
> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
> loading of the db.properties file, however this also affected the 
> DatabaseCreator used by the deploydb procedure to refresh the database. 
> Effectively the db.properies.override is ignored at the moment and the 
> db.proeprties file in utils/conf/ will be used instead. We need to come up 
> with a nice solution for this in the near future, but in the meantime make 
> any custom setting there. Just don’t commit them ;-)
>
> Cheers,
>
> Hugo

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Wei ZHOU]

Please check if you can
create /var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
and mount nfs point.

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

Ugh, I see what I did now.  When I tested that code I changed the code
I didn't do a "mvn install" before running "mvn ... deploydb."  That's
annoying, now I want to figure out why maven doesn't pickup changes
unless I do an install.  So I effectively tested with the old code.

Darren

On Thu, Oct 31, 2013 at 8:52 AM, Darren Shepherd
 wrote:
> I'll fix that.  Gimme ten mintues.  I specifically looked at that code
> and thought I didn't change the behavior, but I guess I screwed it up.
>
> Just a general comment.  There's too many ways to do the same thing in
> CloudStack.  Especially the database.  The way databases are setup for
> developers shouldn't be so different from production.  The way we
> manage the DB at the moment is so problematic in my mind.
>
> Darren
>
>
> On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
>> Heya,
>>
>> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
>> loading of the db.properties file, however this also affected the 
>> DatabaseCreator used by the deploydb procedure to refresh the database. 
>> Effectively the db.properies.override is ignored at the moment and the 
>> db.proeprties file in utils/conf/ will be used instead. We need to come up 
>> with a nice solution for this in the near future, but in the meantime make 
>> any custom setting there. Just don’t commit them ;-)
>>
>> Cheers,
>>
>> Hugo

Re: Review Request 15127: CLOUDSTACK-4495: Fixed TestResetSSHKeyUserRights.test_01_reset_keypair_normal_user

[Girish Shilamkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15127/#review27906
---

Ship it!


Ship It!

- Girish Shilamkar


On Oct. 31, 2013, 1:48 p.m., Gaurav Aradhye wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15127/
> ---
> 
> (Updated Oct. 31, 2013, 1:48 p.m.)
> 
> 
> Review request for cloudstack and Girish Shilamkar.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Fixed test script issue which was checking the VM state by listing the VMs in 
> a particular account and checking the state of first VM in the list.
> It was assuming that there is only one VM in the account, in some cases 
> earlier deleted VM was found in expunged/destroying state in the list.
> 
> Now the VMs are listed by VM id and state is checked.
> 
> Also did some code cleanup.
> 
> 
> Diffs
> -
> 
>   test/integration/component/test_reset_ssh_keypair.py 4d0c45e 
>   tools/marvin/marvin/codes.py 39ef950 
> 
> Diff: https://reviews.apache.org/r/15127/diff/
> 
> 
> Testing
> ---
> 
> Tested locally on KVM setup:
> 
> test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights)
> Verify API resetSSHKeyForVirtualMachine for non admin non root ...
> ==> client.log <==CS_TEST]# tail -f client.log result.log
> 2013-10-31 01:27:19,814 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Check if the VM is 
> properly deployed or
>  not?
> 2013-10-31 01:27:19,834 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Stopping the virtual 
> machine
> 2013-10-31 01:29:25,329 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Creating a new SSH 
> keypair for account:
>  test-5XE7P6
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Created a new keypair 
> with name: W44C6E
> .pem
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Writing the private key 
> to local file
> 2013-10-31 01:29:25,402 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - File path: 
> /tmp/W44C6E.pem
> 2013-10-31 01:29:25,410 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Resetting the SSH key 
> pair for instance
> : 452be90f-d0e9-4b28-afcd-251eea1c23c3
> 2013-10-31 01:29:55,601 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - Starting the virtual 
> machine after rese
> tting the keypair
> 2013-10-31 01:30:05,725 - DEBUG - test_01_reset_keypair_normal_user 
> (test_reset_ssh_keypair.TestResetSSHKeyUserRights) - SSHing with new keypair
> 
> ==> result.log <==
> ok
> 
> 
> Thanks,
> 
> Gaurav Aradhye
> 
>

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

This is fixed now.  Sorry about that.  I really do go through extra
length to test my changes.  I'm finding it difficult to ensure quality
because there is so much variation.  You have to test everything in so
many contexts.  It shouldn't be like that I hope to help reduced the
amount of variation in this platform.  This was part of why I made the
db.properties change to begin with.  I wanted to change MacAddress to
read from db.properties, but then I found out that that properties
file is loaded in probably 10 different places in all slightly
different ways.  Some places would decrypt, some wouldn't, some
ignored if the file was not found, some would just throw a NPE.  So I
consolidated all that code.

Darren


On Thu, Oct 31, 2013 at 9:07 AM, Darren Shepherd
 wrote:
> Ugh, I see what I did now.  When I tested that code I changed the code
> I didn't do a "mvn install" before running "mvn ... deploydb."  That's
> annoying, now I want to figure out why maven doesn't pickup changes
> unless I do an install.  So I effectively tested with the old code.
>
> Darren
>
> On Thu, Oct 31, 2013 at 8:52 AM, Darren Shepherd
>  wrote:
>> I'll fix that.  Gimme ten mintues.  I specifically looked at that code
>> and thought I didn't change the behavior, but I guess I screwed it up.
>>
>> Just a general comment.  There's too many ways to do the same thing in
>> CloudStack.  Especially the database.  The way databases are setup for
>> developers shouldn't be so different from production.  The way we
>> manage the DB at the moment is so problematic in my mind.
>>
>> Darren
>>
>>
>> On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
>>> Heya,
>>>
>>> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
>>> loading of the db.properties file, however this also affected the 
>>> DatabaseCreator used by the deploydb procedure to refresh the database. 
>>> Effectively the db.properies.override is ignored at the moment and the 
>>> db.proeprties file in utils/conf/ will be used instead. We need to come up 
>>> with a nice solution for this in the near future, but in the meantime make 
>>> any custom setting there. Just don’t commit them ;-)
>>>
>>> Cheers,
>>>
>>> Hugo

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

No, that's not the problem.  The problem is that the DownloadCommand
is supposed to go to the SSVM, but it sending it to the
LocalHostEndPoint which is the mgmt server.  So the mgmt server can
not create /var/cloudstack/mnt/
secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
so that it does not have sudo access.  A standard cloudstack install
has sudo access so it will create those folders and mount on the mgmt
server, but it shouldn't.

Please don't tell me we made a change so that the mgmt server is
expected to have NFS access to all secondary storages.

Darren

On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU  wrote:
> Please check if you can
> create /var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> and mount nfs point.

Review Request 15132: CLOUDSTACK 4256: Fixed Shared Network test cases using random private ips for shared networks

[Girish Shilamkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15132/
---

Review request for cloudstack and SrikanteswaraRao Talluri.


Bugs: CLOUDSTACK-4256
https://issues.apache.org/jira/browse/CLOUDSTACK-4256


Repository: cloudstack-git


Description
---

 Fixed Shared Network test cases using random private ips for shared networks


Diffs
-

  test/integration/component/test_shared_networks.py 732e144 

Diff: https://reviews.apache.org/r/15132/diff/


Testing
---


Log: Ran two test cases to check if the randomization works:

Test Shared Network with scope domain ... ok
test_createSharedNetwork_projectSpecific 
(test_shared_networks.TestSharedNetworks)
Test Shared Network with scope project ... ok
test_createSharedNetwork_usedVlan (test_shared_networks.TestSharedNetworks)


Thanks,

Girish Shilamkar

Re: note to devcloud users on db.properties.override

[Hugo Trippaers]

On 31 okt. 2013, at 17:27, Darren Shepherd  wrote:

> This is fixed now.  Sorry about that.  I really do go through extra
> length to test my changes.  I'm finding it difficult to ensure quality
> because there is so much variation.  You have to test everything in so
> many contexts.  It shouldn't be like that I hope to help reduced the
> amount of variation in this platform.  This was part of why I made the
> db.properties change to begin with.  I wanted to change MacAddress to
> read from db.properties, but then I found out that that properties
> file is loaded in probably 10 different places in all slightly
> different ways.  Some places would decrypt, some wouldn't, some
> ignored if the file was not found, some would just throw a NPE.  So I
> consolidated all that code.

Nice! I like the consolidation :-)

Cheers,

Hugo

> 
> Darren
> 
> 
> On Thu, Oct 31, 2013 at 9:07 AM, Darren Shepherd
>  wrote:
>> Ugh, I see what I did now.  When I tested that code I changed the code
>> I didn't do a "mvn install" before running "mvn ... deploydb."  That's
>> annoying, now I want to figure out why maven doesn't pickup changes
>> unless I do an install.  So I effectively tested with the old code.
>> 
>> Darren
>> 
>> On Thu, Oct 31, 2013 at 8:52 AM, Darren Shepherd
>>  wrote:
>>> I'll fix that.  Gimme ten mintues.  I specifically looked at that code
>>> and thought I didn't change the behavior, but I guess I screwed it up.
>>> 
>>> Just a general comment.  There's too many ways to do the same thing in
>>> CloudStack.  Especially the database.  The way databases are setup for
>>> developers shouldn't be so different from production.  The way we
>>> manage the DB at the moment is so problematic in my mind.
>>> 
>>> Darren
>>> 
>>> 
>>> On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
 Heya,
 
 With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
 loading of the db.properties file, however this also affected the 
 DatabaseCreator used by the deploydb procedure to refresh the database. 
 Effectively the db.properies.override is ignored at the moment and the 
 db.proeprties file in utils/conf/ will be used instead. We need to come up 
 with a nice solution for this in the near future, but in the meantime make 
 any custom setting there. Just don’t commit them ;-)
 
 Cheers,
 
 Hugo

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Min Chen]

Hi Darren,

The logic of sending command to Local in case of no SSVM is up is to
support S3 as secondary storage. In case of S3 as secondary storage,
system vm template is automatically downloaded to S3 when S3 is added into
CloudStack, unlike NFS secondary storage case where we used a script to
pre-populate system vm template. The thing I don't get is: what scenario
are you trying to do to trigger DownloadCommand when SSVM is not up?

Thanks
-min

On 10/31/13 9:32 AM, "Darren Shepherd"  wrote:

>No, that's not the problem.  The problem is that the DownloadCommand
>is supposed to go to the SSVM, but it sending it to the
>LocalHostEndPoint which is the mgmt server.  So the mgmt server can
>not create /var/cloudstack/mnt/
>secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
>so that it does not have sudo access.  A standard cloudstack install
>has sudo access so it will create those folders and mount on the mgmt
>server, but it shouldn't.
>
>Please don't tell me we made a change so that the mgmt server is
>expected to have NFS access to all secondary storages.
>
>Darren
>
>On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU  wrote:
>> Please check if you can
>> create 
>>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> and mount nfs point.

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

Yeah I think it would be great to talk about this at CCC.  I'm
hesitant to further narrow down the definition of the network.  For
example, I think OpenStack's Neutron is fundamentally flawed because
they defined a network as a L2 segment.  During my time that I was
away from CloudStack the physical network concept was added as a first
class object in CloudStack.  I don't know if I fully agree with it.  I
have to dig deeper and just haven't had the time to.  My inclination
is that is was one step too far and that it isn't really needed.  I
know you need to associate traffic with essentially a bridge on the
hypervisor, but that can be done with just labelling the network
offering that created the network.  So I need to dig into this
further.

In the world of SDN, I think its even more important to keep the
definition of the a network loose.  SDN has the capability of
completely changing the way we look at L2 and L3.  Currently in
networking we group things by L3 and L2 concepts as that is how
routers and switches are laid out today.  As SDN matures and you see
more flow oriented design it won't make sense to group things using L2
and L3 concepts (as those become more a physical fabric technology),
the groups becomes more loose and thus the definition of a network
should be loose.

Now that's not to say that a network can't provide L2 and L3
information.  You should be able to create a network in CloudStack and
based on the configuration you know that it is a single L2 or L3.  It
is just that the core orchestration system can't make that fundamental
assumption.  I'd be interested in furthering the model and maybe
adding a concept of a L2 network such that a network guru when
designing a network, can define multiple l2networks and associate them
with the generic network that was created.  That idea I'm still
toiling with.

For example, when configuring DHCP on the systemvm.  DHCP is a L2
based serviced.  So to configure DHCP you really need to know for each
nic, what is the L2 its attached to and what are the VMs associated
with that L2.  Today, since there is no first class concept of a L2
network, you have to look at the implied definition of L2.  For basic
networks, the L2 is the Pod, so you need to list all VMs in that Pod.
For guest/VPC networks, the L2 is the network object, so you need to
list all VMs associated with the network.  It would be nice if when
the guru designed the network, it also defined the l2networks, and
then when a VM starts the guru the reserve() method could associate
the l2network to the nic.  So the nic object would have a network_id
and a l2_network_id.

I've gone off on a total tangent, but yeah, we should talk.

Darren

On Thu, Oct 31, 2013 at 2:53 AM, Hugo Trippaers  wrote:
>
> On 31 okt. 2013, at 03:48, Darren Shepherd  
> wrote:
>
>> Its important to remember that the definition of a Network in
>> CloudStack is quite purposefully very loose.  A network in the
>> simplest terms is just a group of VMs.  A guru creates and manages
>> that network.  A network is not strictly defined as a union of
>> traffictype/isolationtype/networktype/guesttype.  So, for example,
>> there need not be a 1-to-1 mapping between Guru and IsolationType.
>> You should be able to have multiple gurus per any context.  That's the
>> theory and vision.
>
> I think that vision was put down somewhere before we entered the era of 
> software defined networking. At the moment it is simply not true as long as 
> the guru is responsible for creating the overlay networks there is a 1 on 1 
> relation with the guru and the supported isolation types in several cases.
>
>>
>> Now the reality is that the current selection logic of the guru in the
>> orchestrator doesn't fully back up that vision because the selection
>> logic too easily allows two Guru's to implement the same network.
>>
>> So I think the current behavior of setupNetwork is flawed.  This
>> change, I feel, muddies to a certain degree the definition of a
>> network in CloudStack and attempts to say a network is strict union of
>> traffic/isolation/network/guest-type.  If we merge this change, does
>> it hurt anything?  No, its more just conceptual disagreement.  When I
>> get to the point that I feel setupNetwork is limiting what I want to
>> do, I'll propose a change.  Right now, I just accept that its not
>> right in my mind.
>
> I’ve fixed the definition of setupNetwork so the function returns a single 
> network when called. This was already the effective behavior, but now its put 
> down in code as well. I’ll push that change to the branch today.
>
>
>>
>> So I'd have to say I'm neutral on this change.  I don't care too much
>> either way as I think down the line this all needs to change.
>
> Let’s find a moment to sit down at CCCEU and discuss this further. I have 
> some ideas of where i want to go with the networking bit as well and it would 
> be nice if we are more or less on the same page before we start working on 
> it. Saying that you

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Alex Ough]

All,

While I'm waiting for the permissions, I'd like to bring up a discussion on
the overall architecture to support this feature.

There can be 2 different approaches as below.
1. master - slave architecture : the manual changes are allowed only in one
master management server, and those in other servers are either prohibited
or discarded.
2. multiple source architecture : all management servers allow manual
changes and any change in any server will be propagated to the rest of
servers.

I'm not sure if we can impose the restriction of #1.
If not, #2 is a little more complicated, but the only issue is to
distinguish the events from either manual jobs or triggered automatic
processing, which I'm currently working on.

Your comments/recommendations will be very appreciated.
Thanks
Alex Ough


On Wed, Oct 30, 2013 at 12:19 PM, Alex Ough  wrote:

> And I'd like to write the design document in the wiki page, but I don't
> seem to have a permission to create pages.
> So can anyone give me the permission?
>
> My account in the wiki is alex.o...@sungard.com
>
> Thanks in advance.
> Alex Ough
>
>
> On Tue, Oct 29, 2013 at 3:38 PM, Alex Ough  wrote:
>
>> I created a jira for this feature.
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-4992
>>
>> But it doesn't allow for me to assign it to myself, so any permission do
>> I need for this?
>> If so, can anyone give me this permission?
>>
>> If there is anything missing, let me know.
>> Thanks
>> Alex Ough
>>
>>
>> On Fri, Oct 18, 2013 at 9:30 AM, Kishan Kavala 
>> wrote:
>>
>>> > -Original Message-
>>> > From: Alex Ough [mailto:alex.o...@sungard.com]
>>> > Sent: Thursday, 17 October 2013 11:25 PM
>>> > To: dev@cloudstack.apache.org; u...@cloudstack.apache.org
>>> > Subject: Fwd: [DISCUSS] Domain/Account/User Sync Up Among Multiple
>>> > Regions
>>> >
>>> > All,
>>> >
>>> > Currently, under the environment of cloudstack with multiple regions,
>>> each
>>> > region has its own management server running with a separate database.
>>> So if
>>> > we want to support multiple regions and provide one point of entry for
>>> a
>>> > customer, we need to duplicate domain/account/user information of that
>>> > customer to all of the databases of regions the customer accesses,
>>> which will
>>> > cause data discrepancies when users update those data independently in
>>> each
>>> > management server.
>>> >
>>> > So I'd like to provide a way to sync up the data using the messaging
>>> system
>>> > introduced in 4.1.0. Using the events from each management server,
>>> updates
>>> > from each region can be propagated to the rest regions and they can be
>>> > executed accordingly.
>>> >
>>> > I hope you guys have a chance to think about this and give some
>>> feedbacks if
>>> > interested.
>>> > Thanks in advance.
>>> > Alex Ough
>>>
>>> [KK] Alex, it was discussed sometime back. Related thread [1]. Sync up
>>> using messaging system is the right way to go.
>>>
>>>
>>> [1]
>>> http://www.mail-archive.com/cloudstack-dev@incubator.apache.org/msg20193.html
>>>
>>>
>>
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

It's happening when I first create the zone.  I'll debug it further to tell
you exactly what its happening.  But I don't understand how the following
code in DefaultEndpointSelector could be correct.  If I have a zone, and I
delete/stop/crash the SSVM, and at the same time register a new template,
it seems the DownloadCommand will go to LocalHostEndPoint.

protected EndPoint findEndpointForImageStorage(DataStore store) {
Long dcId = null;
Scope storeScope = store.getScope();
if (storeScope.getScopeType() == ScopeType.ZONE) {
dcId = storeScope.getScopeId();
}
// find ssvm that can be used to download data to store. For
zone-wide
// image store, use SSVM for that zone. For region-wide store,
// we can arbitrarily pick one ssvm to do that task
List ssAHosts =
listUpAndConnectingSecondaryStorageVmHost(dcId);
if (ssAHosts == null || ssAHosts.isEmpty()) {
s_logger.info("No running ssvm is found, so command will be
sent to LocalHostEndPoint");
return LocalHostEndpoint.getEndpoint(); // use local host as
endpoint in
// case of no ssvm existing
}
Collections.shuffle(ssAHosts);
HostVO host = ssAHosts.get(0);
return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
host.getPrivateIpAddress(),
host.getPublicIpAddress());
}

Darren


On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:

> Hi Darren,
>
> The logic of sending command to Local in case of no SSVM is up is
> to
> support S3 as secondary storage. In case of S3 as secondary storage,
> system vm template is automatically downloaded to S3 when S3 is added into
> CloudStack, unlike NFS secondary storage case where we used a script to
> pre-populate system vm template. The thing I don't get is: what scenario
> are you trying to do to trigger DownloadCommand when SSVM is not up?
>
> Thanks
> -min
>
> On 10/31/13 9:32 AM, "Darren Shepherd" 
> wrote:
>
> >No, that's not the problem.  The problem is that the DownloadCommand
> >is supposed to go to the SSVM, but it sending it to the
> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
> >not create /var/cloudstack/mnt/
> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
> >so that it does not have sudo access.  A standard cloudstack install
> >has sudo access so it will create those folders and mount on the mgmt
> >server, but it shouldn't.
> >
> >Please don't tell me we made a change so that the mgmt server is
> >expected to have NFS access to all secondary storages.
> >
> >Darren
> >
> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU  wrote:
> >> Please check if you can
> >> create
> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> and mount nfs point.
>
>

Re: ACS + Riak-CS, works?

[Darren Shepherd]

My template is less than 5gb and it still uses multipart.  I actually don't
think this is a CloudStack issue but more of an issue with Riak-CS.  I was
just wondering if anybody has specifically ran the setup or knows more
about Riak-CS and can tell me whats up.  I've been trying off and on for
the last week to try to get this working with 4.2 but it just keeps
failing.  I think I'll just try ceph rados gateway if I can't get this
working.

Darren


On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
sanjeev.neelar...@citrix.com> wrote:

> Hi,
>
> Cloudstack with Riak-CS works fine. If we register template more than 5GB
> we use multipart upload to push the files to Riak. But this does not work
> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
> registration it should work fine.
>
> Thanks,
> Sanjeev
>
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
>
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
> what I try when it goes to download and save the template, after its posted
> a series of multi part uploads it ends up sending a 403.  The access log
> looks like
>
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
>
> And then in the error log I always get
>
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
> record for s3 failed. Reason: disconnected
>
> I'm able to interact with Riak-CS with s3cmd just fine but its not using
> multipart. I think I'm going to try to download a new s3cmd that does
> multipart and see if even that will work.
>
> Darren
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Min Chen]

Yes, that is true, it is a bug in that case. The logic of choosing
endpoint to send Command will not work properly for destroying/Stopping VM
case. Our assumption is that registering template is done when ssvm is up
and running. Otherwise, it will send to LocalHost to execute command. In
your opinion, what should be expected behavior if user is trying to
register template when ssvm is crashed? Directly reporting error instead
of trying anything.

Thanks
-min 

On 10/31/13 10:18 AM, "Darren Shepherd" 
wrote:

>It's happening when I first create the zone.  I'll debug it further to
>tell
>you exactly what its happening.  But I don't understand how the following
>code in DefaultEndpointSelector could be correct.  If I have a zone, and I
>delete/stop/crash the SSVM, and at the same time register a new template,
>it seems the DownloadCommand will go to LocalHostEndPoint.
>
>protected EndPoint findEndpointForImageStorage(DataStore store) {
>Long dcId = null;
>Scope storeScope = store.getScope();
>if (storeScope.getScopeType() == ScopeType.ZONE) {
>dcId = storeScope.getScopeId();
>}
>// find ssvm that can be used to download data to store. For
>zone-wide
>// image store, use SSVM for that zone. For region-wide store,
>// we can arbitrarily pick one ssvm to do that task
>List ssAHosts =
>listUpAndConnectingSecondaryStorageVmHost(dcId);
>if (ssAHosts == null || ssAHosts.isEmpty()) {
>s_logger.info("No running ssvm is found, so command will be
>sent to LocalHostEndPoint");
>return LocalHostEndpoint.getEndpoint(); // use local host as
>endpoint in
>// case of no ssvm existing
>}
>Collections.shuffle(ssAHosts);
>HostVO host = ssAHosts.get(0);
>return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>host.getPrivateIpAddress(),
>host.getPublicIpAddress());
>}
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:
>
>> Hi Darren,
>>
>> The logic of sending command to Local in case of no SSVM is up
>>is
>> to
>> support S3 as secondary storage. In case of S3 as secondary storage,
>> system vm template is automatically downloaded to S3 when S3 is added
>>into
>> CloudStack, unlike NFS secondary storage case where we used a script to
>> pre-populate system vm template. The thing I don't get is: what scenario
>> are you trying to do to trigger DownloadCommand when SSVM is not up?
>>
>> Thanks
>> -min
>>
>> On 10/31/13 9:32 AM, "Darren Shepherd" 
>> wrote:
>>
>> >No, that's not the problem.  The problem is that the DownloadCommand
>> >is supposed to go to the SSVM, but it sending it to the
>> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
>> >not create /var/cloudstack/mnt/
>> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
>> >so that it does not have sudo access.  A standard cloudstack install
>> >has sudo access so it will create those folders and mount on the mgmt
>> >server, but it shouldn't.
>> >
>> >Please don't tell me we made a change so that the mgmt server is
>> >expected to have NFS access to all secondary storages.
>> >
>> >Darren
>> >
>> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU 
>>wrote:
>> >> Please check if you can
>> >> create
>> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
>> >> and mount nfs point.
>>
>>

Re: ACS + Riak-CS, works?

[John Burwell]

Darren,

Since S3/Riak CS can receive the contents of a file out of order and 
re-assemble upon completion of transfer of all parts, the underlying driver 
uses the TransferManager which automatically breaks files up into multi-part 
uploads to increase parallelism to decrease the time required to upload to the 
object store.  Therefore, multi-part uploads are not employed solely to upload 
files larger than 5 GB. 

Have you tried to upload the file to Riak CS using s3cmd outside of CloudStack?

Thanks,
-John

On Oct 31, 2013, at 1:23 PM, Darren Shepherd  
wrote:

> My template is less than 5gb and it still uses multipart.  I actually don't 
> think this is a CloudStack issue but more of an issue with Riak-CS.  I was 
> just wondering if anybody has specifically ran the setup or knows more about 
> Riak-CS and can tell me whats up.  I've been trying off and on for the last 
> week to try to get this working with 4.2 but it just keeps failing.  I think 
> I'll just try ceph rados gateway if I can't get this working.
> 
> Darren
> 
> 
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
>  wrote:
> Hi,
> 
> Cloudstack with Riak-CS works fine. If we register template more than 5GB we 
> use multipart upload to push the files to Riak. But this does not work for 
> snapshots and volumes. This will be fixed in CS4.2.1 . But for template 
> registration it should work fine.
> 
> Thanks,
> Sanjeev
> 
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
> 
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I 
> try when it goes to download and save the template, after its posted a series 
> of multi part uploads it ends up sending a 403.  The access log looks like
> 
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 
> And then in the error log I always get
> 
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record 
> for s3 failed. Reason: disconnected
> 
> I'm able to interact with Riak-CS with s3cmd just fine but its not using 
> multipart. I think I'm going to try to download a new s3cmd that does 
> multipart and see if even that will work.
> 
> Darren
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: S3 Secondary Storage fails with large files

[Andrei Mikhailovsky]

Yes, indeed, this is how i am trying to create the template. I do get the same 
error as I get while trying to snapshot. 

Is there a workaround or do i need to wait for the 4.2.1 to come out? 

Thanks 

Andrei 

- Original Message -

From: "Min Chen"  
To: dev@cloudstack.apache.org, us...@cloudstack.apache.org 
Sent: Wednesday, 30 October, 2013 5:39:15 PM 
Subject: Re: S3 Secondary Storage fails with large files 

So you are doing create template from volume, right? I think that it 
should go through the same code used in backup snapshot, so will have the 
same limit. Would you mind paste the stack trace you got when you got that 
error in this case? 

Thanks 
-min 

On 10/30/13 10:29 AM, "Andrei Mikhailovsky"  wrote: 

>Hi, 
> 
>I've just tried creating a template from a 10GB disk image file and it 
>has failed with the same error - EntityTooLarge. So, neither snapshots 
>nor templates over 5GB in size are working for me. Could someone confirm 
>if I am not the only one? 
> 
>Thanks 
> 
>Andrei 
>- Original Message - 
> 
>From: "Min Chen"  
>To: dev@cloudstack.apache.org, us...@cloudstack.apache.org 
>Sent: Wednesday, 30 October, 2013 4:40:48 PM 
>Subject: Re: S3 Secondary Storage fails with large files 
> 
>For 4.2.0, you should be able to register template > 5GB. But we are 
>aware 
>of an issue you mentioned here: backup snapshot > 5GB will fail since 
>that 
>part of code is still using S3 single part upload api. This bug has been 
>fixed in latest 4.2 branch, and will be available in 4.2.1 release. 
> 
>Thanks 
>-min 
> 
>On 10/30/13 9:18 AM, "Andrei Mikhailovsky"  wrote: 
> 
>>Hello Guys , 
>> 
>>I am using ACS 4.2.0 with KVM hypervisor. While trying to create 
>>templates and snapshots from the existing vms I am getting an error 
>>message / exception on the management logs: 
>> 
>>2013-10-30 13:29:07,264 DEBUG [storage.snapshot.SnapshotManagerImpl] 
>>(Job-Executor-12:job-41 = [ 85659739-df2e-4afe-b638-2d25e804063f ]) 
>>Failed to create snapshot 
>>com.cloud.utils.exception.CloudRuntimeException: Status Code: 400, AWS 
>>Service: Amazon S3, AWS Request ID: null, AWS Error Code: 
>>EntityTooLarge, 
>>AWS Error Message: null, S3 Extended Request ID: null 
>>at 
>>com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient. 
>>j 
>>ava:610) 
>>at 
>>com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:3 
>>1 
>>0) 
>>at 
>>com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164) 
>>at 
>>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906) 
>> 
>>at 
>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:11 
>>2 
>>3) 
>>at 
>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:97 
>>9 
>>) 
>>at com.cloud.utils.S3Utils.putFile(S3Utils.java:143) 
>>at 
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToS3(KVMStorageP 
>>r 
>>ocessor.java:556) 
>>at 
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToObjectStore(KV 
>>M 
>>StorageProcessor.java:582) 
>>at 
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshotForObj 
>>e 
>>ctStore(KVMStorageProcessor.java:622) 
>>at 
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMSt 
>>o 
>>rageProcessor.java:634) 
>>at 
>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(Sto 
>>r 
>>ageSubsystemCommandHandlerBase.java:84) 
>>at 
>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStora 
>>g 
>>eCommands(StorageSubsystemCommandHandlerBase.java:49) 
>>at 
>>com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest 
>>( 
>>LibvirtComputingResource.java:1286) 
>>at com.cloud.agent.Agent.processRequest(Agent.java:525) 
>>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) 
>>at com.cloud.utils.nio.Task.run(Task.java:83) 
>>at 
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java 
>>: 
>>1146) 
>>at 
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav 
>>a 
>>:615) 
>>at java.lang.Thread.run(Thread.java:679) 
>> 
>>at 
>>org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot 
>>( 
>>SnapshotServiceImpl.java:280) 
>>at 
>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSn 
>>a 
>>pshot(XenserverSnapshotStrategy.java:138) 
>>at 
>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnap 
>>s 
>>hot(XenserverSnapshotStrategy.java:264) 
>>at 
>>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManag 
>>e 
>>rImpl.java:1013) 
>>at 
>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interceptor 
>>D 
>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) 
>>at 
>>org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Volum 
>>e 
>>ServiceImpl.java:1307) 
>>at 
>>com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2 
>>7 
>>19) 
>>at 
>>org.apache.c

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

The behaviour should be to delay until there is a SSVM available.  I don't
know if that is even possible in CloudStack, I don't want you to sleep().
Is there any way to delay?  Something else in the system should be ensuring
that a SSVM is running and available (what I don't know...).  If you can't
delay, it is far better in my mind to send an error to the user than to try
to do the operation on the mgmt server.

I think it is very dangerous to have this code that routes to localhost
because in a production setup ACS will have the sudo access to
mount/mkdir.  But the server may not have network access, so you run the
chance of mounting the NFS and having it hang almost indefinitely which can
cause system instability for the mgmt server.

The specific issue I have right now is that for some reason on master, the
allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
restart the mgmt stack, on host connect it tries to download templates so
it sends a command and it goes to LocalHostEndPoint.

Darren


On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:

> Yes, that is true, it is a bug in that case. The logic of choosing
> endpoint to send Command will not work properly for destroying/Stopping VM
> case. Our assumption is that registering template is done when ssvm is up
> and running. Otherwise, it will send to LocalHost to execute command. In
> your opinion, what should be expected behavior if user is trying to
> register template when ssvm is crashed? Directly reporting error instead
> of trying anything.
>
> Thanks
> -min
>
> On 10/31/13 10:18 AM, "Darren Shepherd" 
> wrote:
>
> >It's happening when I first create the zone.  I'll debug it further to
> >tell
> >you exactly what its happening.  But I don't understand how the following
> >code in DefaultEndpointSelector could be correct.  If I have a zone, and I
> >delete/stop/crash the SSVM, and at the same time register a new template,
> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >
> >protected EndPoint findEndpointForImageStorage(DataStore store) {
> >Long dcId = null;
> >Scope storeScope = store.getScope();
> >if (storeScope.getScopeType() == ScopeType.ZONE) {
> >dcId = storeScope.getScopeId();
> >}
> >// find ssvm that can be used to download data to store. For
> >zone-wide
> >// image store, use SSVM for that zone. For region-wide store,
> >// we can arbitrarily pick one ssvm to do that task
> >List ssAHosts =
> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> >if (ssAHosts == null || ssAHosts.isEmpty()) {
> >s_logger.info("No running ssvm is found, so command will be
> >sent to LocalHostEndPoint");
> >return LocalHostEndpoint.getEndpoint(); // use local host as
> >endpoint in
> >// case of no ssvm existing
> >}
> >Collections.shuffle(ssAHosts);
> >HostVO host = ssAHosts.get(0);
> >return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >host.getPrivateIpAddress(),
> >host.getPublicIpAddress());
> >}
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:
> >
> >> Hi Darren,
> >>
> >> The logic of sending command to Local in case of no SSVM is up
> >>is
> >> to
> >> support S3 as secondary storage. In case of S3 as secondary storage,
> >> system vm template is automatically downloaded to S3 when S3 is added
> >>into
> >> CloudStack, unlike NFS secondary storage case where we used a script to
> >> pre-populate system vm template. The thing I don't get is: what scenario
> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 9:32 AM, "Darren Shepherd" 
> >> wrote:
> >>
> >> >No, that's not the problem.  The problem is that the DownloadCommand
> >> >is supposed to go to the SSVM, but it sending it to the
> >> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
> >> >not create /var/cloudstack/mnt/
> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
> >> >so that it does not have sudo access.  A standard cloudstack install
> >> >has sudo access so it will create those folders and mount on the mgmt
> >> >server, but it shouldn't.
> >> >
> >> >Please don't tell me we made a change so that the mgmt server is
> >> >expected to have NFS access to all secondary storages.
> >> >
> >> >Darren
> >> >
> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU 
> >>wrote:
> >> >> Please check if you can
> >> >> create
> >> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> >> and mount nfs point.
> >>
> >>
>
>

Re: ACS + Riak-CS, works?

[Darren Shepherd]

Uploading without multipart works with s3cmd.  The s3cmd I was using didn't
have multipart support, so I'll download a newer one and try it tonight.

Would you happen to know what the below means?  I get a 200 response for
parts 1,2,3,4,etc and then randomly one part will fail with the below.

riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3
failed. Reason: disconnected

Darren



On Thu, Oct 31, 2013 at 10:30 AM, John Burwell  wrote:

> Darren,
>
> Since S3/Riak CS can receive the contents of a file out of order and
> re-assemble upon completion of transfer of all parts, the underlying driver
> uses the TransferManager which automatically breaks files up into
> multi-part uploads to increase parallelism to decrease the time required to
> upload to the object store.  Therefore, multi-part uploads are not employed
> solely to upload files larger than 5 GB.
>
> Have you tried to upload the file to Riak CS using s3cmd outside of
> CloudStack?
>
> Thanks,
> -John
>
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
> wrote:
>
> My template is less than 5gb and it still uses multipart.  I actually
> don't think this is a CloudStack issue but more of an issue with Riak-CS.
> I was just wondering if anybody has specifically ran the setup or knows
> more about Riak-CS and can tell me whats up.  I've been trying off and on
> for the last week to try to get this working with 4.2 but it just keeps
> failing.  I think I'll just try ceph rados gateway if I can't get this
> working.
>
> Darren
>
>
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
> sanjeev.neelar...@citrix.com> wrote:
>
>> Hi,
>>
>> Cloudstack with Riak-CS works fine. If we register template more than 5GB
>> we use multipart upload to push the files to Riak. But this does not work
>> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
>> registration it should work fine.
>>
>> Thanks,
>> Sanjeev
>>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>>
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>> what I try when it goes to download and save the template, after its posted
>> a series of multi part uploads it ends up sending a 403.  The access log
>> looks like
>>
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>>
>> And then in the error log I always get
>>
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
>> record for s3 failed. Reason: disconnected
>>
>> I'm able to interact with Riak-CS with s3cmd just fine but its not using
>> multipart. I think I'm going to try to download a new s3cmd that does
>> multipart and see if even that will work.
>>
>> Darren
>>
>
>
>

why system.vm.use.local.storage?

[Darren Shepherd]

Why do we have the configuration system.vm.use.local.storage?  Why would
somebody care if they systemvm is on local storage?  I'm guessing the idea
here is if it's on local storage we can't bring it back in a host failure
situation?  But don't we just recreate the systemvm?

Darren

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Min Chen]

Delaying them until ssvm is up may not that easy to handle in current
CloudStack. We may be able to send an error in this case by distinguishing
S3 register template case from other scenarios. Would you mind filing a
JIRA bug on this? Your issue seems different from the issue filed by Daan
(https://issues.apache.org/jira/browse/CLOUDSTACK-4759).

Thanks
-min


On 10/31/13 10:38 AM, "Darren Shepherd" 
wrote:

>The behaviour should be to delay until there is a SSVM available.  I don't
>know if that is even possible in CloudStack, I don't want you to sleep().
>Is there any way to delay?  Something else in the system should be
>ensuring
>that a SSVM is running and available (what I don't know...).  If you can't
>delay, it is far better in my mind to send an error to the user than to
>try
>to do the operation on the mgmt server.
>
>I think it is very dangerous to have this code that routes to localhost
>because in a production setup ACS will have the sudo access to
>mount/mkdir.  But the server may not have network access, so you run the
>chance of mounting the NFS and having it hang almost indefinitely which
>can
>cause system instability for the mgmt server.
>
>The specific issue I have right now is that for some reason on master, the
>allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
>restart the mgmt stack, on host connect it tries to download templates so
>it sends a command and it goes to LocalHostEndPoint.
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:
>
>> Yes, that is true, it is a bug in that case. The logic of choosing
>> endpoint to send Command will not work properly for destroying/Stopping
>>VM
>> case. Our assumption is that registering template is done when ssvm is
>>up
>> and running. Otherwise, it will send to LocalHost to execute command. In
>> your opinion, what should be expected behavior if user is trying to
>> register template when ssvm is crashed? Directly reporting error instead
>> of trying anything.
>>
>> Thanks
>> -min
>>
>> On 10/31/13 10:18 AM, "Darren Shepherd" 
>> wrote:
>>
>> >It's happening when I first create the zone.  I'll debug it further to
>> >tell
>> >you exactly what its happening.  But I don't understand how the
>>following
>> >code in DefaultEndpointSelector could be correct.  If I have a zone,
>>and I
>> >delete/stop/crash the SSVM, and at the same time register a new
>>template,
>> >it seems the DownloadCommand will go to LocalHostEndPoint.
>> >
>> >protected EndPoint findEndpointForImageStorage(DataStore store) {
>> >Long dcId = null;
>> >Scope storeScope = store.getScope();
>> >if (storeScope.getScopeType() == ScopeType.ZONE) {
>> >dcId = storeScope.getScopeId();
>> >}
>> >// find ssvm that can be used to download data to store. For
>> >zone-wide
>> >// image store, use SSVM for that zone. For region-wide store,
>> >// we can arbitrarily pick one ssvm to do that task
>> >List ssAHosts =
>> >listUpAndConnectingSecondaryStorageVmHost(dcId);
>> >if (ssAHosts == null || ssAHosts.isEmpty()) {
>> >s_logger.info("No running ssvm is found, so command will be
>> >sent to LocalHostEndPoint");
>> >return LocalHostEndpoint.getEndpoint(); // use local host
>>as
>> >endpoint in
>> >// case of no ssvm existing
>> >}
>> >Collections.shuffle(ssAHosts);
>> >HostVO host = ssAHosts.get(0);
>> >return 
>>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>> >host.getPrivateIpAddress(),
>> >host.getPublicIpAddress());
>> >}
>> >
>> >Darren
>> >
>> >
>> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:
>> >
>> >> Hi Darren,
>> >>
>> >> The logic of sending command to Local in case of no SSVM is
>>up
>> >>is
>> >> to
>> >> support S3 as secondary storage. In case of S3 as secondary storage,
>> >> system vm template is automatically downloaded to S3 when S3 is added
>> >>into
>> >> CloudStack, unlike NFS secondary storage case where we used a script
>>to
>> >> pre-populate system vm template. The thing I don't get is: what
>>scenario
>> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
>> >>
>> >> Thanks
>> >> -min
>> >>
>> >> On 10/31/13 9:32 AM, "Darren Shepherd" 
>> >> wrote:
>> >>
>> >> >No, that's not the problem.  The problem is that the DownloadCommand
>> >> >is supposed to go to the SSVM, but it sending it to the
>> >> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
>> >> >not create /var/cloudstack/mnt/
>> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt
>>setup
>> >> >so that it does not have sudo access.  A standard cloudstack install
>> >> >has sudo access so it will create those folders and mount on the
>>mgmt
>> >> >server, but it shouldn't.
>> >> >
>> >> >Please don't tell me we made a change so that the mgmt server is
>> >> >expected to have NFS access to all secon

Re: ACS + Riak-CS, works?

[John Burwell]

Darren,

Your are receiving a 403 which indicates that the ACLs/user is not properly 
configured.  Which version of Riak CS are you using?  Can you send your 
app.config and vm.args for Riak and Riak CS?  Personally, I use the 
vagrant-riak-cs-cluster [1] project to build local instances for development 
and testing.  

Thanks,
-John

[1]: https://github.com/basho/vagrant-riak-cs-cluster


On Oct 31, 2013, at 1:23 PM, Darren Shepherd  
wrote:

> My template is less than 5gb and it still uses multipart.  I actually don't 
> think this is a CloudStack issue but more of an issue with Riak-CS.  I was 
> just wondering if anybody has specifically ran the setup or knows more about 
> Riak-CS and can tell me whats up.  I've been trying off and on for the last 
> week to try to get this working with 4.2 but it just keeps failing.  I think 
> I'll just try ceph rados gateway if I can't get this working.
> 
> Darren
> 
> 
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
>  wrote:
> Hi,
> 
> Cloudstack with Riak-CS works fine. If we register template more than 5GB we 
> use multipart upload to push the files to Riak. But this does not work for 
> snapshots and volumes. This will be fixed in CS4.2.1 . But for template 
> registration it should work fine.
> 
> Thanks,
> Sanjeev
> 
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
> 
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I 
> try when it goes to download and save the template, after its posted a series 
> of multi part uploads it ends up sending a 403.  The access log looks like
> 
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 
> And then in the error log I always get
> 
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record 
> for s3 failed. Reason: disconnected
> 
> I'm able to interact with Riak-CS with s3cmd just fine but its not using 
> multipart. I think I'm going to try to download a new s3cmd that does 
> multipart and see if even that will work.
> 
> Darren
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: ACS + Riak-CS, works?

[Min Chen]

Hi John,
I have a related question on this issue for you regarding RiakCS. In 4.2.0 
release, backup snapshot to S3 is done through your s3xen plugin, where S3 
single part upload API is used. If customer is using RiakCS as their S3 object 
store, can they still backup snapshot larger than 5GB? Based on your comments, 
it seems that they should be able to?

Thanks
-min

From: John Burwell mailto:jburw...@basho.com>>
Reply-To: mailto:dev@cloudstack.apache.org>>
Date: Thursday, October 31, 2013 10:30 AM
To: Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>>
Cc: "dev@cloudstack.apache.org" 
mailto:dev@cloudstack.apache.org>>
Subject: Re: ACS + Riak-CS, works?

Darren,

Since S3/Riak CS can receive the contents of a file out of order and 
re-assemble upon completion of transfer of all parts, the underlying driver 
uses the TransferManager which automatically breaks files up into multi-part 
uploads to increase parallelism to decrease the time required to upload to the 
object store.  Therefore, multi-part uploads are not employed solely to upload 
files larger than 5 GB.

Have you tried to upload the file to Riak CS using s3cmd outside of CloudStack?

Thanks,
-John

On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>> wrote:

My template is less than 5gb and it still uses multipart.  I actually don't 
think this is a CloudStack issue but more of an issue with Riak-CS.  I was just 
wondering if anybody has specifically ran the setup or knows more about Riak-CS 
and can tell me whats up.  I've been trying off and on for the last week to try 
to get this working with 4.2 but it just keeps failing.  I think I'll just try 
ceph rados gateway if I can't get this working.

Darren


On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
mailto:sanjeev.neelar...@citrix.com>> wrote:
Hi,

Cloudstack with Riak-CS works fine. If we register template more than 5GB we 
use multipart upload to push the files to Riak. But this does not work for 
snapshots and volumes. This will be fixed in CS4.2.1 . But for template 
registration it should work fine.

Thanks,
Sanjeev

-Original Message-
From: Darren Shepherd 
[mailto:darren.s.sheph...@gmail.com]
Sent: Thursday, October 31, 2013 11:18 AM
To: dev@cloudstack.apache.org
Subject: ACS + Riak-CS, works?

Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I 
try when it goes to download and save the template, after its posted a series 
of multi part uploads it ends up sending a 403.  The access log looks like

127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
OpenJDK_64-Bit_Server_VM/20.0-b12"

And then in the error log I always get

2013-10-31 05:41:51.867 [error]
<0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for 
s3 failed. Reason: disconnected

I'm able to interact with Riak-CS with s3cmd just fine but its not using 
multipart. I think I'm going to try to download a new s3cmd that does multipart 
and see if even that will work.

Darren

Re: why system.vm.use.local.storage?

[Ahmad Emneina]

you might not have shared local storage in your cloud. so having to add
shared storage for the sole purpose of supporting system vm's would be
painful.


On Thu, Oct 31, 2013 at 10:47 AM, Darren Shepherd <
darren.s.sheph...@gmail.com> wrote:

> Why do we have the configuration system.vm.use.local.storage?  Why would
> somebody care if they systemvm is on local storage?  I'm guessing the idea
> here is if it's on local storage we can't bring it back in a host failure
> situation?  But don't we just recreate the systemvm?
>
> Darren
>

Re: why system.vm.use.local.storage?

[Kelcey Jamison Damage]

That's a good point. I know the system VMs are auto created if the system is 
aware of their state being either 'Destroyed', 'Expunged'. If the state is 
'Starting', or 'Started'/'Running' then no actions are taken. 

In your scenario, the system VM would still be 'Started' in the database even 
though communication is lost to the host, so no automatic action would be 
taken. 

- Original Message -

From: "Darren Shepherd"  
To: dev@cloudstack.apache.org 
Sent: Thursday, October 31, 2013 10:47:41 AM 
Subject: why system.vm.use.local.storage? 

Why do we have the configuration system.vm.use.local.storage? Why would 
somebody care if they systemvm is on local storage? I'm guessing the idea 
here is if it's on local storage we can't bring it back in a host failure 
situation? But don't we just recreate the systemvm? 

Darren 

Re: why system.vm.use.local.storage?

[Ahmad Emneina]

'shared local storage' should be read as shared storage :)


On Thu, Oct 31, 2013 at 10:51 AM, Ahmad Emneina  wrote:

> you might not have shared local storage in your cloud. so having to add
> shared storage for the sole purpose of supporting system vm's would be
> painful.
>
>
> On Thu, Oct 31, 2013 at 10:47 AM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> Why do we have the configuration system.vm.use.local.storage?  Why would
>> somebody care if they systemvm is on local storage?  I'm guessing the idea
>> here is if it's on local storage we can't bring it back in a host failure
>> situation?  But don't we just recreate the systemvm?
>>
>> Darren
>>
>
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

Does this issue exist in 4.2/4.2.1?  I'm going to put the bug as blocker,
if you disagree we can lower it.  I really think it is quite a bad scenario
to mount NFS to the mgmt server.

Darren


On Thu, Oct 31, 2013 at 10:48 AM, Min Chen  wrote:

> Delaying them until ssvm is up may not that easy to handle in current
> CloudStack. We may be able to send an error in this case by distinguishing
> S3 register template case from other scenarios. Would you mind filing a
> JIRA bug on this? Your issue seems different from the issue filed by Daan
> (https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
>
> Thanks
> -min
>
>
> On 10/31/13 10:38 AM, "Darren Shepherd" 
> wrote:
>
> >The behaviour should be to delay until there is a SSVM available.  I don't
> >know if that is even possible in CloudStack, I don't want you to sleep().
> >Is there any way to delay?  Something else in the system should be
> >ensuring
> >that a SSVM is running and available (what I don't know...).  If you can't
> >delay, it is far better in my mind to send an error to the user than to
> >try
> >to do the operation on the mgmt server.
> >
> >I think it is very dangerous to have this code that routes to localhost
> >because in a production setup ACS will have the sudo access to
> >mount/mkdir.  But the server may not have network access, so you run the
> >chance of mounting the NFS and having it hang almost indefinitely which
> >can
> >cause system instability for the mgmt server.
> >
> >The specific issue I have right now is that for some reason on master, the
> >allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
> >restart the mgmt stack, on host connect it tries to download templates so
> >it sends a command and it goes to LocalHostEndPoint.
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:
> >
> >> Yes, that is true, it is a bug in that case. The logic of choosing
> >> endpoint to send Command will not work properly for destroying/Stopping
> >>VM
> >> case. Our assumption is that registering template is done when ssvm is
> >>up
> >> and running. Otherwise, it will send to LocalHost to execute command. In
> >> your opinion, what should be expected behavior if user is trying to
> >> register template when ssvm is crashed? Directly reporting error instead
> >> of trying anything.
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 10:18 AM, "Darren Shepherd" 
> >> wrote:
> >>
> >> >It's happening when I first create the zone.  I'll debug it further to
> >> >tell
> >> >you exactly what its happening.  But I don't understand how the
> >>following
> >> >code in DefaultEndpointSelector could be correct.  If I have a zone,
> >>and I
> >> >delete/stop/crash the SSVM, and at the same time register a new
> >>template,
> >> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >> >
> >> >protected EndPoint findEndpointForImageStorage(DataStore store) {
> >> >Long dcId = null;
> >> >Scope storeScope = store.getScope();
> >> >if (storeScope.getScopeType() == ScopeType.ZONE) {
> >> >dcId = storeScope.getScopeId();
> >> >}
> >> >// find ssvm that can be used to download data to store. For
> >> >zone-wide
> >> >// image store, use SSVM for that zone. For region-wide store,
> >> >// we can arbitrarily pick one ssvm to do that task
> >> >List ssAHosts =
> >> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> >> >if (ssAHosts == null || ssAHosts.isEmpty()) {
> >> >s_logger.info("No running ssvm is found, so command will
> be
> >> >sent to LocalHostEndPoint");
> >> >return LocalHostEndpoint.getEndpoint(); // use local host
> >>as
> >> >endpoint in
> >> >// case of no ssvm existing
> >> >}
> >> >Collections.shuffle(ssAHosts);
> >> >HostVO host = ssAHosts.get(0);
> >> >return
> >>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >> >host.getPrivateIpAddress(),
> >> >host.getPublicIpAddress());
> >> >}
> >> >
> >> >Darren
> >> >
> >> >
> >> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:
> >> >
> >> >> Hi Darren,
> >> >>
> >> >> The logic of sending command to Local in case of no SSVM is
> >>up
> >> >>is
> >> >> to
> >> >> support S3 as secondary storage. In case of S3 as secondary storage,
> >> >> system vm template is automatically downloaded to S3 when S3 is added
> >> >>into
> >> >> CloudStack, unlike NFS secondary storage case where we used a script
> >>to
> >> >> pre-populate system vm template. The thing I don't get is: what
> >>scenario
> >> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
> >> >>
> >> >> Thanks
> >> >> -min
> >> >>
> >> >> On 10/31/13 9:32 AM, "Darren Shepherd" 
> >> >> wrote:
> >> >>
> >> >> >No, that's not the problem.  The problem is that the DownloadCommand
> >> >> >is supposed to go to the SSVM, but it sending it to the
> >> >> >LocalHostEndPoin

Re: ACS + Riak-CS, works?

[John Burwell]

Min,

I apologize for the confusions.  Files greater than 5 GB must uploaded using 
multipart and no part can exceed 5 GB in size.  The original S3 implementation 
in 4.1.0 didn't support multi-part uploads for templates.  Therefore, I didn't 
implement it in the s3xen plugin.  Now that we support it for all other 
operations, the s3xen plugin needs to be retrofitted for multipart.

Thanks,
-John
  
On Oct 31, 2013, at 1:50 PM, Min Chen  wrote:

> Hi John,
> I have a related question on this issue for you regarding RiakCS. In 4.2.0 
> release, backup snapshot to S3 is done through your s3xen plugin, where S3 
> single part upload API is used. If customer is using RiakCS as their S3 
> object store, can they still backup snapshot larger than 5GB? Based on your 
> comments, it seems that they should be able to?
> 
> Thanks
> -min
> 
> From: John Burwell mailto:jburw...@basho.com>>
> Reply-To: mailto:dev@cloudstack.apache.org>>
> Date: Thursday, October 31, 2013 10:30 AM
> To: Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>>
> Cc: "dev@cloudstack.apache.org" 
> mailto:dev@cloudstack.apache.org>>
> Subject: Re: ACS + Riak-CS, works?
> 
> Darren,
> 
> Since S3/Riak CS can receive the contents of a file out of order and 
> re-assemble upon completion of transfer of all parts, the underlying driver 
> uses the TransferManager which automatically breaks files up into multi-part 
> uploads to increase parallelism to decrease the time required to upload to 
> the object store.  Therefore, multi-part uploads are not employed solely to 
> upload files larger than 5 GB.
> 
> Have you tried to upload the file to Riak CS using s3cmd outside of 
> CloudStack?
> 
> Thanks,
> -John
> 
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>> wrote:
> 
> My template is less than 5gb and it still uses multipart.  I actually don't 
> think this is a CloudStack issue but more of an issue with Riak-CS.  I was 
> just wondering if anybody has specifically ran the setup or knows more about 
> Riak-CS and can tell me whats up.  I've been trying off and on for the last 
> week to try to get this working with 4.2 but it just keeps failing.  I think 
> I'll just try ceph rados gateway if I can't get this working.
> 
> Darren
> 
> 
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
> mailto:sanjeev.neelar...@citrix.com>> wrote:
> Hi,
> 
> Cloudstack with Riak-CS works fine. If we register template more than 5GB we 
> use multipart upload to push the files to Riak. But this does not work for 
> snapshots and volumes. This will be fixed in CS4.2.1 . But for template 
> registration it should work fine.
> 
> Thanks,
> Sanjeev
> 
> -Original Message-
> From: Darren Shepherd 
> [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
> 
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I 
> try when it goes to download and save the template, after its posted a series 
> of multi part uploads it ends up sending a 403.  The access log looks like
> 
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 
> And then in the error log I always get
> 
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record 
> for s3 fai

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Min Chen]

Yes, this issue exist in 4.2/4.2.1 since DefaultEndpointSelector was
introduced in 4.2.

Thanks
-min

On 10/31/13 10:56 AM, "Darren Shepherd" 
wrote:

>Does this issue exist in 4.2/4.2.1?  I'm going to put the bug as blocker,
>if you disagree we can lower it.  I really think it is quite a bad
>scenario
>to mount NFS to the mgmt server.
>
>Darren
>
>
>On Thu, Oct 31, 2013 at 10:48 AM, Min Chen  wrote:
>
>> Delaying them until ssvm is up may not that easy to handle in current
>> CloudStack. We may be able to send an error in this case by
>>distinguishing
>> S3 register template case from other scenarios. Would you mind filing a
>> JIRA bug on this? Your issue seems different from the issue filed by
>>Daan
>> (https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
>>
>> Thanks
>> -min
>>
>>
>> On 10/31/13 10:38 AM, "Darren Shepherd" 
>> wrote:
>>
>> >The behaviour should be to delay until there is a SSVM available.  I
>>don't
>> >know if that is even possible in CloudStack, I don't want you to
>>sleep().
>> >Is there any way to delay?  Something else in the system should be
>> >ensuring
>> >that a SSVM is running and available (what I don't know...).  If you
>>can't
>> >delay, it is far better in my mind to send an error to the user than to
>> >try
>> >to do the operation on the mgmt server.
>> >
>> >I think it is very dangerous to have this code that routes to localhost
>> >because in a production setup ACS will have the sudo access to
>> >mount/mkdir.  But the server may not have network access, so you run
>>the
>> >chance of mounting the NFS and having it hang almost indefinitely which
>> >can
>> >cause system instability for the mgmt server.
>> >
>> >The specific issue I have right now is that for some reason on master,
>>the
>> >allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
>> >restart the mgmt stack, on host connect it tries to download templates
>>so
>> >it sends a command and it goes to LocalHostEndPoint.
>> >
>> >Darren
>> >
>> >
>> >On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:
>> >
>> >> Yes, that is true, it is a bug in that case. The logic of choosing
>> >> endpoint to send Command will not work properly for
>>destroying/Stopping
>> >>VM
>> >> case. Our assumption is that registering template is done when ssvm
>>is
>> >>up
>> >> and running. Otherwise, it will send to LocalHost to execute
>>command. In
>> >> your opinion, what should be expected behavior if user is trying to
>> >> register template when ssvm is crashed? Directly reporting error
>>instead
>> >> of trying anything.
>> >>
>> >> Thanks
>> >> -min
>> >>
>> >> On 10/31/13 10:18 AM, "Darren Shepherd" 
>> >> wrote:
>> >>
>> >> >It's happening when I first create the zone.  I'll debug it further
>>to
>> >> >tell
>> >> >you exactly what its happening.  But I don't understand how the
>> >>following
>> >> >code in DefaultEndpointSelector could be correct.  If I have a zone,
>> >>and I
>> >> >delete/stop/crash the SSVM, and at the same time register a new
>> >>template,
>> >> >it seems the DownloadCommand will go to LocalHostEndPoint.
>> >> >
>> >> >protected EndPoint findEndpointForImageStorage(DataStore store)
>>{
>> >> >Long dcId = null;
>> >> >Scope storeScope = store.getScope();
>> >> >if (storeScope.getScopeType() == ScopeType.ZONE) {
>> >> >dcId = storeScope.getScopeId();
>> >> >}
>> >> >// find ssvm that can be used to download data to store. For
>> >> >zone-wide
>> >> >// image store, use SSVM for that zone. For region-wide
>>store,
>> >> >// we can arbitrarily pick one ssvm to do that task
>> >> >List ssAHosts =
>> >> >listUpAndConnectingSecondaryStorageVmHost(dcId);
>> >> >if (ssAHosts == null || ssAHosts.isEmpty()) {
>> >> >s_logger.info("No running ssvm is found, so command will
>> be
>> >> >sent to LocalHostEndPoint");
>> >> >return LocalHostEndpoint.getEndpoint(); // use local
>>host
>> >>as
>> >> >endpoint in
>> >> >// case of no ssvm existing
>> >> >}
>> >> >Collections.shuffle(ssAHosts);
>> >> >HostVO host = ssAHosts.get(0);
>> >> >return
>> >>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
>> >> >host.getPrivateIpAddress(),
>> >> >host.getPublicIpAddress());
>> >> >}
>> >> >
>> >> >Darren
>> >> >
>> >> >
>> >> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen 
>>wrote:
>> >> >
>> >> >> Hi Darren,
>> >> >>
>> >> >> The logic of sending command to Local in case of no SSVM
>>is
>> >>up
>> >> >>is
>> >> >> to
>> >> >> support S3 as secondary storage. In case of S3 as secondary
>>storage,
>> >> >> system vm template is automatically downloaded to S3 when S3 is
>>added
>> >> >>into
>> >> >> CloudStack, unlike NFS secondary storage case where we used a
>>script
>> >>to
>> >> >> pre-populate system vm template. The thing I don't get is: what
>> >>scenario
>> >> >> are you trying to do to trigger DownloadCommand when

Re: ACS + Riak-CS, works?

[Min Chen]

Thanks for your clarification, John. Yes, we have fixed in 4.2.1.

-min

On 10/31/13 10:58 AM, "John Burwell"  wrote:

>Min,
>
>I apologize for the confusions.  Files greater than 5 GB must uploaded
>using multipart and no part can exceed 5 GB in size.  The original S3
>implementation in 4.1.0 didn't support multi-part uploads for templates.
>Therefore, I didn't implement it in the s3xen plugin.  Now that we
>support it for all other operations, the s3xen plugin needs to be
>retrofitted for multipart.
>
>Thanks,
>-John
>  
>On Oct 31, 2013, at 1:50 PM, Min Chen  wrote:
>
>> Hi John,
>> I have a related question on this issue for you regarding RiakCS. In
>>4.2.0 release, backup snapshot to S3 is done through your s3xen plugin,
>>where S3 single part upload API is used. If customer is using RiakCS as
>>their S3 object store, can they still backup snapshot larger than 5GB?
>>Based on your comments, it seems that they should be able to?
>> 
>> Thanks
>> -min
>> 
>> From: John Burwell mailto:jburw...@basho.com>>
>> Reply-To: mailto:dev@cloudstack.apache.org>>
>> Date: Thursday, October 31, 2013 10:30 AM
>> To: Darren Shepherd
>>mailto:darren.s.sheph...@gmail.com>>
>> Cc: "dev@cloudstack.apache.org"
>>mailto:dev@cloudstack.apache.org>>
>> Subject: Re: ACS + Riak-CS, works?
>> 
>> Darren,
>> 
>> Since S3/Riak CS can receive the contents of a file out of order and
>>re-assemble upon completion of transfer of all parts, the underlying
>>driver uses the TransferManager which automatically breaks files up into
>>multi-part uploads to increase parallelism to decrease the time required
>>to upload to the object store.  Therefore, multi-part uploads are not
>>employed solely to upload files larger than 5 GB.
>> 
>> Have you tried to upload the file to Riak CS using s3cmd outside of
>>CloudStack?
>> 
>> Thanks,
>> -John
>> 
>> On Oct 31, 2013, at 1:23 PM, Darren Shepherd
>>mailto:darren.s.sheph...@gmail.com>> wrote:
>> 
>> My template is less than 5gb and it still uses multipart.  I actually
>>don't think this is a CloudStack issue but more of an issue with
>>Riak-CS.  I was just wondering if anybody has specifically ran the setup
>>or knows more about Riak-CS and can tell me whats up.  I've been trying
>>off and on for the last week to try to get this working with 4.2 but it
>>just keeps failing.  I think I'll just try ceph rados gateway if I can't
>>get this working.
>> 
>> Darren
>> 
>> 
>> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu
>>mailto:sanjeev.neelar...@citrix.com>>
>>wrote:
>> Hi,
>> 
>> Cloudstack with Riak-CS works fine. If we register template more than
>>5GB we use multipart upload to push the files to Riak. But this does not
>>work for snapshots and volumes. This will be fixed in CS4.2.1 . But for
>>template registration it should work fine.
>> 
>> Thanks,
>> Sanjeev
>> 
>> -Original Message-
>> From: Darren Shepherd
>>[mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>> 
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>>what I try when it goes to download and save the template, after its
>>posted a series of multi part uploads it ends up sending a 403.  The
>>access log looks like
>> 
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic

Re: S3 Secondary Storage fails with large files

[Min Chen]

Sorry, but you may have to wait for 4.2.1 to come out.

Thanks
-min

On 10/31/13 10:37 AM, "Andrei Mikhailovsky"  wrote:

>
>
>Yes, indeed, this is how i am trying to create the template. I do get the
>same error as I get while trying to snapshot.
>
>Is there a workaround or do i need to wait for the 4.2.1 to come out?
>
>Thanks 
>
>Andrei 
>
>- Original Message -
>
>From: "Min Chen" 
>To: dev@cloudstack.apache.org, us...@cloudstack.apache.org
>Sent: Wednesday, 30 October, 2013 5:39:15 PM
>Subject: Re: S3 Secondary Storage fails with large files
>
>So you are doing create template from volume, right? I think that it
>should go through the same code used in backup snapshot, so will have the
>same limit. Would you mind paste the stack trace you got when you got
>that 
>error in this case?
>
>Thanks 
>-min 
>
>On 10/30/13 10:29 AM, "Andrei Mikhailovsky"  wrote:
>
>>Hi, 
>> 
>>I've just tried creating a template from a 10GB disk image file and it
>>has failed with the same error - EntityTooLarge. So, neither snapshots
>>nor templates over 5GB in size are working for me. Could someone confirm
>>if I am not the only one?
>> 
>>Thanks 
>> 
>>Andrei 
>>- Original Message -
>> 
>>From: "Min Chen" 
>>To: dev@cloudstack.apache.org, us...@cloudstack.apache.org
>>Sent: Wednesday, 30 October, 2013 4:40:48 PM
>>Subject: Re: S3 Secondary Storage fails with large files
>> 
>>For 4.2.0, you should be able to register template > 5GB. But we are
>>aware 
>>of an issue you mentioned here: backup snapshot > 5GB will fail since
>>that 
>>part of code is still using S3 single part upload api. This bug has been
>>fixed in latest 4.2 branch, and will be available in 4.2.1 release.
>> 
>>Thanks 
>>-min 
>> 
>>On 10/30/13 9:18 AM, "Andrei Mikhailovsky"  wrote:
>> 
>>>Hello Guys , 
>>> 
>>>I am using ACS 4.2.0 with KVM hypervisor. While trying to create
>>>templates and snapshots from the existing vms I am getting an error
>>>message / exception on the management logs:
>>> 
>>>2013-10-30 13:29:07,264 DEBUG [storage.snapshot.SnapshotManagerImpl]
>>>(Job-Executor-12:job-41 = [ 85659739-df2e-4afe-b638-2d25e804063f ])
>>>Failed to create snapshot
>>>com.cloud.utils.exception.CloudRuntimeException: Status Code: 400, AWS
>>>Service: Amazon S3, AWS Request ID: null, AWS Error Code:
>>>EntityTooLarge, 
>>>AWS Error Message: null, S3 Extended Request ID: null
>>>at 
>>>com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient
>>>. 
>>>j 
>>>ava:610) 
>>>at 
>>>com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:
>>>3 
>>>1 
>>>0) 
>>>at 
>>>com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>>>at 
>>>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906
>>>) 
>>> 
>>>at 
>>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1
>>>1 
>>>2 
>>>3) 
>>>at 
>>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:9
>>>7 
>>>9 
>>>) 
>>>at com.cloud.utils.S3Utils.putFile(S3Utils.java:143)
>>>at 
>>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToS3(KVMStorage
>>>P 
>>>r 
>>>ocessor.java:556)
>>>at 
>>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToObjectStore(K
>>>V 
>>>M 
>>>StorageProcessor.java:582)
>>>at 
>>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshotForOb
>>>j 
>>>e 
>>>ctStore(KVMStorageProcessor.java:622)
>>>at 
>>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMS
>>>t 
>>>o 
>>>rageProcessor.java:634)
>>>at 
>>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(St
>>>o 
>>>r 
>>>ageSubsystemCommandHandlerBase.java:84)
>>>at 
>>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStor
>>>a 
>>>g 
>>>eCommands(StorageSubsystemCommandHandlerBase.java:49)
>>>at 
>>>com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeReques
>>>t 
>>>( 
>>>LibvirtComputingResource.java:1286)
>>>at com.cloud.agent.Agent.processRequest(Agent.java:525)
>>>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>>>at com.cloud.utils.nio.Task.run(Task.java:83)
>>>at 
>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>a 
>>>: 
>>>1146) 
>>>at 
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>v 
>>>a 
>>>:615) 
>>>at java.lang.Thread.run(Thread.java:679)
>>> 
>>>at 
>>>org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapsho
>>>t 
>>>( 
>>>SnapshotServiceImpl.java:280)
>>>at 
>>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupS
>>>n 
>>>a 
>>>pshot(XenserverSnapshotStrategy.java:138)
>>>at 
>>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSna
>>>p 
>>>s 
>>>hot(XenserverSnapshotStrategy.java:264)
>>>at 
>>>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMana
>>>g 
>>>e 
>>>rImpl.java:1013)
>>>at 
>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercepto
>>>r 
>>>D

Re: ACS + Riak-CS, works?

[Darren Shepherd]

This is actually running in docker and I followed the quickstart [1] to set
it up.  I'll send the config tonight, I don't have access to the machine at
the moment.  I know the 403 seems to indicate the setup is wrong, but all
parts prior to part 18 got a 200 (as you can see in the access log).  So it
would seem that the config is right.  I just don't get why it would
randomly send a 403.  But based on the error message it would seem like it
lost connection to riak to get to user info.

Darren

[1] http://docs.basho.com/riakcs/latest/tutorials/quick-start-riak-cs/

On Thu, Oct 31, 2013 at 10:48 AM, John Burwell  wrote:

> Darren,
>
> Your are receiving a 403 which indicates that the ACLs/user is not
> properly configured.  Which version of Riak CS are you using?  Can you send
> your app.config and vm.args for Riak and Riak CS?  Personally, I use the
> vagrant-riak-cs-cluster [1] project to build local instances for
> development and testing.
>
> Thanks,
> -John
>
> [1]: https://github.com/basho/vagrant-riak-cs-cluster
>
>
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
> wrote:
>
> My template is less than 5gb and it still uses multipart.  I actually
> don't think this is a CloudStack issue but more of an issue with Riak-CS.
> I was just wondering if anybody has specifically ran the setup or knows
> more about Riak-CS and can tell me whats up.  I've been trying off and on
> for the last week to try to get this working with 4.2 but it just keeps
> failing.  I think I'll just try ceph rados gateway if I can't get this
> working.
>
> Darren
>
>
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
> sanjeev.neelar...@citrix.com> wrote:
>
>> Hi,
>>
>> Cloudstack with Riak-CS works fine. If we register template more than 5GB
>> we use multipart upload to push the files to Riak. But this does not work
>> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
>> registration it should work fine.
>>
>> Thanks,
>> Sanjeev
>>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>>
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>> what I try when it goes to download and save the template, after its posted
>> a series of multi part uploads it ends up sending a 403.  The access log
>> looks like
>>
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>>
>> And then in the error log I always get
>>
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
>> record for s3 failed. Reason: disconnected
>>
>> I'm able to interact with Riak-CS with s3cmd just fine but its not using
>> multipart. I think I'm going to try to download a new s3cmd that does
>> multipart and see if even that will work.
>>
>> Darren
>>
>
>
>

Re: ACS + Riak-CS, works?

[John Burwell]

Darren,

One silly question, how much disk space is free in the container?  
Additionally, how much RAM have allocated?  I recommend a minimum of 2 GB for 
testing (memory is consumed largely by the bitcask backend holding keys in 
memory).  When disk space or memory runs out, odd errors can result.  

If you can, I would highly suggest given the vagrant project a go (increase the 
memory in vagrant-overrides.conf to 2 GB).  In addition to be a known good 
configuration, it is a common reference point for support.

Thanks,
-John

On Oct 31, 2013, at 2:14 PM, Darren Shepherd  
wrote:

> This is actually running in docker and I followed the quickstart [1] to set 
> it up.  I'll send the config tonight, I don't have access to the machine at 
> the moment.  I know the 403 seems to indicate the setup is wrong, but all 
> parts prior to part 18 got a 200 (as you can see in the access log).  So it 
> would seem that the config is right.  I just don't get why it would randomly 
> send a 403.  But based on the error message it would seem like it lost 
> connection to riak to get to user info.
> 
> Darren
> 
> [1] http://docs.basho.com/riakcs/latest/tutorials/quick-start-riak-cs/
> 
> On Thu, Oct 31, 2013 at 10:48 AM, John Burwell  wrote:
> Darren,
> 
> Your are receiving a 403 which indicates that the ACLs/user is not properly 
> configured.  Which version of Riak CS are you using?  Can you send your 
> app.config and vm.args for Riak and Riak CS?  Personally, I use the 
> vagrant-riak-cs-cluster [1] project to build local instances for development 
> and testing.  
> 
> Thanks,
> -John
> 
> [1]: https://github.com/basho/vagrant-riak-cs-cluster
> 
> 
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd  
> wrote:
> 
>> My template is less than 5gb and it still uses multipart.  I actually don't 
>> think this is a CloudStack issue but more of an issue with Riak-CS.  I was 
>> just wondering if anybody has specifically ran the setup or knows more about 
>> Riak-CS and can tell me whats up.  I've been trying off and on for the last 
>> week to try to get this working with 4.2 but it just keeps failing.  I think 
>> I'll just try ceph rados gateway if I can't get this working.
>> 
>> Darren
>> 
>> 
>> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
>>  wrote:
>> Hi,
>> 
>> Cloudstack with Riak-CS works fine. If we register template more than 5GB we 
>> use multipart upload to push the files to Riak. But this does not work for 
>> snapshots and volumes. This will be fixed in CS4.2.1 . But for template 
>> registration it should work fine.
>> 
>> Thanks,
>> Sanjeev
>> 
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>> 
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter what 
>> I try when it goes to download and save the template, after its posted a 
>> series of multi part uploads it ends up sending a 403.  The access log looks 
>> like
>> 
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 
>> And then in the error log I always get
>> 
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record 
>> for s3 failed. Reason: disconnected
>> 
>> I'm able to interact with Riak-CS with s3cmd just fine but its not using 
>> multipart. I think I'm going to try to do

[SOLVED]: RE: Issues deploying with marvin

[Santhosh Edukulla]

Instead of going through these many steps,  I believe the other quick solution 
would be to change "sdist" argument to "install" wherever its referencing 
python setup.py, under cs/tools/marvin/pom.xml and run maven command with clean 
install and other arguments. I am assuming it should fix the problem.

Also, added tag solved to the subject line as well.

Santhosh

From: Will Stevens [williamstev...@gmail.com]
Sent: Wednesday, October 30, 2013 12:55 PM
To: dev@cloudstack.apache.org
Cc: Sebastien Goasguen
Subject: Re: Issues deploying with marvin

...Workaround included...

The issue that I run into by not using the 'developer' profile when I do a
'clean install' is that the 'py' files do not get updated in the
 tools/marvin/marvin/cloudstackAPI/ directory.  This is only a problem when
there are changes, but that happens often enough to warrant a work around...

This leads to the original error in this thread when this command runs: "python
../marvin/marvin/deployDataCenter.py -i devcloud.cfg"
Throws an error similar to: "NameError: global name 'addImageStore' is not
defined"
(because the file 'tools/marvin/marvin/cloudstackAPI/addImageStore.py' is
not present)

The following workflow works for me to get around this issue...

Run:
mvn -P developer,systemvm -Dnonoss clean install
(this will fail because of the hard linking of the files between the guest
os and the host os's file system, however, it seems to populate the 'py'
files or at least sets everything up so they will be populated correctly...)

Next run (same thing without the developer profile):
mvn -P systemvm -Dnonoss clean install
(this finishes and gives you a clean build)

Now you can do the deploy db with the developer profile:
mvn -P developer -pl developer,tools/devcloud -Ddeploydb
(this works)

Then startup the server:
export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -Xdebug
-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
mvn -pl :cloud-client-ui jetty:run -D nonoss
(wait for the server to come up)

Then run deploy server with the developer profile:
export MAVEN_OPTS=""
mvn -P developer -pl tools/devcloud -Ddeploysvr
(this now finds everything it needs and completes)

I hope this helps others get past this if they have similar problems...

Cheers...


On Wed, Oct 30, 2013 at 8:56 AM, Will Stevens wrote:

> Yes. I believe the linking is failing because the host os and the guest os
> are different.
>
> Ws
>
> On Wednesday, October 30, 2013, Santhosh Edukulla wrote:
>
>> Based upon the error log below, it seems hard link creation is failing.
>>
>>
>>
>>  hard linking CHANGES.txt -> Marvin-0.1.0
>>
>> > 
>>
>> >  error: Operation not permitted
>>
>>
>>
>> It seems some body earlier has faced a similar issue when using the
>> similar workflow and were not able to create hard links on mounted devices.
>>   They were able to overcome this as well. Check the link below.
>>
>>
>>
>>
>> http://stackoverflow.com/questions/7719380/python-setup-py-sdist-error-operation-not-permitted
>>
>>
>>
>> Thanks!
>>
>> Santhosh
>>
>>
>>
>> -Original Message-
>> From: Prasanna Santhanam [mailto:t...@apache.org]
>> Sent: Wednesday, October 30, 2013 3:48 PM
>> To: Sebastien Goasguen
>> Cc: dev@cloudstack.apache.org; Prasanna Santhanam
>> Subject: Re: Issues deploying with marvin
>>
>>
>>
>> Never tried this with shared dir on vmware fusion but I expect the
>> failure is outside marvin really as Will mentioned. I'm sure this would
>> fail with vbox shared dirs as well.
>>
>>
>>
>> mvn is simply calling 'python setup.py sdist' when doing the build as the
>> user who runs mvn. If that user has permissions to write to your
>> shared-fusion dir, this should ideally have worked.
>>
>>
>>
>> On Wed, Oct 30, 2013 at 04:05:39AM -0400, Sebastien Goasguen wrote:
>>
>> > I am adding Prasanna to the thread as he is the main Marvin developer
>>
>> >
>>
>> > -sebastien
>>
>> >
>>
>> > On Oct 29, 2013, at 1:02 PM, Will Stevens > > wrote:
>>
>> >
>>
>> > > I believe the issues I am running into stem from the fact that the
>>
>> > > CS code is in a shared directory from my Mac to VMware Fusion.
>>
>> > > Because of this the 'ln' command is failing when building marvin...
>>
>> > >
>>
>> > > I may have to find a work around for this problem in order to build
>>
>> > > the developer profile.
>>
>> > >
>>
>> > > Will
>>
>> > >
>>
>> > >
>>
>> > > On Tue, Oct 29, 2013 at 11:59 AM, Will Stevens > >wrote:
>>
>> > >
>>
>> > >> I will pull and make this change because apparently this has been
>>
>> > >> in for a little while...
>>
>> > >>
>>
>> > >> I will update this if with more news when I have it.
>>
>> > >>
>>
>> > >>
>>
>> > >> On Tue, Oct 29, 2013 at 11:57 AM, Will Stevens <
>> wstev...@cloudops.com>wrote:
>>
>> > >>
>>
>> > >>> I know that I will have to change the 'nonoss' flag when I pull in
>>
>> > >>> the latest mas

race conditions in VolumeServiceImpl.createBaseImageAsync() creates NPE

[Darren Shepherd]

The following code results in a NPE in bad situations

templatePoolRef =
_tmpltPoolDao.acquireInLockTable(templatePoolRefId,
storagePoolMaxWaitSeconds);

if (templatePoolRef == null) {
if (s_logger.isDebugEnabled()) {
s_logger.info("Unable to acquire lock on
VMTemplateStoragePool " + templatePoolRefId);
}
templatePoolRef =
_tmpltPoolDao.findByPoolTemplate(dataStore.getId(), template.getId());
if (templatePoolRef.getState() ==
ObjectInDataStoreStateMachine.State.Ready ) {
s_logger.info("Unable to acquire lock on
VMTemplateStoragePool " + templatePoolRefId + ", But Template " +
template.getUniqueName() + " is already copied to primary storage, skip
copying");
createVolumeFromBaseImageAsync(volume,
templateOnPrimaryStoreObj, dataStore, future);
return;
}
throw new CloudRuntimeException("Unable to acquire lock on
VMTemplateStoragePool: " + templatePoolRefId);
}

If two threads are trying to stage the same template thread one gets the
lock, thread two will wait.  If thread one fails to stage the template it
will delete the templatePoolRef from the database.  Thread two will now get
the lock in op_lock, but the internal findById will not find a
templatePoolRef because it has been deleted and return null from
acquireInLockTable().  Technically thread two has the lock, but the ref
templatePoolRef wasn't found.  The subsequent line "templatePoolRef =
_tmpltPoolDao.findByPoolTemplate(...)" will return null, because it doesn't
exist and then on the next line templatePoolRef.getState() will throw a NPE.

Darren

Re: why system.vm.use.local.storage?

[Darren Shepherd]

I just find it annoying setting up a zone with local storage that I have to 
turn this setting on.  If I have no shared storage, then system VMs won't 
deploy with out this parameter.  It's just seems like a useless setting that 
might have been added because there is something else in the system that isn't 
working right.

Darren

> On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage 
>  wrote:
> 
> That's a good point. I know the system VMs are auto created if the system is 
> aware of their state being either 'Destroyed', 'Expunged'. If the state is 
> 'Starting', or 'Started'/'Running' then no actions are taken. 
> 
> In your scenario, the system VM would still be 'Started' in the database even 
> though communication is lost to the host, so no automatic action would be 
> taken. 
> 
> - Original Message -
> 
> From: "Darren Shepherd"  
> To: dev@cloudstack.apache.org 
> Sent: Thursday, October 31, 2013 10:47:41 AM 
> Subject: why system.vm.use.local.storage? 
> 
> Why do we have the configuration system.vm.use.local.storage? Why would 
> somebody care if they systemvm is on local storage? I'm guessing the idea 
> here is if it's on local storage we can't bring it back in a host failure 
> situation? But don't we just recreate the systemvm? 
> 
> Darren 
> 

Re: why system.vm.use.local.storage?

[Kelcey Jamison Damage]

Well perhaps it should be a checkbox on the first zone wizard page, have it 
show up when the user selects 'use local storage'. it could say 'allow system 
VMs on local storage'. 


- Original Message -

From: "Darren Shepherd"  
To: dev@cloudstack.apache.org 
Cc: dev@cloudstack.apache.org 
Sent: Thursday, October 31, 2013 11:46:32 AM 
Subject: Re: why system.vm.use.local.storage? 

I just find it annoying setting up a zone with local storage that I have to 
turn this setting on. If I have no shared storage, then system VMs won't deploy 
with out this parameter. It's just seems like a useless setting that might have 
been added because there is something else in the system that isn't working 
right. 

Darren 

> On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage 
>  wrote: 
> 
> That's a good point. I know the system VMs are auto created if the system is 
> aware of their state being either 'Destroyed', 'Expunged'. If the state is 
> 'Starting', or 'Started'/'Running' then no actions are taken. 
> 
> In your scenario, the system VM would still be 'Started' in the database even 
> though communication is lost to the host, so no automatic action would be 
> taken. 
> 
> - Original Message - 
> 
> From: "Darren Shepherd"  
> To: dev@cloudstack.apache.org 
> Sent: Thursday, October 31, 2013 10:47:41 AM 
> Subject: why system.vm.use.local.storage? 
> 
> Why do we have the configuration system.vm.use.local.storage? Why would 
> somebody care if they systemvm is on local storage? I'm guessing the idea 
> here is if it's on local storage we can't bring it back in a host failure 
> situation? But don't we just recreate the systemvm? 
> 
> Darren 
> 

Re: ACS + Riak-CS, works?

[Darren Shepherd]

Memory is 4gb and disk is about 1gb free (kinda low on disk space).  But the 
system is overtaxed and slow and swapping a bit.  I'll give the vagrant setup a 
try.  

Darren

> On Oct 31, 2013, at 11:29 AM, John Burwell  wrote:
> 
> Darren,
> 
> One silly question, how much disk space is free in the container?  
> Additionally, how much RAM have allocated?  I recommend a minimum of 2 GB for 
> testing (memory is consumed largely by the bitcask backend holding keys in 
> memory).  When disk space or memory runs out, odd errors can result.  
> 
> If you can, I would highly suggest given the vagrant project a go (increase 
> the memory in vagrant-overrides.conf to 2 GB).  In addition to be a known 
> good configuration, it is a common reference point for support.
> 
> Thanks,
> -John
> 
>> On Oct 31, 2013, at 2:14 PM, Darren Shepherd  
>> wrote:
>> 
>> This is actually running in docker and I followed the quickstart [1] to set 
>> it up.  I'll send the config tonight, I don't have access to the machine at 
>> the moment.  I know the 403 seems to indicate the setup is wrong, but all 
>> parts prior to part 18 got a 200 (as you can see in the access log).  So it 
>> would seem that the config is right.  I just don't get why it would randomly 
>> send a 403.  But based on the error message it would seem like it lost 
>> connection to riak to get to user info.
>> 
>> Darren
>> 
>> [1] http://docs.basho.com/riakcs/latest/tutorials/quick-start-riak-cs/
>> 
>>> On Thu, Oct 31, 2013 at 10:48 AM, John Burwell  wrote:
>>> Darren,
>>> 
>>> Your are receiving a 403 which indicates that the ACLs/user is not properly 
>>> configured.  Which version of Riak CS are you using?  Can you send your 
>>> app.config and vm.args for Riak and Riak CS?  Personally, I use the 
>>> vagrant-riak-cs-cluster [1] project to build local instances for 
>>> development and testing.  
>>> 
>>> Thanks,
>>> -John
>>> 
>>> [1]: https://github.com/basho/vagrant-riak-cs-cluster
>>> 
>>> 
 On Oct 31, 2013, at 1:23 PM, Darren Shepherd  
 wrote:
 
 My template is less than 5gb and it still uses multipart.  I actually 
 don't think this is a CloudStack issue but more of an issue with Riak-CS.  
 I was just wondering if anybody has specifically ran the setup or knows 
 more about Riak-CS and can tell me whats up.  I've been trying off and on 
 for the last week to try to get this working with 4.2 but it just keeps 
 failing.  I think I'll just try ceph rados gateway if I can't get this 
 working.
 
 Darren
 
 
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
>  wrote:
> Hi,
> 
> Cloudstack with Riak-CS works fine. If we register template more than 5GB 
> we use multipart upload to push the files to Riak. But this does not work 
> for snapshots and volumes. This will be fixed in CS4.2.1 . But for 
> template registration it should work fine.
> 
> Thanks,
> Sanjeev
> 
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
> 
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter 
> what I try when it goes to download and save the template, after its 
> posted a series of multi part uploads it ends up sending a 403.  The 
> access log looks like
> 
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZnc

Re: Review Request 15108: Grab MacAddress in a more deterministic fashion

[Darren Shepherd]

> On Oct. 31, 2013, 8:29 a.m., Laszlo Hornyak wrote:
> > I haven't had a chance to test it yet, but I prefer this solution over my 
> > own patchset, since I was only trying to fix some broken functionality 
> > without radically changing the behavior.
> > One minor problem I see in there is that still lots of code in static 
> > initializer block, it is hard to write unit tests for it. I think we can 
> > solve that later.

I can't remove the static initialization atm but we can move the contents of 
the static block to a method findDefaultMacAddress that returns the MacAddress. 
 Then the static block would just be the assignment.  That should help with 
unit testing. I wasn't too sure how to unit test this code.  If you want to 
move around the code and see if you can write some tests that would be great.


- Darren


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15108/#review27875
---


On Oct. 31, 2013, 12:18 a.m., Darren Shepherd wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15108/
> ---
> 
> (Updated Oct. 31, 2013, 12:18 a.m.)
> 
> 
> Review request for cloudstack, Alex Huang and Laszlo Hornyak.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> The MAC address is determined in the following order
> 
> 1. Look for MAC that matches interface of cluster.node.IP
> 2. Look for first interface alphabetically that has a non-linklocal address
> 3. Look for first interface alphabetically
> 4. Randomly generate a mac address
> 
> 
> Diffs
> -
> 
>   utils/src/com/cloud/utils/net/MacAddress.java 15350c8 
> 
> Diff: https://reviews.apache.org/r/15108/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Darren Shepherd
> 
>

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Chip Childers]

Permission added.

On Wed, Oct 30, 2013 at 12:19:23PM -0500, Alex Ough wrote:
> And I'd like to write the design document in the wiki page, but I don't
> seem to have a permission to create pages.
> So can anyone give me the permission?
> 
> My account in the wiki is alex.o...@sungard.com
> 
> Thanks in advance.
> Alex Ough
> 
> 
> On Tue, Oct 29, 2013 at 3:38 PM, Alex Ough  wrote:
> 
> > I created a jira for this feature.
> >
> > https://issues.apache.org/jira/browse/CLOUDSTACK-4992
> >
> > But it doesn't allow for me to assign it to myself, so any permission do I
> > need for this?
> > If so, can anyone give me this permission?
> >
> > If there is anything missing, let me know.
> > Thanks
> > Alex Ough
> >
> >
> > On Fri, Oct 18, 2013 at 9:30 AM, Kishan Kavala 
> > wrote:
> >
> >> > -Original Message-
> >> > From: Alex Ough [mailto:alex.o...@sungard.com]
> >> > Sent: Thursday, 17 October 2013 11:25 PM
> >> > To: dev@cloudstack.apache.org; u...@cloudstack.apache.org
> >> > Subject: Fwd: [DISCUSS] Domain/Account/User Sync Up Among Multiple
> >> > Regions
> >> >
> >> > All,
> >> >
> >> > Currently, under the environment of cloudstack with multiple regions,
> >> each
> >> > region has its own management server running with a separate database.
> >> So if
> >> > we want to support multiple regions and provide one point of entry for a
> >> > customer, we need to duplicate domain/account/user information of that
> >> > customer to all of the databases of regions the customer accesses,
> >> which will
> >> > cause data discrepancies when users update those data independently in
> >> each
> >> > management server.
> >> >
> >> > So I'd like to provide a way to sync up the data using the messaging
> >> system
> >> > introduced in 4.1.0. Using the events from each management server,
> >> updates
> >> > from each region can be propagated to the rest regions and they can be
> >> > executed accordingly.
> >> >
> >> > I hope you guys have a chance to think about this and give some
> >> feedbacks if
> >> > interested.
> >> > Thanks in advance.
> >> > Alex Ough
> >>
> >> [KK] Alex, it was discussed sometime back. Related thread [1]. Sync up
> >> using messaging system is the right way to go.
> >>
> >>
> >> [1]
> >> http://www.mail-archive.com/cloudstack-dev@incubator.apache.org/msg20193.html
> >>
> >>
> >

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Chip Childers]

(dropping users@ - try not to cross post please)

On Thu, Oct 31, 2013 at 12:13:18PM -0500, Alex Ough wrote:
> All,
> 
> While I'm waiting for the permissions, I'd like to bring up a discussion on
> the overall architecture to support this feature.
> 
> There can be 2 different approaches as below.
> 1. master - slave architecture : the manual changes are allowed only in one
> master management server, and those in other servers are either prohibited
> or discarded.
> 2. multiple source architecture : all management servers allow manual
> changes and any change in any server will be propagated to the rest of
> servers.
> 
> I'm not sure if we can impose the restriction of #1.
> If not, #2 is a little more complicated, but the only issue is to
> distinguish the events from either manual jobs or triggered automatic
> processing, which I'm currently working on.
> 
> Your comments/recommendations will be very appreciated.
> Thanks
> Alex Ough

Marcus - you might want to track this discussion, since I know that you
were surprised that there wasn't a native sync between regions as part
of the initial region feature.

-chip

Re: why system.vm.use.local.storage?

[Ahmad Emneina]

I agree, it was probably an afterthought. I was just justifying the desired
effect not the flag/workaround.


On Thu, Oct 31, 2013 at 11:46 AM, Darren Shepherd <
darren.s.sheph...@gmail.com> wrote:

> I just find it annoying setting up a zone with local storage that I have
> to turn this setting on.  If I have no shared storage, then system VMs
> won't deploy with out this parameter.  It's just seems like a useless
> setting that might have been added because there is something else in the
> system that isn't working right.
>
> Darren
>
> > On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage <
> kel...@backbonetechnology.com> wrote:
> >
> > That's a good point. I know the system VMs are auto created if the
> system is aware of their state being either 'Destroyed', 'Expunged'. If the
> state is 'Starting', or 'Started'/'Running' then no actions are taken.
> >
> > In your scenario, the system VM would still be 'Started' in the database
> even though communication is lost to the host, so no automatic action would
> be taken.
> >
> > - Original Message -
> >
> > From: "Darren Shepherd" 
> > To: dev@cloudstack.apache.org
> > Sent: Thursday, October 31, 2013 10:47:41 AM
> > Subject: why system.vm.use.local.storage?
> >
> > Why do we have the configuration system.vm.use.local.storage? Why would
> > somebody care if they systemvm is on local storage? I'm guessing the idea
> > here is if it's on local storage we can't bring it back in a host failure
> > situation? But don't we just recreate the systemvm?
> >
> > Darren
> >
>

Re: why system.vm.use.local.storage?

[Marcus Sorensen]

Well, certainly you don't want system vms using local storage by
default. How would you live migrate them for host maintenance? I
assume you mean that when you set up a zone and check 'local storage',
and it pops up the big long thing stating "make sure you set the
system vms to local as well in the global config", that pop up should
not exist and happen by default. But the user has already selected
local storage in that case.

On Thu, Oct 31, 2013 at 1:09 PM, Ahmad Emneina  wrote:
> I agree, it was probably an afterthought. I was just justifying the desired
> effect not the flag/workaround.
>
>
> On Thu, Oct 31, 2013 at 11:46 AM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> I just find it annoying setting up a zone with local storage that I have
>> to turn this setting on.  If I have no shared storage, then system VMs
>> won't deploy with out this parameter.  It's just seems like a useless
>> setting that might have been added because there is something else in the
>> system that isn't working right.
>>
>> Darren
>>
>> > On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage <
>> kel...@backbonetechnology.com> wrote:
>> >
>> > That's a good point. I know the system VMs are auto created if the
>> system is aware of their state being either 'Destroyed', 'Expunged'. If the
>> state is 'Starting', or 'Started'/'Running' then no actions are taken.
>> >
>> > In your scenario, the system VM would still be 'Started' in the database
>> even though communication is lost to the host, so no automatic action would
>> be taken.
>> >
>> > - Original Message -
>> >
>> > From: "Darren Shepherd" 
>> > To: dev@cloudstack.apache.org
>> > Sent: Thursday, October 31, 2013 10:47:41 AM
>> > Subject: why system.vm.use.local.storage?
>> >
>> > Why do we have the configuration system.vm.use.local.storage? Why would
>> > somebody care if they systemvm is on local storage? I'm guessing the idea
>> > here is if it's on local storage we can't bring it back in a host failure
>> > situation? But don't we just recreate the systemvm?
>> >
>> > Darren
>> >
>>

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Alex Ough]

Thanks Chip, and can you also give a permission in Jira so that I can
assign myself in its jira?

Alex Ough


On Thu, Oct 31, 2013 at 2:00 PM, Chip Childers wrote:

> Permission added.
>
> On Wed, Oct 30, 2013 at 12:19:23PM -0500, Alex Ough wrote:
> > And I'd like to write the design document in the wiki page, but I don't
> > seem to have a permission to create pages.
> > So can anyone give me the permission?
> >
> > My account in the wiki is alex.o...@sungard.com
> >
> > Thanks in advance.
> > Alex Ough
> >
> >
> > On Tue, Oct 29, 2013 at 3:38 PM, Alex Ough 
> wrote:
> >
> > > I created a jira for this feature.
> > >
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4992
> > >
> > > But it doesn't allow for me to assign it to myself, so any permission
> do I
> > > need for this?
> > > If so, can anyone give me this permission?
> > >
> > > If there is anything missing, let me know.
> > > Thanks
> > > Alex Ough
> > >
> > >
> > > On Fri, Oct 18, 2013 at 9:30 AM, Kishan Kavala <
> kishan.kav...@citrix.com>wrote:
> > >
> > >> > -Original Message-
> > >> > From: Alex Ough [mailto:alex.o...@sungard.com]
> > >> > Sent: Thursday, 17 October 2013 11:25 PM
> > >> > To: dev@cloudstack.apache.org; u...@cloudstack.apache.org
> > >> > Subject: Fwd: [DISCUSS] Domain/Account/User Sync Up Among Multiple
> > >> > Regions
> > >> >
> > >> > All,
> > >> >
> > >> > Currently, under the environment of cloudstack with multiple
> regions,
> > >> each
> > >> > region has its own management server running with a separate
> database.
> > >> So if
> > >> > we want to support multiple regions and provide one point of entry
> for a
> > >> > customer, we need to duplicate domain/account/user information of
> that
> > >> > customer to all of the databases of regions the customer accesses,
> > >> which will
> > >> > cause data discrepancies when users update those data independently
> in
> > >> each
> > >> > management server.
> > >> >
> > >> > So I'd like to provide a way to sync up the data using the messaging
> > >> system
> > >> > introduced in 4.1.0. Using the events from each management server,
> > >> updates
> > >> > from each region can be propagated to the rest regions and they can
> be
> > >> > executed accordingly.
> > >> >
> > >> > I hope you guys have a chance to think about this and give some
> > >> feedbacks if
> > >> > interested.
> > >> > Thanks in advance.
> > >> > Alex Ough
> > >>
> > >> [KK] Alex, it was discussed sometime back. Related thread [1]. Sync up
> > >> using messaging system is the right way to go.
> > >>
> > >>
> > >> [1]
> > >>
> http://www.mail-archive.com/cloudstack-dev@incubator.apache.org/msg20193.html
> > >>
> > >>
> > >
>
>

Re: haPlanners error in fresh 4.2 checkout

[Srikanteswararao Talluri]

I too faced this issue today. Is there any resolution for this?

Thanks,
~Talluri

On 31/10/13 2:35 pm, "sebgoa"  wrote:

>
>On Oct 29, 2013, at 7:56 PM, Nitin Mehta  wrote:
>
>> Yeah, that should have cleaned the workspace. Did you see what Prachi
>> recommended below ?
>
>Yes and it's there. I do a fresh 4.2 checkout
>
>it compiles fine with:
>
>mvn -Pdeveloper -Dsimulator -DskipTests clean install
>
>But when I run with:
>
>mvn -pl client jetty:run
>
>I get this:
>
>2013-10-31 10:01:31.403:INFO::Started SelectChannelConnector@0.0.0.0:8080
>[INFO] Started Jetty Server
>Exception in thread "Timer-2"
>org.springframework.beans.factory.BeanExpressionException: Expression
>parsing failed; nested exception is
>org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos
>0): Field or property 'haPlanners' cannot be found on object of type
>'org.springframework.beans.factory.config.BeanExpressionContext'
>   at 
>org.springframework.context.expression.StandardBeanExpressionResolver.eval
>uate(StandardBeanExpressionResolver.java:142)
>   at 
>org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBean
>DefinitionString(AbstractBeanFactory.java:1299)
>   at 
>org.springframework.beans.factory.support.BeanDefinitionValueResolver.eval
>uate(BeanDefinitionValueResolver.java:210)
>   at 
>org.springframework.beans.factory.support.BeanDefinitionValueResolver.reso
>lveValueIfNecessary(BeanDefinitionValueResolver.java:182)
>   at 
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto
>ry.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
>   at 
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto
>ry.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
>   at 
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto
>ry.configureBean(AbstractAutowireCapableBeanFactory.java:314)
>   at 
>com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(Compone
>ntContext.java:71)
>   at 
>com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:54)
>   at java.util.TimerThread.mainLoop(Timer.java:555)
>   at java.util.TimerThread.run(Timer.java:505)
>Caused by: org.springframework.expression.spel.SpelEvaluationException:
>EL1008E:(pos 0): Field or property 'haPlanners' cannot be found on object
>of type 'org.springframework.beans.factory.config.BeanExpressionContext'
>   at 
>org.springframework.expression.spel.ast.PropertyOrFieldReference.readPrope
>rty(PropertyOrFieldReference.java:208)
>   at 
>org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueI
>nternal(PropertyOrFieldReference.java:72)
>   at 
>org.springframework.expression.spel.ast.CompoundExpression.getValueInterna
>l(CompoundExpression.java:52)
>   at 
>org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl
>.java:93)
>   at 
>org.springframework.expression.spel.standard.SpelExpression.getValue(SpelE
>xpression.java:88)
>   at 
>org.springframework.context.expression.StandardBeanExpressionResolver.eval
>uate(StandardBeanExpressionResolver.java:139)
>   ... 10 more
>ERROR [cloud.api.ApiServlet] (770998563@qtp-534473248-6:) unknown
>exception writing api response
>java.lang.NullPointerException
>   at com.cloud.api.ApiServer.loginUser(ApiServer.java:770)
>   at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:211)
>   at com.cloud.api.ApiServlet.doPost(ApiServlet.java:71)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>   at 
>org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>   at 
>org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216
>)
>   at 
>org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>   at 
>org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>   at 
>org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo
>llection.java:230)
>   at 
>org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
>114)
>   at 
>org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>   at org.mortbay.jetty.Server.handle(Server.java:326)
>   at 
>org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>   at 
>org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.jav
>a:945)
>   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>   at 
>org.mortbay.io.nio.SelectChannelEndPoin

Re: why system.vm.use.local.storage?

[Ahmad Emneina]

basic networks dont really require that level of flexibility. Sure you
suffer a small dns/dhcp outage... So it definitely depends on the purpose
of the cloud, and services its going to provide. We cant negate the
flexibility local storage system vm's provides.


On Thu, Oct 31, 2013 at 12:24 PM, Marcus Sorensen wrote:

> Well, certainly you don't want system vms using local storage by
> default. How would you live migrate them for host maintenance? I
> assume you mean that when you set up a zone and check 'local storage',
> and it pops up the big long thing stating "make sure you set the
> system vms to local as well in the global config", that pop up should
> not exist and happen by default. But the user has already selected
> local storage in that case.
>
> On Thu, Oct 31, 2013 at 1:09 PM, Ahmad Emneina  wrote:
> > I agree, it was probably an afterthought. I was just justifying the
> desired
> > effect not the flag/workaround.
> >
> >
> > On Thu, Oct 31, 2013 at 11:46 AM, Darren Shepherd <
> > darren.s.sheph...@gmail.com> wrote:
> >
> >> I just find it annoying setting up a zone with local storage that I have
> >> to turn this setting on.  If I have no shared storage, then system VMs
> >> won't deploy with out this parameter.  It's just seems like a useless
> >> setting that might have been added because there is something else in
> the
> >> system that isn't working right.
> >>
> >> Darren
> >>
> >> > On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage <
> >> kel...@backbonetechnology.com> wrote:
> >> >
> >> > That's a good point. I know the system VMs are auto created if the
> >> system is aware of their state being either 'Destroyed', 'Expunged'. If
> the
> >> state is 'Starting', or 'Started'/'Running' then no actions are taken.
> >> >
> >> > In your scenario, the system VM would still be 'Started' in the
> database
> >> even though communication is lost to the host, so no automatic action
> would
> >> be taken.
> >> >
> >> > - Original Message -
> >> >
> >> > From: "Darren Shepherd" 
> >> > To: dev@cloudstack.apache.org
> >> > Sent: Thursday, October 31, 2013 10:47:41 AM
> >> > Subject: why system.vm.use.local.storage?
> >> >
> >> > Why do we have the configuration system.vm.use.local.storage? Why
> would
> >> > somebody care if they systemvm is on local storage? I'm guessing the
> idea
> >> > here is if it's on local storage we can't bring it back in a host
> failure
> >> > situation? But don't we just recreate the systemvm?
> >> >
> >> > Darren
> >> >
> >>
>

Re: Tiered Quality

[Daan Hoogland]

one note on testing guys,

I see that the analysis site give lines-coverage and branch-coverage.
I don't see anything on distinct paths. What I mean is that the
program
if(a)
 A
else
 B
if(b)
 C
else
 D
if(c)
 E
else
 F
has eight (2^3) distict paths. It is not enough to show that
A,B,C,D,E,F are all hit and hance every line and branch. Also all
combinations of a/!a and b/!b and c/!c need to be hit.

Now I am not saying that we should not score our code if not in this
way but it is kind of kidding ourselves if we don't face up to the
fact that coverage of lines of code or branches is not a completeness
criterium of some kind. I don't know if any of the mentioned tools
does analysis this thorough. But if any does we should go for that
one.

€0,02
Daan

On Tue, Oct 29, 2013 at 2:21 AM, Darren Shepherd
 wrote:
> Starting with the honor system might be good.  It's not so easy some times to 
> relate lines of code to functionality.  Also just because it hits a line of 
> code doesn't mean it's really tested.
>
> Can't we just get people to just put a check mark on some table in the wiki?
>
> Darren
>
>> On Oct 28, 2013, at 12:08 PM, Santhosh Edukulla 
>>  wrote:
>>
>> 1.It seems we already have a code coverage numbers using sonar as below. It 
>> currently shows only the numbers for unit tests.
>>
>> https://analysis.apache.org/dashboard/index/100206
>>
>> 2. The below link has an explanation for using it for both integration and 
>> unit tests.
>>
>> http://docs.codehaus.org/display/SONAR/Code+Coverage+by+Integration+Tests+for+Java+Project
>>
>> 3. Many links suggests it has good decision coverage facility compared to 
>> other coverage tools.
>>
>> http://onlysoftware.wordpress.com/2012/12/19/code-coverage-tools-jacoco-cobertura-emma-comparison-in-sonar/
>>
>> Regards,
>> Santhosh
>> 
>> From: Laszlo Hornyak [laszlo.horn...@gmail.com]
>> Sent: Monday, October 28, 2013 1:43 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Tiered Quality
>>
>> Sonar already tracks the unit test coverage. It is also able to track the
>> integration test coverage, however this might be a bit more sophisticated
>> in CS since not all hardware/software requirements are available in the
>> jenkins environment. However, this could be a problem in any environment.
>>
>>
>>> On Mon, Oct 28, 2013 at 5:53 AM, Prasanna Santhanam  wrote:
>>>
>>> We need a way to check coverage of (unit+integration) tests. How many
>>> lines of code hit on a deployed system that corresponds to the
>>> component donated/committed. We don't have that for existing tests so
>>> it makes it hard to judge if a feature that comes with tests covers
>>> enough of itself.
>>>
 On Sun, Oct 27, 2013 at 11:00:46PM +0100, Laszlo Hornyak wrote:
 Ok, makes sense, but that sounds like even more work :) Can you share the
 plan on how will this work?


 On Sun, Oct 27, 2013 at 7:54 PM, Darren Shepherd <
 darren.s.sheph...@gmail.com> wrote:

> I think it can't be at a component level because components are too
>>> large.
> It needs to be at a feature for implementation level.  For example,
>>> live
> storage migration for xen and live storage migration for kvm (don't
>>> know if
> that's a real thing) would be two separate items.
>
> Darren
>
>> On Oct 27, 2013, at 10:57 AM, Laszlo Hornyak <
>>> laszlo.horn...@gmail.com>
> wrote:
>>
>> I believe this will be very useful for users.
>> As far as I understand someone will have to qualify components. What
>>> will
>> be the method for qualification? I do not think simply the test
>>> coverage
>> would be right. But then if you want to go deeper, then you need a
>>> bigger
>> effort testing the components.
>>
>>
>>
>> On Sun, Oct 27, 2013 at 4:51 PM, Darren Shepherd <
>> darren.s.sheph...@gmail.com> wrote:
>>
>>> I don't know if a similar thing has been talked about before but I
>>> thought I'd just throws this out there.  The ultimate way to ensure
>>> quality is that we have unit test and integration test coverage on
>>> all
>>> functionality.  That way somebody authors some code, commits to, for
>>> example, 4.2, but then when we release 4.3, 4.4, etc they aren't on
>>> the hook to manually tests the functionality with each release.  The
>>> obvious nature of a community project is that people come and go.
>>> If
>>> a contributor wants to ensure the long term viability of the
>>> component, they should ensure that there are unit+integration tests.
>>>
>>> Now, for whatever reason whether good or bad, it's not always
>>> possible
>>> to have full integration tests.  I don't want to throw down the
>>> gamut
>>> and say everything must have coverage because that will mean some
>>> useful code/feature will not get in because of some coverage wasn't
>>> possible at the time.
>>>
>>> What I propose is that for e

Re: [DISCUSS] Leaky abstractions [was review requests 13238, 13896, 14320]

[Daan Hoogland]

Nicolas,

Please continue the discussion either in the original thread or start a new
one. I think the subject deserves more discussion.

regards,
Daan


On Tue, Oct 29, 2013 at 2:56 PM,  wrote:

> Hello,
>
> Concerning the last message
> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201310.mbox/%3ccagqtxvyvzplcycge1vmwuxfvt5px0xdxwxmb5bwv5edcewu...@mail.gmail.com%3e
>
> and the following patches
> https://reviews.apache.org/r/13238/
>
> https://reviews.apache.org/r/14320/
>
> https://reviews.apache.org/r/13896/
> a wiki discussion seems to be launched and/or done.
>
>
> Does a such discussion exist on the wiki ? And where could we access it ?
> Otherwise, how could we start it ?
>
> Have a good day, 
>
> ** **
>
> Best regards,
>
>
> 
>
> [image: cid:image001.gif@01CDFFC9.76D78E80]
>
> **
>
> *Nicolas Foata*
>
> O/OF/DSIF/DFY/SDFY/RnB
>
> Sophia Antipolis
> FRANCE
>
> [image: cid:image002.gif@01CDFFC9.76D78E80]
>
> ** **
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

Re: Review Request 15050: Add Palo Alto Networks Firewall Integration

[Will Stevens]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15050/
---

(Updated Oct. 31, 2013, 8:45 p.m.)


Review request for cloudstack and Sheng Yang.


Changes
---

Turned off the detailed unit test logging by default.  Fixed a couple issues I 
thought had gotten in the original patch.


Bugs: CLOUDSTACK-1275
https://issues.apache.org/jira/browse/CLOUDSTACK-1275


Repository: cloudstack-git


Description
---

More information in the FS: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Palo+Alto+Firewall+Integration

This patch adds a network plugin which adds support for the Palo Alto Networks 
firewall (their appliance and their VM series firewall).

Features supported are:
- List/Add/Delete Palo Alto service provider
- List/Add/Delete Palo Alto network service offering
- List/Add/Delete Palo Alto network with above service offering
- Add instance to the new network (creates the public IP and private 
gateway/cidr on the PA as well as the source nat rule)
- List/Add/Delete Ingress Firewall rule
- List/Add/Delete Egress Firewall rule
- List/Add/Delete Port Forwarding rule
- List/Add/Delete Static Nat rule
- Supports Palo Alto Networks 'Log Forwarding' profile globally per device 
(additional docs to come)
- Supports Palo Alto Networks 'Security Profile Groups' functionality globally 
per device (additional docs to come)

Knowns limitations:
- Only supports one public IP range in CloudStack.
- Currently not verifying SSL certificates when creating a connection between 
CloudStack and the Palo Alto Networks firewall.
- Currently not tracking usage on Public IPs.


Diffs (updated)
-

  api/src/com/cloud/network/Network.java 49f380b 
  
api/src/org/apache/cloudstack/api/command/admin/network/AddNetworkDeviceCmd.java
 4983255 
  
api/src/org/apache/cloudstack/api/command/admin/network/ListNetworkDeviceCmd.java
 0b7836d 
  api/src/org/apache/cloudstack/network/ExternalNetworkDeviceManager.java 
29ce2e3 
  client/WEB-INF/classes/resources/messages.properties c075bf8 
  client/pom.xml fd1f13a 
  client/tomcatconf/commands.properties.in 96e841a 
  client/tomcatconf/nonossComponentContext.xml.in 0502bbc 
  plugins/network-elements/palo-alto/pom.xml PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddExternalFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddPaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ConfigurePaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeleteExternalFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeletePaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListExternalFirewallsCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallNetworksCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallsCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/response/PaloAltoFirewallResponse.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoExternalFirewallElement.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoFirewallElementService.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/resource/PaloAltoResource.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/utils/HttpClientWrapper.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/test/com/cloud/network/resource/MockablePaloAltoResource.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/test/com/cloud/network/resource/PaloAltoResourceTest.java
 PRE-CREATION 
  plugins/pom.xml ca41dff 
  server/src/com/cloud/api/ApiResponseHelper.java f4ca112 
  server/src/com/cloud/configuration/ConfigurationManagerImpl.java 4fda3b1 
  ui/dictionary.jsp 80aab6f 
  ui/scripts/docs.js 3a4f8ca 
  ui/scripts/system.js 0af3952 

Diff: https://reviews.apache.org/r/15050/diff/


Testing
---

I have tested all of the functionality listed under 'supported features'.


Thanks,

Will Stevens

Re: Review Request 15050: Add Palo Alto Networks Firewall Integration

[Will Stevens]

> On Oct. 30, 2013, 5:34 p.m., Sheng Yang wrote:
> > Ship It!
> 
> Sheng Yang wrote:
> No shipping for now.
> 
> Sorry, press the wrong one. I meant to ship the other patch.
> 
> Will, since it's a big change, could you send out a notice to the 
> community about this merge?
> 
> Thanks!
> 
> Will Stevens wrote:
> Yes, no problem...  What type of information would you like to be 
> included in this notice?
> 
> Thx...
> 
> Sheng Yang wrote:
> It would looks like e.g. 
> http://www.mail-archive.com/dev@cloudstack.apache.org/msg17259.html , and the 
> history of patch/related discussion is needed as well.
> 
> The purpose is to let community notice that there is a new feature would 
> be in, and check on it.
> 
> Community would vote on it and after the vote pass we can merge it.
> 
> And I think it would need some test cases or unit tests for such a big 
> merge.
> 
>

Great!  Thank you.  I will put that together.


- Will


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15050/#review27794
---


On Oct. 31, 2013, 8:45 p.m., Will Stevens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15050/
> ---
> 
> (Updated Oct. 31, 2013, 8:45 p.m.)
> 
> 
> Review request for cloudstack and Sheng Yang.
> 
> 
> Bugs: CLOUDSTACK-1275
> https://issues.apache.org/jira/browse/CLOUDSTACK-1275
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> More information in the FS: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Palo+Alto+Firewall+Integration
> 
> This patch adds a network plugin which adds support for the Palo Alto 
> Networks firewall (their appliance and their VM series firewall).
> 
> Features supported are:
> - List/Add/Delete Palo Alto service provider
> - List/Add/Delete Palo Alto network service offering
> - List/Add/Delete Palo Alto network with above service offering
> - Add instance to the new network (creates the public IP and private 
> gateway/cidr on the PA as well as the source nat rule)
> - List/Add/Delete Ingress Firewall rule
> - List/Add/Delete Egress Firewall rule
> - List/Add/Delete Port Forwarding rule
> - List/Add/Delete Static Nat rule
> - Supports Palo Alto Networks 'Log Forwarding' profile globally per device 
> (additional docs to come)
> - Supports Palo Alto Networks 'Security Profile Groups' functionality 
> globally per device (additional docs to come)
> 
> Knowns limitations:
> - Only supports one public IP range in CloudStack.
> - Currently not verifying SSL certificates when creating a connection between 
> CloudStack and the Palo Alto Networks firewall.
> - Currently not tracking usage on Public IPs.
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/network/Network.java 49f380b 
>   
> api/src/org/apache/cloudstack/api/command/admin/network/AddNetworkDeviceCmd.java
>  4983255 
>   
> api/src/org/apache/cloudstack/api/command/admin/network/ListNetworkDeviceCmd.java
>  0b7836d 
>   api/src/org/apache/cloudstack/network/ExternalNetworkDeviceManager.java 
> 29ce2e3 
>   client/WEB-INF/classes/resources/messages.properties c075bf8 
>   client/pom.xml fd1f13a 
>   client/tomcatconf/commands.properties.in 96e841a 
>   client/tomcatconf/nonossComponentContext.xml.in 0502bbc 
>   plugins/network-elements/palo-alto/pom.xml PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddExternalFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddPaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ConfigurePaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeleteExternalFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeletePaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListExternalFirewallsCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallNetworksCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallsCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/response/PaloAltoFirewallResponse.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoExternalFirewallElement.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoFirewallElementService.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/

Re: Review Request 14549: Rename net.juniper.contrail to org.apache.cloudstack.network.contrail

[Pedro Marques]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14549/
---

(Updated Oct. 31, 2013, 8:46 p.m.)


Review request for cloudstack and Hugo Trippaers.


Changes
---

Adding Hugo as reviewer.


Repository: cloudstack-git


Description
---

Rename net.juniper.contrail to org.apache.cloudstack.network.contrail.


Diffs
-

  api/src/com/cloud/network/Network.java 49f380b 
  client/pom.xml fd1f13a 
  client/tomcatconf/commands.properties.in 0296de0 
  client/tomcatconf/componentContext.xml.in df5b002 
  
plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/CitrixResourceBase.java
 f16a6f5 
  plugins/network-elements/juniper-contrail/pom.xml PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/command/CreateServiceInstanceCmd.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/response/ServiceInstanceResponse.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElement.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElementImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailGuru.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManager.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManagerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/DBSyncGeneric.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/EventUtils.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ManagementNetworkGuru.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ModelDatabase.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSync.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSyncImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandler.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandlerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManager.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManagerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceVirtualMachine.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpPoolModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/InstanceIpModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelController.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObject.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObjectBase.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ServiceInstanceModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VMInterfaceModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VirtualMachineModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VirtualNetworkModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/management/ApiConnectorMockito.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/management/IntegrationTestConfiguration.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/manage

Re: Review Request 15050: Add Palo Alto Networks Firewall Integration

[Will Stevens]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15050/
---

(Updated Oct. 31, 2013, 8:48 p.m.)


Review request for cloudstack and Sheng Yang.


Bugs: CLOUDSTACK-1275
https://issues.apache.org/jira/browse/CLOUDSTACK-1275


Repository: cloudstack-git


Description
---

More information in the FS: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Palo+Alto+Firewall+Integration

This patch adds a network plugin which adds support for the Palo Alto Networks 
firewall (their appliance and their VM series firewall).

Features supported are:
- List/Add/Delete Palo Alto service provider
- List/Add/Delete Palo Alto network service offering
- List/Add/Delete Palo Alto network with above service offering
- Add instance to the new network (creates the public IP and private 
gateway/cidr on the PA as well as the source nat rule)
- List/Add/Delete Ingress Firewall rule
- List/Add/Delete Egress Firewall rule
- List/Add/Delete Port Forwarding rule
- List/Add/Delete Static Nat rule
- Supports Palo Alto Networks 'Log Forwarding' profile globally per device 
(additional docs to come)
- Supports Palo Alto Networks 'Security Profile Groups' functionality globally 
per device (additional docs to come)

Knowns limitations:
- Only supports one public IP range in CloudStack.
- Currently not verifying SSL certificates when creating a connection between 
CloudStack and the Palo Alto Networks firewall.
- Currently not tracking usage on Public IPs.


Diffs
-

  api/src/com/cloud/network/Network.java 49f380b 
  
api/src/org/apache/cloudstack/api/command/admin/network/AddNetworkDeviceCmd.java
 4983255 
  
api/src/org/apache/cloudstack/api/command/admin/network/ListNetworkDeviceCmd.java
 0b7836d 
  api/src/org/apache/cloudstack/network/ExternalNetworkDeviceManager.java 
29ce2e3 
  client/WEB-INF/classes/resources/messages.properties c075bf8 
  client/pom.xml fd1f13a 
  client/tomcatconf/commands.properties.in 96e841a 
  client/tomcatconf/nonossComponentContext.xml.in 0502bbc 
  plugins/network-elements/palo-alto/pom.xml PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddExternalFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddPaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ConfigurePaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeleteExternalFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeletePaloAltoFirewallCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListExternalFirewallsCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallNetworksCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallsCmd.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/api/response/PaloAltoFirewallResponse.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoExternalFirewallElement.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/element/PaloAltoFirewallElementService.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/resource/PaloAltoResource.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/src/com/cloud/network/utils/HttpClientWrapper.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/test/com/cloud/network/resource/MockablePaloAltoResource.java
 PRE-CREATION 
  
plugins/network-elements/palo-alto/test/com/cloud/network/resource/PaloAltoResourceTest.java
 PRE-CREATION 
  plugins/pom.xml ca41dff 
  server/src/com/cloud/api/ApiResponseHelper.java f4ca112 
  server/src/com/cloud/configuration/ConfigurationManagerImpl.java 4fda3b1 
  ui/dictionary.jsp 80aab6f 
  ui/scripts/docs.js 3a4f8ca 
  ui/scripts/system.js 0af3952 

Diff: https://reviews.apache.org/r/15050/diff/


Testing (updated)
---

I have tested all of the functionality listed under 'supported features'.

Unit tests are in file 24 and 25 of this patch.  The details of the tests can 
be found in the functional spec.


Thanks,

Will Stevens

[Merge] Palo Alto Networks firewall integration to master

[Will Stevens]

Hi,

I would like to merge support for Palo Alto Network's firewall appliances
to the master branch.  Development for this has been done by Will Stevens
at CloudOps on branch [1].

There was an introduction [2], a proposal [3], and a discussion [4] on the
mailing list regarding this feature.

Checklist:
Jira ticket for the feature is here [5].
The FS can be found at [6].
Unit tests for the feature are available at [7] and [8].  I have developed
the unit tests with a flag to output additional detail in the console [9].
 Here is the result of the tests without detail [10] and here is the result
of the tests with detail [11].

This plugin communicates to the Palo Alto Networks firewall appliances
through an API documented at [12] with a training manual [13].
This plugin depends on a modification to core to remove a limitation which
was discussed here [14], with this jira issue [15] and has been approved
here [16].
This plugin is being reviewed at [17] according to this patch [18].

There are no 3rd party libraries needed for this plugin, however it does
depend on a 3rd party API [12][13] to orchestrate the configuration on the
appliance.  The plugin is currently being built via the 'nonoss' flag.  It
should be moved into either the 'noredist' or core because it appears that
'nonoss' will be going away [19] and 'noredist' has been merged [20].  I
would appreciate input on which build this should be put into given its
interaction with an 3rd party appliance.

Here are the slides for a presentation [21] given about this integration at
the CloudStack Collaboration Conference in Santa Clara, CA.

[1] https://github.com/cloudops/cs_palo_alto/tree/palo_alto
[2]
http://markmail.org/message/hukydzwkec3dwuxq?q=list:org%2Eapache%2Eincubator%2Ecloudstack-%2A+Palo+Alto
[3]
http://markmail.org/message/odbg2icft7esj3ut?q=list:org%2Eapache%2Eincubator%2Ecloudstack-%2A+Palo+Alto
[4]
http://markmail.org/message/n5276i4hfh7ek57o?q=list:org%2Eapache%2Eincubator%2Ecloudstack-%2A+Palo+Alto
[5] https://issues.apache.org/jira/browse/CLOUDSTACK-1275
[6]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Palo+Alto+Firewall+Integration
[7]
https://github.com/cloudops/cs_palo_alto/blob/palo_alto/plugins/network-elements/palo-alto/test/com/cloud/network/resource/PaloAltoResourceTest.java
[8]
https://github.com/cloudops/cs_palo_alto/blob/palo_alto/plugins/network-elements/palo-alto/test/com/cloud/network/resource/MockablePaloAltoResource.java
[9]
https://github.com/cloudops/cs_palo_alto/blob/palo_alto/plugins/network-elements/palo-alto/test/com/cloud/network/resource/PaloAltoResourceTest.java#L156
[10]
https://cwiki.apache.org/confluence/download/attachments/30753712/palo_alto_tests_without_logging.txt?version=1&modificationDate=1383248404474
[11]
https://cwiki.apache.org/confluence/download/attachments/30753712/palo_alto_tests_with_logging.txt?version=1&modificationDate=1383248432061
[12]
https://cwiki.apache.org/confluence/download/attachments/30753712/XML-API-5-1.0-RevA.pdf?version=1&modificationDate=1366305634000
[13]
https://cwiki.apache.org/confluence/download/attachments/30753712/XML_API_Training.pdf?version=1&modificationDate=1366305635000
[14]
http://markmail.org/message/374hyn7ko6zrb2cf?q=list:org%2Eapache%2Eincubator%2Ecloudstack-%2A+supported+source+nat+types
[15] https://issues.apache.org/jira/browse/CLOUDSTACK-4991
[16] https://reviews.apache.org/r/15047/
[17] https://reviews.apache.org/r/15050/
[18] https://reviews.apache.org/r/15050/diff/
[19]
http://markmail.org/message/37qcg4lgudmf57ws?q=DISCUSS%5D+rename+nonoss+to+noredist
[20]
http://markmail.org/message/zqkiuod5qabcyra6?q=%5BMERGE%5D+changing+nonoss+to+noredist
[21]
https://cwiki.apache.org/confluence/download/attachments/30753712/CS_PA_Integration.pptx?version=1&modificationDate=1383250830719

Cheers,

Will

Re: Review Request 15050: Add Palo Alto Networks Firewall Integration

[Will Stevens]

> On Oct. 30, 2013, 5:34 p.m., Sheng Yang wrote:
> > Ship It!
> 
> Sheng Yang wrote:
> No shipping for now.
> 
> Sorry, press the wrong one. I meant to ship the other patch.
> 
> Will, since it's a big change, could you send out a notice to the 
> community about this merge?
> 
> Thanks!
> 
> Will Stevens wrote:
> Yes, no problem...  What type of information would you like to be 
> included in this notice?
> 
> Thx...
> 
> Sheng Yang wrote:
> It would looks like e.g. 
> http://www.mail-archive.com/dev@cloudstack.apache.org/msg17259.html , and the 
> history of patch/related discussion is needed as well.
> 
> The purpose is to let community notice that there is a new feature would 
> be in, and check on it.
> 
> Community would vote on it and after the vote pass we can merge it.
> 
> And I think it would need some test cases or unit tests for such a big 
> merge.
> 
>
> 
> Will Stevens wrote:
> Great!  Thank you.  I will put that together.

I have put together these details.  They can be found at: 
http://markmail.org/message/m7ynuc2tbkyxojv3?q=list:org%2Eapache%2Eincubator%2Ecloudstack-%2A+%5BMerge%5D+Palo+Alto+Networks+firewall+integration+to+master


- Will


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15050/#review27794
---


On Oct. 31, 2013, 8:48 p.m., Will Stevens wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15050/
> ---
> 
> (Updated Oct. 31, 2013, 8:48 p.m.)
> 
> 
> Review request for cloudstack and Sheng Yang.
> 
> 
> Bugs: CLOUDSTACK-1275
> https://issues.apache.org/jira/browse/CLOUDSTACK-1275
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> More information in the FS: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Palo+Alto+Firewall+Integration
> 
> This patch adds a network plugin which adds support for the Palo Alto 
> Networks firewall (their appliance and their VM series firewall).
> 
> Features supported are:
> - List/Add/Delete Palo Alto service provider
> - List/Add/Delete Palo Alto network service offering
> - List/Add/Delete Palo Alto network with above service offering
> - Add instance to the new network (creates the public IP and private 
> gateway/cidr on the PA as well as the source nat rule)
> - List/Add/Delete Ingress Firewall rule
> - List/Add/Delete Egress Firewall rule
> - List/Add/Delete Port Forwarding rule
> - List/Add/Delete Static Nat rule
> - Supports Palo Alto Networks 'Log Forwarding' profile globally per device 
> (additional docs to come)
> - Supports Palo Alto Networks 'Security Profile Groups' functionality 
> globally per device (additional docs to come)
> 
> Knowns limitations:
> - Only supports one public IP range in CloudStack.
> - Currently not verifying SSL certificates when creating a connection between 
> CloudStack and the Palo Alto Networks firewall.
> - Currently not tracking usage on Public IPs.
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/network/Network.java 49f380b 
>   
> api/src/org/apache/cloudstack/api/command/admin/network/AddNetworkDeviceCmd.java
>  4983255 
>   
> api/src/org/apache/cloudstack/api/command/admin/network/ListNetworkDeviceCmd.java
>  0b7836d 
>   api/src/org/apache/cloudstack/network/ExternalNetworkDeviceManager.java 
> 29ce2e3 
>   client/WEB-INF/classes/resources/messages.properties c075bf8 
>   client/pom.xml fd1f13a 
>   client/tomcatconf/commands.properties.in 96e841a 
>   client/tomcatconf/nonossComponentContext.xml.in 0502bbc 
>   plugins/network-elements/palo-alto/pom.xml PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddExternalFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/AddPaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ConfigurePaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeleteExternalFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/DeletePaloAltoFirewallCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListExternalFirewallsCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallNetworksCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/commands/ListPaloAltoFirewallsCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/api/response/PaloAltoFirewallResponse.java
>  PRE-CREATION 
>   
> plugins/network-elements/palo-alto/src/com/cloud/netwo

Re: Mailing list search

[Daan Hoogland]

So far I haven't heard back from them. I will check next morning
making [Rr] but last week they where still at incubator in the
markmail indeces. I don't know if that means the markmail subscription
is also on the incubator list

Daan

On Tue, Oct 29, 2013 at 11:37 PM, Chip Childers  wrote:
> Noah,
>
> The last time I checked, markmail still had all traffic against the incubator 
> list names. If that has changed, great. If not, then incubator needs to 
> remain the target of the search.
>
>
>
>> On Oct 29, 2013, at 6:13 PM, Noah Slater  wrote:
>>
>> Bump.
>>
>>> On 25 October 2013 20:58, Noah Slater  wrote:
>>> Hi,
>>>
>>> Mailing list search on our site still points to the incubator:
>>>
>>> http://cloudstack.apache.org/mailing-lists.html
>>>
>>> I recall discussing this in the past with someone and was asked to
>>> hold back, as the Incubator lists still had most of the traffic.
>>>
>>> Is it time we switched this over?
>>>
>>> Thanks,
>>>
>>> --
>>> Noah Slater
>>> https://twitter.com/nslater
>>
>>
>>
>> --
>> Noah Slater
>> https://twitter.com/nslater

Re: Review Request 14549: Rename net.juniper.contrail to org.apache.cloudstack.network.contrail

[Hugo Trippaers]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14549/#review27960
---


Just had a nice chat with Pedro on Skype. We ran through all the review remarks 
i posted and we agreed to a couple of minor changes and Pedro will submit an 
updated diff. 

I'm ok with the patch status as is and we will leave the concerns on the guru 
selection for a other patch. So I intend to "ship it" the patch when the new 
diff is in.

- Hugo Trippaers


On Oct. 31, 2013, 8:46 p.m., Pedro Marques wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14549/
> ---
> 
> (Updated Oct. 31, 2013, 8:46 p.m.)
> 
> 
> Review request for cloudstack and Hugo Trippaers.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Rename net.juniper.contrail to org.apache.cloudstack.network.contrail.
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/network/Network.java 49f380b 
>   client/pom.xml fd1f13a 
>   client/tomcatconf/commands.properties.in 0296de0 
>   client/tomcatconf/componentContext.xml.in df5b002 
>   
> plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/CitrixResourceBase.java
>  f16a6f5 
>   plugins/network-elements/juniper-contrail/pom.xml PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/command/CreateServiceInstanceCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/response/ServiceInstanceResponse.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElement.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElementImpl.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailGuru.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManager.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManagerImpl.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/DBSyncGeneric.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/EventUtils.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ManagementNetworkGuru.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ModelDatabase.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSync.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSyncImpl.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandler.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandlerImpl.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManager.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManagerImpl.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceVirtualMachine.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpModel.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpPoolModel.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/InstanceIpModel.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelController.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObject.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObjectBase.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ServiceInst

Question: Error when creating templates from ROOT volumes in the Simulator

[David Grizzanti]

Hi All,

I have been seeing an issue with the Simulator that I was hoping someone
could provide some insight into.

Some background. I'm running from the 4.2.0 tag on rhel 6.3, building from
source/enabling the simulator.

Overall, most operations seem to work fine, however, I noticed that when
creating a template from the ROOT volume on a VM, the template can't then
be used to create a subsequent VM.  The reason for this seems to be that
the "size" of the template is set to 0.  After digging through the database
a bit, I discovered that the size and physical_size columns in
the template_store_ref table are set to 0 for these templates.  If I
correct those values for my entry after the fact, everything appears to
work.

I dug around in the code a bit as well, but nothing obvious was jumping out
at me as to why this was happening.

Any thoughts?

Thanks!

-- 
David Grizzanti
Software Engineer
Sungard Availability Services

e: david.grizza...@sungard.com
w: 215.446.1431
c: 570.575.0315

Re: CS 4.2 VM snapshot is not enabled for hypervisor type: KVM

[Nux!]

On 31.10.2013 19:32, motty cruz wrote:

Hello,
I'm having issues with snapshot, when trying to take a snapshot I get 
the

following error: "VM snapshot is not enabled for hypervisor type: VKM"

but I do have it enable in Global settings, am I missing a step?
kvm.snapshot.enabled = true,


Thanks,


Yes, I'm seeing the same. You can however snapshot the individual 
root/data disks if you go in the Storage section which is a bit "wtf".
CC-ing "dev". Anyone knows why we can't snapshot KVM VMs from the 
instance's "Details" menu? We should be able to perform live, disk-only 
snapshots from this menu.



management-server.log says:
==> /var/log/cloudstack/management/management-server.log <==
2013-10-31 21:16:24,934 DEBUG [cloud.api.ApiServlet] 
(catalina-exec-1:null) ===END===  1.2.3.4 -- GET  
command=listOsTypes&response=json&sessionkey=teaa9DZoOgNS38zwc%2FY2VwoUjoU%3D&_=1383254184625
2013-10-31 21:16:26,868 DEBUG [cloud.server.StatsCollector] 
(StatsCollector-3:null) VmStatsCollector is running...
2013-10-31 21:16:26,947 DEBUG [agent.transport.Request] 
(StatsCollector-3:null) Seq 1-1578762257: Received:  { Ans: , MgmtId: 
144350851819, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2013-10-31 21:16:27,034 DEBUG [cloud.server.StatsCollector] 
(StatsCollector-3:null) StorageCollector is running...
2013-10-31 21:16:27,103 DEBUG [agent.transport.Request] 
(StatsCollector-3:null) Seq 3-852819979: Received:  { Ans: , MgmtId: 
144350851819, via: 3, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2013-10-31 21:16:27,171 DEBUG [agent.transport.Request] 
(StatsCollector-3:null) Seq 1-1578762258: Received:  { Ans: , MgmtId: 
144350851819, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2013-10-31 21:16:28,525 DEBUG [cloud.server.StatsCollector] 
(StatsCollector-2:null) HostStatsCollector is running...
2013-10-31 21:16:28,551 DEBUG [agent.manager.AgentManagerImpl] 
(AgentManager-Handler-13:null) SeqA 2-37914: Processing Seq 2-37914:  { 
Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n 
 \"connections\": []\n}","wait":0}}] }
2013-10-31 21:16:28,626 DEBUG [agent.manager.AgentManagerImpl] 
(AgentManager-Handler-13:null) SeqA 2-37914: Sending Seq 2-37914:  { 
Ans: , MgmtId: 144350851819, via: 2, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-10-31 21:16:29,147 DEBUG [agent.transport.Request] 
(StatsCollector-2:null) Seq 1-1578762259: Received:  { Ans: , MgmtId: 
144350851819, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2013-10-31 21:16:37,968 DEBUG [cloud.api.ApiServlet] 
(catalina-exec-15:null) ===START===  1.2.3.4 -- GET  
command=createVMSnapshot&virtualmachineid=67dd260a-d951-427c-90f0-350f2f31114d&snapshotmemory=false&name=&response=json&sessionkey=teaa9DZoOgNS38zwc%2FY2VwoUjoU%3D&_=1383254197913
2013-10-31 21:16:37,991 INFO  [cloud.api.ApiServer] 
(catalina-exec-15:null) VM snapshot is not enabled for hypervisor type: 
KVM




Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: [ACS 421] Upgrade testing

[Daan Hoogland]

H Sudha,

We will at Schuberg Philis but probably not before next friday. We
will be upgrading from 4.1.1. I hope we can free up some people to
test that scenario soon.

Daan

On Thu, Oct 31, 2013 at 3:51 AM, Sudha Ponnaganti
 wrote:
> Hi All,
>
> Want to check if anyone is testing upgrades from prior versions to 4.2.1. As 
> Release date is getting close, If there are gaps, would like to cover some 
> scenarios in our lab.
>
> Thanks
> /sudha

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Daan Hoogland]

you are added to jira, Alex

On Thu, Oct 31, 2013 at 8:31 PM, Alex Ough  wrote:
> Thanks Chip, and can you also give a permission in Jira so that I can
> assign myself in its jira?
>
> Alex Ough
>
>
> On Thu, Oct 31, 2013 at 2:00 PM, Chip Childers wrote:
>
>> Permission added.
>>
>> On Wed, Oct 30, 2013 at 12:19:23PM -0500, Alex Ough wrote:
>> > And I'd like to write the design document in the wiki page, but I don't
>> > seem to have a permission to create pages.
>> > So can anyone give me the permission?
>> >
>> > My account in the wiki is alex.o...@sungard.com
>> >
>> > Thanks in advance.
>> > Alex Ough
>> >
>> >
>> > On Tue, Oct 29, 2013 at 3:38 PM, Alex Ough 
>> wrote:
>> >
>> > > I created a jira for this feature.
>> > >
>> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4992
>> > >
>> > > But it doesn't allow for me to assign it to myself, so any permission
>> do I
>> > > need for this?
>> > > If so, can anyone give me this permission?
>> > >
>> > > If there is anything missing, let me know.
>> > > Thanks
>> > > Alex Ough
>> > >
>> > >
>> > > On Fri, Oct 18, 2013 at 9:30 AM, Kishan Kavala <
>> kishan.kav...@citrix.com>wrote:
>> > >
>> > >> > -Original Message-
>> > >> > From: Alex Ough [mailto:alex.o...@sungard.com]
>> > >> > Sent: Thursday, 17 October 2013 11:25 PM
>> > >> > To: dev@cloudstack.apache.org; u...@cloudstack.apache.org
>> > >> > Subject: Fwd: [DISCUSS] Domain/Account/User Sync Up Among Multiple
>> > >> > Regions
>> > >> >
>> > >> > All,
>> > >> >
>> > >> > Currently, under the environment of cloudstack with multiple
>> regions,
>> > >> each
>> > >> > region has its own management server running with a separate
>> database.
>> > >> So if
>> > >> > we want to support multiple regions and provide one point of entry
>> for a
>> > >> > customer, we need to duplicate domain/account/user information of
>> that
>> > >> > customer to all of the databases of regions the customer accesses,
>> > >> which will
>> > >> > cause data discrepancies when users update those data independently
>> in
>> > >> each
>> > >> > management server.
>> > >> >
>> > >> > So I'd like to provide a way to sync up the data using the messaging
>> > >> system
>> > >> > introduced in 4.1.0. Using the events from each management server,
>> > >> updates
>> > >> > from each region can be propagated to the rest regions and they can
>> be
>> > >> > executed accordingly.
>> > >> >
>> > >> > I hope you guys have a chance to think about this and give some
>> > >> feedbacks if
>> > >> > interested.
>> > >> > Thanks in advance.
>> > >> > Alex Ough
>> > >>
>> > >> [KK] Alex, it was discussed sometime back. Related thread [1]. Sync up
>> > >> using messaging system is the right way to go.
>> > >>
>> > >>
>> > >> [1]
>> > >>
>> http://www.mail-archive.com/cloudstack-dev@incubator.apache.org/msg20193.html
>> > >>
>> > >>
>> > >
>>
>>

Re: Review Request 15128: added group and domain params to importLdapUsers api call

[Ian Duffy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15128/#review27966
---

Ship it!


Ship It!

- Ian Duffy


On Oct. 31, 2013, 1:24 p.m., Rajani Karuturi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15128/
> ---
> 
> (Updated Oct. 31, 2013, 1:24 p.m.)
> 
> 
> Review request for cloudstack, Abhinandan Prateek and Ian Duffy.
> 
> 
> Bugs: CLOUDSTACK-4866
> https://issues.apache.org/jira/browse/CLOUDSTACK-4866
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> added group and domain params to importLdapUsers api call.
> 
> If a group param is present, importUsers will import ldap users only from 
> that group.
> If a domain param is present, it will create the users in that domain. it 
> will fallback to group or OU of the ldap user or the ROOT domain in the order
> 
> 
> Diffs
> -
> 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/api/command/LdapImportUsersCmd.java
>  f872247 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapConfiguration.java
>  a08dccb 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapManager.java
>  683822d 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapManagerImpl.java
>  90a79b3 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapUserManager.java
>  47697c9 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapConfigurationSpec.groovy
>  66b4673 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapCreateAccountCmdSpec.groovy
>  cc849de 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapImportUsersCmdSpec.groovy
>  d04b094 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapManagerImplSpec.groovy
>  321e1af 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapUserManagerSpec.groovy
>  339923e 
>   server/src/com/cloud/configuration/Config.java 3fdc343 
>   setup/db/db/schema-421to430.sql 0de9dfd 
> 
> Diff: https://reviews.apache.org/r/15128/diff/
> 
> 
> Testing
> ---
> 
> Yes. Added unit tests.
> 
> 
> Thanks,
> 
> Rajani Karuturi
> 
>

Re: [ACS 421] Upgrade testing

[Nux!]

On 31.10.2013 02:51, Sudha Ponnaganti wrote:

Hi All,

Want to check if anyone is testing upgrades from prior versions to
4.2.1. As Release date is getting close, If there are gaps, would like
to cover some scenarios in our lab.

Thanks
/sudha


I'd love to test it, anyone got an RPM repo with it?

Lucian


--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

parallel mvn builds

[John Kinsella]

I've been tinkering with adding -T to builds on my laptop - e.g. "mvn -T 2C 
install" (Means run two build threads on each CPU core the system has)

I'm seeing a 20-30% improvement in build time vs a single-thread build, and 
things seem to work so far, but I do get this warning a few times during the 
build:

[WARNING] *
[WARNING] * Your build is requesting parallel execution, but project  *
[WARNING] * contains the following plugin(s) that are not marked as   *
[WARNING] * @threadSafe to support parallel building. *
[WARNING] * While this /may/ work fine, please look for plugin updates*
[WARNING] * and/or request plugins be made thread-safe.   *
[WARNING] * If reporting an issue, report it against the plugin in*
[WARNING] * question, not against maven-core  *
[WARNING] *
[WARNING] The following plugins are not marked @threadSafe in Apache CloudStack:
[WARNING] org.apache.maven.plugins:maven-site-plugin:3.1
[WARNING] *

Curious if anybody else has tried this or if we should recommend it to folks?

RE: [ACS 421] Upgrade testing

[Sudha Ponnaganti]

Thanks Daan 

-Original Message-
From: Daan Hoogland [mailto:daan.hoogl...@gmail.com] 
Sent: Thursday, October 31, 2013 2:33 PM
To: dev
Subject: Re: [ACS 421] Upgrade testing

H Sudha,

We will at Schuberg Philis but probably not before next friday. We will be 
upgrading from 4.1.1. I hope we can free up some people to test that scenario 
soon.

Daan

On Thu, Oct 31, 2013 at 3:51 AM, Sudha Ponnaganti  
wrote:
> Hi All,
>
> Want to check if anyone is testing upgrades from prior versions to 4.2.1. As 
> Release date is getting close, If there are gaps, would like to cover some 
> scenarios in our lab.
>
> Thanks
> /sudha

Re: [DISCUSS] Domain/Account/User Sync Up Among Multiple Regions

[Alex Ough]

Great! Thanks a lot, Daan.


On Thu, Oct 31, 2013 at 4:58 PM, Daan Hoogland wrote:

> you are added to jira, Alex
>
> On Thu, Oct 31, 2013 at 8:31 PM, Alex Ough  wrote:
> > Thanks Chip, and can you also give a permission in Jira so that I can
> > assign myself in its jira?
> >
> > Alex Ough
> >
> >
> > On Thu, Oct 31, 2013 at 2:00 PM, Chip Childers  >wrote:
> >
> >> Permission added.
> >>
> >> On Wed, Oct 30, 2013 at 12:19:23PM -0500, Alex Ough wrote:
> >> > And I'd like to write the design document in the wiki page, but I
> don't
> >> > seem to have a permission to create pages.
> >> > So can anyone give me the permission?
> >> >
> >> > My account in the wiki is alex.o...@sungard.com
> >> >
> >> > Thanks in advance.
> >> > Alex Ough
> >> >
> >> >
> >> > On Tue, Oct 29, 2013 at 3:38 PM, Alex Ough 
> >> wrote:
> >> >
> >> > > I created a jira for this feature.
> >> > >
> >> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4992
> >> > >
> >> > > But it doesn't allow for me to assign it to myself, so any
> permission
> >> do I
> >> > > need for this?
> >> > > If so, can anyone give me this permission?
> >> > >
> >> > > If there is anything missing, let me know.
> >> > > Thanks
> >> > > Alex Ough
> >> > >
> >> > >
> >> > > On Fri, Oct 18, 2013 at 9:30 AM, Kishan Kavala <
> >> kishan.kav...@citrix.com>wrote:
> >> > >
> >> > >> > -Original Message-
> >> > >> > From: Alex Ough [mailto:alex.o...@sungard.com]
> >> > >> > Sent: Thursday, 17 October 2013 11:25 PM
> >> > >> > To: dev@cloudstack.apache.org; u...@cloudstack.apache.org
> >> > >> > Subject: Fwd: [DISCUSS] Domain/Account/User Sync Up Among
> Multiple
> >> > >> > Regions
> >> > >> >
> >> > >> > All,
> >> > >> >
> >> > >> > Currently, under the environment of cloudstack with multiple
> >> regions,
> >> > >> each
> >> > >> > region has its own management server running with a separate
> >> database.
> >> > >> So if
> >> > >> > we want to support multiple regions and provide one point of
> entry
> >> for a
> >> > >> > customer, we need to duplicate domain/account/user information of
> >> that
> >> > >> > customer to all of the databases of regions the customer
> accesses,
> >> > >> which will
> >> > >> > cause data discrepancies when users update those data
> independently
> >> in
> >> > >> each
> >> > >> > management server.
> >> > >> >
> >> > >> > So I'd like to provide a way to sync up the data using the
> messaging
> >> > >> system
> >> > >> > introduced in 4.1.0. Using the events from each management
> server,
> >> > >> updates
> >> > >> > from each region can be propagated to the rest regions and they
> can
> >> be
> >> > >> > executed accordingly.
> >> > >> >
> >> > >> > I hope you guys have a chance to think about this and give some
> >> > >> feedbacks if
> >> > >> > interested.
> >> > >> > Thanks in advance.
> >> > >> > Alex Ough
> >> > >>
> >> > >> [KK] Alex, it was discussed sometime back. Related thread [1].
> Sync up
> >> > >> using messaging system is the right way to go.
> >> > >>
> >> > >>
> >> > >> [1]
> >> > >>
> >>
> http://www.mail-archive.com/cloudstack-dev@incubator.apache.org/msg20193.html
> >> > >>
> >> > >>
> >> > >
> >>
> >>
>
>

Re: parallel mvn builds

[Darren Shepherd]

I use it all the time with -T 1.5C and it makes my build twice as fast.
But I do it with the assumption that "it may void your warranty."
Sometimes things blow up, and then I go back to single threaded.  So it
kinda works, but it does blow up.

I wouldn't recommend most people use it.  For people who are closer to full
time ACS developers it can be helpful.  For people that don't use ACS much,
its not worth the hassle because its not guaranteed to work and I have seen
it fail.

Darren


On Thu, Oct 31, 2013 at 3:36 PM, John Kinsella  wrote:

> I've been tinkering with adding -T to builds on my laptop - e.g. "mvn -T
> 2C install" (Means run two build threads on each CPU core the system has)
>
> I'm seeing a 20-30% improvement in build time vs a single-thread build,
> and things seem to work so far, but I do get this warning a few times
> during the build:
>
> [WARNING] *
> [WARNING] * Your build is requesting parallel execution, but project  *
> [WARNING] * contains the following plugin(s) that are not marked as   *
> [WARNING] * @threadSafe to support parallel building. *
> [WARNING] * While this /may/ work fine, please look for plugin updates*
> [WARNING] * and/or request plugins be made thread-safe.   *
> [WARNING] * If reporting an issue, report it against the plugin in*
> [WARNING] * question, not against maven-core  *
> [WARNING] *
> [WARNING] The following plugins are not marked @threadSafe in Apache
> CloudStack:
> [WARNING] org.apache.maven.plugins:maven-site-plugin:3.1
> [WARNING] *
>
> Curious if anybody else has tried this or if we should recommend it to
> folks?
>
>

Re: Review Request 14549: Rename net.juniper.contrail to org.apache.cloudstack.network.contrail

[Pedro Marques]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14549/
---

(Updated Nov. 1, 2013, 12:19 a.m.)


Review request for cloudstack and Hugo Trippaers.


Changes
---

Update the diff according to code review comments.


Repository: cloudstack-git


Description
---

Rename net.juniper.contrail to org.apache.cloudstack.network.contrail.


Diffs (updated)
-

  api/src/com/cloud/network/Network.java 49f380b 
  client/pom.xml 3e08a9a 
  client/tomcatconf/commands.properties.in e92596c 
  
plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/CitrixResourceBase.java
 3df28ed 
  plugins/network-elements/juniper-contrail/pom.xml PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/command/CreateServiceInstanceCmd.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/response/ServiceInstanceResponse.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElement.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElementImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailGuru.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManager.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailManagerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/DBSyncGeneric.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/EventUtils.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ManagementNetworkGuru.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ModelDatabase.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSync.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSyncImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandler.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerEventHandlerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManager.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManagerImpl.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceVirtualMachine.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/FloatingIpPoolModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/InstanceIpModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelController.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObject.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ModelObjectBase.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ServiceInstanceModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VMInterfaceModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VirtualMachineModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/VirtualNetworkModel.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/management/ApiConnectorMockito.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/management/IntegrationTestConfiguration.java
 PRE-CREATION 
  
plugins/network-elements/juniper-contrail/test/org/apache/cloudstack/network/contrail/management/ManagementSe

Re: race conditions in VolumeServiceImpl.createBaseImageAsync() creates NPE

[Min Chen]

Darren, I just checked the code, you are right. In case of one thread throws 
exception in downloading template to primary, it will delete the entry in 
template_store_ref, causing the second thread failing with NPE. We need to fix 
this in 4.3. Please file a bug for this.

Thanks
-min

From: Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>>
Date: Thursday, October 31, 2013 11:39 AM
To: "dev@cloudstack.apache.org" 
mailto:dev@cloudstack.apache.org>>, Min Chen 
mailto:min.c...@citrix.com>>
Subject: race conditions in VolumeServiceImpl.createBaseImageAsync() creates NPE

The following code results in a NPE in bad situations

templatePoolRef = _tmpltPoolDao.acquireInLockTable(templatePoolRefId, 
storagePoolMaxWaitSeconds);

if (templatePoolRef == null) {
if (s_logger.isDebugEnabled()) {
s_logger.info("Unable to acquire lock on 
VMTemplateStoragePool " + templatePoolRefId);
}
templatePoolRef = 
_tmpltPoolDao.findByPoolTemplate(dataStore.getId(), template.getId());
if (templatePoolRef.getState() == 
ObjectInDataStoreStateMachine.State.Ready ) {
s_logger.info("Unable to acquire lock on 
VMTemplateStoragePool " + templatePoolRefId + ", But Template " + 
template.getUniqueName() + " is already copied to primary storage, skip 
copying");
createVolumeFromBaseImageAsync(volume, 
templateOnPrimaryStoreObj, dataStore, future);
return;
}
throw new CloudRuntimeException("Unable to acquire lock on 
VMTemplateStoragePool: " + templatePoolRefId);
}

If two threads are trying to stage the same template thread one gets the lock, 
thread two will wait.  If thread one fails to stage the template it will delete 
the templatePoolRef from the database.  Thread two will now get the lock in 
op_lock, but the internal findById will not find a templatePoolRef because it 
has been deleted and return null from acquireInLockTable().  Technically thread 
two has the lock, but the ref templatePoolRef wasn't found.  The subsequent 
line "templatePoolRef = _tmpltPoolDao.findByPoolTemplate(...)" will return 
null, because it doesn't exist and then on the next line 
templatePoolRef.getState() will throw a NPE.

Darren

Re: Review Request 15068: Change labels for VLAN to vNet

[Toshiaki Hatano]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15068/#review27992
---


I suppose this should be reviewed by UI devs too.
Could you add UI devs in reviewers?

- Toshiaki Hatano


On Oct. 30, 2013, 8:46 p.m., Chris Cameron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15068/
> ---
> 
> (Updated Oct. 30, 2013, 8:46 p.m.)
> 
> 
> Review request for cloudstack and Toshiaki Hatano.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> We would like to change the labels for VLAN to vNet to make the term more 
> generic for VXLAN and VLAN. This relates to the work being done to add in 
> VXLAN support to Cloudstack.
> 
> 
> Diffs
> -
> 
>   client/WEB-INF/classes/resources/messages.properties 3210aca 
>   ui/dictionary.jsp 35cba22 
>   ui/scripts/network.js 12e5389 
>   ui/scripts/system.js 479883c 
> 
> Diff: https://reviews.apache.org/r/15068/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chris Cameron
> 
>

Re: parallel mvn builds

[John Kinsella]

Makes sense, and now documented in the mailing list. :)

On Oct 31, 2013, at 5:00 PM, Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>> wrote:

I use it all the time with -T 1.5C and it makes my build twice as fast.
But I do it with the assumption that "it may void your warranty."
Sometimes things blow up, and then I go back to single threaded.  So it
kinda works, but it does blow up.

I wouldn't recommend most people use it.  For people who are closer to full
time ACS developers it can be helpful.  For people that don't use ACS much,
its not worth the hassle because its not guaranteed to work and I have seen
it fail.

Darren


On Thu, Oct 31, 2013 at 3:36 PM, John Kinsella 
mailto:j...@stratosec.co>> wrote:

I've been tinkering with adding -T to builds on my laptop - e.g. "mvn -T
2C install" (Means run two build threads on each CPU core the system has)

I'm seeing a 20-30% improvement in build time vs a single-thread build,
and things seem to work so far, but I do get this warning a few times
during the build:

[WARNING] *
[WARNING] * Your build is requesting parallel execution, but project  *
[WARNING] * contains the following plugin(s) that are not marked as   *
[WARNING] * @threadSafe to support parallel building. *
[WARNING] * While this /may/ work fine, please look for plugin updates*
[WARNING] * and/or request plugins be made thread-safe.   *
[WARNING] * If reporting an issue, report it against the plugin in*
[WARNING] * question, not against maven-core  *
[WARNING] *
[WARNING] The following plugins are not marked @threadSafe in Apache
CloudStack:
[WARNING] org.apache.maven.plugins:maven-site-plugin:3.1
[WARNING] *

Curious if anybody else has tried this or if we should recommend it to
folks?



Stratosec - Compliance as a Service
o: 415.315.9385
@johnlkinsella

Re: Review Request 15068: Change labels for VLAN to vNet

[Toshiaki Hatano]

> On Nov. 1, 2013, 1:28 a.m., Toshiaki Hatano wrote:
> > I suppose this should be reviewed by UI devs too.
> > Could you add UI devs in reviewers?

Other than that, this patch looks OK for me.
But this patch makes changes in UI, so I suppose it's better to check with UI 
ppl.


- Toshiaki


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15068/#review27992
---


On Oct. 30, 2013, 8:46 p.m., Chris Cameron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15068/
> ---
> 
> (Updated Oct. 30, 2013, 8:46 p.m.)
> 
> 
> Review request for cloudstack and Toshiaki Hatano.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> We would like to change the labels for VLAN to vNet to make the term more 
> generic for VXLAN and VLAN. This relates to the work being done to add in 
> VXLAN support to Cloudstack.
> 
> 
> Diffs
> -
> 
>   client/WEB-INF/classes/resources/messages.properties 3210aca 
>   ui/dictionary.jsp 35cba22 
>   ui/scripts/network.js 12e5389 
>   ui/scripts/system.js 479883c 
> 
> Diff: https://reviews.apache.org/r/15068/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chris Cameron
> 
>

Review Request 15173: Dynamic compute offering.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15173/
---

Review request for cloudstack, Kishan Kavala and Koushik Das.


Bugs: CLOUDSTACK-4738
https://issues.apache.org/jira/browse/CLOUDSTACK-4738


Repository: cloudstack-git


Description
---

https://issues.apache.org/jira/browse/CLOUDSTACK-4738
Dynamic compute Offering.

Still need to test this. Facing some auto wiring problems when UsageEventUtils 
bean is created. 


Diffs
-

  api/src/com/cloud/offering/ServiceOffering.java 9f7bf8e 
  api/src/com/cloud/vm/UserVmService.java 0b142e8 
  api/src/org/apache/cloudstack/api/ApiConstants.java b1bfcfb 
  
api/src/org/apache/cloudstack/api/command/admin/offering/CreateServiceOfferingCmd.java
 decac29 
  api/src/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java 8a6cea7 
  
engine/api/src/org/apache/cloudstack/engine/orchestration/service/VolumeOrchestrationService.java
 a773ac4 
  
engine/api/src/org/apache/cloudstack/engine/service/api/OrchestrationService.java
 64ef063 
  engine/components-api/src/com/cloud/configuration/ConfigurationManager.java 
03a549f 
  engine/components-api/src/com/cloud/event/UsageEventUtils.java b44ed32 
  engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java b74b4c5 
  
engine/orchestration/src/org/apache/cloudstack/engine/orchestration/CloudOrchestrator.java
 2fd10b6 
  
engine/orchestration/src/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java
 0821c81 
  engine/schema/src/com/cloud/event/UsageEventDetailsVO.java PRE-CREATION 
  engine/schema/src/com/cloud/event/dao/UsageEventDetailsDao.java PRE-CREATION 
  engine/schema/src/com/cloud/event/dao/UsageEventDetailsDaoImpl.java 
PRE-CREATION 
  engine/schema/src/com/cloud/service/ServiceOfferingVO.java 9a262c5 
  engine/schema/src/com/cloud/service/dao/ServiceOfferingDao.java 7da7208 
  engine/schema/src/com/cloud/service/dao/ServiceOfferingDaoImpl.java f807f0d 
  
plugins/network-elements/internal-loadbalancer/src/org/apache/cloudstack/network/lb/InternalLoadBalancerVMManagerImpl.java
 b6269eb 
  
server/src/com/cloud/agent/manager/allocator/impl/UserConcentratedAllocator.java
 0da2c92 
  server/src/com/cloud/api/query/QueryManagerImpl.java f9d9c4f 
  server/src/com/cloud/capacity/CapacityManagerImpl.java 72905a7 
  server/src/com/cloud/configuration/ConfigurationManagerImpl.java e3aa4fa 
  server/src/com/cloud/consoleproxy/ConsoleProxyManagerImpl.java e82aaba 
  server/src/com/cloud/network/NetworkModelImpl.java df9f651 
  server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java 
a93480b 
  server/src/com/cloud/vm/UserVmManagerImpl.java 80a4036 
  server/test/com/cloud/capacity/CapacityManagerTest.java 3faa32f 
  server/test/com/cloud/vpc/MockConfigurationManagerImpl.java 3147f1f 
  server/test/resources/createNetworkOffering.xml 9d684ba 
  setup/db/db/schema-421to430.sql 0de9dfd 

Diff: https://reviews.apache.org/r/15173/diff/


Testing
---

Not tested.


Thanks,

bharat kumar