date:20131024

Whenever I use marvin on master I get

Traceback (most recent call last):
  File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "./tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,

So it looks like cloudConnection needs a logger passed to the
constructor and its not being passed?  I've just been hacking up
./tools/marvin/marvin/cloudstackConnection.py to work around this.

Darren

Re: how to use hashes on c.a.o?

But how does one validate it?  I just wrote a dumb script to
concatenation, remove whitespace, lowercase and then pass to
"sha512sum -c."  I've never seen anyone provide SHAs in that format.
I wouldn't expect many people to know how to use them.  Why can't we
use the good old GNU coreutils style?

Darren

On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella  wrote:
> This is the output of gpg -v --print-md SHA512, generated as part of the 
> release procedure [1] by tools/build/build_asf.sh
>
> 1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure
>
>
> On Oct 17, 2013, at 7:56 PM, Darren Shepherd  
> wrote:
>
>> The hashes that are on c.a.o for the releases have a format like
>>
>> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
>>
>> apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
>> 684E0094 F1666F57 5D694916 CF74ED98
>> 9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
>> EA82D42C 78880EDB 04F2532A 61376537
>>
>> I've never seen this.  Is this some hip new format I'm not aware of,
>> and I'm the uncool kid still using GNU coreutils?
>>
>> Darren
>
>
>

Re: Can't create cluster in CS 4.2

2013-10-24 Thread Ryan Lei

JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-4943

I'm encountering the same problem using CloudStack 4.2.0 + VMware 5.0.
Trapped for a day.
I just found a potential bug causing this. I have commented in JIRA.

Basically the Advanced Zone wizard now automatically appends VMware traffic
labels to appear like:

[Public] vSwitch0,,vmwaresvs
[Guest] vSwitch0,,vmwaresvs
[Management] vSwitch0,
[Storage] vSwitch1,

Thus, CloudStack cannot match the same virtual switches on vCenter.
The wizard cannot go all the way to the last step. You have to manually
rename these labels in
Zone -> Physical Network, and add the remaining primary/secondary storages
later.

Can you or someone else confirm this? Is this auto-renaming a bug?

---
Yu-Heng (Ryan) Lei, Associate Researcher
Chunghwa Telecom Laboratories / Cloud Computing Laboratory
ryan...@cht.com.tw
or
ryanlei750...@gmail.com



On Wed, Oct 23, 2013 at 6:40 PM, Denis Finko wrote:

> Hello CloudStack community,
>
> Could you please help found source of following issue and how to fix it.
> I am tring to configure CloudStack 4.2 (installed from cloudstack repo
> baseurl=http://cloudstack.apt-get.eu/rhel/4.2/) with VMware 5.1. During
> adding new zone into UI I have got following error:
>
> "Unable to add the external cluster"
>
> We have following networking configuration in vSphere Client:
> Standard Switch: vSwitch0
> -Virtual Machine Port Group (VM Network)  --vmnic0
> -VMkernel Port (Management Network)
>
> Standard Switch: vSwitch1
> -VMkernel Port (Storage1)  --vmnic2
>
>
> management-server.log:
> 2013-10-23 05:36:03,534 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-22:null) ===START===  193.142.124.9 -- GET
>
> command=addCluster&zoneId=4f28654f-e767-4be2-b32f-39f634a01746&hypervisor=VMware&clustertype=ExternalManaged&podId=94e55d0c-3ea9-4dbc-800b-854262fd2674&username=Administrator&url=http%3A%2F%2F10.199.0.40%2Fcs4%2Fcs4it&clustername=10.199.0.40%2Fcs4%2Fcs4it&response=json&sessionkey=uAl68djojOPwUBfhwxw8OwodZuE%3D&_=1382520964075
> 2013-10-23 05:36:03,643 INFO  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Discover host. dc: 1, pod: 1, cluster: 1, uri
> host: 10.199.0.40
> 2013-10-23 05:36:03,689 INFO  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Detected private network label : vSwitch0,
> 2013-10-23 05:36:03,689 DEBUG [vmware.resource.VmwareContextFactory]
> (catalina-exec-22:null) initialize VmwareContext. url:
> https://10.199.0.40/sdk/vimService, username: Administrator, password:
> r***
> 2013-10-23 05:36:03,817 INFO  [vmware.util.VmwareContext]
> (catalina-exec-22:null) New VmwareContext object, current outstanding
> count: 1
> 2013-10-23 05:36:03,952 INFO  [vmware.manager.VmwareManagerImpl]
> (catalina-exec-22:null) Preparing network on host
> com.cloud.hypervisor.vmware.util.VmwareContext@60e2e3d4 for vSwitch0,
> 2013-10-23 05:36:03,999 ERROR [vmware.mo.HypervisorHostHelper]
> (catalina-exec-22:null) Unable to find vSwitchvSwitch0,
> 2013-10-23 05:36:03,999 WARN  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Unable to connect to Vmware vSphere server.
> service address: 10.199.0.40
> 2013-10-23 05:36:04,012 WARN  [cloud.resource.ResourceManagerImpl]
> (catalina-exec-22:null) Unable to find the server resources at
> http://10.199.0.40/cs4/cs4it
> 2013-10-23 05:36:04,014 INFO  [utils.exception.CSExceptionErrorCode]
> (catalina-exec-22:null) Could not find exception:
> com.cloud.exception.DiscoveryException in error code list for exceptions
> 2013-10-23 05:36:04,095 WARN  [admin.cluster.AddClusterCmd]
> (catalina-exec-22:null) Exception:
> com.cloud.exception.DiscoveryException: Unable to add the external cluster
> at
>
> com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.java:533)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> org.apache.cloudstack.api.command.admin.cluster.AddClusterCmd.execute(AddClusterCmd.java:181)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:514)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:372)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:305)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
>
> org.apache.catalina.core

RE: is marvin on master broken?

2013-10-24 Thread Santhosh Edukulla

Hello Darren,

Some trace is still missing i believe. i could not see the last stack frame in 
the below trace as what lead to this trace?

I just pulled the latest from master branch and used marvin  to deploy few 
cloudstack entities and it worked. Can you please provide the command you are 
using to run marvin tests? or what command lead to the below trace?  

Regards,
Santhosh

From: Darren Shepherd [darren.s.sheph...@gmail.com]
Sent: Thursday, October 24, 2013 3:48 AM
To: dev@cloudstack.apache.org
Subject: is marvin on master broken?

Whenever I use marvin on master I get

Traceback (most recent call last):
  File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "./tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,

So it looks like cloudConnection needs a logger passed to the
constructor and its not being passed?  I've just been hacking up
./tools/marvin/marvin/cloudstackConnection.py to work around this.

Darren

Re: How does HA work?

2013-10-24 Thread Nux!


On 24.10.2013 01:42, Chiradeep Vittal wrote:

https://cwiki.apache.org/confluence/x/dwn8AQ


Thanks!

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: Command sequence logic in agent code

2013-10-24 Thread Koushik Das

Created https://issues.apache.org/jira/browse/CLOUDSTACK-4944 to track this 
issue.

Alex, Any reason for adding requests based on sequence and not doing FIFO? Do 
you see any issues if request always gets added to the end of the queue?


On 23-Oct-2013, at 6:26 PM, Koushik Das  wrote:

> I was looking at the command sequencing logic in the agent code.
> 
> Each agent maintains a sequence that gets initialised based on following logic
> 
>private static final Random s_rand = new 
> Random(System.currentTimeMillis());
>_nextSequence = s_rand.nextInt(Short.MAX_VALUE) << 48;
> 
> For every command that gets processed by the agent the sequence is 
> incremented by 1. If commands are to be executed in sequence then they are 
> queued up based on this sequence
> 
>protected synchronized void addRequest(Request req) {
>int index = findRequest(req);
>assert (index < 0) : "How can we get index again? " + index + ":" + 
> req.toString();
>_requests.add(-index - 1, req);
>}
> 
> The above works fine in case of a single MS scenario. In case of a clustered 
> MS setup things change slightly.
> 
> The command can originate at any MS and based on the ownership of the agent, 
> it gets forwarded to the correct MS which then handles the command. Now 
> command sequences are local to individual agents in MS. In this case the 
> originating MS agent tags the request with a sequence. This gets forwarded to 
> the owning MS and based on if 'executedInSequence' flag is set, gets added to 
> the list based on the sequence number. Now here lies the problem, commands 
> are not inserted in the order in which they arrive but based on the sequence 
> number. In case of a forwarded command the sequence is different from the 
> local sequence. If the starting sequence of forwarded commands is much less 
> than that of the locally generated commands then there is a possibility of 
> local commands getting starved if there is a steady arrival of forwarded 
> commands. Similarly it can also happen the other way round. Also if the the 
> starting sequence for a agent in local and peer MS is not spread far apart 
> then there may be overlaps and a new request will override the old one.
> 
> Not sure if anyone encountered any issues due to this. The correct way looks 
> like to implement the queue model rather than doing a add based on the above 
> code.
> 
> Comments?
> 
> -Koushik
> 
> 
> 
> 
>

Re: Review Request 14875: Fixed few pep8 compliant issues

2013-10-24 Thread Prasanna Santhanam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14875/#review27452
---

Ship it!


Ship It!

- Prasanna Santhanam


On Oct. 23, 2013, 1:57 p.m., Santhosh Edukulla wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14875/
> ---
> 
> (Updated Oct. 23, 2013, 1:57 p.m.)
> 
> 
> Review request for cloudstack and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> There were few pep8 changes, fixed it. Added new patch on top of existing one.
> 
> 
> Diffs
> -
> 
>   tools/marvin/marvin/marvinPlugin.py aded17c 
> 
> Diff: https://reviews.apache.org/r/14875/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Santhosh Edukulla
> 
>

Re: [ANNOUNCE] New Committer: Shilamkar, Girish

2013-10-24 Thread Jayapal Reddy Uradi

Congrats Girish!

On 24-Oct-2013, at 12:45 AM, Rayees Namathponnan 
 wrote:

> Congrats Girsih 
> 
> -Original Message-
> From: Ahmad Emneina [mailto:aemne...@gmail.com] 
> Sent: Wednesday, October 23, 2013 10:34 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Shilamkar, Girish
> 
> Nice work Girish!
> 
> 
> On Wed, Oct 23, 2013 at 9:59 AM, Sangeetha Hariharan < 
> sangeetha.hariha...@citrix.com> wrote:
> 
>> Congrats Girish!!
>> 
>> -Original Message-
>> From: Radhika Puthiyetath [mailto:radhika.puthiyet...@citrix.com]
>> Sent: Wednesday, October 23, 2013 3:59 AM
>> To: dev@cloudstack.apache.org
>> Subject: RE: [ANNOUNCE] New Committer: Shilamkar, Girish
>> 
>> Congratz, Girish
>> 
>> -Original Message-
>> From: Prasanna Santhanam [mailto:t...@apache.org]
>> Sent: Wednesday, October 23, 2013 4:08 PM
>> To: CloudStack Dev
>> Subject: [ANNOUNCE] New Committer: Shilamkar, Girish
>> 
>> The Project Management Committee (PMC) for Apache CloudStack has asked 
>> Girish Shilamkar to become a committer and we are pleased to announce 
>> that they have accepted.
>> 
>> Being a committer allows many contributors to contribute more 
>> autonomously. For developers, it makes it easier to submit changes and 
>> eliminates the need to have contributions reviewed via the patch 
>> submission process. Whether contributions are development-related or 
>> otherwise, it is a recognition of a contributor's participation in the 
>> project and commitment to the project and the Apache Way.
>> 
>> Please join me in congratulating Girish!
>> 
>> --
>> Prasanna.,
>> on behalf of the Apache CloudStack PMC
>> 
>> 
>> Powered by BigRock.com
>> 
>>

Re: Review Request 14874: Marvin Plugin Changes: Adding timestamp and timetaken in seconds for a given testcase run to console

2013-10-24 Thread Prasanna Santhanam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14874/#review27453
---

Ship it!


0780604 master

- Prasanna Santhanam


On Oct. 23, 2013, 1:53 p.m., Santhosh Edukulla wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14874/
> ---
> 
> (Updated Oct. 23, 2013, 1:53 p.m.)
> 
> 
> Review request for cloudstack and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Adding fix for the bug 4885. This will  add timestamp and timetaken in 
> seconds for a given testcase run.
> 
> 
> Diffs
> -
> 
>   tools/marvin/marvin/marvinPlugin.py aded17c 
> 
> Diff: https://reviews.apache.org/r/14874/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Santhosh Edukulla
> 
>

Review Request 14905: Add global configuration parameters for SMTP timeout

2013-10-24 Thread Anshul Gangwar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14905/
---

Review request for cloudstack, Devdeep Singh, Likitha Shetty, and Sateesh 
Chodapuneedi.


Bugs: CLOUDSTACK-4909
https://issues.apache.org/jira/browse/CLOUDSTACK-4909


Repository: cloudstack-git


Description
---

This will make the SMTP timeout parameters configurable. Default timeout is 30 
seconds.


Diffs
-

  server/src/com/cloud/alert/AlertManagerImpl.java 3aa9695 
  server/src/com/cloud/configuration/Config.java cefcdd5 

Diff: https://reviews.apache.org/r/14905/diff/


Testing
---

tested on local setup


Thanks,

Anshul Gangwar

Re: Can't create cluster in CS 4.2

2013-10-24 Thread Denis Finko

Ryan, thank you for your advice! It was helpful and I finally added a
Cluster.

In my case traffic labels were:

Public: vSwitch0,3,vmwaresvs
Guest: vSwitch0,3,vmwaresvs
Management Network: vSwitch0,

Storage: vSwitch1,

I have renamed it to:
vSwitch0
vSwitch0
vSwitch0
vSwitch1
and cluster was added.

So it looks like a real bug in CloudStack 4.2 wizard.


On 10/24/2013 10:55 AM, Ryan Lei wrote:
> JIRA ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-4943
>
> I'm encountering the same problem using CloudStack 4.2.0 + VMware 5.0.
> Trapped for a day.
> I just found a potential bug causing this. I have commented in JIRA.
>
> Basically the Advanced Zone wizard now automatically appends VMware
> traffic labels to appear like:
>
> [Public] vSwitch0,,vmwaresvs
> [Guest] vSwitch0,,vmwaresvs
> [Management] vSwitch0,
> [Storage] vSwitch1,
>
> Thus, CloudStack cannot match the same virtual switches on vCenter.
> The wizard cannot go all the way to the last step. You have to
> manually rename these labels in
> Zone -> Physical Network, and add the remaining primary/secondary
> storages later.
>
> Can you or someone else confirm this? Is this auto-renaming a bug?
>
> ---
> Yu-Heng (Ryan) Lei, Associate Researcher
> Chunghwa Telecom Laboratories / Cloud Computing Laboratory
> ryan...@cht.com.tw
> 
>  or
> ryanlei750...@gmail.com 
>
>
>
> On Wed, Oct 23, 2013 at 6:40 PM, Denis Finko
> mailto:denis.fi...@ecommerce.com>> wrote:
>
> Hello CloudStack community,
>
> Could you please help found source of following issue and how to
> fix it.
> I am tring to configure CloudStack 4.2 (installed from cloudstack repo
> baseurl=http://cloudstack.apt-get.eu/rhel/4.2/) with VMware 5.1.
> During
> adding new zone into UI I have got following error:
>
> "Unable to add the external cluster"
>
> We have following networking configuration in vSphere Client:
> Standard Switch: vSwitch0
> -Virtual Machine Port Group (VM Network)  --vmnic0
> -VMkernel Port (Management Network)
>
> Standard Switch: vSwitch1
> -VMkernel Port (Storage1)  --vmnic2
>
>
> management-server.log:
> 2013-10-23 05:36:03,534 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-22:null) ===START===  193.142.124.9 -- GET
> 
> command=addCluster&zoneId=4f28654f-e767-4be2-b32f-39f634a01746&hypervisor=VMware&clustertype=ExternalManaged&podId=94e55d0c-3ea9-4dbc-800b-854262fd2674&username=Administrator&url=http%3A%2F%2F10.199.0.40%2Fcs4%2Fcs4it&clustername=10.199.0.40%2Fcs4%2Fcs4it&response=json&sessionkey=uAl68djojOPwUBfhwxw8OwodZuE%3D&_=1382520964075
> 2013-10-23 05:36:03,643 INFO
>  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Discover host. dc: 1, pod: 1, cluster: 1, uri
> host: 10.199.0.40
> 2013-10-23 05:36:03,689 INFO
>  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Detected private network label : vSwitch0,
> 2013-10-23 05:36:03,689 DEBUG [vmware.resource.VmwareContextFactory]
> (catalina-exec-22:null) initialize VmwareContext. url:
> https://10.199.0.40/sdk/vimService, username: Administrator, password:
> r***
> 2013-10-23 05:36:03,817 INFO  [vmware.util.VmwareContext]
> (catalina-exec-22:null) New VmwareContext object, current outstanding
> count: 1
> 2013-10-23 05:36:03,952 INFO  [vmware.manager.VmwareManagerImpl]
> (catalina-exec-22:null) Preparing network on host
> com.cloud.hypervisor.vmware.util.VmwareContext@60e2e3d4 for vSwitch0,
> 2013-10-23 05:36:03,999 ERROR [vmware.mo.HypervisorHostHelper]
> (catalina-exec-22:null) Unable to find vSwitchvSwitch0,
> 2013-10-23 05:36:03,999 WARN
>  [hypervisor.vmware.VmwareServerDiscoverer]
> (catalina-exec-22:null) Unable to connect to Vmware vSphere server.
> service address: 10.199.0.40
> 2013-10-23 05:36:04,012 WARN  [cloud.resource.ResourceManagerImpl]
> (catalina-exec-22:null) Unable to find the server resources at
> http://10.199.0.40/cs4/cs4it
> 2013-10-23 05:36:04,014 INFO  [utils.exception.CSExceptionErrorCode]
> (catalina-exec-22:null) Could not find exception:
> com.cloud.exception.DiscoveryException in error code list for
> exceptions
> 2013-10-23 05:36:04,095 WARN  [admin.cluster.AddClusterCmd]
> (catalina-exec-22:null) Exception:
> com.cloud.exception.DiscoveryException: Unable to add the external
> cluster
> at
> 
> com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.java:533)
> at
> 
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125

Re: LXC and SSVM/CPVM on the host

2013-10-24 Thread Francois Gaudreault

If this is the case, then you should remove the ability to create LXC 
zones or clarify the documentation about that.


According to the wiki page:

Each of the different hypervisors currently have their own System VMs. 
These system VM images are used to run a console proxy, secondary 
storage, and router VMs.


We discussed the possibility of creating System VMs for LXC. There was 
concern with the complexity and potential issues involving iptables for 
the router inside an LXC container. As an intermediate solution we are 
going to use KVM System VMs inside the LXC Cluster.


So we need a KVM cluster to run the VMs? (Added the author of the feature)

Francois

On 10/22/2013, 1:24 AM, Chiradeep Vittal wrote:

As far as I understand, in an LXC scenario, the system vms are expected to
run on real hypervisors.
You can always use the QuickCloud way to not use system vms at all.

On 10/21/13 1:45 PM, "Francois Gaudreault" 
wrote:


Ok I think we have to look at this further. I'll stop hijacking other
threads.

I am trying to get the SSVM/CPVM to run on a LXC host. The SSVM/CPVM
starts, get IPs, but then CloudStack kill them for some reason. Yes, I
use the 4.2 images :

2013-10-21 16:19:21,605 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 73--1: Processing Seq 73--1:  { Cmd ,
MgmtId: -1, via: 73, Ver: v1, Flags: 111,
[{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","wait":0}}] }
2013-10-21 16:19:21,605 INFO  [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) Host 73 has informed us that it is
shutting down with reason sig.kill and detail null
2013-10-21 16:19:21,606 INFO  [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Host 73 is disconnecting with event
ShutdownRequested
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) The next status of agent 73is Disconnected,
current status is Up
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Deregistering link for 73 with state Disconnected
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Remove Agent : 73
2013-10-21 16:19:21,609 DEBUG [agent.manager.ConnectedAgentAttache]
(AgentTaskPool-11:null) Processing Disconnect.

I transferred the host to KVM, and now the same SSVM/CPVM images are
running fine for the last 30min ( so I assume it works fine...).
Something seems to be wrong with the LXC side :S

Anyone wants to invest some time to troubleshoot? I'll open a ticket also.

--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_






--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_

Metadata URL location

Hi Guys,

CloudStack metadata services are on the default gateway while on EC2,
its at 169.254.169.254. Am curious to know why CloudStack does not
use a link local address for meta data services.

A search of the Wiki
(https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearch=true&queryString=metadata)
didn’t seem to list any doc related to the design of this service.

TIA.

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre,
Bangalore - 560 055

CloudStack Bootcamp Training on 27/28 November, Bangalore
http://www.shapeblue.com/cloudstack-training/

This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Shape Blue Ltd or related companies. If you are not the
intended recipient of this email, you must neither take any action based upon
its contents, nor copy or show it to anyone. Please contact the sender if you
believe you have received this email in error. Shape Blue Ltd is a company
incorporated in England & Wales. ShapeBlue Services India LLP is a company
incorporated in India and is operated under license from Shape Blue Ltd. Shape
Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is
operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: git commit: Add support for using username and password to cloudmonkey

On 16-Oct-2013, at 8:47 am, chirad...@apache.org wrote:

> Updated Branches:
>  refs/heads/username_password_support [created] 61901f20d
>
>
> Add support for using username and password to cloudmonkey.
> The interactive CLI user can call 'login' and 'logout' to start and terminate 
> a session.
> If the apikey and secretkey are not supplied in the config, then the username 
> and password are used.
> If the session has expired, the CLI will automatically login and obtain a 
> session

Thanks. Solves a TON of automation problems. :)

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
Bangalore - 560 055

CloudStack Bootcamp Training on 27/28 November, Bangalore
http://www.shapeblue.com/cloudstack-training/

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not the 
intended recipient of this email, you must neither take any action based upon 
its contents, nor copy or show it to anyone. Please contact the sender if you 
believe you have received this email in error. Shape Blue Ltd is a company 
incorporated in England & Wales. ShapeBlue Services India LLP is a company 
incorporated in India and is operated under license from Shape Blue Ltd. Shape 
Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: [ASF4.2.1] Progress

2013-10-24 Thread Abhinandan Prateek

Cloudstack-4828 is targeted for 4.2.1. And as of now is resolved.

On 21/10/13 10:07 pm, "Patrick Miller" 
mailto:patrick.mil...@sungard.com>> wrote:

 Hi,
   I posted a question on dev Friday regarding Cloudstack-4828.  This was 
broken between 4.1 and 4.2.
   What is the corrrect process to get this in the queue for 4.2.1.

 Thanks for all the great work all of you have put in to this.

 Patrick


On Sun, Oct 20, 2013 at 8:42 PM, Abhinandan Prateek 
mailto:abhinandan.prat...@citrix.com>> wrote:
With active help of community and dedicated engineers from Citrix we have made 
some tremendous progress in fixing blockers and criticals targeted for this 
maintenance release. In last two weeks around 70 tickets were resolved (~38 
blockers and criticals).

There are roughly 15 criticals/blockers ticket open now. Our target is to be 
code complete early this week and we are on our way to meet it. A timely 
maintenance release will also help the next big release.

-abhi

Re: [ASF4.2.1] default to 64-bit system VM template

2013-10-24 Thread Abhinandan Prateek

Looked at the scripts found minor differences have fixed those commit
499a8c0915dd25b3d9c813aa1b715ba9ba865ffb.

On 24/10/13 5:47 am, "Darren Shepherd"  wrote:

>Slightly off topic but I've noticed that the scripts used to create the
>vm templates are slightly out of sync between 32bit and 64bit.  I just
>assumed nobody used 64bit and that's why it got out of sync.
>
>Darren
>
>> On Oct 23, 2013, at 7:21 AM, Marty Sweet  wrote:
>> 
>> What would be the main reasoning behind this change? Surely the
>> minimalistic specifications that SystemVMs are assigned suit 32-bit
>>better
>> in terms of Memory?
>> 
>> "The main disadvantage of 64-bit architectures is that, relative to
>>32-bit
>> architectures, the same data occupies more space in memory (due to
>>longer
>> pointers and possibly other types, and alignment padding). This
>>increases
>> the memory requirements of a given process and can have implications for
>> efficient processor cache utilization."
>> 
>> 
>> On Wed, Oct 23, 2013 at 1:48 PM, Abhinandan Prateek <
>> abhinandan.prat...@citrix.com> wrote:
>> 
 On 23/10/13 3:48 pm, "David Nalley"  wrote:
 
 On Wed, Oct 23, 2013 at 1:32 AM, Abhinandan Prateek
  wrote:
> 
>  We are planning to  make 64-bit system vm templates as default
> offering in 4.2.1.
> This is an initial email to have thoughts from the community on this.
> 
> -abhi
 
 -1
 Who is this we?
 I agree with Wido We adhere to semver, and accordingly 4.2.1 should be
 a bugfix release.
>>> 
>>> We were some of the fellow committers, even I was of the same thinking
>>>but
>>> thought it will be good to have a general opinion.
>>> 
>>> -abhi
>>> 
>>>

Re: Review Request 14319: CLOUDSTACK 2238: Automation - Non Contiguous VLAN Ranges

2013-10-24 Thread Gaurav Aradhye


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14319/
---

(Updated Oct. 24, 2013, 7:37 p.m.)


Review request for cloudstack, bharat kumar and sanjeev n.


Changes
---

Review Changes.


Repository: cloudstack-git


Description
---

Adding Automation test cases for feature - Non Contiguous VLAN ranges
CLOUDSTACk 2238.


Diffs (updated)
-

  test/integration/component/test_non_contiguous_vlan.py PRE-CREATION 

Diff: https://reviews.apache.org/r/14319/diff/


Testing (updated)
---

Tested again after review changes:

Log:

test_01_add_non_contiguous_ranges 
(test_non_contiguous_vlan.TestNonContiguousVLANRanges) ... ok
test_02_add_existing_vlan_range 
(test_non_contiguous_vlan.TestNonContiguousVLANRanges) ... ok
test_03_extend_contiguous_range 
(test_non_contiguous_vlan.TestNonContiguousVLANRanges) ... ok
test_04_remove_unused_range 
(test_non_contiguous_vlan.TestNonContiguousVLANRanges) ... ok
test_05_remove_used_range 
(test_non_contiguous_vlan.TestNonContiguousVLANRanges) ... ok

--
Ran 5 tests in 345.586s

OK


Thanks,

Gaurav Aradhye

Re: Review Request 14905: Add global configuration parameters for SMTP timeout

2013-10-24 Thread Likitha Shetty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14905/#review27455
---

Ship it!


Ship It!

- Likitha Shetty


On Oct. 24, 2013, 10:57 a.m., Anshul Gangwar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14905/
> ---
> 
> (Updated Oct. 24, 2013, 10:57 a.m.)
> 
> 
> Review request for cloudstack, Devdeep Singh, Likitha Shetty, and Sateesh 
> Chodapuneedi.
> 
> 
> Bugs: CLOUDSTACK-4909
> https://issues.apache.org/jira/browse/CLOUDSTACK-4909
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> This will make the SMTP timeout parameters configurable. Default timeout is 
> 30 seconds.
> 
> 
> Diffs
> -
> 
>   server/src/com/cloud/alert/AlertManagerImpl.java 3aa9695 
>   server/src/com/cloud/configuration/Config.java cefcdd5 
> 
> Diff: https://reviews.apache.org/r/14905/diff/
> 
> 
> Testing
> ---
> 
> tested on local setup
> 
> 
> Thanks,
> 
> Anshul Gangwar
> 
>

Re: is marvin on master broken?

Traceback (most recent call last):
  File "tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,
AttributeError: 'NoneType' object has no attribute 'debug'

On Thu, Oct 24, 2013 at 1:21 AM, Santhosh Edukulla
 wrote:
> Hello Darren,
>
> Some trace is still missing i believe. i could not see the last stack frame 
> in the below trace as what lead to this trace?
>
> I just pulled the latest from master branch and used marvin  to deploy few 
> cloudstack entities and it worked. Can you please provide the command you are 
> using to run marvin tests? or what command lead to the below trace?
>
> Regards,
> Santhosh
> 
> From: Darren Shepherd [darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 3:48 AM
> To: dev@cloudstack.apache.org
> Subject: is marvin on master broken?
>
> Whenever I use marvin on master I get
>
> Traceback (most recent call last):
>   File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
> deploy.deploy()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
> self.loadCfg()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 492, in 
> registerApiKey
> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 505, in listUsers
> response = self.connection.marvinRequest(command,
> response_type=response, method=method)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
> line 269, in marvinRequest
> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
>
> So it looks like cloudConnection needs a logger passed to the
> constructor and its not being passed?  I've just been hacking up
> ./tools/marvin/marvin/cloudstackConnection.py to work around this.
>
> Darren

Re: Review Request 14867: api call to import ldap users to the same domains in cloudstack

2013-10-24 Thread Ian Duffy



> On Oct. 23, 2013, 6:22 a.m., Prasanna Santhanam wrote:
> > How is the import behaviour when OUs contain other OUs?
> 
> Rajani Karuturi wrote:
> That is still not done. (Its the second part of TODO:1, nested hierarchy)
> 
> Prasanna Santhanam wrote:
> Ah, didn't notice the TODO. Is there a spec file on how this is planned 
> to be handled? Right now it seems you will just create domains under ROOT 
> behind-the-scenes?
> 
> Rajani Karuturi wrote:
> Nothing yet. I will update the wiki page created by Ian 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/LDAP+implementation+improvement+and+user+provisioning
> Yes, right now it just imports the same hierarchy(1-level) to cloudstack 
> under ROOT with the specified account type.
> 
> Ian Duffy wrote:
> Thanks Rajani.
> I will try look at this tonight, currently a bit tied with college 
> assignments.

This looks OK to me, other than the highlighted todos.
It would be nice to include a ou/nested-on in the flat ldif 
file(plugins/user-authenticators/ldap/test/resources) that the embedded 
apacheds server uses to populate.

@aprateek can I sign off on this or do you wish to look at it aswell?


- Ian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14867/#review27362
---


On Oct. 23, 2013, 5:26 a.m., Rajani Karuturi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14867/
> ---
> 
> (Updated Oct. 23, 2013, 5:26 a.m.)
> 
> 
> Review request for cloudstack, Abhinandan Prateek and Ian Duffy.
> 
> 
> Bugs: CLOUDSTACK-4866
> https://issues.apache.org/jira/browse/CLOUDSTACK-4866
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> Added an api call to import all the ldap users to the same domains(ou's) in 
> cloudstack
> 
> TODO:
> 1. error handling of no domains present, nested hierarchy
> 2. handling the case when the api call fails for a specific user/users
> 3. test cases for LdapUserManager
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/user/DomainService.java 7c302e3 
>   client/tomcatconf/commands.properties.in 0296de0 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/api/command/LdapImportUsersCmd.java
>  PRE-CREATION 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/api/response/LdapUserResponse.java
>  9b21c8f 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapConfiguration.java
>  0cfb37c 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapManagerImpl.java
>  87406ad 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapUser.java 
> 18ad7d9 
>   
> plugins/user-authenticators/ldap/src/org/apache/cloudstack/ldap/LdapUserManager.java
>  7494346 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapConfigurationSpec.groovy
>  c593959 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapImportUsersCmdSpec.groovy
>  PRE-CREATION 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapListUsersCmdSpec.groovy
>  5039443 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapManagerImplSpec.groovy
>  d681eac 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapSearchUserCmdSpec.groovy
>  fce299d 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapUserResponseSpec.groovy
>  f1978fa 
>   
> plugins/user-authenticators/ldap/test/groovy/org/apache/cloudstack/ldap/LdapUserSpec.groovy
>  8fd1ccc 
>   server/src/com/cloud/user/DomainManagerImpl.java b885c48 
>   server/test/com/cloud/user/MockDomainManagerImpl.java 616e12d 
> 
> Diff: https://reviews.apache.org/r/14867/diff/
> 
> 
> Testing
> ---
> 
> testing is done except for LdapUserManager(for which i am facing some issues 
> locally) and DomainService(for which no test cases exist currently)
> 
> 
> Thanks,
> 
> Rajani Karuturi
> 
>

Re: Metadata URL location

My guess, I don't really know, would be because its hard.  The VR uses
link local for the control network so 169.254/16 is bound to the wrong
nic.  To fix this you just need some ip routing magic in linux (credit
goes to Kris Lindgren who showed me how to do this).  Add the below to
a file, substitute eth0 for the guest network nic, run "ip -b "
The below effectively creates a routing table dedicated to the IP
169.254.169.254 that sets it default route to go out the guest network
nic.

rule add from 169.254.169.254 table 70
rule add to 169.254.169.254 dev eth0 table 70
route flush table 70
route add default dev eth0 src 169.254.169.254 table 70
route flush cache

Darren

On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
 wrote:
> Hi Guys,
>
> CloudStack metadata services are on the default gateway while on EC2,
> its at 169.254.169.254. Am curious to know why CloudStack does not
> use a link local address for meta data services.
>
> A search of the Wiki 
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearch=true&queryString=metadata)
>  didn’t seem to list any doc related to the design of this service.
>
> TIA.
>
> --
> @shankerbalan
>
> M: +91 98860 60539 | O: +91 (80) 67935867
> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
> Bangalore - 560 055
>
> CloudStack Bootcamp Training on 27/28 November, Bangalore
> http://www.shapeblue.com/cloudstack-training/
>
>
>
>
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark.

Re: Metadata URL location

So additionally you need to do

ip addr add dev eth0 169.254.169.254/0

On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren  wrote:
> You would also need to supernet 169.254.169.254 on the virtual router
> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it will
> still arp to servers that are calling it that have real ip addresses.
>
> Additionally we had some other iptables rules in there that would change
> the the ip address of the incoming request to metadata based upon the mac
> address that was hitting it.  This was to prevent spoofing of another vm's
> IP and getting someone else's metadata (at least in our metadata
> implementation we keyed off of the VM IP calling into metadata).  This
> also allowed a user to set whatever ipaddress they wanted, but as long as
> the mac address was the same and they still had a zeroconfig route on the
> VM, they still got only their metadata.
> 
>
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
>
> This email message and any attachment(s) hereto are intended for use only
> by its intended recipient(s) and may contain confidential information. If
> you have received this email in error, please immediately notify the
> sender and permanently delete the original and any copy of this message
> and its attachments.
>
>
>
>
>
>
>
> On 10/24/13 9:12 AM, "Darren Shepherd"  wrote:
>
>>My guess, I don't really know, would be because its hard.  The VR uses
>>link local for the control network so 169.254/16 is bound to the wrong
>>nic.  To fix this you just need some ip routing magic in linux (credit
>>goes to Kris Lindgren who showed me how to do this).  Add the below to
>>a file, substitute eth0 for the guest network nic, run "ip -b "
>>The below effectively creates a routing table dedicated to the IP
>>169.254.169.254 that sets it default route to go out the guest network
>>nic.
>>
>>rule add from 169.254.169.254 table 70
>>rule add to 169.254.169.254 dev eth0 table 70
>>route flush table 70
>>route add default dev eth0 src 169.254.169.254 table 70
>>route flush cache
>>
>>Darren
>>
>>On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>> wrote:
>>> Hi Guys,
>>>
>>> CloudStack metadata services are on the default gateway while on EC2,
>>> its at 169.254.169.254. Am curious to know why CloudStack does not
>>> use a link local address for meta data services.
>>>
>>> A search of the Wiki
>>>(https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK
>>>&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearc
>>>h=true&queryString=metadata) didn¹t seem to list any doc related to the
>>>design of this service.
>>>
>>> TIA.
>>>
>>> --
>>> @shankerbalan
>>>
>>> M: +91 98860 60539 | O: +91 (80) 67935867
>>> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
>>> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
>>>Centre, Bangalore - 560 055
>>>
>>> CloudStack Bootcamp Training on 27/28 November, Bangalore
>>> http://www.shapeblue.com/cloudstack-training/
>>>
>>>
>>>
>>>
>>> This email and any attachments to it may be confidential and are
>>>intended solely for the use of the individual to whom it is addressed.
>>>Any views or opinions expressed are solely those of the author and do
>>>not necessarily represent those of Shape Blue Ltd or related companies.
>>>If you are not the intended recipient of this email, you must neither
>>>take any action based upon its contents, nor copy or show it to anyone.
>>>Please contact the sender if you believe you have received this email in
>>>error. Shape Blue Ltd is a company incorporated in England & Wales.
>>>ShapeBlue Services India LLP is a company incorporated in India and is
>>>operated under license from Shape Blue Ltd. Shape Blue Brasil
>>>Consultoria Ltda is a company incorporated in Brasil and is operated
>>>under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

RE: Command sequence logic in agent code

2013-10-24 Thread Alex Huang

I only took a quick look here.  I think when commands originate from the 
management server, it doesn't go through this code.  This is if the command 
came from the agent which doesn't matter if there's multiple management servers.

--Alex

> -Original Message-
> From: Koushik Das
> Sent: Thursday, October 24, 2013 2:11 AM
> To: 
> Cc: Alex Huang
> Subject: Re: Command sequence logic in agent code
> 
> Created https://issues.apache.org/jira/browse/CLOUDSTACK-4944 to track
> this issue.
> 
> Alex, Any reason for adding requests based on sequence and not doing FIFO?
> Do you see any issues if request always gets added to the end of the queue?
> 
> 
> On 23-Oct-2013, at 6:26 PM, Koushik Das  wrote:
> 
> > I was looking at the command sequencing logic in the agent code.
> >
> > Each agent maintains a sequence that gets initialised based on following
> logic
> >
> >private static final Random s_rand = new
> Random(System.currentTimeMillis());
> >_nextSequence = s_rand.nextInt(Short.MAX_VALUE) << 48;
> >
> > For every command that gets processed by the agent the sequence is
> incremented by 1. If commands are to be executed in sequence then they
> are queued up based on this sequence
> >
> >protected synchronized void addRequest(Request req) {
> >int index = findRequest(req);
> >assert (index < 0) : "How can we get index again? " + index + ":" +
> req.toString();
> >_requests.add(-index - 1, req);
> >}
> >
> > The above works fine in case of a single MS scenario. In case of a clustered
> MS setup things change slightly.
> >
> > The command can originate at any MS and based on the ownership of the
> agent, it gets forwarded to the correct MS which then handles the command.
> Now command sequences are local to individual agents in MS. In this case
> the originating MS agent tags the request with a sequence. This gets
> forwarded to the owning MS and based on if 'executedInSequence' flag is
> set, gets added to the list based on the sequence number. Now here lies the
> problem, commands are not inserted in the order in which they arrive but
> based on the sequence number. In case of a forwarded command the
> sequence is different from the local sequence. If the starting sequence of
> forwarded commands is much less than that of the locally generated
> commands then there is a possibility of local commands getting starved if
> there is a steady arrival of forwarded commands. Similarly it can also happen
> the other way round. Also if the the starting sequence for a agent in local 
> and
> peer MS is not spread far apart then there may be overlaps and a new
> request will override the old one.
> >
> > Not sure if anyone encountered any issues due to this. The correct way
> looks like to implement the queue model rather than doing a add based on
> the above code.
> >
> > Comments?
> >
> > -Koushik
> >
> >
> >
> >
> >

Re: LXC and SSVM/CPVM on the host

2013-10-24 Thread Francois Gaudreault

If it's designed to do that, then something is wrong with how CS deals 
with it.


When I was trying to get the KVM images to work, they were starting, 
getting IPs, but then something was killing the VM. I though for 
sometime that libvirt was the issue, so I tried Ubuntu 13.10, 12.04 and 
CentOS with the same results. I then switched the hypervisor type in CS 
from LXC to KVM (rebuilt the zone), keep the same settings on my host, 
and the System VMs are running fine since then.


Anyone have time to help me troubleshoot? I mean, this is not a blocker, 
but I can't get standalone LXC cluster to work...


Francois

On 10/24/2013, 11:53 AM, Phong Nguyen wrote:
> So we need a KVM cluster to run the VMs? (Added the author of the 
feature)


As it was originally discussed and implemented, the decision was to 
use KVM as the system VM for LXC clusters instead of creating an LXC 
system VM. A zone with only LXC clusters will deploy a KVM system VM 
on a host running an LXC agent. Behind the scenes, this is possible 
because both KVM and LXC agents use libvirt for provisioning (and that 
the setup of an LXC agent is almost identical to KVM and perfectly 
capable of running KVM VMs).


-Phong


On Thu, Oct 24, 2013 at 8:57 AM, Francois Gaudreault 
mailto:fgaudrea...@cloudops.com>> wrote:


If this is the case, then you should remove the ability to create
LXC zones or clarify the documentation about that.

According to the wiki page:

Each of the different hypervisors currently have their own System
VMs. These system VM images are used to run a console proxy,
secondary storage, and router VMs.

We discussed the possibility of creating System VMs for LXC. There
was concern with the complexity and potential issues involving
iptables for the router inside an LXC container. As an
intermediate solution we are going to use KVM System VMs inside
the LXC Cluster.

So we need a KVM cluster to run the VMs? (Added the author of the
feature)

Francois

On 10/22/2013, 1:24 AM, Chiradeep Vittal wrote:

As far as I understand, in an LXC scenario, the system vms are
expected to
run on real hypervisors.
You can always use the QuickCloud way to not use system vms at
all.

On 10/21/13 1:45 PM, "Francois Gaudreault"
mailto:fgaudrea...@cloudops.com>>
wrote:

Ok I think we have to look at this further. I'll stop
hijacking other
threads.

I am trying to get the SSVM/CPVM to run on a LXC host. The
SSVM/CPVM
starts, get IPs, but then CloudStack kill them for some
reason. Yes, I
use the 4.2 images :

2013-10-21 16:19:21,605 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 73--1: Processing Seq
73--1:  { Cmd ,
MgmtId: -1, via: 73, Ver: v1, Flags: 111,

[{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","wait":0}}]
}
2013-10-21 16:19:21,605 INFO  [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) Host 73 has informed us that
it is
shutting down with reason sig.kill and detail null
2013-10-21 16:19:21,606 INFO  [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Host 73 is disconnecting with event
ShutdownRequested
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) The next status of agent 73is
Disconnected,
current status is Up
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Deregistering link for 73 with
state Disconnected
2013-10-21 16:19:21,609 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Remove Agent : 73
2013-10-21 16:19:21,609 DEBUG
[agent.manager.ConnectedAgentAttache]
(AgentTaskPool-11:null) Processing Disconnect.

I transferred the host to KVM, and now the same SSVM/CPVM
images are
running fine for the last 30min ( so I assume it works
fine...).
Something seems to be wrong with the LXC side :S

Anyone wants to invest some time to troubleshoot? I'll
open a ticket also.

-- 
Francois Gaudreault

Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com 
514-629-6775 
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com 
@CloudOps_




-- 
Francois Gaudreault

Architecte de Solution Cloud | Cloud Solutions Architect
fgaudrea...@cloudops.com

Re: Review Request 14889: Expose as much type information to commands.xml as well as to the generated python clasesss for Marvin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14889/#review27461
---

Ship it!


Looks cool to me, will give Prasanna time to respond.


- Marcus Sorensen


On Oct. 23, 2013, 9:38 p.m., Ryan Dietrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14889/
> ---
> 
> (Updated Oct. 23, 2013, 9:38 p.m.)
> 
> 
> Review request for cloudstack, Marcus Sorensen and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> I have a requirement for some work I am doing to "fail early".  The existing 
> marvin cloudstackAPI classes give me both the "legal" and "required" elements 
> for each API call as well as the response attributes.  However, they do not 
> tell me the types of each value.  I've gone ahead and exposed that type 
> information to python and to commands.xml as well.  I decided to try and 
> expose things so we're being as descriptive as possible.  It's a little 
> deflating how the Request/Response annotations aren't equal (Requests can 
> expose "strings" as "uuid's", allowing me to auto generate regex's for 
> validation).  Responses however, simply fall back to the specific java type, 
> (Integer, Long, String, Boolean).
> 
> I believe cloudmonkey could take advantage of this, by stopping invalid 
> parameters from being sent to Cloudstack in the first place (client side 
> validation is good, right?)
> I also think the UI could use this as part of their validation routines, if 
> they're using commands.xml as part of their build process.
> 
> 
> Diffs
> -
> 
>   server/src/com/cloud/api/doc/ApiXmlDocWriter.java c3c0cab 
>   server/src/com/cloud/api/doc/Argument.java 29c361e 
>   tools/marvin/marvin/codegenerator.py 96729f6 
> 
> Diff: https://reviews.apache.org/r/14889/diff/
> 
> 
> Testing
> ---
> 
> mvn clean
> mvn
> cd tools/apidoc
> mvn
> inspect target/commands.xml
> cd ../tools/marvin
> mvn
> inspect python classes
> 
> 
> Thanks,
> 
> Ryan Dietrich
> 
>

Re: Command sequence logic in agent code

2013-10-24 Thread Koushik Das

I came across some MS logs which showed that commands originating locally are 
getting starved (timed out and then cancelled since it never got a chance to 
execute) and the forwarded commands from another MS gets executed even though 
the local commands got scheduled earlier. A huge number of forwarded commands 
arrived in a quick succession. The sequence numbers of the forwarded commands 
were much less than the local commands and so the forwarded commands always got 
added ahead of the local ones.
  
On 24-Oct-2013, at 9:34 PM, Alex Huang  wrote:

> I only took a quick look here.  I think when commands originate from the 
> management server, it doesn't go through this code.  This is if the command 
> came from the agent which doesn't matter if there's multiple management 
> servers.
> 
> --Alex
> 
>> -Original Message-
>> From: Koushik Das
>> Sent: Thursday, October 24, 2013 2:11 AM
>> To: 
>> Cc: Alex Huang
>> Subject: Re: Command sequence logic in agent code
>> 
>> Created https://issues.apache.org/jira/browse/CLOUDSTACK-4944 to track
>> this issue.
>> 
>> Alex, Any reason for adding requests based on sequence and not doing FIFO?
>> Do you see any issues if request always gets added to the end of the queue?
>> 
>> 
>> On 23-Oct-2013, at 6:26 PM, Koushik Das  wrote:
>> 
>>> I was looking at the command sequencing logic in the agent code.
>>> 
>>> Each agent maintains a sequence that gets initialised based on following
>> logic
>>> 
>>>   private static final Random s_rand = new
>> Random(System.currentTimeMillis());
>>>   _nextSequence = s_rand.nextInt(Short.MAX_VALUE) << 48;
>>> 
>>> For every command that gets processed by the agent the sequence is
>> incremented by 1. If commands are to be executed in sequence then they
>> are queued up based on this sequence
>>> 
>>>   protected synchronized void addRequest(Request req) {
>>>   int index = findRequest(req);
>>>   assert (index < 0) : "How can we get index again? " + index + ":" +
>> req.toString();
>>>   _requests.add(-index - 1, req);
>>>   }
>>> 
>>> The above works fine in case of a single MS scenario. In case of a clustered
>> MS setup things change slightly.
>>> 
>>> The command can originate at any MS and based on the ownership of the
>> agent, it gets forwarded to the correct MS which then handles the command.
>> Now command sequences are local to individual agents in MS. In this case
>> the originating MS agent tags the request with a sequence. This gets
>> forwarded to the owning MS and based on if 'executedInSequence' flag is
>> set, gets added to the list based on the sequence number. Now here lies the
>> problem, commands are not inserted in the order in which they arrive but
>> based on the sequence number. In case of a forwarded command the
>> sequence is different from the local sequence. If the starting sequence of
>> forwarded commands is much less than that of the locally generated
>> commands then there is a possibility of local commands getting starved if
>> there is a steady arrival of forwarded commands. Similarly it can also happen
>> the other way round. Also if the the starting sequence for a agent in local 
>> and
>> peer MS is not spread far apart then there may be overlaps and a new
>> request will override the old one.
>>> 
>>> Not sure if anyone encountered any issues due to this. The correct way
>> looks like to implement the queue model rather than doing a add based on
>> the above code.
>>> 
>>> Comments?
>>> 
>>> -Koushik
>>> 
>>> 
>>> 
>>> 
>>> 
>

RE: Metadata URL location

2013-10-24 Thread Alex Huang

In order to use an link local address inside the end user vm, that metadata 
service must be setup on every hypervisor's dom0 or it has to be proxied out of 
the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to serve 
the data, which works for all three hypervisors.  

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 8:13 AM
> To: dev@cloudstack.apache.org
> Cc: klindg...@godaddy.com
> Subject: Re: Metadata URL location
> 
> My guess, I don't really know, would be because its hard.  The VR uses link
> local for the control network so 169.254/16 is bound to the wrong nic.  To fix
> this you just need some ip routing magic in linux (credit goes to Kris 
> Lindgren
> who showed me how to do this).  Add the below to a file, substitute eth0 for
> the guest network nic, run "ip -b "
> The below effectively creates a routing table dedicated to the IP
> 169.254.169.254 that sets it default route to go out the guest network nic.
> 
> rule add from 169.254.169.254 table 70
> rule add to 169.254.169.254 dev eth0 table 70 route flush table 70 route add
> default dev eth0 src 169.254.169.254 table 70 route flush cache
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>  wrote:
> > Hi Guys,
> >
> > CloudStack metadata services are on the default gateway while on EC2,
> > its at 169.254.169.254. Am curious to know why CloudStack does not use
> > a link local address for meta data services.
> >
> > A search of the Wiki
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDST
> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
> spaceSearch=true&queryString=metadata) didn't seem to list any doc
> related to the design of this service.
> >
> > TIA.
> >
> > --
> > @shankerbalan
> >
> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.ba...@shapeblue.com
> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
> >
> > CloudStack Bootcamp Training on 27/28 November, Bangalore
> > http://www.shapeblue.com/cloudstack-training/
> >
> >
> >
> >
> > This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender if
> you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape
> Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.

Re: Metadata URL location

On 24-Oct-2013, at 9:21 pm, Darren Shepherd  wrote:

> So additionally you need to do
>
> ip addr add dev eth0 169.254.169.254/0

Thanks Kris and Darren. Thats very useful information.

The reason I ask is currently there is a bit of heuristics involved
in obtaining the meta data server IP from the DHCP lease files.

Scripts are not portable across different OSes and distribution
as they use different DHCP clients.

Having a consistent well known meta data server IP 169.254.169.254 would be 
nice. :)

>
> On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren  
> wrote:
>> You would also need to supernet 169.254.169.254 on the virtual router
>> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it will
>> still arp to servers that are calling it that have real ip addresses.
>>
>> Additionally we had some other iptables rules in there that would change
>> the the ip address of the incoming request to metadata based upon the mac
>> address that was hitting it.  This was to prevent spoofing of another vm's
>> IP and getting someone else's metadata (at least in our metadata
>> implementation we keyed off of the VM IP calling into metadata).  This
>> also allowed a user to set whatever ipaddress they wanted, but as long as
>> the mac address was the same and they still had a zeroconfig route on the
>> VM, they still got only their metadata.
>> 
>>
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy, LLC.
>>
>>
>> This email message and any attachment(s) hereto are intended for use only
>> by its intended recipient(s) and may contain confidential information. If
>> you have received this email in error, please immediately notify the
>> sender and permanently delete the original and any copy of this message
>> and its attachments.
>>
>>
>>
>>
>>
>>
>>
>> On 10/24/13 9:12 AM, "Darren Shepherd"  wrote:
>>
>>> My guess, I don't really know, would be because its hard.  The VR uses
>>> link local for the control network so 169.254/16 is bound to the wrong
>>> nic.  To fix this you just need some ip routing magic in linux (credit
>>> goes to Kris Lindgren who showed me how to do this).  Add the below to
>>> a file, substitute eth0 for the guest network nic, run "ip -b "
>>> The below effectively creates a routing table dedicated to the IP
>>> 169.254.169.254 that sets it default route to go out the guest network
>>> nic.
>>>
>>> rule add from 169.254.169.254 table 70
>>> rule add to 169.254.169.254 dev eth0 table 70
>>> route flush table 70
>>> route add default dev eth0 src 169.254.169.254 table 70
>>> route flush cache
>>>
>>> Darren
>>>
>>> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>>>  wrote:
 Hi Guys,

 CloudStack metadata services are on the default gateway while on EC2,
 its at 169.254.169.254. Am curious to know why CloudStack does not
 use a link local address for meta data services.

 A search of the Wiki
 (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK
 &tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearc
 h=true&queryString=metadata) didn¹t seem to list any doc related to the
 design of this service.

 TIA.

 --
 @shankerbalan

 M: +91 98860 60539 | O: +91 (80) 67935867
 shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
 ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
 Centre, Bangalore - 560 055

 CloudStack Bootcamp Training on 27/28 November, Bangalore
 http://www.shapeblue.com/cloudstack-training/

 This email and any attachments to it may be confidential and are
 intended solely for the use of the individual to whom it is addressed.
 Any views or opinions expressed are solely those of the author and do
 not necessarily represent those of Shape Blue Ltd or related companies.
 If you are not the intended recipient of this email, you must neither
 take any action based upon its contents, nor copy or show it to anyone.
 Please contact the sender if you believe you have received this email in
 error. Shape Blue Ltd is a company incorporated in England & Wales.
 ShapeBlue Services India LLP is a company incorporated in India and is
 operated under license from Shape Blue Ltd. Shape Blue Brasil
 Consultoria Ltda is a company incorporated in Brasil and is operated
 under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

--
@shankerbalan

M: +91 98860 60539 | O: +91 (80) 67935867
shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
Bangalore - 560 055

CloudStack Bootcamp Training on 27/28 November, Bangalore
http://www.shapeblue.com/cloudstack-training/

This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. A

Re: Review Request 14843: CLOUDSTACK-1049 : git commit by using command cloudstack-sccs from installed setup

2013-10-24 Thread ASF Subversion and Git Services


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14843/#review27466
---


Commit 435689c9ea9959492068c235cb73e7745541124e in branch refs/heads/4.2 from 
rayeesn
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=435689c ]

CLOUDSTACK-1049 : git commit from installed cloudstack setup


- ASF Subversion and Git Services


On Oct. 23, 2013, 12:12 a.m., Rayees Namathponnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14843/
> ---
> 
> (Updated Oct. 23, 2013, 12:12 a.m.)
> 
> 
> Review request for cloudstack, Chip Childers, Frank Zhang, and Hugo Trippaers.
> 
> 
> Bugs: CLOUDSTACK-1049
> https://issues.apache.org/jira/browse/CLOUDSTACK-1049
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> The generation of sccs-info was done by WAF but has been broken from 4.2 
> onwords;
> 
> updated cloud.spec file to generate git commit and package with rpm
> 
> if you are building outside git tree, this file will be blank; and packaging 
> works fine
> 
> 
> Diffs
> -
> 
>   packaging/centos63/cloud.spec 17fb2b1 
>   packaging/centos63/cloudstack-sccs PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/14843/diff/
> 
> 
> Testing
> ---
> 
> Tested
> 
> 
> File Attachments
> 
> 
> We need to merge with 4.2 branch; submitting patch for this
>   
> https://reviews.apache.org/media/uploaded/files/2013/10/23/0001-CLOUDSTACK-1049-git-commit-from-installed-cloudstack.patch
> 
> 
> Thanks,
> 
> Rayees Namathponnan
> 
>

RE: is marvin on master broken?

2013-10-24 Thread Santhosh Edukulla

Hi Darren,

1. I just ran again using  the latest nosetests-2.7 with  marvin-plugin, 
marvin-config=, it went fine with no 
issues. 
 
2. Please check whether the below contents are available under configuration 
file passed to marvin or nosetests. Check advanced.cfg under ( setup/dev/ ) for 
reference. In short, it seems as per trace, under loadCfg , testClientLogFile 
which is getting loaded based upon below values from config is None, leading to 
further issues and so below trace.

For passing the correct config :

 If we are using nosetests to run the marvin tests, please use 
--marvin-config=""  ( or  )

 If we are using deployAndRun, please use --config ="".

"logger": [
{
"name": "TestClient",
"file": "/tmp/testclient.log"
},
{
"name": "TestCase",
"file": "/tmp/testcase.log"
}
],
 
3. If issue persists for some reason, please let us know.

4.  Ideally, marvin should have gracefully handled to continue further or 
gracefully exited with relevant help text to console if there are any 
dependencies. It seems there were other checks also missing.  In any case, we 
are planning to do some changes to logging facility under marvin. 

Thanks!
Santhosh

From: Darren Shepherd [darren.s.sheph...@gmail.com]
Sent: Thursday, October 24, 2013 11:00 AM
To: dev@cloudstack.apache.org
Subject: Re: is marvin on master broken?

Traceback (most recent call last):
  File "tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,
AttributeError: 'NoneType' object has no attribute 'debug'

On Thu, Oct 24, 2013 at 1:21 AM, Santhosh Edukulla
 wrote:
> Hello Darren,
>
> Some trace is still missing i believe. i could not see the last stack frame 
> in the below trace as what lead to this trace?
>
> I just pulled the latest from master branch and used marvin  to deploy few 
> cloudstack entities and it worked. Can you please provide the command you are 
> using to run marvin tests? or what command lead to the below trace?
>
> Regards,
> Santhosh
> 
> From: Darren Shepherd [darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 3:48 AM
> To: dev@cloudstack.apache.org
> Subject: is marvin on master broken?
>
> Whenever I use marvin on master I get
>
> Traceback (most recent call last):
>   File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
> deploy.deploy()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
> self.loadCfg()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 492, in 
> registerApiKey
> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 505, in listUsers
> response = self.connection.marvinRequest(command,
> response_type=response, method=method)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
> line 269, in marvinRequest
> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
>
> So it looks like cloudConnection needs a logger passed to the
> constructor and its not being passed?  I've just been hacking up
> ./tools/marvin/marvin/cloudstackConnection.py to work around this.
>
> Darren

Anyone ever get this exception?

Hi,

I'm running in master.

I had to reboot the only host (KVM) in my CloudStack dev environment. As
you'd expect, it was running my system VMs.

At the same time, I had to restart my CS MS.

I've noticed this exception before when I first start up the CS MS and it
has to restart SSVM. SSVM doesn't start and I have to restart the CS MS to
get it to work (usually starts up SSVM the second time without any
exceptions):

WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
exception There's no support for locking yet
com.cloud.utils.exception.CloudRuntimeException: There's no support for
locking yet
at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
at
com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
at
com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Thanks!

-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: Metadata URL location

2013-10-24 Thread Kris G. Lindgren

You would also need to supernet 169.254.169.254 on the virtual router
(assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it will
still arp to servers that are calling it that have real ip addresses.

Additionally we had some other iptables rules in there that would change
the the ip address of the incoming request to metadata based upon the mac
address that was hitting it.  This was to prevent spoofing of another vm's
IP and getting someone else's metadata (at least in our metadata
implementation we keyed off of the VM IP calling into metadata).  This
also allowed a user to set whatever ipaddress they wanted, but as long as
the mac address was the same and they still had a zeroconfig route on the
VM, they still got only their metadata.

Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

This email message and any attachment(s) hereto are intended for use only
by its intended recipient(s) and may contain confidential information. If
you have received this email in error, please immediately notify the
sender and permanently delete the original and any copy of this message
and its attachments.

On 10/24/13 9:12 AM, "Darren Shepherd"  wrote:

>My guess, I don't really know, would be because its hard.  The VR uses
>link local for the control network so 169.254/16 is bound to the wrong
>nic.  To fix this you just need some ip routing magic in linux (credit
>goes to Kris Lindgren who showed me how to do this).  Add the below to
>a file, substitute eth0 for the guest network nic, run "ip -b "
>The below effectively creates a routing table dedicated to the IP
>169.254.169.254 that sets it default route to go out the guest network
>nic.
>
>rule add from 169.254.169.254 table 70
>rule add to 169.254.169.254 dev eth0 table 70
>route flush table 70
>route add default dev eth0 src 169.254.169.254 table 70
>route flush cache
>
>Darren
>
>On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
> wrote:
>> Hi Guys,
>>
>> CloudStack metadata services are on the default gateway while on EC2,
>> its at 169.254.169.254. Am curious to know why CloudStack does not
>> use a link local address for meta data services.
>>
>> A search of the Wiki
>>(https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK
>>&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearc
>>h=true&queryString=metadata) didn¹t seem to list any doc related to the
>>design of this service.
>>
>> TIA.
>>
>> --
>> @shankerbalan
>>
>> M: +91 98860 60539 | O: +91 (80) 67935867
>> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
>> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
>>Centre, Bangalore - 560 055
>>
>> CloudStack Bootcamp Training on 27/28 November, Bangalore
>> http://www.shapeblue.com/cloudstack-training/
>>
>>
>>
>>
>> This email and any attachments to it may be confidential and are
>>intended solely for the use of the individual to whom it is addressed.
>>Any views or opinions expressed are solely those of the author and do
>>not necessarily represent those of Shape Blue Ltd or related companies.
>>If you are not the intended recipient of this email, you must neither
>>take any action based upon its contents, nor copy or show it to anyone.
>>Please contact the sender if you believe you have received this email in
>>error. Shape Blue Ltd is a company incorporated in England & Wales.
>>ShapeBlue Services India LLP is a company incorporated in India and is
>>operated under license from Shape Blue Ltd. Shape Blue Brasil
>>Consultoria Ltda is a company incorporated in Brasil and is operated
>>under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: LXC and SSVM/CPVM on the host

2013-10-24 Thread Phong Nguyen

> So we need a KVM cluster to run the VMs? (Added the author of the feature)

As it was originally discussed and implemented, the decision was to use KVM
as the system VM for LXC clusters instead of creating an LXC system VM. A
zone with only LXC clusters will deploy a KVM system VM on a host running
an LXC agent. Behind the scenes, this is possible because both KVM and LXC
agents use libvirt for provisioning (and that the setup of an LXC agent is
almost identical to KVM and perfectly capable of running KVM VMs).

-Phong


On Thu, Oct 24, 2013 at 8:57 AM, Francois Gaudreault <
fgaudrea...@cloudops.com> wrote:

> If this is the case, then you should remove the ability to create LXC
> zones or clarify the documentation about that.
>
> According to the wiki page:
>
> Each of the different hypervisors currently have their own System VMs.
> These system VM images are used to run a console proxy, secondary storage,
> and router VMs.
>
> We discussed the possibility of creating System VMs for LXC. There was
> concern with the complexity and potential issues involving iptables for the
> router inside an LXC container. As an intermediate solution we are going to
> use KVM System VMs inside the LXC Cluster.
>
> So we need a KVM cluster to run the VMs? (Added the author of the feature)
>
> Francois
>
> On 10/22/2013, 1:24 AM, Chiradeep Vittal wrote:
>
>> As far as I understand, in an LXC scenario, the system vms are expected to
>> run on real hypervisors.
>> You can always use the QuickCloud way to not use system vms at all.
>>
>> On 10/21/13 1:45 PM, "Francois Gaudreault" 
>> wrote:
>>
>>  Ok I think we have to look at this further. I'll stop hijacking other
>>> threads.
>>>
>>> I am trying to get the SSVM/CPVM to run on a LXC host. The SSVM/CPVM
>>> starts, get IPs, but then CloudStack kill them for some reason. Yes, I
>>> use the 4.2 images :
>>>
>>> 2013-10-21 16:19:21,605 DEBUG [agent.manager.**AgentManagerImpl]
>>> (AgentManager-Handler-9:null) SeqA 73--1: Processing Seq 73--1:  { Cmd ,
>>> MgmtId: -1, via: 73, Ver: v1, Flags: 111,
>>> [{"com.cloud.agent.api.**ShutdownCommand":{"reason":"**sig.kill","wait":0}}]
>>> }
>>> 2013-10-21 16:19:21,605 INFO  [agent.manager.**AgentManagerImpl]
>>> (AgentManager-Handler-9:null) Host 73 has informed us that it is
>>> shutting down with reason sig.kill and detail null
>>> 2013-10-21 16:19:21,606 INFO  [agent.manager.**AgentManagerImpl]
>>> (AgentTaskPool-11:null) Host 73 is disconnecting with event
>>> ShutdownRequested
>>> 2013-10-21 16:19:21,609 DEBUG [agent.manager.**AgentManagerImpl]
>>> (AgentTaskPool-11:null) The next status of agent 73is Disconnected,
>>> current status is Up
>>> 2013-10-21 16:19:21,609 DEBUG [agent.manager.**AgentManagerImpl]
>>> (AgentTaskPool-11:null) Deregistering link for 73 with state Disconnected
>>> 2013-10-21 16:19:21,609 DEBUG [agent.manager.**AgentManagerImpl]
>>> (AgentTaskPool-11:null) Remove Agent : 73
>>> 2013-10-21 16:19:21,609 DEBUG [agent.manager.**ConnectedAgentAttache]
>>> (AgentTaskPool-11:null) Processing Disconnect.
>>>
>>> I transferred the host to KVM, and now the same SSVM/CPVM images are
>>> running fine for the last 30min ( so I assume it works fine...).
>>> Something seems to be wrong with the LXC side :S
>>>
>>> Anyone wants to invest some time to troubleshoot? I'll open a ticket
>>> also.
>>>
>>> --
>>> Francois Gaudreault
>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>> fgaudrea...@cloudops.com
>>> 514-629-6775
>>> - - -
>>> CloudOps
>>> 420 rue Guy
>>> Montréal QC  H3J 1S6
>>> www.cloudops.com
>>> @CloudOps_
>>>
>>>
>>
>
> --
> Francois Gaudreault
> Architecte de Solution Cloud | Cloud Solutions Architect
> fgaudrea...@cloudops.com
> 514-629-6775
> - - -
> CloudOps
> 420 rue Guy
> Montréal QC  H3J 1S6
> www.cloudops.com
> @CloudOps_
>
>

RE: [VOTE] Accept the donation of RDP client code into Apache CloudStack

2013-10-24 Thread Donal Lafferty

It's been a bit over 72 hours, and I would like to now close voting on the RDP 
client code donation.

We have an acceptance with the following results:

+1
Animesh Chaturvedi 
Devdeep Singh
Chiradeep Vittal
Ahmad Emneina
Rajesh Battala
Kelven Yang
Chip Childers
Clayton Weise
John Kinsella
Sateesh Chodapuneedi
Ryan Shafer

+0 
Wido den Hollander
Daan hoogland

The next stage is to put in place IP clearance paper work in advance before the 
code can go into a branch of master.  I will contact a PMC member to arrange 
for this paperwork.

Thanks for participating everyone!


> -Original Message-
> From: Donal Lafferty [mailto:donal.laffe...@citrix.com]
> Sent: 21 October 2013 19:12
> To: dev@cloudstack.apache.org
> Subject: [VOTE] Accept the donation of RDP client code into Apache
> CloudStack
> 
> As stated in a previous thread [1], Citrix is proposing the donation of source
> for an RDP client.  After donation, the client would be integrated with the
> console system VM in order to provide access to Hyper-V based VMs.
> 
> The client's source is in the diff attached to the Review Board submission
> https://reviews.apache.org/r/14701/
> 
> [1] http://markmail.org/thread/q6sfqrhosmirm3bg
> 
> I would like to call a vote here, so that we have a formal consensus on
> accepting the code into the project.  I suggest that it be accepted into a
> branch, and then we work through any technical concerns / reviews /
> changes prior to a master branch merge.
> 
> VOTING will be left open for 72 hours.
> 
> This is a technical decision, which means committer and PMC votes are
> binding.
> 
> 
> DL

Modularized Spring Modules

Darren,

I’m switching my plugin over to use the new modularized Spring stuff you just 
merged and there is something I’m still battling with. I have other beans that 
were previously instantiated before my DataStoreProvider which get injected 
into the provider, lifecycle, etc. So, those beans need to be instantiated 
before the DataStoreProvider. How can I ensure those beans are created and 
setup before the DataStoreProvider does?

-Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

Re: Modularized Spring Modules

So, I kind of figured it out…

In the past, I was creating my DataStoreLifeCycle with 
ComponentContext.inject() and it had some members which were @Injecte(d) - 
those were the guys that were failing to be instantiated. When I switched to 
simply @Inject my DataStoreLifeCycle into my DataStoreProvider, everything 
works fine.

So, with this new system, is there some reason that using 
ComponentContext.inject() would fail to load beans while @Inject would succeed?

-Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 24, 2013, at 4:04 PM, SuichII, Christopher  
wrote:

> Darren,
> 
> I’m switching my plugin over to use the new modularized Spring stuff you just 
> merged and there is something I’m still battling with. I have other beans 
> that were previously instantiated before my DataStoreProvider which get 
> injected into the provider, lifecycle, etc. So, those beans need to be 
> instantiated before the DataStoreProvider. How can I ensure those beans are 
> created and setup before the DataStoreProvider does?
> 
> -Chris
> -- 
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>

Re: Unable to create volume - volume size xxxx, but the maximum size allowed is 0 Gb.

Hey Chris,

I've been having to hard code this in my sandbox to something large enough
(like 2000).

private long _maxVolumeSizeInGb = 2000;

Has anyone ever figured out why this value is not initialized in master?

Thanks!


On Wed, Aug 21, 2013 at 1:12 PM, SuichII, Christopher <
chris.su...@netapp.com> wrote:

> It looks like in 4.2 it is set in VolumeApiServiceImpl.configure(), but
> that method does not exist in master (and the history only goes back 8
> days).
>
> On Aug 21, 2013, at 3:08 PM, "SuichII, Christopher" <
> chris.su...@netapp.com> wrote:
>
> > Yeah, I'm on master. That setting is set to 2000 (as found through the
> UI).
> >
> > On Aug 21, 2013, at 2:12 PM, Koushik Das 
> > wrote:
> >
> >> In 4.2 branch it gets initialized from config property
> "storage.max.volume.size".
> >> Storage  DEFAULT management-server   storage.max.volume.size
> 2000The maximum size for a volume (in GB).
> >>
> >> Which branch are you in? Master. Maybe some missing commit in master.
> >>
> >>> -Original Message-
> >>> From: SuichII, Christopher [mailto:chris.su...@netapp.com]
> >>> Sent: Wednesday, August 21, 2013 11:28 PM
> >>> To: 
> >>> Subject: Unable to create volume - volume size , but the maximum
> size
> >>> allowed is 0 Gb.
> >>>
> >>> I get the error message: 'volume size , but the maximum size
> allowed is 0
> >>> Gb.' when I attempt to create a new volume. It is thrown from
> >>> VolumeApiServiceImpl.validateVolumeSizeRange(int size).
> >>>
> >>> Unless I'm missing something, it looks like '_maxVolumeSizeInGb' is
> never
> >>> initialized in VolumeApiServiceImpl.java. Am I missing the
> initialization
> >>> somewhere?
> >>>
> >>> -Chris
> >
>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: [DISCUSS] make commands.properties the exception, not the rule

I’d like to see if we can come up with some solution for this issue. Having 
‘add these lines to commands.properties’ is not really an acceptable 
installation step for plugins/extensions/etc.

So, the ideas I’ve seen are:
-Turn commands.properties into a blacklist instead of a whitelist
-Dynamically discover additional commands.properties

Darren - is there any chance that the Spring modularization stuff you’ve done 
would make the latter any easier?

Are there any others?

-Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 9, 2013, at 3:49 PM, SuichII, Christopher  wrote:

> I just wanted to add a little clarification from a plugin perspective.
> 
> Having commands.properties as a whitelist just adds another place that 
> plugins have to register with CloudStack. For plugins that do not intend on 
> being a part of the CloudStack source, this is actually quite tricky. 
> Currently, to add entries to commands.properties, any plugin like this would 
> either need to tell the CS administrator to manually modify this file (error 
> prone, laborious and an uncommon installation practice) or develop an 
> installation script to modify commands.properties when installing, updating 
> or uninstalling the plugin (also error prone and scary).
> 
> -- 
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
> 
> On Oct 9, 2013, at 1:08 AM, Darren Shepherd  
> wrote:
> 
>> So I'm saying if you want to disable a command you put myBadCmd=0 in
>> the commands.properties.  So yes, a blacklist over a whitelist.  For
>> people paranoid about maybe some command exists that they don't know
>> about, we can even add a "blacklist=false to the command properties.
>> Then the commands.properites becomes the all mighty master of what is
>> allowed (a whitelist).  But by default, I think the file should be
>> empty and default to what is defined by the API annotation.
>> 
>> Darren
>> 
>> On Tue, Oct 8, 2013 at 5:45 PM, SuichII, Christopher
>>  wrote:
>>> Maybe we could consider switching from a whitelist to a blacklist, then. A 
>>> whitelist is certainly easier in terms of a one-step configuration, but a 
>>> blacklist would allow for much easier plugin development, installation and 
>>> removal. Perhaps we could find write a script that generates the complete 
>>> list of APIs to create the blacklist from (I know this API exists 
>>> currently, but not in the format of commands.properties).
>>> 
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>> 
>>> On Oct 8, 2013, at 7:11 PM, Prachi Damle  wrote:
>>> 
 I think commands.properties is not just providing ACL on the API - but it 
 also serves as a whitelist of APIs available on the deployment.
 It can be a one-step configuration option to disable certain functionality.
 
 Prachi
 
 
 -Original Message-
 From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
 Sent: Tuesday, October 08, 2013 3:24 PM
 To: dev@cloudstack.apache.org
 Subject: [DISCUSS] make commands.properties the exception, not the rule
 
 I would like to largely remove commands.properties.  I think most API 
 commands naturally have a default ACL that should be applied.  I think it 
 makes sense to add to the @APICommand flags for user, domain, admin.  
 Then, as an override mechanism, people can edit commands.properties to 
 change the default ACL.  This would make it such that people could add new 
 commands without the need to edit commands.properties.
 
 Thoughts?  How will this play with whatever is being done with rbac?
 
 Darren
>>> 
>

Re: Modularized Spring Modules

Let me clarify some terms first, this might be confusing.  There is
instantiation and then there is initialization.  Instantiation is
calling the constructor.  Initialization is calling the
@PostConstruct.  Beyond that there is what I've been calling
"CloudStack Extended Lifecycle,"  This for bean implementing
ComponentLifecycle or an extensible interface, like DataStoreProvider.
 Since I'm not sure which aspect your asking about, I'll just explain
the whole process.

Spring is now arranged into a hierarchy of contexts.  The parent
contexts are initialized first, and then the children afterwards.  So
the process goes as follows

1. Instantiate all beans in the current context - There is no way to
deterministically control the instantiation of beans within a context.
 This is just a fundamental fact that can't change.  So beans are
randomly instantiated.
2. Inject all dependencies on all beans in current context - This
entails injecting all the dependencies defined by @Inject.  The order
is not really deterministic which beans get injected first.
3. Initialize all beans in graph order - This entails calling all
methods that have @PostConstruct.  Once the beans are all wired up
they form a graph of dependencies.  Child beans are initialized first,
then the parent.  CloudStack has a lot of circular dependencies in the
beans currently, so while this is intended to be deterministic, you
may find issues.  The storage code is particularly bad in this area,
but if you are defining your beans in your own context then this
doesn't apply.  Only if you add beans to the "core" context will you
find issues.  If a class has an initialization dependency that is not
express through its bean dependencies, you can add "depends-on" to the
bean defintion in the XML.  This will ensure that the specified bean
will be initialized first.
4. CloudStack Extended LifeCycle: Call ComponentLifecycle.configure()
on all ComponentLifecycle implementing beans
5. CloudStack Extended LifeCycle: Register extensible types.  This
phase happens really at the same time as step 4.  When an extensible
type is registered nothing typically happens except that its stored in
a list.  Storage is different in that when the DataStoreProvider is
registered, at that time DataStoreProvider.configure() will be called.
 Notice that even though DataStoreProvider.configure() has the same
signature as ComponentLifecycle.configure() it is different.  Honestly
with the new Spring code we can remove DataStoreProvider completely,
but that's a different discussion

After the context is initialized following the steps above, it will
then initialize all the child contexts.  Once all contexts have been
completely initialized, ComponentLifecycle.start() will be call on all
beans starting with the parent context, then children.

A general guide line of when to use ComponentLifecycle.start() vs
@PostConstruct is if your initialization logic starts a background
thread or requires all extensible types to be registered, then you
should use ComponentLifecycle.start().  All other initialization code
should be in @PostConstruct.  ComponentLifecycle.configure() is
largely useless in my mind and should be avoided.

I hope somewhere in all that is something that answered your question.
 If not you can email me directly with your spring config and I can
help, or we can setup a GTM.

Darren

On Thu, Oct 24, 2013 at 1:04 PM, SuichII, Christopher
 wrote:
> Darren,
>
> I’m switching my plugin over to use the new modularized Spring stuff you just 
> merged and there is something I’m still battling with. I have other beans 
> that were previously instantiated before my DataStoreProvider which get 
> injected into the provider, lifecycle, etc. So, those beans need to be 
> instantiated before the DataStoreProvider. How can I ensure those beans are 
> created and setup before the DataStoreProvider does?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>

Re: Modularized Spring Modules

Wow, this was really helpful! I was able to get my extension set up and thing 
seem to be injecting just fine.

-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 24, 2013, at 5:00 PM, Darren Shepherd  
wrote:

> Let me clarify some terms first, this might be confusing.  There is
> instantiation and then there is initialization.  Instantiation is
> calling the constructor.  Initialization is calling the
> @PostConstruct.  Beyond that there is what I've been calling
> "CloudStack Extended Lifecycle,"  This for bean implementing
> ComponentLifecycle or an extensible interface, like DataStoreProvider.
> Since I'm not sure which aspect your asking about, I'll just explain
> the whole process.
> 
> Spring is now arranged into a hierarchy of contexts.  The parent
> contexts are initialized first, and then the children afterwards.  So
> the process goes as follows
> 
> 1. Instantiate all beans in the current context - There is no way to
> deterministically control the instantiation of beans within a context.
> This is just a fundamental fact that can't change.  So beans are
> randomly instantiated.
> 2. Inject all dependencies on all beans in current context - This
> entails injecting all the dependencies defined by @Inject.  The order
> is not really deterministic which beans get injected first.
> 3. Initialize all beans in graph order - This entails calling all
> methods that have @PostConstruct.  Once the beans are all wired up
> they form a graph of dependencies.  Child beans are initialized first,
> then the parent.  CloudStack has a lot of circular dependencies in the
> beans currently, so while this is intended to be deterministic, you
> may find issues.  The storage code is particularly bad in this area,
> but if you are defining your beans in your own context then this
> doesn't apply.  Only if you add beans to the "core" context will you
> find issues.  If a class has an initialization dependency that is not
> express through its bean dependencies, you can add "depends-on" to the
> bean defintion in the XML.  This will ensure that the specified bean
> will be initialized first.
> 4. CloudStack Extended LifeCycle: Call ComponentLifecycle.configure()
> on all ComponentLifecycle implementing beans
> 5. CloudStack Extended LifeCycle: Register extensible types.  This
> phase happens really at the same time as step 4.  When an extensible
> type is registered nothing typically happens except that its stored in
> a list.  Storage is different in that when the DataStoreProvider is
> registered, at that time DataStoreProvider.configure() will be called.
> Notice that even though DataStoreProvider.configure() has the same
> signature as ComponentLifecycle.configure() it is different.  Honestly
> with the new Spring code we can remove DataStoreProvider completely,
> but that's a different discussion
> 
> After the context is initialized following the steps above, it will
> then initialize all the child contexts.  Once all contexts have been
> completely initialized, ComponentLifecycle.start() will be call on all
> beans starting with the parent context, then children.
> 
> A general guide line of when to use ComponentLifecycle.start() vs
> @PostConstruct is if your initialization logic starts a background
> thread or requires all extensible types to be registered, then you
> should use ComponentLifecycle.start().  All other initialization code
> should be in @PostConstruct.  ComponentLifecycle.configure() is
> largely useless in my mind and should be avoided.
> 
> I hope somewhere in all that is something that answered your question.
> If not you can email me directly with your spring config and I can
> help, or we can setup a GTM.
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 1:04 PM, SuichII, Christopher
>  wrote:
>> Darren,
>> 
>> I’m switching my plugin over to use the new modularized Spring stuff you 
>> just merged and there is something I’m still battling with. I have other 
>> beans that were previously instantiated before my DataStoreProvider which 
>> get injected into the provider, lifecycle, etc. So, those beans need to be 
>> instantiated before the DataStoreProvider. How can I ensure those beans are 
>> created and setup before the DataStoreProvider does?
>> 
>> -Chris
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>

Re: Modularized Spring Modules

Yes avoid ComponentContext.inject() at all costs.  It is really bad.
The problem with ComponentContext.inject(), besides that it is a bad
programming pattern, is that it doesn't know which spring context you
are in.  So ComponentContext.inject() will inject only beans that are
defined in the "core" context.  So extensions are in a child of "core"
so it will not find beans in the child.  There are some ways to work
around this because API commands heavily rely on
ComponentContext.inject(), but I'd prefer that that not be used.
Basically, just don't use ComponentContext.inject().

This was also why at some point I mentioned that
List can not be inject to a SnapshotObject.  The
SnapshotObject is setup with ComponentContext.inject() which will not
find the SnapshotStrategys because no SnapshotStrategys exist in the
"core" context.

I will have to work with Edison to eventually remove
ComponentContext.inject() from storage in general.

Darren

On Thu, Oct 24, 2013 at 1:40 PM, SuichII, Christopher
 wrote:
> So, I kind of figured it out…
>
> In the past, I was creating my DataStoreLifeCycle with 
> ComponentContext.inject() and it had some members which were @Injecte(d) - 
> those were the guys that were failing to be instantiated. When I switched to 
> simply @Inject my DataStoreLifeCycle into my DataStoreProvider, everything 
> works fine.
>
> So, with this new system, is there some reason that using 
> ComponentContext.inject() would fail to load beans while @Inject would 
> succeed?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 24, 2013, at 4:04 PM, SuichII, Christopher  
> wrote:
>
>> Darren,
>>
>> I’m switching my plugin over to use the new modularized Spring stuff you 
>> just merged and there is something I’m still battling with. I have other 
>> beans that were previously instantiated before my DataStoreProvider which 
>> get injected into the provider, lifecycle, etc. So, those beans need to be 
>> instantiated before the DataStoreProvider. How can I ensure those beans are 
>> created and setup before the DataStoreProvider does?
>>
>> -Chris
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>

Re: [DISCUSS] make commands.properties the exception, not the rule

The spring stuff doesn't help.  This discussion didn't really go
anywhere and there weren't much comments.  I'd say at this point we
just put together a proposal.  Give me 5 minutes and I'll throw up a
proposal in a different thread.

Darren

On Thu, Oct 24, 2013 at 1:47 PM, SuichII, Christopher
 wrote:
> I’d like to see if we can come up with some solution for this issue. Having 
> ‘add these lines to commands.properties’ is not really an acceptable 
> installation step for plugins/extensions/etc.
>
> So, the ideas I’ve seen are:
> -Turn commands.properties into a blacklist instead of a whitelist
> -Dynamically discover additional commands.properties
>
> Darren - is there any chance that the Spring modularization stuff you’ve done 
> would make the latter any easier?
>
> Are there any others?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 9, 2013, at 3:49 PM, SuichII, Christopher  
> wrote:
>
>> I just wanted to add a little clarification from a plugin perspective.
>>
>> Having commands.properties as a whitelist just adds another place that 
>> plugins have to register with CloudStack. For plugins that do not intend on 
>> being a part of the CloudStack source, this is actually quite tricky. 
>> Currently, to add entries to commands.properties, any plugin like this would 
>> either need to tell the CS administrator to manually modify this file (error 
>> prone, laborious and an uncommon installation practice) or develop an 
>> installation script to modify commands.properties when installing, updating 
>> or uninstalling the plugin (also error prone and scary).
>>
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 9, 2013, at 1:08 AM, Darren Shepherd  
>> wrote:
>>
>>> So I'm saying if you want to disable a command you put myBadCmd=0 in
>>> the commands.properties.  So yes, a blacklist over a whitelist.  For
>>> people paranoid about maybe some command exists that they don't know
>>> about, we can even add a "blacklist=false to the command properties.
>>> Then the commands.properites becomes the all mighty master of what is
>>> allowed (a whitelist).  But by default, I think the file should be
>>> empty and default to what is defined by the API annotation.
>>>
>>> Darren
>>>
>>> On Tue, Oct 8, 2013 at 5:45 PM, SuichII, Christopher
>>>  wrote:
 Maybe we could consider switching from a whitelist to a blacklist, then. A 
 whitelist is certainly easier in terms of a one-step configuration, but a 
 blacklist would allow for much easier plugin development, installation and 
 removal. Perhaps we could find write a script that generates the complete 
 list of APIs to create the blacklist from (I know this API exists 
 currently, but not in the format of commands.properties).

 --
 Chris Suich
 chris.su...@netapp.com
 NetApp Software Engineer
 Data Center Platforms – Cloud Solutions
 Citrix, Cisco & Red Hat

 On Oct 8, 2013, at 7:11 PM, Prachi Damle  wrote:

> I think commands.properties is not just providing ACL on the API - but it 
> also serves as a whitelist of APIs available on the deployment.
> It can be a one-step configuration option to disable certain 
> functionality.
>
> Prachi
>
>
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Tuesday, October 08, 2013 3:24 PM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] make commands.properties the exception, not the rule
>
> I would like to largely remove commands.properties.  I think most API 
> commands naturally have a default ACL that should be applied.  I think it 
> makes sense to add to the @APICommand flags for user, domain, admin.  
> Then, as an override mechanism, people can edit commands.properties to 
> change the default ACL.  This would make it such that people could add 
> new commands without the need to edit commands.properties.
>
> Thoughts?  How will this play with whatever is being done with rbac?
>
> Darren

>>
>

RE: VM snapshots

2013-10-24 Thread Angeline Shen

Hi Mice:

I was informed to inquire with you as you are Developer of VMsnapshot:


According to vmsnapshot FS  
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

VM Snapshot revert

* Revert VM from running/stopped to a disk+memory snapshot, result in 
running state
* Revert VM from running/stopped to a disk snapshot, result in stopped state

Is above result state correct regardless of the VM state (running or stopped) 
before the Revert ?

Thanks


From: Angeline Shen
Sent: Thursday, October 24, 2013 1:42 PM
To: Anthony Xu; Kelven Yang; Chandan Purushothama
Subject: VM snapshots

Hi:

According to vmsnapshot FS  
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

VM Snapshot revert

* Revert VM from running/stopped to a disk+memory snapshot, result in 
running state
* Revert VM from running/stopped to a disk snapshot, result in stopped state

Is above result state correct regardless of the VM state (running or stopped) 
before the Revert ?

[PROPOSAL] Enable "non-ACS distributed" API extension to be added CloudStack w/o modifying commands.properties

Currently any new API extension to CloudStack must edit
commands.properties to add the appropriate ACLs.  This generally works
fine for ACS as we control the contents of that file and distribute
all the code ourself.  The hang up comes when somebody develops code
outside of ACS and want to add their code to an existing ACS
installation.  The Spring work that has been done has made this much
easier, but you are still required to manually edit
commands.properties.  I propose that we add the ACL metadata as an
optional field in @APICommand.  The logic will be as follows.

First check commands.properties for ACL info.  If ACL info exists, use
that to authorize the command.  If no ACL information exists (ie
null), then look at the @APICommand annotation.  The defaults of
@APICommand will provide no ACL info.  If the @APICommand annotation
provides no ACL info, use that.

Effectively what this means is that for all existing "ACS distributed"
code @APICommand will provide no ACL info (as the default is none) so
commands.properties will be used.  If somebody extends ACS, they can
set the ACL info in @APICommand.

The scope of proposal is focused on "non-ACS" distributed code.  In
order for ACS to move to the annotations fully and make
commands.properties optional more things need to change.  Specifically
the documentation and some python utilities look at
command.properties.  It would be a nice future enhancement to make
commands.properties fully optional, but for the time being this
proposal will address the core issue.

Darren

Re: how to use hashes on c.a.o?

2013-10-24 Thread John Kinsella

Instructions for testing the hash are in the release test page [1]. It is also 
documented in the install guide.

It is the way it is I believe because Chip took the release build script from 
CouchDB, as mentioned in the release build page.

1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+procedure

On Oct 24, 2013, at 12:53 AM, Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>> wrote:

But how does one validate it?  I just wrote a dumb script to
concatenation, remove whitespace, lowercase and then pass to
"sha512sum -c."  I've never seen anyone provide SHAs in that format.
I wouldn't expect many people to know how to use them.  Why can't we
use the good old GNU coreutils style?

Darren

On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella 
mailto:j...@stratosec.co>> wrote:
This is the output of gpg -v --print-md SHA512, generated as part of the 
release procedure [1] by tools/build/build_asf.sh

1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure


On Oct 17, 2013, at 7:56 PM, Darren Shepherd 
mailto:darren.s.sheph...@gmail.com>> wrote:

The hashes that are on c.a.o for the releases have a format like

http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha

apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
   684E0094 F1666F57 5D694916 CF74ED98
   9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
   EA82D42C 78880EDB 04F2532A 61376537

I've never seen this.  Is this some hip new format I'm not aware of,
and I'm the uncool kid still using GNU coreutils?

Darren




Stratosec - Compliance as a Service
o: 415.315.9385
@johnlkinsella

Re: how to use hashes on c.a.o?

Chip,

Do you care if we switch to GNU coreutils format for the hashes?  The
hash value is the same it will just be in the format like

file.tbz2 *12b12341b1234b1234b1b2341b234b

And then you just run "sha512sum -c "

Darren

On Thu, Oct 24, 2013 at 2:34 PM, John Kinsella  wrote:
> Instructions for testing the hash are in the release test page [1]. It is 
> also documented in the install guide.
>
> It is the way it is I believe because Chip took the release build script from 
> CouchDB, as mentioned in the release build page.
>
> 1: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+procedure
>
> On Oct 24, 2013, at 12:53 AM, Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>> wrote:
>
> But how does one validate it?  I just wrote a dumb script to
> concatenation, remove whitespace, lowercase and then pass to
> "sha512sum -c."  I've never seen anyone provide SHAs in that format.
> I wouldn't expect many people to know how to use them.  Why can't we
> use the good old GNU coreutils style?
>
> Darren
>
> On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella 
> mailto:j...@stratosec.co>> wrote:
> This is the output of gpg -v --print-md SHA512, generated as part of the 
> release procedure [1] by tools/build/build_asf.sh
>
> 1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure
>
>
> On Oct 17, 2013, at 7:56 PM, Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>> wrote:
>
> The hashes that are on c.a.o for the releases have a format like
>
> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
>
> apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
>684E0094 F1666F57 5D694916 CF74ED98
>9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
>EA82D42C 78880EDB 04F2532A 61376537
>
> I've never seen this.  Is this some hip new format I'm not aware of,
> and I'm the uncool kid still using GNU coreutils?
>
> Darren
>
>
>
>
> Stratosec - Compliance as a Service
> o: 415.315.9385
> @johnlkinsella
>

Re: Metadata URL location

Alex,

I don't think that's correct.  The instructions that I gave in the
earlier email was changes to the VR to serve metadata from the VR.
Regardless of hypervisor, it should work.

Darren

On Thu, Oct 24, 2013 at 10:54 AM, Alex Huang  wrote:
> In order to use an link local address inside the end user vm, that metadata 
> service must be setup on every hypervisor's dom0 or it has to be proxied out 
> of the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to 
> serve the data, which works for all three hypervisors.
>
> --Alex
>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 24, 2013 8:13 AM
>> To: dev@cloudstack.apache.org
>> Cc: klindg...@godaddy.com
>> Subject: Re: Metadata URL location
>>
>> My guess, I don't really know, would be because its hard.  The VR uses link
>> local for the control network so 169.254/16 is bound to the wrong nic.  To 
>> fix
>> this you just need some ip routing magic in linux (credit goes to Kris 
>> Lindgren
>> who showed me how to do this).  Add the below to a file, substitute eth0 for
>> the guest network nic, run "ip -b "
>> The below effectively creates a routing table dedicated to the IP
>> 169.254.169.254 that sets it default route to go out the guest network nic.
>>
>> rule add from 169.254.169.254 table 70
>> rule add to 169.254.169.254 dev eth0 table 70 route flush table 70 route add
>> default dev eth0 src 169.254.169.254 table 70 route flush cache
>>
>> Darren
>>
>> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>>  wrote:
>> > Hi Guys,
>> >
>> > CloudStack metadata services are on the default gateway while on EC2,
>> > its at 169.254.169.254. Am curious to know why CloudStack does not use
>> > a link local address for meta data services.
>> >
>> > A search of the Wiki
>> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDST
>> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
>> spaceSearch=true&queryString=metadata) didn't seem to list any doc
>> related to the design of this service.
>> >
>> > TIA.
>> >
>> > --
>> > @shankerbalan
>> >
>> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.ba...@shapeblue.com
>> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
>> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
>> >
>> > CloudStack Bootcamp Training on 27/28 November, Bangalore
>> > http://www.shapeblue.com/cloudstack-training/
>> >
>> >
>> >
>> >
>> > This email and any attachments to it may be confidential and are intended
>> solely for the use of the individual to whom it is addressed. Any views or
>> opinions expressed are solely those of the author and do not necessarily
>> represent those of Shape Blue Ltd or related companies. If you are not the
>> intended recipient of this email, you must neither take any action based
>> upon its contents, nor copy or show it to anyone. Please contact the sender 
>> if
>> you believe you have received this email in error. Shape Blue Ltd is a
>> company incorporated in England & Wales. ShapeBlue Services India LLP is a
>> company incorporated in India and is operated under license from Shape
>> Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
>> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
>> registered trademark.

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Hey Marcus,

If you're interested in running the simulator against your code + my code,
I have it on GitHub here:

https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072

Thanks!


On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> I'm trying to attach my diff to https://reviews.apache.org/r/13865/, but
> I don't see the necessary buttons.
>
> I wonder if I need to get edit access back again? We had trouble with the
> Wiki. Was this also impacted?
>
>
> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Sure, I can create a diff file and attach it to Review Board.
>>
>>
>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen wrote:
>>
>>> Sure. The majority of it only affects people who are on your storage
>>> anyway. Perhaps you can post a patch and I can run it through the
>>> simulator to verify that the minor change to the existing code hasn't
>>> broken the standard storages. I don't think it is since I've
>>> thoroughly tested the code I posted, but I know there were some
>>> additional changes.
>>>
>>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>>  wrote:
>>> > OK, Marcus, I made the change to detect my volumes and it seems to
>>> work just
>>> > fine.
>>> >
>>> > Perhaps another day of testing and we can check this code in. What do
>>> you
>>> > think?
>>> >
>>> >
>>> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
>>> >  wrote:
>>> >>
>>> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
>>> >>
>>> >> Yes, that must be the root disk for the VM. I can put in code, as you
>>> >> recommend, to handle only my volumes.
>>> >>
>>> >>
>>> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen >> >
>>> >> wrote:
>>> >>>
>>> >>> It should be sending the path info for each disk per the XML of the
>>> >>> VM... so it will send all disks regardless of whether or not your
>>> >>> adaptor manages that disk, and it's up to your adaptor to ignore any
>>> >>> that aren't managed by it. There should be notes to that effect in
>>> the
>>> >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
>>> >>>
>>> >>> // given local path to file/device (per Libvirt XML), 1) check
>>> >>> that device is
>>> >>> // handled by your adaptor, return false if not. 2) clean up
>>> >>> device, return true
>>> >>> public boolean disconnectPhysicalDiskByPath(String localPath);
>>> >>>
>>> >>> Since we only have XML disk definitions when we stop or migrate a VM,
>>> >>> we have to try all adaptors against all defined disks. So in your
>>> >>> disconnectPhysicalDisk you might do something like check that the
>>> path
>>> >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
>>> there's
>>> >>> some way that's more robust like checking the full path against a lun
>>> >>> listing or something). If it doesn't match, then your
>>> >>> disconnectPhysicalDisk just does nothing.
>>> >>>
>>> >>> I assume this is a root disk or some other local storage disk. If
>>> it's
>>> >>> not, then your VM XML is messed up somehow.
>>> >>>
>>> >>> On Wed, Oct 23, 2013 at 5:01 PM, Mike Tutkowski
>>> >>>  wrote:
>>> >>> > I found the problem.
>>> >>> >
>>> >>> > disconnectPhysicalDiskByPath is being passed in (in my situation)
>>> the
>>> >>> > following:
>>> >>> >
>>> >>> > /var/lib/libvirt/images/9887d511-8dc7-4cb4-96f9-01230fe4bbb6
>>> >>> >
>>> >>> > Due to the name of the method, my code was expecting data such as
>>> the
>>> >>> > following:
>>> >>> >
>>> >>> >
>>> >>> >
>>> /dev/disk/by-path/ip-192.168.233.10:3260-iscsi-iqn.2012-03.com.solidfire:volume1-lun-0
>>> >>> >
>>> >>> > Was it intentional to send the data into this method in the current
>>> >>> > way?
>>> >>> >
>>> >>> >
>>> >>> > On Wed, Oct 23, 2013 at 1:57 PM, Mike Tutkowski
>>> >>> >  wrote:
>>> >>> >>
>>> >>> >> You know, I forgot we supposed to be doing that! :) Multi-tasking
>>> too
>>> >>> >> much
>>> >>> >> today, I guess.
>>> >>> >>
>>> >>> >> Anyways, it must not be working because I still had a hypervisor
>>> >>> >> connection after I shut down the VM.
>>> >>> >>
>>> >>> >> Let me investigate.
>>> >>> >>
>>> >>> >>
>>> >>> >> On Wed, Oct 23, 2013 at 1:48 PM, Marcus Sorensen <
>>> shadow...@gmail.com>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> Are we not disconnecting when we stop the vm? There's a method
>>> for
>>> >>> >>> it, we
>>> >>> >>> should be. disconnectPhysicalDiskViaVmSpec
>>> >>> >>>
>>> >>> >>> On Oct 23, 2013 1:28 PM, "Mike Tutkowski"
>>> >>> >>> 
>>> >>> >>> wrote:
>>> >>> 
>>> >>>  I see one problem for us now, Marcus.
>>> >>> 
>>> >>>  * You have a running VM that you attach a volume to.
>>> >>>  * You stop the VM.
>>> >>>  * You detach the volume.
>>> >>>  * You start up the VM.
>>> >>> 
>>> >>>  The VM will not be connected to the volume (which is good), but
>>> the
>>> >>>  hypervisor will still be connected to the volume.
>>> >>> >>>

RE: VM snapshots

2013-10-24 Thread Sudha Ponnaganti

Can you log defect in apache and target for 4.2.1 
Do not specify any internal stuff in the defect or on ACS mailing lists

-Original Message-
From: Angeline Shen [mailto:angeline.s...@citrix.com] 
Sent: Thursday, October 24, 2013 2:19 PM
To: Anthony Xu; Kelven Yang; Chandan Purushothama; 'Mice Xia'
Cc: cloudstack-...@incubator.apache.org
Subject: RE: VM snapshots

Hi Mice:

I was informed to inquire with you as you are Developer of VMsnapshot:


According to vmsnapshot FS  
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

VM Snapshot revert

* Revert VM from running/stopped to a disk+memory snapshot, result in 
running state
* Revert VM from running/stopped to a disk snapshot, result in stopped state

Is above result state correct regardless of the VM state (running or stopped) 
before the Revert ?

Thanks


From: Angeline Shen
Sent: Thursday, October 24, 2013 1:42 PM
To: Anthony Xu; Kelven Yang; Chandan Purushothama
Subject: VM snapshots

Hi:

According to vmsnapshot FS  
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Snapshots

VM Snapshot revert

* Revert VM from running/stopped to a disk+memory snapshot, result in 
running state
* Revert VM from running/stopped to a disk snapshot, result in stopped state

Is above result state correct regardless of the VM state (running or stopped) 
before the Revert ?

Recall: VM snapshots

2013-10-24 Thread Sudha Ponnaganti

Sudha Ponnaganti would like to recall the message, "VM snapshots".

Recall: VM snapshots

2013-10-24 Thread Sudha Ponnaganti

Sudha Ponnaganti would like to recall the message, "VM snapshots".

Re: [PROPOSAL] Enable "non-ACS distributed" API extension to be added CloudStack w/o modifying commands.properties

Chris,

This change was simple enough I just put up a review for it
https://reviews.apache.org/r/14914/  You want to download the patch
and see if that works for you?

Darren

On Thu, Oct 24, 2013 at 2:28 PM, Darren Shepherd
 wrote:
> Currently any new API extension to CloudStack must edit
> commands.properties to add the appropriate ACLs.  This generally works
> fine for ACS as we control the contents of that file and distribute
> all the code ourself.  The hang up comes when somebody develops code
> outside of ACS and want to add their code to an existing ACS
> installation.  The Spring work that has been done has made this much
> easier, but you are still required to manually edit
> commands.properties.  I propose that we add the ACL metadata as an
> optional field in @APICommand.  The logic will be as follows.
>
> First check commands.properties for ACL info.  If ACL info exists, use
> that to authorize the command.  If no ACL information exists (ie
> null), then look at the @APICommand annotation.  The defaults of
> @APICommand will provide no ACL info.  If the @APICommand annotation
> provides no ACL info, use that.
>
> Effectively what this means is that for all existing "ACS distributed"
> code @APICommand will provide no ACL info (as the default is none) so
> commands.properties will be used.  If somebody extends ACS, they can
> set the ACL info in @APICommand.
>
> The scope of proposal is focused on "non-ACS" distributed code.  In
> order for ACS to move to the annotations fully and make
> commands.properties optional more things need to change.  Specifically
> the documentation and some python utilities look at
> command.properties.  It would be a nice future enhancement to make
> commands.properties fully optional, but for the time being this
> proposal will address the core issue.
>
> Darren

Re: Review Request 14843: CLOUDSTACK-1049 : git commit by using command cloudstack-sccs from installed setup

2013-10-24 Thread David Nalley

Rayees:

While it doesn't technically break things, it isn't very refined:
Here's the output from a machine using a release tarball on a machine
without git:
++ git rev-parse HEAD
/var/tmp/rpm-tmp.JvKJtB: line 33: git: command not found
+ echo
E.g. when you actually go to release software, the file will be empty.

When you do the same with a release tarball on a machine with git:
++ git rev-parse HEAD
fatal: Not a git repository (or any of the parent directories): .git
+ echo


Keep in mind we build a release tarball and it strips out all of the
git information. Can we come up with a more refined solution that if
the above git rev-parse fails that we also look elsewhere?

--David

--David

On Tue, Oct 22, 2013 at 8:12 PM, Rayees Namathponnan
 wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14843/
> ---
>
> (Updated Oct. 23, 2013, 12:12 a.m.)
>
>
> Review request for cloudstack, Chip Childers, Frank Zhang, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-1049
> https://issues.apache.org/jira/browse/CLOUDSTACK-1049
>
>
> Repository: cloudstack-git
>
>
> Description
> ---
>
> The generation of sccs-info was done by WAF but has been broken from 4.2 
> onwords;
>
> updated cloud.spec file to generate git commit and package with rpm
>
> if you are building outside git tree, this file will be blank; and packaging 
> works fine
>
>
> Diffs
> -
>
>   packaging/centos63/cloud.spec 17fb2b1
>   packaging/centos63/cloudstack-sccs PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14843/diff/
>
>
> Testing
> ---
>
> Tested
>
>
> File Attachments (updated)
> 
>
> We need to merge with 4.2 branch; submitting patch for this
>   
> https://reviews.apache.org/media/uploaded/files/2013/10/23/0001-CLOUDSTACK-1049-git-commit-from-installed-cloudstack.patch
>
>
> Thanks,
>
> Rayees Namathponnan
>

Re: how to use hashes on c.a.o?

I ran into the issue while trying to write scripts to get CloudStack
in docker.  Here's my hack-tastic work around

HASH=$(echo $(wget -O - -q
http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
|\
 cut -f2 -d: ) | sed 's/ //g' | tr '[:upper:]' '[:lower:]')

Darren

On Thu, Oct 24, 2013 at 3:05 PM, David Nalley  wrote:
> On Thu, Oct 24, 2013 at 5:43 PM, Darren Shepherd
>  wrote:
>> Chip,
>>
>> Do you care if we switch to GNU coreutils format for the hashes?  The
>> hash value is the same it will just be in the format like
>>
>> file.tbz2 *12b12341b1234b1234b1b2341b234b
>>
>> And then you just run "sha512sum -c "
>>
>> Darren
>>
>
> I just ran into this problem writing an ansible playbook to build RPM
> packages. md5sum -c and sha256sum -c don't seem to work with the
> format that we provide which seems odd.
>
> Incidentally the download page no longer has the documentation for
> verifying hashes.
>
> --David

Review Request 14914: Make commands.properties optional for non-ACS code


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14914/
---

Review request for cloudstack and Chris Suich.


Repository: cloudstack-git


Description
---

Currently any new API extension to CloudStack must edit
commands.properties to add the appropriate ACLs.  This generally works
fine for ACS as we control the contents of that file and distribute
all the code ourself.  The hang up comes when somebody develops code
outside of ACS and want to add their code to an existing ACS
installation.  The Spring work that has been done has made this much
easier, but you are still required to manually edit
commands.properties.  This change introduces the following logic.

First check commands.properties for ACL info.  If ACL info exists, use
that to authorize the command.  If no ACL information exists (ie
null), then look at the @APICommand annotation.  The defaults of


Diffs
-

  api/src/org/apache/cloudstack/api/APICommand.java 4d024c1 
  
plugins/acl/static-role-based/src/org/apache/cloudstack/acl/StaticRoleBasedAPIAccessChecker.java
 affcf91 

Diff: https://reviews.apache.org/r/14914/diff/


Testing
---


Thanks,

Darren Shepherd

Re: how to use hashes on c.a.o?

2013-10-24 Thread David Nalley

On Thu, Oct 24, 2013 at 5:43 PM, Darren Shepherd
 wrote:
> Chip,
>
> Do you care if we switch to GNU coreutils format for the hashes?  The
> hash value is the same it will just be in the format like
>
> file.tbz2 *12b12341b1234b1234b1b2341b234b
>
> And then you just run "sha512sum -c "
>
> Darren
>

I just ran into this problem writing an ansible playbook to build RPM
packages. md5sum -c and sha256sum -c don't seem to work with the
format that we provide which seems odd.

Incidentally the download page no longer has the documentation for
verifying hashes.

--David

Re: is marvin on master broken?

Ah thanks, that's my problem.  I removed the logger portion not really
thinking about it.  Oh the joys of DSLs.

Darren

On Thu, Oct 24, 2013 at 11:08 AM, Santhosh Edukulla
 wrote:
> Hi Darren,
>
> 1. I just ran again using  the latest nosetests-2.7 with  marvin-plugin, 
> marvin-config=, it went fine with 
> no issues.
>
> 2. Please check whether the below contents are available under configuration 
> file passed to marvin or nosetests. Check advanced.cfg under ( setup/dev/ ) 
> for reference. In short, it seems as per trace, under loadCfg , 
> testClientLogFile which is getting loaded based upon below values from config 
> is None, leading to further issues and so below trace.
>
> For passing the correct config :
>
>  If we are using nosetests to run the marvin tests, please use 
> --marvin-config=""  ( or  )
>
>  If we are using deployAndRun, please use --config ="".
>
> "logger": [
> {
> "name": "TestClient",
> "file": "/tmp/testclient.log"
> },
> {
> "name": "TestCase",
> "file": "/tmp/testcase.log"
> }
> ],
>
> 3. If issue persists for some reason, please let us know.
>
> 4.  Ideally, marvin should have gracefully handled to continue further or 
> gracefully exited with relevant help text to console if there are any 
> dependencies. It seems there were other checks also missing.  In any case, we 
> are planning to do some changes to logging facility under marvin.
>
> Thanks!
> Santhosh
> 
> From: Darren Shepherd [darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 11:00 AM
> To: dev@cloudstack.apache.org
> Subject: Re: is marvin on master broken?
>
> Traceback (most recent call last):
>   File "tools/marvin/marvin/deployDataCenter.py", line 610, in 
> deploy.deploy()
>   File "tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
> self.loadCfg()
>   File "tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>   File "tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 505, in listUsers
> response = self.connection.marvinRequest(command,
> response_type=response, method=method)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
> line 269, in marvinRequest
> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
> AttributeError: 'NoneType' object has no attribute 'debug'
>
> On Thu, Oct 24, 2013 at 1:21 AM, Santhosh Edukulla
>  wrote:
>> Hello Darren,
>>
>> Some trace is still missing i believe. i could not see the last stack frame 
>> in the below trace as what lead to this trace?
>>
>> I just pulled the latest from master branch and used marvin  to deploy few 
>> cloudstack entities and it worked. Can you please provide the command you 
>> are using to run marvin tests? or what command lead to the below trace?
>>
>> Regards,
>> Santhosh
>> 
>> From: Darren Shepherd [darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 24, 2013 3:48 AM
>> To: dev@cloudstack.apache.org
>> Subject: is marvin on master broken?
>>
>> Whenever I use marvin on master I get
>>
>> Traceback (most recent call last):
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
>> deploy.deploy()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
>> self.loadCfg()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
>> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 492, in 
>> registerApiKey
>> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>>   File 
>> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
>> line 505, in listUsers
>> response = self.connection.marvinRequest(command,
>> response_type=response, method=method)
>>   File 
>> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
>> line 269, in marvinRequest
>> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
>>
>> So it looks like cloudConnection needs a logger passed to the
>> constructor and its not being passed?  I've just been hacking up
>> ./tools/marvin/marvin/cloudstackConnection.py to work around this.
>>
>> Darren

Re: Anyone ever get this exception?

Never seen that, but yeah that's an issue.  It's possible given the
current initialization in ACS to somebody tries to lock something
before the lock manager is created.  That should change let me
look at that.  I can probably fix it on master.

Darren

On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
 wrote:
> Hi,
>
> I'm running in master.
>
> I had to reboot the only host (KVM) in my CloudStack dev environment. As
> you'd expect, it was running my system VMs.
>
> At the same time, I had to restart my CS MS.
>
> I've noticed this exception before when I first start up the CS MS and it
> has to restart SSVM. SSVM doesn't start and I have to restart the CS MS to
> get it to work (usually starts up SSVM the second time without any
> exceptions):
>
> WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
> exception There's no support for locking yet
> com.cloud.utils.exception.CloudRuntimeException: There's no support for
> locking yet
> at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
> at
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
> at
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
> Thanks!
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *™*

Re: how to use hashes on c.a.o?

2013-10-24 Thread David Nalley

On Thu, Oct 24, 2013 at 6:12 PM, Darren Shepherd
 wrote:
> I ran into the issue while trying to write scripts to get CloudStack
> in docker.  Here's my hack-tastic work around
>
> HASH=$(echo $(wget -O - -q
> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
> |\
>  cut -f2 -d: ) | sed 's/ //g' | tr '[:upper:]' '[:lower:]')
>
> Darren
>

Is there a reason not to verify authenticity and validity of the
tarball with the gpg sig?

gpg --import KEYS
gpg --verify acs.tar.bz2.asc

I think that's what I am currently defaulting to.

Re: [PROPOSAL] Enable "non-ACS distributed" API extension to be added CloudStack w/o modifying commands.properties

On first glance it looks great! I’ll look at it first thing in the morning 
since it is getting late here in EST.

Thanks!
Chris
-- 
Chris Suich
chris.su...@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 24, 2013, at 6:08 PM, Darren Shepherd  
wrote:

> Chris,
> 
> This change was simple enough I just put up a review for it
> https://reviews.apache.org/r/14914/  You want to download the patch
> and see if that works for you?
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 2:28 PM, Darren Shepherd
>  wrote:
>> Currently any new API extension to CloudStack must edit
>> commands.properties to add the appropriate ACLs.  This generally works
>> fine for ACS as we control the contents of that file and distribute
>> all the code ourself.  The hang up comes when somebody develops code
>> outside of ACS and want to add their code to an existing ACS
>> installation.  The Spring work that has been done has made this much
>> easier, but you are still required to manually edit
>> commands.properties.  I propose that we add the ACL metadata as an
>> optional field in @APICommand.  The logic will be as follows.
>> 
>> First check commands.properties for ACL info.  If ACL info exists, use
>> that to authorize the command.  If no ACL information exists (ie
>> null), then look at the @APICommand annotation.  The defaults of
>> @APICommand will provide no ACL info.  If the @APICommand annotation
>> provides no ACL info, use that.
>> 
>> Effectively what this means is that for all existing "ACS distributed"
>> code @APICommand will provide no ACL info (as the default is none) so
>> commands.properties will be used.  If somebody extends ACS, they can
>> set the ACL info in @APICommand.
>> 
>> The scope of proposal is focused on "non-ACS" distributed code.  In
>> order for ACS to move to the annotations fully and make
>> commands.properties optional more things need to change.  Specifically
>> the documentation and some python utilities look at
>> command.properties.  It would be a nice future enhancement to make
>> commands.properties fully optional, but for the time being this
>> proposal will address the core issue.
>> 
>> Darren

Re: how to use hashes on c.a.o?

2013-10-24 Thread Chip Childers

Yeah, we can make this change IMO. 

Darren, note that my sungard.com address no longer finds me. Please use my a.o 
or gmail.c address. ;)

> On Oct 24, 2013, at 5:43 PM, Darren Shepherd  
> wrote:
> 
> Chip,
> 
> Do you care if we switch to GNU coreutils format for the hashes?  The
> hash value is the same it will just be in the format like
> 
> file.tbz2 *12b12341b1234b1234b1b2341b234b
> 
> And then you just run "sha512sum -c "
> 
> Darren
> 
>> On Thu, Oct 24, 2013 at 2:34 PM, John Kinsella  wrote:
>> Instructions for testing the hash are in the release test page [1]. It is 
>> also documented in the install guide.
>> 
>> It is the way it is I believe because Chip took the release build script 
>> from CouchDB, as mentioned in the release build page.
>> 
>> 1: 
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+procedure
>> 
>> On Oct 24, 2013, at 12:53 AM, Darren Shepherd 
>> mailto:darren.s.sheph...@gmail.com>> wrote:
>> 
>> But how does one validate it?  I just wrote a dumb script to
>> concatenation, remove whitespace, lowercase and then pass to
>> "sha512sum -c."  I've never seen anyone provide SHAs in that format.
>> I wouldn't expect many people to know how to use them.  Why can't we
>> use the good old GNU coreutils style?
>> 
>> Darren
>> 
>> On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella 
>> mailto:j...@stratosec.co>> wrote:
>> This is the output of gpg -v --print-md SHA512, generated as part of the 
>> release procedure [1] by tools/build/build_asf.sh
>> 
>> 1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure
>> 
>> 
>> On Oct 17, 2013, at 7:56 PM, Darren Shepherd 
>> mailto:darren.s.sheph...@gmail.com>> wrote:
>> 
>> The hashes that are on c.a.o for the releases have a format like
>> 
>> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
>> 
>> apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
>>   684E0094 F1666F57 5D694916 CF74ED98
>>   9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
>>   EA82D42C 78880EDB 04F2532A 61376537
>> 
>> I've never seen this.  Is this some hip new format I'm not aware of,
>> and I'm the uncool kid still using GNU coreutils?
>> 
>> Darren
>> 
>> 
>> 
>> 
>> Stratosec - Compliance as a Service
>> o: 415.315.9385
>> @johnlkinsella
>>

Re: Metadata URL location

2013-10-24 Thread Chiradeep Vittal

For the VPC virtual router case this would this have to be done on all
guest interfaces? 
Could we alias localhost on the VR to 169.254.169.254?
For shared networks, basic zone and networks where the VR is not the
default gateway, we would have to send another (169.254.0.0/16) route in
the DHCP response to point to the VR.

Additionally in basic zone the anti-spoof filters in dom0 would have to be
adjusted to let 169.254.169.254 addresses

On 10/24/13 8:51 AM, "Darren Shepherd"  wrote:

>So additionally you need to do
>
>ip addr add dev eth0 169.254.169.254/0
>
>On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren 
>wrote:
>> You would also need to supernet 169.254.169.254 on the virtual router
>> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it
>>will
>> still arp to servers that are calling it that have real ip addresses.
>>
>> Additionally we had some other iptables rules in there that would change
>> the the ip address of the incoming request to metadata based upon the
>>mac
>> address that was hitting it.  This was to prevent spoofing of another
>>vm's
>> IP and getting someone else's metadata (at least in our metadata
>> implementation we keyed off of the VM IP calling into metadata).  This
>> also allowed a user to set whatever ipaddress they wanted, but as long
>>as
>> the mac address was the same and they still had a zeroconfig route on
>>the
>> VM, they still got only their metadata.
>> 
>>
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy, LLC.
>>
>>
>> This email message and any attachment(s) hereto are intended for use
>>only
>> by its intended recipient(s) and may contain confidential information.
>>If
>> you have received this email in error, please immediately notify the
>> sender and permanently delete the original and any copy of this message
>> and its attachments.
>>
>>
>>
>>
>>
>>
>>
>> On 10/24/13 9:12 AM, "Darren Shepherd" 
>>wrote:
>>
>>>My guess, I don't really know, would be because its hard.  The VR uses
>>>link local for the control network so 169.254/16 is bound to the wrong
>>>nic.  To fix this you just need some ip routing magic in linux (credit
>>>goes to Kris Lindgren who showed me how to do this).  Add the below to
>>>a file, substitute eth0 for the guest network nic, run "ip -b "
>>>The below effectively creates a routing table dedicated to the IP
>>>169.254.169.254 that sets it default route to go out the guest network
>>>nic.
>>>
>>>rule add from 169.254.169.254 table 70
>>>rule add to 169.254.169.254 dev eth0 table 70
>>>route flush table 70
>>>route add default dev eth0 src 169.254.169.254 table 70
>>>route flush cache
>>>
>>>Darren
>>>
>>>On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>>> wrote:
 Hi Guys,

 CloudStack metadata services are on the default gateway while on EC2,
 its at 169.254.169.254. Am curious to know why CloudStack does not
 use a link local address for meta data services.

 A search of the Wiki
(https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTA
CK
&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSea
rc
h=true&queryString=metadata) didn¹t seem to list any doc related to the
design of this service.

 TIA.

 --
 @shankerbalan

 M: +91 98860 60539 | O: +91 (80) 67935867
 shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
 ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
Centre, Bangalore - 560 055

 CloudStack Bootcamp Training on 27/28 November, Bangalore
 http://www.shapeblue.com/cloudstack-training/




 This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
Any views or opinions expressed are solely those of the author and do
not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither
take any action based upon its contents, nor copy or show it to anyone.
Please contact the sender if you believe you have received this email
in
error. Shape Blue Ltd is a company incorporated in England & Wales.
ShapeBlue Services India LLP is a company incorporated in India and is
operated under license from Shape Blue Ltd. Shape Blue Brasil
Consultoria Ltda is a company incorporated in Brasil and is operated
under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>

Re: LXC and SSVM/CPVM on the host

2013-10-24 Thread Chiradeep Vittal

So here's what should work
Create zone
Add a KVM cluster -> add a KVM host -> wait for systemvms to start
Add a LXC cluster -> add a LXC host

On 10/24/13 9:55 AM, "Francois Gaudreault" 
wrote:

>If it's designed to do that, then something is wrong with how CS deals
>with it.
>
>When I was trying to get the KVM images to work, they were starting,
>getting IPs, but then something was killing the VM. I though for
>sometime that libvirt was the issue, so I tried Ubuntu 13.10, 12.04 and
>CentOS with the same results. I then switched the hypervisor type in CS
>from LXC to KVM (rebuilt the zone), keep the same settings on my host,
>and the System VMs are running fine since then.
>
>Anyone have time to help me troubleshoot? I mean, this is not a blocker,
>but I can't get standalone LXC cluster to work...
>
>Francois
>
>On 10/24/2013, 11:53 AM, Phong Nguyen wrote:
>> > So we need a KVM cluster to run the VMs? (Added the author of the
>> feature)
>>
>> As it was originally discussed and implemented, the decision was to
>> use KVM as the system VM for LXC clusters instead of creating an LXC
>> system VM. A zone with only LXC clusters will deploy a KVM system VM
>> on a host running an LXC agent. Behind the scenes, this is possible
>> because both KVM and LXC agents use libvirt for provisioning (and that
>> the setup of an LXC agent is almost identical to KVM and perfectly
>> capable of running KVM VMs).
>>
>> -Phong
>>
>>
>> On Thu, Oct 24, 2013 at 8:57 AM, Francois Gaudreault
>> mailto:fgaudrea...@cloudops.com>> wrote:
>>
>> If this is the case, then you should remove the ability to create
>> LXC zones or clarify the documentation about that.
>>
>> According to the wiki page:
>>
>> Each of the different hypervisors currently have their own System
>> VMs. These system VM images are used to run a console proxy,
>> secondary storage, and router VMs.
>>
>> We discussed the possibility of creating System VMs for LXC. There
>> was concern with the complexity and potential issues involving
>> iptables for the router inside an LXC container. As an
>> intermediate solution we are going to use KVM System VMs inside
>> the LXC Cluster.
>>
>> So we need a KVM cluster to run the VMs? (Added the author of the
>> feature)
>>
>> Francois
>>
>> On 10/22/2013, 1:24 AM, Chiradeep Vittal wrote:
>>
>> As far as I understand, in an LXC scenario, the system vms are
>> expected to
>> run on real hypervisors.
>> You can always use the QuickCloud way to not use system vms at
>> all.
>>
>> On 10/21/13 1:45 PM, "Francois Gaudreault"
>> mailto:fgaudrea...@cloudops.com>>
>> wrote:
>>
>> Ok I think we have to look at this further. I'll stop
>> hijacking other
>> threads.
>>
>> I am trying to get the SSVM/CPVM to run on a LXC host. The
>> SSVM/CPVM
>> starts, get IPs, but then CloudStack kill them for some
>> reason. Yes, I
>> use the 4.2 images :
>>
>> 2013-10-21 16:19:21,605 DEBUG
>>[agent.manager.AgentManagerImpl]
>> (AgentManager-Handler-9:null) SeqA 73--1: Processing Seq
>> 73--1:  { Cmd ,
>> MgmtId: -1, via: 73, Ver: v1, Flags: 111,
>> 
>>[{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","wait":0}}]
>> }
>> 2013-10-21 16:19:21,605 INFO
>>[agent.manager.AgentManagerImpl]
>> (AgentManager-Handler-9:null) Host 73 has informed us that
>> it is
>> shutting down with reason sig.kill and detail null
>> 2013-10-21 16:19:21,606 INFO
>>[agent.manager.AgentManagerImpl]
>> (AgentTaskPool-11:null) Host 73 is disconnecting with event
>> ShutdownRequested
>> 2013-10-21 16:19:21,609 DEBUG
>>[agent.manager.AgentManagerImpl]
>> (AgentTaskPool-11:null) The next status of agent 73is
>> Disconnected,
>> current status is Up
>> 2013-10-21 16:19:21,609 DEBUG
>>[agent.manager.AgentManagerImpl]
>> (AgentTaskPool-11:null) Deregistering link for 73 with
>> state Disconnected
>> 2013-10-21 16:19:21,609 DEBUG
>>[agent.manager.AgentManagerImpl]
>> (AgentTaskPool-11:null) Remove Agent : 73
>> 2013-10-21 16:19:21,609 DEBUG
>> [agent.manager.ConnectedAgentAttache]
>> (AgentTaskPool-11:null) Processing Disconnect.
>>
>> I transferred the host to KVM, and now the same SSVM/CPVM
>> images are
>> running fine for the last 30min ( so I assume it works
>> fine...).
>> Something seems to be wrong with the LXC side :S
>>
>> Anyone wants to invest some time to troubleshoot? I'll
>> open a ticket also.
>>
>> -- 
>> Francois Gaudreault
>> Archit

Build failed in Jenkins: build-master-noredist #1601

2013-10-24 Thread jenkins

See 

Changes:

[brian.federle] Update logo

[darren.s.shepherd] Move LockMasterListener initialization to earlier in the 
code

[brian.federle] Make header width 100%; body stays 1024px centered.

[brian.federle] Browser widget: Modify behavior

[brian.federle] Detail view UI: make view all links have hyperlink style

[brian.federle] Quickview: Fix positioning

[brian.federle] Quickview: Better styling

[Alena Prokharchyk] ResourceDetails:

--
[...truncated 1251 lines...]

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ cloud-engine-components-api 
---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.1:attach-descriptor (attach-descriptor) @ 
cloud-engine-components-api ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ 
cloud-engine-components-api ---
[INFO] Installing 

 to 
/jenkins/.m2/repository/org/apache/cloudstack/cloud-engine-components-api/4.3.0-SNAPSHOT/cloud-engine-components-api-4.3.0-SNAPSHOT.jar
[INFO] Installing 

 to 
/jenkins/.m2/repository/org/apache/cloudstack/cloud-engine-components-api/4.3.0-SNAPSHOT/cloud-engine-components-api-4.3.0-SNAPSHOT.pom
[INFO] 
[INFO] 
[INFO] Building Apache CloudStack Server 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ cloud-server ---
[INFO] Deleting 
 
(includes = [**/*], excludes = [])
[INFO] Deleting 
 (includes 
= [target, dist], excludes = [])
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.3:process (default) @ cloud-server 
---
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (generate-resource) @ cloud-server ---
[INFO] Executing tasks

main:
 [copy] Copying 3 files to 

 [copy] Copying 1 file to 

[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
cloud-server ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 29 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ cloud-server 
---
[INFO] Compiling 348 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
cloud-server ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 14 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ 
cloud-server ---
[INFO] Compiling 73 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ cloud-server ---
[INFO] Surefire report directory: 


---
 T E S T S
---
Running com.cloud.event.EventControlsUnitTest
log4j:WARN No appenders could be found for logger 
(com.cloud.utils.crypt.EncryptionSecretKeyChecker).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.248 sec <<< 
FAILURE!
Running com.cloud.keystore.KeystoreTest
org.apache.cloudstack.api.response.UserVmResponse/null/{"id":"3","securitygroup":[],"nic":[],"tags":[],"affinitygroup":[]}
org.apache.cloudstack.api.response.AlertResponse/null/{"id":"100","description":"Hello"}
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.119 sec
Running com.cloud.alert.AlertControlsUnitTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec <<< 
FAILURE!
Running com.cloud.capacity.CapacityManagerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.414 sec
Running

Re: Anyone ever get this exception?

OK, thanks, Darren.

If it becomes larger than a "simple" fix, I can log a JIRA ticket for it,
too.


On Thu, Oct 24, 2013 at 4:19 PM, Darren Shepherd <
darren.s.sheph...@gmail.com> wrote:

> Never seen that, but yeah that's an issue.  It's possible given the
> current initialization in ACS to somebody tries to lock something
> before the lock manager is created.  That should change let me
> look at that.  I can probably fix it on master.
>
> Darren
>
> On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
>  wrote:
> > Hi,
> >
> > I'm running in master.
> >
> > I had to reboot the only host (KVM) in my CloudStack dev environment. As
> > you'd expect, it was running my system VMs.
> >
> > At the same time, I had to restart my CS MS.
> >
> > I've noticed this exception before when I first start up the CS MS and it
> > has to restart SSVM. SSVM doesn't start and I have to restart the CS MS
> to
> > get it to work (usually starts up SSVM the second time without any
> > exceptions):
> >
> > WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
> > exception There's no support for locking yet
> > com.cloud.utils.exception.CloudRuntimeException: There's no support for
> > locking yet
> > at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
> > at
> >
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
> > at
> >
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
> > at
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
> > at
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
> > at
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
> > at
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
> > at
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > at
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
> > at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:724)
> >
> > Thanks!
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkow...@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud
> > *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: how to use hashes on c.a.o?

> Is there a reason not to verify authenticity and validity of the
> tarball with the gpg sig?

Nope, just laziness.

Darren

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

This is a newer link, Marcus:

https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b

I just rebased on top of master today.


On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Hey Marcus,
>
> If you're interested in running the simulator against your code + my code,
> I have it on GitHub here:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>
> Thanks!
>
>
> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> I'm trying to attach my diff to https://reviews.apache.org/r/13865/, but
>> I don't see the necessary buttons.
>>
>> I wonder if I need to get edit access back again? We had trouble with the
>> Wiki. Was this also impacted?
>>
>>
>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Sure, I can create a diff file and attach it to Review Board.
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen 
>>> wrote:
>>>
 Sure. The majority of it only affects people who are on your storage
 anyway. Perhaps you can post a patch and I can run it through the
 simulator to verify that the minor change to the existing code hasn't
 broken the standard storages. I don't think it is since I've
 thoroughly tested the code I posted, but I know there were some
 additional changes.

 On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
  wrote:
 > OK, Marcus, I made the change to detect my volumes and it seems to
 work just
 > fine.
 >
 > Perhaps another day of testing and we can check this code in. What do
 you
 > think?
 >
 >
 > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
 >  wrote:
 >>
 >> Thanks, Marcus...I hadn't read that note, but that makes sense.
 >>
 >> Yes, that must be the root disk for the VM. I can put in code, as you
 >> recommend, to handle only my volumes.
 >>
 >>
 >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
 shadow...@gmail.com>
 >> wrote:
 >>>
 >>> It should be sending the path info for each disk per the XML of the
 >>> VM... so it will send all disks regardless of whether or not your
 >>> adaptor manages that disk, and it's up to your adaptor to ignore any
 >>> that aren't managed by it. There should be notes to that effect in
 the
 >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
 >>>
 >>> // given local path to file/device (per Libvirt XML), 1) check
 >>> that device is
 >>> // handled by your adaptor, return false if not. 2) clean up
 >>> device, return true
 >>> public boolean disconnectPhysicalDiskByPath(String localPath);
 >>>
 >>> Since we only have XML disk definitions when we stop or migrate a
 VM,
 >>> we have to try all adaptors against all defined disks. So in your
 >>> disconnectPhysicalDisk you might do something like check that the
 path
 >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
 there's
 >>> some way that's more robust like checking the full path against a
 lun
 >>> listing or something). If it doesn't match, then your
 >>> disconnectPhysicalDisk just does nothing.
 >>>
 >>> I assume this is a root disk or some other local storage disk. If
 it's
 >>> not, then your VM XML is messed up somehow.
 >>>
 >>> On Wed, Oct 23, 2013 at 5:01 PM, Mike Tutkowski
 >>>  wrote:
 >>> > I found the problem.
 >>> >
 >>> > disconnectPhysicalDiskByPath is being passed in (in my situation)
 the
 >>> > following:
 >>> >
 >>> > /var/lib/libvirt/images/9887d511-8dc7-4cb4-96f9-01230fe4bbb6
 >>> >
 >>> > Due to the name of the method, my code was expecting data such as
 the
 >>> > following:
 >>> >
 >>> >
 >>> >
 /dev/disk/by-path/ip-192.168.233.10:3260-iscsi-iqn.2012-03.com.solidfire:volume1-lun-0
 >>> >
 >>> > Was it intentional to send the data into this method in the
 current
 >>> > way?
 >>> >
 >>> >
 >>> > On Wed, Oct 23, 2013 at 1:57 PM, Mike Tutkowski
 >>> >  wrote:
 >>> >>
 >>> >> You know, I forgot we supposed to be doing that! :)
 Multi-tasking too
 >>> >> much
 >>> >> today, I guess.
 >>> >>
 >>> >> Anyways, it must not be working because I still had a hypervisor
 >>> >> connection after I shut down the VM.
 >>> >>
 >>> >> Let me investigate.
 >>> >>
 >>> >>
 >>> >> On Wed, Oct 23, 2013 at 1:48 PM, Marcus Sorensen <
 shadow...@gmail.com>
 >>> >> wrote:
 >>> >>>
 >>> >>> Are we not disconnecting when we stop the vm? There's a method
 for
 >>> >>> it, we
 >>> >>> should be. disconnectPhysicalDiskViaVmSpec
 >>> >>>
 >>> >>> On Oct 23, 2013 1:28 PM, "Mike Tutkowski"
 >

RE: Metadata URL location

2013-10-24 Thread Alex Huang

Darren,

I was under the impression that linklocal doesn't go out of the VM but 
Chiradeep just explained to me that it does as long as the default gateway is 
correct and there's no other routes directing it elsewhere.  Then the changes 
make sense.  

--Alex

> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 2:52 PM
> To: dev@cloudstack.apache.org
> Cc: klindg...@godaddy.com
> Subject: Re: Metadata URL location
> 
> Alex,
> 
> I don't think that's correct.  The instructions that I gave in the earlier 
> email
> was changes to the VR to serve metadata from the VR.
> Regardless of hypervisor, it should work.
> 
> Darren
> 
> On Thu, Oct 24, 2013 at 10:54 AM, Alex Huang 
> wrote:
> > In order to use an link local address inside the end user vm, that metadata
> service must be setup on every hypervisor's dom0 or it has to be proxied out
> of the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to
> serve the data, which works for all three hypervisors.
> >
> > --Alex
> >
> >> -Original Message-
> >> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> >> Sent: Thursday, October 24, 2013 8:13 AM
> >> To: dev@cloudstack.apache.org
> >> Cc: klindg...@godaddy.com
> >> Subject: Re: Metadata URL location
> >>
> >> My guess, I don't really know, would be because its hard.  The VR
> >> uses link local for the control network so 169.254/16 is bound to the
> >> wrong nic.  To fix this you just need some ip routing magic in linux
> >> (credit goes to Kris Lindgren who showed me how to do this).  Add the
> >> below to a file, substitute eth0 for the guest network nic, run "ip -b
> "
> >> The below effectively creates a routing table dedicated to the IP
> >> 169.254.169.254 that sets it default route to go out the guest network nic.
> >>
> >> rule add from 169.254.169.254 table 70 rule add to 169.254.169.254
> >> dev eth0 table 70 route flush table 70 route add default dev eth0 src
> >> 169.254.169.254 table 70 route flush cache
> >>
> >> Darren
> >>
> >> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
> >>  wrote:
> >> > Hi Guys,
> >> >
> >> > CloudStack metadata services are on the default gateway while on
> >> > EC2, its at 169.254.169.254. Am curious to know why CloudStack does
> >> > not use a link local address for meta data services.
> >> >
> >> > A search of the Wiki
> >>
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDS
> >> T
> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
> >> spaceSearch=true&queryString=metadata) didn't seem to list any doc
> >> related to the design of this service.
> >> >
> >> > TIA.
> >> >
> >> > --
> >> > @shankerbalan
> >> >
> >> > M: +91 98860 60539 | O: +91 (80) 67935867
> >> > shanker.ba...@shapeblue.com
> >> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India
> >> > | LLP,
> >> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
> >> >
> >> > CloudStack Bootcamp Training on 27/28 November, Bangalore
> >> > http://www.shapeblue.com/cloudstack-training/
> >> >
> >> >
> >> >
> >> >
> >> > This email and any attachments to it may be confidential and are
> >> > intended
> >> solely for the use of the individual to whom it is addressed. Any
> >> views or opinions expressed are solely those of the author and do not
> >> necessarily represent those of Shape Blue Ltd or related companies.
> >> If you are not the intended recipient of this email, you must neither
> >> take any action based upon its contents, nor copy or show it to
> >> anyone. Please contact the sender if you believe you have received
> >> this email in error. Shape Blue Ltd is a company incorporated in
> >> England & Wales. ShapeBlue Services India LLP is a company
> >> incorporated in India and is operated under license from Shape Blue
> >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> >> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.

Re: VolumeApiServiceImpl Question

Figured I'd send this out again.

Any thoughts as to why _maxVolumeSizeInGb is never assigned to in
VolumeApiServiceImpl?

The end result is you cannot create a volume unless you explicitly go into
VolumeApiServiceImpl and do something like assign a high value to
_maxVolumeSizeInGb.

I opened a JIRA ticket for this issue:

https://issues.apache.org/jira/browse/CLOUDSTACK-4812


On Wed, Oct 2, 2013 at 1:48 PM, Mike Tutkowski  wrote:

> Hi,
>
> I'm trying to create a volume from a Disk Offering that specifies a 10 GB
> disk.
>
> When attempting to create a volume from this Disk Offering, I'm told the
> maximum size I can create for a volume is 0 GB.
>
> In VolumeApiServiceImpl, I see a private instance variable called
> _maxVolumeSizeInGb, which appears to be the problem.
>
> It is read from, but never written to (its default value is 0 since it is
> a 'long' member variable).
>
> Am I missing something here?
>
> Thanks!
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the 
> cloud
> *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud
*™*

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

By the way, in case you're looking at the diff and wondering why I took out
a StopCommand check and call to execute in LibvirtComputingResource, there
were two of them.

-} else if (cmd instanceof StopCommand) {

-return execute((StopCommand) cmd);


On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> This is a newer link, Marcus:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>
> I just rebased on top of master today.
>
>
> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Hey Marcus,
>>
>> If you're interested in running the simulator against your code + my
>> code, I have it on GitHub here:
>>
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>>
>> Thanks!
>>
>>
>> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
>>> but I don't see the necessary buttons.
>>>
>>> I wonder if I need to get edit access back again? We had trouble with
>>> the Wiki. Was this also impacted?
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Sure, I can create a diff file and attach it to Review Board.


 On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen 
 wrote:

> Sure. The majority of it only affects people who are on your storage
> anyway. Perhaps you can post a patch and I can run it through the
> simulator to verify that the minor change to the existing code hasn't
> broken the standard storages. I don't think it is since I've
> thoroughly tested the code I posted, but I know there were some
> additional changes.
>
> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>  wrote:
> > OK, Marcus, I made the change to detect my volumes and it seems to
> work just
> > fine.
> >
> > Perhaps another day of testing and we can check this code in. What
> do you
> > think?
> >
> >
> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
> >  wrote:
> >>
> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
> >>
> >> Yes, that must be the root disk for the VM. I can put in code, as
> you
> >> recommend, to handle only my volumes.
> >>
> >>
> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
> shadow...@gmail.com>
> >> wrote:
> >>>
> >>> It should be sending the path info for each disk per the XML of the
> >>> VM... so it will send all disks regardless of whether or not your
> >>> adaptor manages that disk, and it's up to your adaptor to ignore
> any
> >>> that aren't managed by it. There should be notes to that effect in
> the
> >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
> >>>
> >>> // given local path to file/device (per Libvirt XML), 1) check
> >>> that device is
> >>> // handled by your adaptor, return false if not. 2) clean up
> >>> device, return true
> >>> public boolean disconnectPhysicalDiskByPath(String localPath);
> >>>
> >>> Since we only have XML disk definitions when we stop or migrate a
> VM,
> >>> we have to try all adaptors against all defined disks. So in your
> >>> disconnectPhysicalDisk you might do something like check that the
> path
> >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
> there's
> >>> some way that's more robust like checking the full path against a
> lun
> >>> listing or something). If it doesn't match, then your
> >>> disconnectPhysicalDisk just does nothing.
> >>>
> >>> I assume this is a root disk or some other local storage disk. If
> it's
> >>> not, then your VM XML is messed up somehow.
> >>>
> >>> On Wed, Oct 23, 2013 at 5:01 PM, Mike Tutkowski
> >>>  wrote:
> >>> > I found the problem.
> >>> >
> >>> > disconnectPhysicalDiskByPath is being passed in (in my
> situation) the
> >>> > following:
> >>> >
> >>> > /var/lib/libvirt/images/9887d511-8dc7-4cb4-96f9-01230fe4bbb6
> >>> >
> >>> > Due to the name of the method, my code was expecting data such
> as the
> >>> > following:
> >>> >
> >>> >
> >>> >
> /dev/disk/by-path/ip-192.168.233.10:3260-iscsi-iqn.2012-03.com.solidfire:volume1-lun-0
> >>> >
> >>> > Was it intentional to send the data into this method in the
> current
> >>> > way?
> >>> >
> >>> >
> >>> > On Wed, Oct 23, 2013 at 1:57 PM, Mike Tutkowski
> >>> >  wrote:
> >>> >>
> >>> >> You know, I forgot we supposed to be doing that! :)
> Multi-tasking too
> >>> >> much
> >>> >> today, I guess.
> >>> >>
> >>> >> Anyways, it must not

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Here are some of the tests I ran on this code:

attach/detach/attach volume while VM is running: works
volume attached while VM is running, then reboot: works
volume attached while VM is running, then reset: works
volume detached while VM is stopped, then start: works
volume attached while VM is stopped, then start: works
deleted VM with attached volume: works (volume goes into the attachable
state after VM is expunged)


On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> By the way, in case you're looking at the diff and wondering why I took
> out a StopCommand check and call to execute in LibvirtComputingResource,
> there were two of them.
>
> -} else if (cmd instanceof StopCommand) {
>
> -return execute((StopCommand) cmd);
>
>
> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> This is a newer link, Marcus:
>>
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>>
>> I just rebased on top of master today.
>>
>>
>> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Hey Marcus,
>>>
>>> If you're interested in running the simulator against your code + my
>>> code, I have it on GitHub here:
>>>
>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>>>
>>> Thanks!
>>>
>>>
>>> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
 but I don't see the necessary buttons.

 I wonder if I need to get edit access back again? We had trouble with
 the Wiki. Was this also impacted?


 On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Sure, I can create a diff file and attach it to Review Board.
>
>
> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen  > wrote:
>
>> Sure. The majority of it only affects people who are on your storage
>> anyway. Perhaps you can post a patch and I can run it through the
>> simulator to verify that the minor change to the existing code hasn't
>> broken the standard storages. I don't think it is since I've
>> thoroughly tested the code I posted, but I know there were some
>> additional changes.
>>
>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>  wrote:
>> > OK, Marcus, I made the change to detect my volumes and it seems to
>> work just
>> > fine.
>> >
>> > Perhaps another day of testing and we can check this code in. What
>> do you
>> > think?
>> >
>> >
>> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
>> >  wrote:
>> >>
>> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
>> >>
>> >> Yes, that must be the root disk for the VM. I can put in code, as
>> you
>> >> recommend, to handle only my volumes.
>> >>
>> >>
>> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
>> shadow...@gmail.com>
>> >> wrote:
>> >>>
>> >>> It should be sending the path info for each disk per the XML of
>> the
>> >>> VM... so it will send all disks regardless of whether or not your
>> >>> adaptor manages that disk, and it's up to your adaptor to ignore
>> any
>> >>> that aren't managed by it. There should be notes to that effect
>> in the
>> >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
>> >>>
>> >>> // given local path to file/device (per Libvirt XML), 1) check
>> >>> that device is
>> >>> // handled by your adaptor, return false if not. 2) clean up
>> >>> device, return true
>> >>> public boolean disconnectPhysicalDiskByPath(String localPath);
>> >>>
>> >>> Since we only have XML disk definitions when we stop or migrate a
>> VM,
>> >>> we have to try all adaptors against all defined disks. So in your
>> >>> disconnectPhysicalDisk you might do something like check that the
>> path
>> >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
>> there's
>> >>> some way that's more robust like checking the full path against a
>> lun
>> >>> listing or something). If it doesn't match, then your
>> >>> disconnectPhysicalDisk just does nothing.
>> >>>
>> >>> I assume this is a root disk or some other local storage disk. If
>> it's
>> >>> not, then your VM XML is messed up somehow.
>> >>>
>> >>> On Wed, Oct 23, 2013 at 5:01 PM, Mike Tutkowski
>> >>>  wrote:
>> >>> > I found the problem.
>> >>> >
>> >>> > disconnectPhysicalDiskByPath is being passed in (in my
>> situation) the
>> >>> > following:
>> >>> >
>> >>> > /var/lib/libvirt/images/9887d511-8dc7-4cb4-96f9-01230fe4bbb6
>> >>> >
>

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Having trouble with nfs based storage (or storage tags), but I'm using
your older link, may have been fixed in master... let me try again.

On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski
 wrote:
> Here are some of the tests I ran on this code:
>
> attach/detach/attach volume while VM is running: works
> volume attached while VM is running, then reboot: works
> volume attached while VM is running, then reset: works
> volume detached while VM is stopped, then start: works
> volume attached while VM is stopped, then start: works
> deleted VM with attached volume: works (volume goes into the attachable
> state after VM is expunged)
>
>
> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski
>  wrote:
>>
>> By the way, in case you're looking at the diff and wondering why I took
>> out a StopCommand check and call to execute in LibvirtComputingResource,
>> there were two of them.
>>
>> -} else if (cmd instanceof StopCommand) {
>>
>> -return execute((StopCommand) cmd);
>>
>>
>>
>> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski
>>  wrote:
>>>
>>> This is a newer link, Marcus:
>>>
>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>>>
>>> I just rebased on top of master today.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski
>>>  wrote:

 Hey Marcus,

 If you're interested in running the simulator against your code + my
 code, I have it on GitHub here:


 https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072

 Thanks!


 On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski
  wrote:
>
> I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
> but I don't see the necessary buttons.
>
> I wonder if I need to get edit access back again? We had trouble with
> the Wiki. Was this also impacted?
>
>
> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski
>  wrote:
>>
>> Sure, I can create a diff file and attach it to Review Board.
>>
>>
>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen
>>  wrote:
>>>
>>> Sure. The majority of it only affects people who are on your storage
>>> anyway. Perhaps you can post a patch and I can run it through the
>>> simulator to verify that the minor change to the existing code hasn't
>>> broken the standard storages. I don't think it is since I've
>>> thoroughly tested the code I posted, but I know there were some
>>> additional changes.
>>>
>>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>>  wrote:
>>> > OK, Marcus, I made the change to detect my volumes and it seems to
>>> > work just
>>> > fine.
>>> >
>>> > Perhaps another day of testing and we can check this code in. What
>>> > do you
>>> > think?
>>> >
>>> >
>>> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
>>> >  wrote:
>>> >>
>>> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
>>> >>
>>> >> Yes, that must be the root disk for the VM. I can put in code, as
>>> >> you
>>> >> recommend, to handle only my volumes.
>>> >>
>>> >>
>>> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen
>>> >> 
>>> >> wrote:
>>> >>>
>>> >>> It should be sending the path info for each disk per the XML of
>>> >>> the
>>> >>> VM... so it will send all disks regardless of whether or not your
>>> >>> adaptor manages that disk, and it's up to your adaptor to ignore
>>> >>> any
>>> >>> that aren't managed by it. There should be notes to that effect
>>> >>> in the
>>> >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
>>> >>>
>>> >>> // given local path to file/device (per Libvirt XML), 1)
>>> >>> check
>>> >>> that device is
>>> >>> // handled by your adaptor, return false if not. 2) clean up
>>> >>> device, return true
>>> >>> public boolean disconnectPhysicalDiskByPath(String
>>> >>> localPath);
>>> >>>
>>> >>> Since we only have XML disk definitions when we stop or migrate a
>>> >>> VM,
>>> >>> we have to try all adaptors against all defined disks. So in your
>>> >>> disconnectPhysicalDisk you might do something like check that the
>>> >>> path
>>> >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
>>> >>> there's
>>> >>> some way that's more robust like checking the full path against a
>>> >>> lun
>>> >>> listing or something). If it doesn't match, then your
>>> >>> disconnectPhysicalDisk just does nothing.
>>> >>>
>>> >>> I assume this is a root disk or some other local storage disk. If
>>> >>> it's
>>> >>> not, then your VM XML is messed up somehow.
>>> >>>
>>> >>> On Wed, Oct 23, 2013 at 5:01 PM, Mike Tutkowski
>>> >>>  wrote:
>>> >>> > I found the

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Also, the way I determined if the attach/detach worked was to use fdisk -l
and see if the device was present or not in the hypervisor and VM instance.


On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Here are some of the tests I ran on this code:
>
> attach/detach/attach volume while VM is running: works
> volume attached while VM is running, then reboot: works
> volume attached while VM is running, then reset: works
> volume detached while VM is stopped, then start: works
> volume attached while VM is stopped, then start: works
> deleted VM with attached volume: works (volume goes into the attachable
> state after VM is expunged)
>
>
> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> By the way, in case you're looking at the diff and wondering why I took
>> out a StopCommand check and call to execute in LibvirtComputingResource,
>> there were two of them.
>>
>> -} else if (cmd instanceof StopCommand) {
>>
>> -return execute((StopCommand) cmd);
>>
>>
>> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> This is a newer link, Marcus:
>>>
>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>>>
>>> I just rebased on top of master today.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Hey Marcus,

 If you're interested in running the simulator against your code + my
 code, I have it on GitHub here:


 https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072

 Thanks!


 On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
> but I don't see the necessary buttons.
>
> I wonder if I need to get edit access back again? We had trouble with
> the Wiki. Was this also impacted?
>
>
> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Sure, I can create a diff file and attach it to Review Board.
>>
>>
>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
>> shadow...@gmail.com> wrote:
>>
>>> Sure. The majority of it only affects people who are on your storage
>>> anyway. Perhaps you can post a patch and I can run it through the
>>> simulator to verify that the minor change to the existing code hasn't
>>> broken the standard storages. I don't think it is since I've
>>> thoroughly tested the code I posted, but I know there were some
>>> additional changes.
>>>
>>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>>  wrote:
>>> > OK, Marcus, I made the change to detect my volumes and it seems to
>>> work just
>>> > fine.
>>> >
>>> > Perhaps another day of testing and we can check this code in. What
>>> do you
>>> > think?
>>> >
>>> >
>>> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
>>> >  wrote:
>>> >>
>>> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
>>> >>
>>> >> Yes, that must be the root disk for the VM. I can put in code, as
>>> you
>>> >> recommend, to handle only my volumes.
>>> >>
>>> >>
>>> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
>>> shadow...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> It should be sending the path info for each disk per the XML of
>>> the
>>> >>> VM... so it will send all disks regardless of whether or not your
>>> >>> adaptor manages that disk, and it's up to your adaptor to ignore
>>> any
>>> >>> that aren't managed by it. There should be notes to that effect
>>> in the
>>> >>> code near the disconnectPhysicalDisk interface in StorageAdaptor:
>>> >>>
>>> >>> // given local path to file/device (per Libvirt XML), 1)
>>> check
>>> >>> that device is
>>> >>> // handled by your adaptor, return false if not. 2) clean up
>>> >>> device, return true
>>> >>> public boolean disconnectPhysicalDiskByPath(String
>>> localPath);
>>> >>>
>>> >>> Since we only have XML disk definitions when we stop or migrate
>>> a VM,
>>> >>> we have to try all adaptors against all defined disks. So in your
>>> >>> disconnectPhysicalDisk you might do something like check that
>>> the path
>>> >>> starts with '/dev/disk/by-path' and contains 'iscs-iqn' (maybe
>>> there's
>>> >>> some way that's more robust like checking the full path against
>>> a lun
>>> >>> listing or something). If it doesn't match, then your
>>> >>> disconnectPhysicalDisk just does nothing.
>>> >>>
>>> >>> I assume this is a root disk or some other local storage disk.
>

Re: Metadata URL location

Chiradeep,

Linux distros set 169.254/16 route on the primary interface.  It's just there 
now.  I'm not sure if that's because of ec2 or if it's always been that way, 
but all modern distros will assign it if you have a standard base install.

In the VPC case I think we would have to use network namespaces to bind the 
same IP to multiple NICs.  You could bind the IP to loopback, but then it won't 
ARP.  So since linux distros already have the route, they won't send to the 
gateway.  

Yes systemvm will need to be allowed to talk over 169.254, so that's needs to 
change.

Again, I said the reason I thought this wasn't done is because it's hard.  But 
still doable.  I'd like to see us do this change.  At least in the KVM case, it 
would be real nice to be able to pickup the standard ubuntu (or fedora) cloud 
qcow and have it run in CloudStack.

Darren

> On Oct 24, 2013, at 3:57 PM, Chiradeep Vittal  
> wrote:
> 
> For the VPC virtual router case this would this have to be done on all
> guest interfaces? 
> Could we alias localhost on the VR to 169.254.169.254?
> For shared networks, basic zone and networks where the VR is not the
> default gateway, we would have to send another (169.254.0.0/16) route in
> the DHCP response to point to the VR.
> 
> Additionally in basic zone the anti-spoof filters in dom0 would have to be
> adjusted to let 169.254.169.254 addresses
> 
>> On 10/24/13 8:51 AM, "Darren Shepherd"  wrote:
>> 
>> So additionally you need to do
>> 
>> ip addr add dev eth0 169.254.169.254/0
>> 
>> On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren 
>> wrote:
>>> You would also need to supernet 169.254.169.254 on the virtual router
>>> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it
>>> will
>>> still arp to servers that are calling it that have real ip addresses.
>>> 
>>> Additionally we had some other iptables rules in there that would change
>>> the the ip address of the incoming request to metadata based upon the
>>> mac
>>> address that was hitting it.  This was to prevent spoofing of another
>>> vm's
>>> IP and getting someone else's metadata (at least in our metadata
>>> implementation we keyed off of the VM IP calling into metadata).  This
>>> also allowed a user to set whatever ipaddress they wanted, but as long
>>> as
>>> the mac address was the same and they still had a zeroconfig route on
>>> the
>>> VM, they still got only their metadata.
>>> 
>>> 
>>> Kris Lindgren
>>> Senior Linux Systems Engineer
>>> GoDaddy, LLC.
>>> 
>>> 
>>> This email message and any attachment(s) hereto are intended for use
>>> only
>>> by its intended recipient(s) and may contain confidential information.
>>> If
>>> you have received this email in error, please immediately notify the
>>> sender and permanently delete the original and any copy of this message
>>> and its attachments.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 10/24/13 9:12 AM, "Darren Shepherd" 
>>> wrote:
>>> 
 My guess, I don't really know, would be because its hard.  The VR uses
 link local for the control network so 169.254/16 is bound to the wrong
 nic.  To fix this you just need some ip routing magic in linux (credit
 goes to Kris Lindgren who showed me how to do this).  Add the below to
 a file, substitute eth0 for the guest network nic, run "ip -b "
 The below effectively creates a routing table dedicated to the IP
 169.254.169.254 that sets it default route to go out the guest network
 nic.
 
 rule add from 169.254.169.254 table 70
 rule add to 169.254.169.254 dev eth0 table 70
 route flush table 70
 route add default dev eth0 src 169.254.169.254 table 70
 route flush cache
 
 Darren
 
 On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
  wrote:
> Hi Guys,
> 
> CloudStack metadata services are on the default gateway while on EC2,
> its at 169.254.169.254. Am curious to know why CloudStack does not
> use a link local address for meta data services.
> 
> A search of the Wiki
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTA
> CK
> &tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSea
> rc
> h=true&queryString=metadata) didn¹t seem to list any doc related to the
> design of this service.
> 
> TIA.
> 
> --
> @shankerbalan
> 
> M: +91 98860 60539 | O: +91 (80) 67935867
> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
> Centre, Bangalore - 560 055
> 
> CloudStack Bootcamp Training on 27/28 November, Bangalore
> http://www.shapeblue.com/cloudstack-training/
> 
> 
> 
> 
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Hey Marcus,

I assume you use Virtual Machine Manager with KVM?

I was wondering, is there a way when you stop a VM to have it not disappear
from the GUI?

In XenCenter and VI Client, stopped VMs just show up with a different icon,
but are still easy to interact with.

Thanks!


On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:

> Also, the way I determined if the attach/detach worked was to use fdisk -l
> and see if the device was present or not in the hypervisor and VM instance.
>
>
> On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Here are some of the tests I ran on this code:
>>
>> attach/detach/attach volume while VM is running: works
>> volume attached while VM is running, then reboot: works
>> volume attached while VM is running, then reset: works
>> volume detached while VM is stopped, then start: works
>> volume attached while VM is stopped, then start: works
>> deleted VM with attached volume: works (volume goes into the attachable
>> state after VM is expunged)
>>
>>
>> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> By the way, in case you're looking at the diff and wondering why I took
>>> out a StopCommand check and call to execute in LibvirtComputingResource,
>>> there were two of them.
>>>
>>> -} else if (cmd instanceof StopCommand) {
>>>
>>> -return execute((StopCommand) cmd);
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 This is a newer link, Marcus:


 https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b

 I just rebased on top of master today.


 On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Hey Marcus,
>
> If you're interested in running the simulator against your code + my
> code, I have it on GitHub here:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>
> Thanks!
>
>
> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
>> but I don't see the necessary buttons.
>>
>> I wonder if I need to get edit access back again? We had trouble with
>> the Wiki. Was this also impacted?
>>
>>
>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Sure, I can create a diff file and attach it to Review Board.
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
>>> shadow...@gmail.com> wrote:
>>>
 Sure. The majority of it only affects people who are on your storage
 anyway. Perhaps you can post a patch and I can run it through the
 simulator to verify that the minor change to the existing code
 hasn't
 broken the standard storages. I don't think it is since I've
 thoroughly tested the code I posted, but I know there were some
 additional changes.

 On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
  wrote:
 > OK, Marcus, I made the change to detect my volumes and it seems
 to work just
 > fine.
 >
 > Perhaps another day of testing and we can check this code in.
 What do you
 > think?
 >
 >
 > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
 >  wrote:
 >>
 >> Thanks, Marcus...I hadn't read that note, but that makes sense.
 >>
 >> Yes, that must be the root disk for the VM. I can put in code,
 as you
 >> recommend, to handle only my volumes.
 >>
 >>
 >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
 shadow...@gmail.com>
 >> wrote:
 >>>
 >>> It should be sending the path info for each disk per the XML of
 the
 >>> VM... so it will send all disks regardless of whether or not
 your
 >>> adaptor manages that disk, and it's up to your adaptor to
 ignore any
 >>> that aren't managed by it. There should be notes to that effect
 in the
 >>> code near the disconnectPhysicalDisk interface in
 StorageAdaptor:
 >>>
 >>> // given local path to file/device (per Libvirt XML), 1)
 check
 >>> that device is
 >>> // handled by your adaptor, return false if not. 2) clean up
 >>> device, return true
 >>> public boolean disconnectPhysicalDiskByPath(String
 localPath);
 >>>
 >>> Since we only have XML disk definitions when we stop or migrate
 a VM,
 >>> we have to try all adapto

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Are you talking cloudstack VMS or your own?  If a vm is 'created' it is
ephemeral, the definition will disappear when stopped. This is what
cloudstack does so that a KVM host that crashes or loses power won't
remember and try to start VMS that were previously on it (they are likely
running elsewhere now). However, for your desktop and your own VMS you
usually 'define' VMS, which saves the XML to a file and leaves them
persistent. I use vmm occasionally, but usually virsh, the CLI version.
Both just talk to libvirt.
On Oct 24, 2013 6:08 PM, "Mike Tutkowski" 
wrote:

> Hey Marcus,
>
> I assume you use Virtual Machine Manager with KVM?
>
> I was wondering, is there a way when you stop a VM to have it not
> disappear from the GUI?
>
> In XenCenter and VI Client, stopped VMs just show up with a different
> icon, but are still easy to interact with.
>
> Thanks!
>
>
> On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Also, the way I determined if the attach/detach worked was to use fdisk
>> -l and see if the device was present or not in the hypervisor and VM
>> instance.
>>
>>
>> On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Here are some of the tests I ran on this code:
>>>
>>> attach/detach/attach volume while VM is running: works
>>> volume attached while VM is running, then reboot: works
>>> volume attached while VM is running, then reset: works
>>> volume detached while VM is stopped, then start: works
>>> volume attached while VM is stopped, then start: works
>>> deleted VM with attached volume: works (volume goes into the attachable
>>> state after VM is expunged)
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 By the way, in case you're looking at the diff and wondering why I took
 out a StopCommand check and call to execute in LibvirtComputingResource,
 there were two of them.

 -} else if (cmd instanceof StopCommand) {

 -return execute((StopCommand) cmd);


 On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> This is a newer link, Marcus:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>
> I just rebased on top of master today.
>
>
> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Hey Marcus,
>>
>> If you're interested in running the simulator against your code + my
>> code, I have it on GitHub here:
>>
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>>
>> Thanks!
>>
>>
>> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
>>> but I don't see the necessary buttons.
>>>
>>> I wonder if I need to get edit access back again? We had trouble
>>> with the Wiki. Was this also impacted?
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Sure, I can create a diff file and attach it to Review Board.


 On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
 shadow...@gmail.com> wrote:

> Sure. The majority of it only affects people who are on your
> storage
> anyway. Perhaps you can post a patch and I can run it through the
> simulator to verify that the minor change to the existing code
> hasn't
> broken the standard storages. I don't think it is since I've
> thoroughly tested the code I posted, but I know there were some
> additional changes.
>
> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>  wrote:
> > OK, Marcus, I made the change to detect my volumes and it seems
> to work just
> > fine.
> >
> > Perhaps another day of testing and we can check this code in.
> What do you
> > think?
> >
> >
> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
> >  wrote:
> >>
> >> Thanks, Marcus...I hadn't read that note, but that makes sense.
> >>
> >> Yes, that must be the root disk for the VM. I can put in code,
> as you
> >> recommend, to handle only my volumes.
> >>
> >>
> >> On Wed, Oct 23, 2013 at 5:37 PM, Marcus Sorensen <
> shadow...@gmail.com>
> >> wrote:
> >>>
> >>> It should be sending the path info for each disk per the XML
> of the
> >>> VM... so it will send all disks regardless of whether or not
> your
> >>> adaptor ma

About component.xml and applicationContext.xml.in

2013-10-24 Thread Rayees Namathponnan

In 4.2,  component.xml and applicationContext.xml.in using to load required 
plugin in management server.

component.xml and applicationContext.xml.in are removed from master branch ?  
this case,   where I can enable or disable plugin from cloudstack ?

Regards,
Rayees

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

OK, thanks for that info, Marcus...I was not aware that CloudStack treated
VMs ephemerally with KVM.

I guess I should try to stop a VM on XenServer and ESX and see what
happens. I was under the impression they remained accessible using
XenCenter or VI Client, but perhaps I am wrong about that.

Since my KVM VM's root disk was on local storage, I expected it to remain
accessible in VMM after I had stopped it via CS.


On Thu, Oct 24, 2013 at 6:16 PM, Marcus Sorensen wrote:

> Are you talking cloudstack VMS or your own?  If a vm is 'created' it is
> ephemeral, the definition will disappear when stopped. This is what
> cloudstack does so that a KVM host that crashes or loses power won't
> remember and try to start VMS that were previously on it (they are likely
> running elsewhere now). However, for your desktop and your own VMS you
> usually 'define' VMS, which saves the XML to a file and leaves them
> persistent. I use vmm occasionally, but usually virsh, the CLI version.
> Both just talk to libvirt.
> On Oct 24, 2013 6:08 PM, "Mike Tutkowski" 
> wrote:
>
>> Hey Marcus,
>>
>> I assume you use Virtual Machine Manager with KVM?
>>
>> I was wondering, is there a way when you stop a VM to have it not
>> disappear from the GUI?
>>
>> In XenCenter and VI Client, stopped VMs just show up with a different
>> icon, but are still easy to interact with.
>>
>> Thanks!
>>
>>
>> On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Also, the way I determined if the attach/detach worked was to use fdisk
>>> -l and see if the device was present or not in the hypervisor and VM
>>> instance.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Here are some of the tests I ran on this code:

 attach/detach/attach volume while VM is running: works
 volume attached while VM is running, then reboot: works
 volume attached while VM is running, then reset: works
 volume detached while VM is stopped, then start: works
 volume attached while VM is stopped, then start: works
 deleted VM with attached volume: works (volume goes into the attachable
 state after VM is expunged)


 On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> By the way, in case you're looking at the diff and wondering why I
> took out a StopCommand check and call to execute in
> LibvirtComputingResource, there were two of them.
>
> -} else if (cmd instanceof StopCommand) {
>
> -return execute((StopCommand) cmd);
>
>
> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> This is a newer link, Marcus:
>>
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>>
>> I just rebased on top of master today.
>>
>>
>> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Hey Marcus,
>>>
>>> If you're interested in running the simulator against your code + my
>>> code, I have it on GitHub here:
>>>
>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>>>
>>> Thanks!
>>>
>>>
>>> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 I'm trying to attach my diff to https://reviews.apache.org/r/13865/,
 but I don't see the necessary buttons.

 I wonder if I need to get edit access back again? We had trouble
 with the Wiki. Was this also impacted?


 On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Sure, I can create a diff file and attach it to Review Board.
>
>
> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
> shadow...@gmail.com> wrote:
>
>> Sure. The majority of it only affects people who are on your
>> storage
>> anyway. Perhaps you can post a patch and I can run it through the
>> simulator to verify that the minor change to the existing code
>> hasn't
>> broken the standard storages. I don't think it is since I've
>> thoroughly tested the code I posted, but I know there were some
>> additional changes.
>>
>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>  wrote:
>> > OK, Marcus, I made the change to detect my volumes and it seems
>> to work just
>> > fine.
>> >
>> > Perhaps another day of testing and we can check this code in.
>> What do you
>> > think?
>> >
>> >
>> > On Wed, Oct 23, 2013 at 9:14 PM, Mike Tutkowski
>> >  wr

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

That is probably true, but there is no xencenter or vsphere equivalent for
KVM, no central authority. Its cloudstack.
On Oct 24, 2013 6:22 PM, "Mike Tutkowski" 
wrote:

> OK, thanks for that info, Marcus...I was not aware that CloudStack treated
> VMs ephemerally with KVM.
>
> I guess I should try to stop a VM on XenServer and ESX and see what
> happens. I was under the impression they remained accessible using
> XenCenter or VI Client, but perhaps I am wrong about that.
>
> Since my KVM VM's root disk was on local storage, I expected it to remain
> accessible in VMM after I had stopped it via CS.
>
>
> On Thu, Oct 24, 2013 at 6:16 PM, Marcus Sorensen wrote:
>
>> Are you talking cloudstack VMS or your own?  If a vm is 'created' it is
>> ephemeral, the definition will disappear when stopped. This is what
>> cloudstack does so that a KVM host that crashes or loses power won't
>> remember and try to start VMS that were previously on it (they are likely
>> running elsewhere now). However, for your desktop and your own VMS you
>> usually 'define' VMS, which saves the XML to a file and leaves them
>> persistent. I use vmm occasionally, but usually virsh, the CLI version.
>> Both just talk to libvirt.
>>  On Oct 24, 2013 6:08 PM, "Mike Tutkowski" 
>> wrote:
>>
>>> Hey Marcus,
>>>
>>> I assume you use Virtual Machine Manager with KVM?
>>>
>>> I was wondering, is there a way when you stop a VM to have it not
>>> disappear from the GUI?
>>>
>>> In XenCenter and VI Client, stopped VMs just show up with a different
>>> icon, but are still easy to interact with.
>>>
>>> Thanks!
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Also, the way I determined if the attach/detach worked was to use fdisk
 -l and see if the device was present or not in the hypervisor and VM
 instance.


 On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Here are some of the tests I ran on this code:
>
> attach/detach/attach volume while VM is running: works
> volume attached while VM is running, then reboot: works
> volume attached while VM is running, then reset: works
> volume detached while VM is stopped, then start: works
> volume attached while VM is stopped, then start: works
> deleted VM with attached volume: works (volume goes into the
> attachable state after VM is expunged)
>
>
> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> By the way, in case you're looking at the diff and wondering why I
>> took out a StopCommand check and call to execute in
>> LibvirtComputingResource, there were two of them.
>>
>> -} else if (cmd instanceof StopCommand) {
>>
>> -return execute((StopCommand) cmd);
>>
>>
>> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> This is a newer link, Marcus:
>>>
>>>
>>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>>>
>>> I just rebased on top of master today.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Hey Marcus,

 If you're interested in running the simulator against your code +
 my code, I have it on GitHub here:


 https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072

 Thanks!


 On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> I'm trying to attach my diff to
> https://reviews.apache.org/r/13865/, but I don't see the
> necessary buttons.
>
> I wonder if I need to get edit access back again? We had trouble
> with the Wiki. Was this also impacted?
>
>
> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Sure, I can create a diff file and attach it to Review Board.
>>
>>
>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
>> shadow...@gmail.com> wrote:
>>
>>> Sure. The majority of it only affects people who are on your
>>> storage
>>> anyway. Perhaps you can post a patch and I can run it through the
>>> simulator to verify that the minor change to the existing code
>>> hasn't
>>> broken the standard storages. I don't think it is since I've
>>> thoroughly tested the code I posted, but I know there were some
>>> additional changes.
>>>
>>> On Wed, Oct 23, 2013 at 10:35 PM, Mike Tutkowski
>>>  wrote:
>>> > OK, Marcus, I made the cha

Re: [VOTE] Accept the donation of RDP client code into Apache CloudStack

2013-10-24 Thread Chip Childers



> On Oct 24, 2013, at 3:20 PM, Donal Lafferty  wrote:
> 
> It's been a bit over 72 hours, and I would like to now close voting on the 
> RDP client code donation.
> 
> We have an acceptance with the following results:
> 
> +1
> Animesh Chaturvedi 
> Devdeep Singh
> Chiradeep Vittal
> Ahmad Emneina
> Rajesh Battala
> Kelven Yang
> Chip Childers
> Clayton Weise
> John Kinsella
> Sateesh Chodapuneedi
> Ryan Shafer
> 
> +0 
> Wido den Hollander
> Daan hoogland
> 
> The next stage is to put in place IP clearance paper work in advance before 
> the code can go into a branch of master.  I will contact a PMC member to 
> arrange for this paperwork.

No need to contact any PMC members off list (unless there are legal concerns, 
in which case email to private@cs.a.o, not individuals). I just need an ack 
from secretary@ saying that a properly signed Software Grant Agreement has been 
received by the ASF to move through the procedural steps. Citrix has done these 
before, so they should be known to your legal dept. 

Thanks!


> 
> Thanks for participating everyone!
> 
> 
>> -Original Message-
>> From: Donal Lafferty [mailto:donal.laffe...@citrix.com]
>> Sent: 21 October 2013 19:12
>> To: dev@cloudstack.apache.org
>> Subject: [VOTE] Accept the donation of RDP client code into Apache
>> CloudStack
>> 
>> As stated in a previous thread [1], Citrix is proposing the donation of 
>> source
>> for an RDP client.  After donation, the client would be integrated with the
>> console system VM in order to provide access to Hyper-V based VMs.
>> 
>> The client's source is in the diff attached to the Review Board submission
>> https://reviews.apache.org/r/14701/
>> 
>> [1] http://markmail.org/thread/q6sfqrhosmirm3bg
>> 
>> I would like to call a vote here, so that we have a formal consensus on
>> accepting the code into the project.  I suggest that it be accepted into a
>> branch, and then we work through any technical concerns / reviews /
>> changes prior to a master branch merge.
>> 
>> VOTING will be left open for 72 hours.
>> 
>> This is a technical decision, which means committer and PMC votes are
>> binding.
>> 
>> 
>> DL
>

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Well...XenCenter is "just" the GUI (like VI Client is for vSphere). As far
as I know, XenServer has no equivalent to vCenter Server.


On Thu, Oct 24, 2013 at 6:52 PM, Marcus Sorensen wrote:

> That is probably true, but there is no xencenter or vsphere equivalent for
> KVM, no central authority. Its cloudstack.
> On Oct 24, 2013 6:22 PM, "Mike Tutkowski" 
> wrote:
>
>> OK, thanks for that info, Marcus...I was not aware that CloudStack
>> treated VMs ephemerally with KVM.
>>
>> I guess I should try to stop a VM on XenServer and ESX and see what
>> happens. I was under the impression they remained accessible using
>> XenCenter or VI Client, but perhaps I am wrong about that.
>>
>> Since my KVM VM's root disk was on local storage, I expected it to remain
>> accessible in VMM after I had stopped it via CS.
>>
>>
>> On Thu, Oct 24, 2013 at 6:16 PM, Marcus Sorensen wrote:
>>
>>> Are you talking cloudstack VMS or your own?  If a vm is 'created' it is
>>> ephemeral, the definition will disappear when stopped. This is what
>>> cloudstack does so that a KVM host that crashes or loses power won't
>>> remember and try to start VMS that were previously on it (they are likely
>>> running elsewhere now). However, for your desktop and your own VMS you
>>> usually 'define' VMS, which saves the XML to a file and leaves them
>>> persistent. I use vmm occasionally, but usually virsh, the CLI version.
>>> Both just talk to libvirt.
>>>  On Oct 24, 2013 6:08 PM, "Mike Tutkowski" 
>>> wrote:
>>>
 Hey Marcus,

 I assume you use Virtual Machine Manager with KVM?

 I was wondering, is there a way when you stop a VM to have it not
 disappear from the GUI?

 In XenCenter and VI Client, stopped VMs just show up with a different
 icon, but are still easy to interact with.

 Thanks!


 On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Also, the way I determined if the attach/detach worked was to use
> fdisk -l and see if the device was present or not in the hypervisor and VM
> instance.
>
>
> On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Here are some of the tests I ran on this code:
>>
>> attach/detach/attach volume while VM is running: works
>> volume attached while VM is running, then reboot: works
>> volume attached while VM is running, then reset: works
>> volume detached while VM is stopped, then start: works
>> volume attached while VM is stopped, then start: works
>> deleted VM with attached volume: works (volume goes into the
>> attachable state after VM is expunged)
>>
>>
>> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> By the way, in case you're looking at the diff and wondering why I
>>> took out a StopCommand check and call to execute in
>>> LibvirtComputingResource, there were two of them.
>>>
>>> -} else if (cmd instanceof StopCommand) {
>>>
>>> -return execute((StopCommand) cmd);
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 This is a newer link, Marcus:


 https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b

 I just rebased on top of master today.


 On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> Hey Marcus,
>
> If you're interested in running the simulator against your code +
> my code, I have it on GitHub here:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>
> Thanks!
>
>
> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> I'm trying to attach my diff to
>> https://reviews.apache.org/r/13865/, but I don't see the
>> necessary buttons.
>>
>> I wonder if I need to get edit access back again? We had trouble
>> with the Wiki. Was this also impacted?
>>
>>
>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Sure, I can create a diff file and attach it to Review Board.
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:40 PM, Marcus Sorensen <
>>> shadow...@gmail.com> wrote:
>>>
 Sure. The majority of it only affects people who are on your
 storage
 anyway. Perhaps you can post a patch and I can run it through
 the
 simulator to verify that the minor change to the existing code

Re: Review Request 14381: KVM: add connect/disconnect capabilities to StorageAdaptors so that external storage services can attach/detach devices on-demand

Oh, xencenter doesn't remember VMS and allow you to start one if the host
it was on is down? I haven't played with it in a year but I thought it
synced with each xen server.
On Oct 24, 2013 7:06 PM, "Mike Tutkowski" 
wrote:

> Well...XenCenter is "just" the GUI (like VI Client is for vSphere). As far
> as I know, XenServer has no equivalent to vCenter Server.
>
>
> On Thu, Oct 24, 2013 at 6:52 PM, Marcus Sorensen wrote:
>
>> That is probably true, but there is no xencenter or vsphere equivalent
>> for KVM, no central authority. Its cloudstack.
>>  On Oct 24, 2013 6:22 PM, "Mike Tutkowski" 
>> wrote:
>>
>>> OK, thanks for that info, Marcus...I was not aware that CloudStack
>>> treated VMs ephemerally with KVM.
>>>
>>> I guess I should try to stop a VM on XenServer and ESX and see what
>>> happens. I was under the impression they remained accessible using
>>> XenCenter or VI Client, but perhaps I am wrong about that.
>>>
>>> Since my KVM VM's root disk was on local storage, I expected it to
>>> remain accessible in VMM after I had stopped it via CS.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 6:16 PM, Marcus Sorensen wrote:
>>>
 Are you talking cloudstack VMS or your own?  If a vm is 'created' it is
 ephemeral, the definition will disappear when stopped. This is what
 cloudstack does so that a KVM host that crashes or loses power won't
 remember and try to start VMS that were previously on it (they are likely
 running elsewhere now). However, for your desktop and your own VMS you
 usually 'define' VMS, which saves the XML to a file and leaves them
 persistent. I use vmm occasionally, but usually virsh, the CLI version.
 Both just talk to libvirt.
  On Oct 24, 2013 6:08 PM, "Mike Tutkowski" <
 mike.tutkow...@solidfire.com> wrote:

> Hey Marcus,
>
> I assume you use Virtual Machine Manager with KVM?
>
> I was wondering, is there a way when you stop a VM to have it not
> disappear from the GUI?
>
> In XenCenter and VI Client, stopped VMs just show up with a different
> icon, but are still easy to interact with.
>
> Thanks!
>
>
> On Thu, Oct 24, 2013 at 5:46 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Also, the way I determined if the attach/detach worked was to use
>> fdisk -l and see if the device was present or not in the hypervisor and 
>> VM
>> instance.
>>
>>
>> On Thu, Oct 24, 2013 at 5:42 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> Here are some of the tests I ran on this code:
>>>
>>> attach/detach/attach volume while VM is running: works
>>> volume attached while VM is running, then reboot: works
>>> volume attached while VM is running, then reset: works
>>> volume detached while VM is stopped, then start: works
>>> volume attached while VM is stopped, then start: works
>>> deleted VM with attached volume: works (volume goes into the
>>> attachable state after VM is expunged)
>>>
>>>
>>> On Thu, Oct 24, 2013 at 5:40 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 By the way, in case you're looking at the diff and wondering why I
 took out a StopCommand check and call to execute in
 LibvirtComputingResource, there were two of them.

 -} else if (cmd instanceof StopCommand) {

 -return execute((StopCommand) cmd);


 On Thu, Oct 24, 2013 at 5:26 PM, Mike Tutkowski <
 mike.tutkow...@solidfire.com> wrote:

> This is a newer link, Marcus:
>
>
> https://github.com/mike-tutkowski/incubator-cloudstack/commit/e84de7577ad38fea88d1492d4949672d1989889b
>
> I just rebased on top of master today.
>
>
> On Thu, Oct 24, 2013 at 3:46 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Hey Marcus,
>>
>> If you're interested in running the simulator against your code +
>> my code, I have it on GitHub here:
>>
>>
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/776f3eb9dda45745f43e6765b026d34fbc6be072
>>
>> Thanks!
>>
>>
>> On Wed, Oct 23, 2013 at 11:06 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> I'm trying to attach my diff to
>>> https://reviews.apache.org/r/13865/, but I don't see the
>>> necessary buttons.
>>>
>>> I wonder if I need to get edit access back again? We had trouble
>>> with the Wiki. Was this also impacted?
>>>
>>>
>>> On Wed, Oct 23, 2013 at 10:47 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
 Sure, I can create a diff file and attach it to Review Board.

Q: Cloudstack S3 performance

2013-10-24 Thread Zoran Rajic

Hello fellow Cloudstack users,

This is my first post to this mailing list, so please excuse me if I'm not
following the proper etiquette.

My name is Zoran, and I'm a developer working for a DDN company.  While
investigating the Cloudstack S3 and its performance, I encountered a bit
weird behavior of the Cloudstack S3 server, so I wanted to verify my
findings with you guys.

To test the S3 performance I used Python and BOTO libraries to create an
S3 client that is adding random content/name keys into the Cloudstack S3
and a single bucket.  To my surprise, the Cloudstack S3 buckets were
getting more and more unresponsive.  For example, at about 20'000 keys it
was taking about 10 seconds to "get a bucket" (BOTO ref
http://boto.s3.amazonaws.com/ref/s3.html#boto.s3.connection.S3Connection.ge
t_bucket, AWS ref 
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html).

Did someone else also notice this significant slow-downs, or is it perhaps
just my environment and possibly misconfiguration?


Not to leave it at this, I tried to locate the delay on the Cloudstack S3
server-side, and I may have found two potential issues with
SObjectDaoImpl.listBucketObjects()  (ref
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=awsapi/sr
c/com/cloud/bridge/persist/dao/SObjectDaoImpl.java;h=6d23757b8b57ded9443bfe
61aaa3742590b21c49;hb=master#l71):

1) the maxKeys parameter seems to be ignored, so all 20'000+ keys (ie.
full bucket content) was being inspected instead of normally just first
1'000 keys, which is a default maxKeys value

2) the way the object's data is extracted form the MySQL seems to be using
sub-queries instead of JOINs, so something similar to this:
>  objList = SQL("select * from SObject where SBucketID='xxx' ");
>  for (ObjectVO obj : objList) {
> objItem = SQL("select * from SObject_Item where SObjectID='yyy' ");
>  }

Note that the data can be retrieved "in one go" and lot more efficiently
if one used JOIN on the database-side, ie.
>  select * from sobject so LEFT JOIN sobject_item si on
>so.ID=si.SObjectID where SBucketID='xxx';
However, I am not sure if the Cloudstack DAO and *VO-objects abstraction
supports database JOINs.

Can someone confirm if these are actual code issues?

Thank you in advance!


Best regards,
Zoran

RE: Review Request 14843: CLOUDSTACK-1049 : git commit by using command cloudstack-sccs from installed setup

2013-10-24 Thread Rayees Namathponnan

Hi David,

This will be useful with RPM installation;  you need to execute the command 
"cloudstack-sccs" to get the git commit after installing RPM.

This is similar like command  "cloud-sccs" before 4.1 release.

Regards,
Rayees 

-Original Message-
From: David Nalley [mailto:da...@gnsa.us] 
Sent: Thursday, October 24, 2013 3:12 PM
To: dev@cloudstack.apache.org; Rayees Namathponnan
Cc: Chip Childers; Frank Zhang; Hugo Trippaers; ASF Subversion and Git Services
Subject: Re: Review Request 14843: CLOUDSTACK-1049 : git commit by using 
command cloudstack-sccs from installed setup

Rayees:

While it doesn't technically break things, it isn't very refined:
Here's the output from a machine using a release tarball on a machine without 
git:
++ git rev-parse HEAD
/var/tmp/rpm-tmp.JvKJtB: line 33: git: command not found
+ echo
E.g. when you actually go to release software, the file will be empty.

When you do the same with a release tarball on a machine with git:
++ git rev-parse HEAD
fatal: Not a git repository (or any of the parent directories): .git
+ echo


Keep in mind we build a release tarball and it strips out all of the git 
information. Can we come up with a more refined solution that if the above git 
rev-parse fails that we also look elsewhere?

--David

--David

On Tue, Oct 22, 2013 at 8:12 PM, Rayees Namathponnan 
 wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14843/
> ---
>
> (Updated Oct. 23, 2013, 12:12 a.m.)
>
>
> Review request for cloudstack, Chip Childers, Frank Zhang, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-1049
> https://issues.apache.org/jira/browse/CLOUDSTACK-1049
>
>
> Repository: cloudstack-git
>
>
> Description
> ---
>
> The generation of sccs-info was done by WAF but has been broken from 
> 4.2 onwords;
>
> updated cloud.spec file to generate git commit and package with rpm
>
> if you are building outside git tree, this file will be blank; and 
> packaging works fine
>
>
> Diffs
> -
>
>   packaging/centos63/cloud.spec 17fb2b1
>   packaging/centos63/cloudstack-sccs PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14843/diff/
>
>
> Testing
> ---
>
> Tested
>
>
> File Attachments (updated)
> 
>
> We need to merge with 4.2 branch; submitting patch for this
>   
> https://reviews.apache.org/media/uploaded/files/2013/10/23/0001-CLOUDS
> TACK-1049-git-commit-from-installed-cloudstack.patch
>
>
> Thanks,
>
> Rayees Namathponnan
>

Re: Anyone ever get this exception?

It was simple enough I committed
d178b25daa484dda44bb34e63cb719d1c3cc1519 for this.

Darren

On Thu, Oct 24, 2013 at 4:07 PM, Mike Tutkowski
 wrote:
> OK, thanks, Darren.
>
> If it becomes larger than a "simple" fix, I can log a JIRA ticket for it,
> too.
>
>
> On Thu, Oct 24, 2013 at 4:19 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> Never seen that, but yeah that's an issue.  It's possible given the
>> current initialization in ACS to somebody tries to lock something
>> before the lock manager is created.  That should change let me
>> look at that.  I can probably fix it on master.
>>
>> Darren
>>
>> On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
>>  wrote:
>> > Hi,
>> >
>> > I'm running in master.
>> >
>> > I had to reboot the only host (KVM) in my CloudStack dev environment. As
>> > you'd expect, it was running my system VMs.
>> >
>> > At the same time, I had to restart my CS MS.
>> >
>> > I've noticed this exception before when I first start up the CS MS and it
>> > has to restart SSVM. SSVM doesn't start and I have to restart the CS MS
>> to
>> > get it to work (usually starts up SSVM the second time without any
>> > exceptions):
>> >
>> > WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
>> > exception There's no support for locking yet
>> > com.cloud.utils.exception.CloudRuntimeException: There's no support for
>> > locking yet
>> > at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
>> > at
>> >
>> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
>> > at
>> >
>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
>> > at
>> >
>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
>> > at
>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
>> > at
>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
>> > at
>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
>> > at
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
>> > at
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > at
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:724)
>> >
>> > Thanks!
>> >
>> > --
>> > *Mike Tutkowski*
>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > e: mike.tutkow...@solidfire.com
>> > o: 303.746.7302
>> > Advancing the way the world uses the
>> > cloud
>>

Re: About component.xml and applicationContext.xml.in

Refer to
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions

In short your can either add/remove jars from the classpath or use the
*exclude properties to disable elements.  Exclude properties are
usually the easiest as most everything is enabled by default.

Darren

On Thu, Oct 24, 2013 at 5:17 PM, Rayees Namathponnan
 wrote:
> In 4.2,  component.xml and applicationContext.xml.in using to load required 
> plugin in management server.
>
> component.xml and applicationContext.xml.in are removed from master branch ?  
> this case,   where I can enable or disable plugin from cloudstack ?
>
> Regards,
> Rayees
>

Re: Anyone ever get this exception?

Great - thanks!


On Thu, Oct 24, 2013 at 11:01 PM, Darren Shepherd <
darren.s.sheph...@gmail.com> wrote:

> It was simple enough I committed
> d178b25daa484dda44bb34e63cb719d1c3cc1519 for this.
>
> Darren
>
> On Thu, Oct 24, 2013 at 4:07 PM, Mike Tutkowski
>  wrote:
> > OK, thanks, Darren.
> >
> > If it becomes larger than a "simple" fix, I can log a JIRA ticket for it,
> > too.
> >
> >
> > On Thu, Oct 24, 2013 at 4:19 PM, Darren Shepherd <
> > darren.s.sheph...@gmail.com> wrote:
> >
> >> Never seen that, but yeah that's an issue.  It's possible given the
> >> current initialization in ACS to somebody tries to lock something
> >> before the lock manager is created.  That should change let me
> >> look at that.  I can probably fix it on master.
> >>
> >> Darren
> >>
> >> On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
> >>  wrote:
> >> > Hi,
> >> >
> >> > I'm running in master.
> >> >
> >> > I had to reboot the only host (KVM) in my CloudStack dev environment.
> As
> >> > you'd expect, it was running my system VMs.
> >> >
> >> > At the same time, I had to restart my CS MS.
> >> >
> >> > I've noticed this exception before when I first start up the CS MS
> and it
> >> > has to restart SSVM. SSVM doesn't start and I have to restart the CS
> MS
> >> to
> >> > get it to work (usually starts up SSVM the second time without any
> >> > exceptions):
> >> >
> >> > WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6)
> Unexpected
> >> > exception There's no support for locking yet
> >> > com.cloud.utils.exception.CloudRuntimeException: There's no support
> for
> >> > locking yet
> >> > at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
> >> > at
> >> >
> >>
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
> >> > at
> >> >
> >>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> >> > at
> >> >
> >>
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
> >> > at
> >> >
> >>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> >> > at
> >> >
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
> >> > at
> >> >
> >>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> >> > at
> >> >
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
> >> > at
> >> >
> >>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
> >> > at
> >> >
> >>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
> >> > at
> >> >
> >>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
> >> > at
> >> >
> >>
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
> >> > at
> >> >
> >>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> >> > at
> >> >
> >>
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> >> > at
> >> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
> >> > at
> >> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> >> > at
> >>
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
> >> > at
> >> >
> >>
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
> >> > at
> >> >
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> > at
> >> >
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> > at
> >> >
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> > at
> >> >
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> > at
> >> >
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> > at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> >> > at
> >> >
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >> > at
> >> >
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> > at
>

[Question] CloudStack support tool: cloud-bugtool

2013-10-24 Thread Gavin Lee

Unlike the CS4.2 release notes say, I could not find cloud-bugtool in
source tree or installed/upgrade CS4.2 platform.
This script can only be found in CloudPlatform 4.2.

I think the support tool like this is very helpful, does anyone know the
detailed info?
Thanks!

-- 
Gavin

Re: Review Request 14889: Expose as much type information to commands.xml as well as to the generated python clasesss for Marvin

2013-10-24 Thread Prasanna Santhanam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14889/#review27513
---


I actually don't quite see how marvin uses the typeInfo. Will it only act as a 
carrier for this information? 

How would a test case author use this? It would've been nice if we could use 
the type info to deserialize the response returned by the server so each field 
gets its appropriate type. But in most tests we don't use the type information 
afaik.

- Prasanna Santhanam


On Oct. 23, 2013, 9:38 p.m., Ryan Dietrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14889/
> ---
> 
> (Updated Oct. 23, 2013, 9:38 p.m.)
> 
> 
> Review request for cloudstack, Marcus Sorensen and Prasanna Santhanam.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> I have a requirement for some work I am doing to "fail early".  The existing 
> marvin cloudstackAPI classes give me both the "legal" and "required" elements 
> for each API call as well as the response attributes.  However, they do not 
> tell me the types of each value.  I've gone ahead and exposed that type 
> information to python and to commands.xml as well.  I decided to try and 
> expose things so we're being as descriptive as possible.  It's a little 
> deflating how the Request/Response annotations aren't equal (Requests can 
> expose "strings" as "uuid's", allowing me to auto generate regex's for 
> validation).  Responses however, simply fall back to the specific java type, 
> (Integer, Long, String, Boolean).
> 
> I believe cloudmonkey could take advantage of this, by stopping invalid 
> parameters from being sent to Cloudstack in the first place (client side 
> validation is good, right?)
> I also think the UI could use this as part of their validation routines, if 
> they're using commands.xml as part of their build process.
> 
> 
> Diffs
> -
> 
>   server/src/com/cloud/api/doc/ApiXmlDocWriter.java c3c0cab 
>   server/src/com/cloud/api/doc/Argument.java 29c361e 
>   tools/marvin/marvin/codegenerator.py 96729f6 
> 
> Diff: https://reviews.apache.org/r/14889/diff/
> 
> 
> Testing
> ---
> 
> mvn clean
> mvn
> cd tools/apidoc
> mvn
> inspect target/commands.xml
> cd ../tools/marvin
> mvn
> inspect python classes
> 
> 
> Thanks,
> 
> Ryan Dietrich
> 
>

ISO/Template Registration - OS Type Dropdown List