Re: ssh keypair value overridden by template meta data

[Harikrishna Patnala]

Hi Hiroki,

We should not override the ssh key pair provided in the deployVM API with the 
ssh key in template metadata.
I just able to reproduce this. Please create a ticket for this issue.

Thanks
Harikrishna

On 06-Jun-2014, at 5:18 pm, Hiroki Ohashi  wrote:

> Dear guys
> 
> I encountered a problem that a ssh public key of ssh_keypairs for a
> newly created instance is overridden by another ssh key in template
> meta data. I think this leads to security vulnerability because a
> template owner can login to other user's instance created from the
> template. So, could you fix this issue?
> 
> This behavior is caused by meta data import at commitUserVm method. A
> ssh key value specified by an instance owner is set to a UserVmVO
> object at line 2986-2988 of
> server/src/com/cloud/vm/UserVmManagerImpl.java in 4.3 branch [1], but
> this value is overridden at line 3035-3038 by template meta data.
> 
> Please note a database contains meta data entries related to a
> template which you created from an instance in cloud.template_view
> like [2]. 2nd row has detail_name and detail_value about ssh key and
> CloudStack override the user specified ssh key value by the owner's
> detail value in this situation. It results in delivery of the template
> owner's ssh key for the instance created from the template to virtual
> router inspite of specification of instance owner's ssh key.
> 
> You can reproduce this phenomenon like below.
> 
>1. Deploy an instance with a ssh key A by specifying 'keypair'
>   value.
>2. Create a template from this instance.
>3. Deploy an instance with another ssh key B by specifying
>   'keypair' value.
> 
> 
> [1] server/src/com/cloud/vm/UserVmManagerImpl.java
> 
>   2971 private UserVmVO commitUserVm(final DataCenter zone, final
> VirtualMachineTemplate template, final String hostName, final String
> displayName, final Account owner,
>   2972 final Long diskOfferingId, final Long diskSize, final
> String userData, final HypervisorType hypervisor, final Account
> caller, final Boolean isDisplayVmEnabled,
>   2973 final String keyboard, final long accountId, final
> ServiceOfferingVO offering, final boolean isIso, final String
> sshPublicKey,
>   2974 final LinkedHashMap networkNicMap,
> final long id, final String instanceName, final String uuidName, final
> HypervisorType hypervisorType,
>   2975 final Map customParameters) throws
> InsufficientCapacityException {
>   2976 return Transaction.execute(new
> TransactionCallbackWithException InsufficientCapacityException>() {
>   2977 @Override
>   2978 public UserVmVO doInTransaction(TransactionStatus
> status) throws InsufficientCapacityException {
>   2979 UserVmVO vm = new UserVmVO(id, instanceName,
> displayName,
>   2980 template.getId(), hypervisorType,
> template.getGuestOSId(),
>   2981 offering.getOfferHA(),
> offering.getLimitCpuUse(),
>   2982 owner.getDomainId(), owner.getId(),
> offering.getId(), userData,
>   2983 hostName, diskOfferingId);
>   2984 vm.setUuid(uuidName);
>   2985
> vm.setDynamicallyScalable(template.isDynamicallyScalable());
>   2986 if (sshPublicKey != null) {
>   2987 vm.setDetail("SSH.PublicKey", sshPublicKey);
>   2988 }
>   2989
>   2990 if (keyboard != null && !keyboard.isEmpty())
>   2991 vm.setDetail(VmDetailConstants.KEYBOARD, keyboard);
>   2992
>   2993 if (isIso) {
>   2994 vm.setIsoId(template.getId());
>   2995 }
>   2996
>   2997 if(isDisplayVmEnabled != null){
>   2998 if(!_accountMgr.isRootAdmin(caller.getType())){
>   2999 throw new PermissionDeniedException(
> "Cannot update parameter displayvm, only admin permitted ");
>   3000 }
>   3001 vm.setDisplayVm(isDisplayVmEnabled);
>   3002 }else {
>   3003 vm.setDisplayVm(true);
>   3004 }
>   3005
>   3006 // If hypervisor is vSphere, check for clone
> type setting.
>   3007 if (hypervisorType.equals(HypervisorType.VMware)) {
>   3008 // retrieve clone flag.
>   3009 UserVmCloneType cloneType = UserVmCloneType.linked;
>   3010 String value =
> _configDao.getValue(Config.VmwareCreateFullClone.key());
>   3011 if (value != null) {
>   3012 if (Boolean.parseBoolean(value) == true)
>   3013 cloneType = UserVmCloneType.full;
>   3014 }
>   3015 UserVmCloneSettingVO vmCloneSettingVO =
> new UserVmCloneSettingVO(id, cloneType.toString());
>   3016   

RE: KVM + LXC on the same host

[Kishan Kavala]

> -Original Message-
> From: ilya musayev [mailto:ilya.mailing.li...@gmail.com]
> Sent: Saturday, 7 June 2014 1:50 AM
> To: dev@cloudstack.apache.org
> Subject: Re: KVM + LXC on the same host
> 
> Tuna
> 
> Thanks for the feedback, conceptually, i was not trying to address the issue 
> of
> sysvms not running on LXC.
> 
> Here is the use case as i see it.
> 
> Assume you roll out a farm of  10 KVM servers and your density with KVM is
> 200 virtual machines. Most of these VMs dont require independent kernel
> and additional layers of virtualization abstraction. As the result, you can 
> place
> 400 LXC machines and 20 fully virtualized Linux or even Windows Servers. If
> you do chargeback, you can offer the LXC machines for much lower price point
> since we can place more LXC containers.
> 
> Your density becomes much greater with LXC and yet you still cover the
> corner case when end-user needs KVM instance.
> 
> Pierre-Luc
> 
> If I will get to try this scenario - i may have to alter the KVM/LXC agent
> somewhat to make it work with CloudStack, i will let you know.
Ilya, 
 Some minor changes to KVM/LXC agent will be required to make this work. Below 
piece of code in createVMFromSpec (LibvirtComputingResource), deploys systems 
Vms in KVM and user Vms in LXC.
If we include some flag in VirtualMachineTO, it should be possible to specify 
which userVm has to be deployed (KVM/LXC).

if (HypervisorType.LXC == _hypervisorType && VirtualMachine.Type.User 
== vmTO.getType()) {
// LXC domain is only valid for user VMs. Use KVM for system VMs.
guest.setGuestType(GuestDef.guestType.LXC);
vm.setHvsType(HypervisorType.LXC.toString().toLowerCase());
} else {
guest.setGuestType(GuestDef.guestType.KVM);
vm.setHvsType(HypervisorType.KVM.toString().toLowerCase());
vm.setLibvirtVersion(_hypervisorLibvirtVersion);
vm.setQemuVersion(_hypervisorQemuVersion);
}



> Alternative solution would be to run separate LXC and KVM clusters, which is
> also a possibility - but usage and distribution will be uneven.
> 
> Regards
> ilya
> 
> The concept of combining both technologies under one hypervisor On 6/6/14,
> 8:42 AM, Pierre-Luc Dion wrote:
> > ilya,
> >
> > Let us know how it goes your hybrid of LXC+KVM. I'm interested to know
> > how it's going,  I might try that too on the side.
> >
> >
> >
> >
> > Pierre-Luc Dion
> > Architecte de Solution Cloud | Cloud Solutions Architect 855-OK-CLOUD
> > (855-652-5683) x1101
> > - - -
> >
> > *CloudOps*420 rue Guy
> > Montréal QC  H3J 1S6
> > www.cloudops.com
> > @CloudOps_
> >
> >
> > On Fri, Jun 6, 2014 at 10:53 AM, Nguyen Anh Tu  wrote:
> >
> >> That should be a good idea, Ilya. At that moment, i'm working on
> >> Docker support. When it's done, we can run only Docker hosts, no need
> >> to use KVM for hosting system vms.
> >>
> >> Cheers,
> >> --Tuna
> >>
> >> Sent from my GT-N7000
> >> On Jun 5, 2014 7:58 AM, "ilya musayev" 
> >> wrote:
> >>
> >>> We are considering running KVM and LXC on the same host and
> >>> hopefully control both through cloudstack.
> >>>
> >>> I know there are agents involved for each component, i dont know if
> >>> we
> >> can
> >>> have a hybrid of LXC+KVM.
> >>>
> >>> The use case is simple, we would like the end user to pick
> >>> LXC/Docker for performance, or KVM instance if he really needed all
> >>> bells and whistles
> >> of
> >>> dedicated kernel in fully virtualized environment.
> >>>
> >>> Is anyone aware why we should not mix 2 workloads on the same host?
> >>> Is it possible at this point in time to mix LXC, KVM and CloudStack,
> >>> i assume
> >> the
> >>> answer is no, but perhaps there is a hack i can try.
> >>>
> >>> Thanks
> >>> ilya
> >>>

Re: Review Request 22364: Fixed 26 issues reported by coverity

[Rajani Karuturi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22364/
---

(Updated June 9, 2014, 7:09 a.m.)


Review request for cloudstack and Kelven Yang.


Changes
---

Fixed checkstyle issue


Repository: cloudstack-git


Description
---

NPEs, unused code or dead code, unwritten field access and self assignment


Diffs (updated)
-

  engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java 25c67db 

Diff: https://reviews.apache.org/r/22364/diff/


Testing
---


Thanks,

Rajani Karuturi

Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[sebgoa]

On Jun 8, 2014, at 7:25 PM, Mike Tutkowski  wrote:

> Yes, there appears to be at least two lines of thought on x.y-forward
> branches (specifically using 4.4-forward as an example here).
> 
> 1) 4.4-forward and 4.4 should eventually be the same. Once the 4.4 release
> goes out the door, 4.4-forward should be removed and changes for 4.4.1,
> should such a release ever be made, should go into 4.4.

That's what I tried to say. I you commit to 4.4-forward that means that you 
want it in 4.4.0, otherwise go to master.

> 
> 2) 4.4-forward contains changes that might go into 4.4 (if a cherry pick is
> requested) and changes that would go into 4.4.1, should such a release ever
> be made.
> 
> In both cases: Most all changes that go into 4.4-forward would need to go
> into master (unless that part of the codebase in master has been modified
> in such a way as to no longer make this 4.4-forward change relevant for
> master).
> 
> We should gain some consensus on what x.y-forward means.
> 
> 
> On Sun, Jun 8, 2014 at 10:51 AM, Daan Hoogland 
> wrote:
> 
>> As I read your comments you actually don't agree with Sebastien,
>> Sudha. 4.4-forward was cut of at code freeze. Since then fixes are
>> committed there and cherry-pick are done by the RM. It seems that this
>> is exactly what Sebastien meant it to be for.
>> 
>> @Sebastien: in Shengs words I read that some of the things in
>> 4.4-forward would only go in 4.4 after the release (thing with lower
>> priority then critical)
>> 
>> We all don't agree, let's agree on that.;}
>> 
>> On Sun, Jun 8, 2014 at 2:27 PM, Sudha Ponnaganti
>>  wrote:
>>> Agree with Sebastien. 4.4 is cut too early. By cherry picking and
>> missing some of the critical fixes in to 4.4, isn't it safe to baseline
>> with 4.4-forward which has majority of the fixes. It is not like new
>> features are added to 4.4 forward. Looks like most of them are fixes to
>> existing functionality on 4.4.
>>> 
>>> Thanks
>>> /Sudha
>>> 
>>> -Original Message-
>>> From: sebgoa [mailto:run...@gmail.com]
>>> Sent: Sunday, June 08, 2014 4:52 AM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?
>>> 
>>> 
>>> On Jun 8, 2014, at 5:31 AM, Sheng Yang  wrote:
>>> 
 To my understanding 4.4-forward is a staging tree for 4.4.1 release
 currently, and only issues critical enough would go for 4.4 branch,
>>> 
>>> I disagree. That makes things messy, 4.4-forward should only exist till
>> 4.4 is out, really everything in 4.4-forward should go in 4.4.0.
>>> then folks commit to 4.4 for future bug fix releases...
>>> 
>>> -sebastien
>>> 
>>> 
 and
 author would ask for pick up in that case. I think that's what's
 happened with 4.3-forward branch.
 
 --Sheng
 
 
 On Sat, Jun 7, 2014 at 3:05 PM, Sudha Ponnaganti <
 sudha.ponnaga...@citrix.com> wrote:
 
> Wondering why not baseline with 4.4-forward branch.
> 
> -Original Message-
> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> Sent: Saturday, June 07, 2014 2:53 AM
> To: dev
> Subject: [ACS44] 112 unpicked cherries in 4.4-forward. why?
> 
> H,
> 
> To start with my personal finger pointing: especially test authors
> and UI devs seem to be to blame for the difference (and me of course).
> please take a moment to make sure your important work is not going to
> be lost for future generations.
> 
> I am sure I did it way more complicated then needed but here it is:
> 
> git config pretty.blame "format:blame %H '%aN <%aE>' - '%cn <%cE>' -
>> '%s'"
> git cherry 4.4 4.4-forward | grep -e ^+ | cut -f 2 -d ' ' - | xargs
> -L
> 1 git show --pretty=blame | grep -e ^blame
> 
> this gave me a nice list of commits that are not going into 4.4 as
> things are.
> There is some noise in there that might be filtered out by even more
> clever scripting on the output with ^- in the first grep command:
> 
> 1. things that where picked the other way around for instance
>  fc52e641d8f69d8c0b552119203b0a2bc58e488f 'Daan Hoogland <
> d...@onecht.net>' - 'Daan Hoogland ' -
> 'try-with-resource to prevent resource leaks'
> 2. things that were reverted in the branch (8 occurs so that is a
> minus 16 dropping the number under 100)
> 
> this leaves a lot of things though.
> 
> Please have a look in the list below or execute the commands your
> self and decide what to do about them:
> 
> blame 617826d16b4d5220bb3b51ed511b3c065d0e8926 'Koushik Das <
> kous...@apache.org>' - 'Daan Hoogland ' -
> 'CLOUDSTACK-6445: Simulator enhancements Refer FS -
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Simulator+enha
> ncements
> '
> blame 953167808c3e9a9c40a6f7be6e16ebf32c608a4c 'Jessica Wang <
> jessicaw...@apache.org>' - 'Jessica Wang ' -
> 'BUG-ID: CS-19795: UI > Add LDAP Account - fix a bug that a LDA

Re: Review Request 22354: CLOUDSTACK-6850: Return cpu cores, cpu speed and memory in listUsageRecords

[Sebastien Goasguen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22354/#review45061
---


Hi, is it a bug or a new feature ? I see that it alters the usage db schema, 
will this affects upgrades ?

The jira entry says that it applies to 4.3 as well, does you patch applies to 
4.3 ?

- Sebastien Goasguen


On June 8, 2014, 4:40 p.m., Olivier Lemasle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22354/
> ---
> 
> (Updated June 8, 2014, 4:40 p.m.)
> 
> 
> Review request for cloudstack and Kishan Kavala.
> 
> 
> Bugs: CLOUDSTACK-6850
> https://issues.apache.org/jira/browse/CLOUDSTACK-6850
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> This patch updates VMInstanceUsageParser to process usage details (cpu cores, 
> cpu speed and memory) for dynamic compute offering usages, in order to save 
> them in table cloud_usage.cloud_usage and return them with the API 
> (listUsageRecords).
> 
> See CLOUDSTACK-6850.
> 
> A change in database model is required for this patch (three new columns in 
> cloud_usage.cloud_usage).
> 
> The patch should apply on master and 4.4.
> 
> 
> Diffs
> -
> 
>   api/src/org/apache/cloudstack/api/response/UsageRecordResponse.java 5e2e85d 
>   api/src/org/apache/cloudstack/usage/Usage.java 20dc189 
>   engine/schema/src/com/cloud/usage/UsageVMInstanceVO.java 06d4876 
>   engine/schema/src/com/cloud/usage/UsageVO.java 67014ef 
>   engine/schema/src/com/cloud/usage/dao/UsageVMInstanceDaoImpl.java b94e12f 
>   server/src/com/cloud/api/ApiResponseHelper.java 119f56d 
>   setup/db/db/schema-430to440.sql d149a65 
>   usage/src/com/cloud/usage/parser/VMInstanceUsageParser.java ba0d4bf 
> 
> Diff: https://reviews.apache.org/r/22354/diff/
> 
> 
> Testing
> ---
> 
> Packaged in RPMs, installed and manually tested.
> 
> 1. Create a VM with compute offering "Small instance".
> 2. Stop the VM.
> 3. Change service offering to a custom offering, with cpu_number=1, 
> cpu_speed=500 MHz, memory=512 MB
> 4. Start the VM, wait... Stop the VM.
> 6. Change service offering to an other custom offering, with cpu_number=2, 
> cpu_speed=800 MHz, memory=1024 MB
> 7. Start the VM, wait... Stop the VM.
> 8. Change service offering to the first custom offering, with same "details" 
> (cpu_number=2, cpu_speed=800 MHz, memory=1024 MB)
> 9. Start the VM, wait... Stop the VM.
> 10. Change service offering to "Medium instance".
> 
> Usages and details are reflected in database and in API response. There is no 
> regression found for other usage types.
> Details are returned (and saved in database) only for RUNNING_VM and 
> ALLOCATED_VM, only if the service offering is a custom one.
> 
> Example of response:
> 
> 
> admin
> 184a6564-edab-11e3-b629-0050569ed71c
> fbc7578a-edaa-11e3-b629-0050569ed71c
> e5047089-c911-40ea-8251-4a76810ac1e0
> california allocated (ServiceOffering: 12) (Template: 
> 5)
> 0.098078 Hrs
> 2
> 0.098078
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> california
> b05b64e9-8e0f-4335-b0e0-acc5bad81938
> fbca2564-edaa-11e3-b629-0050569ed71c
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> XenServer
> 1
> 500
> 512
> 2014-06-08'T'17:05:52+02:00
> 2014-06-08'T'17:15:52+02:00
> 
> 
> admin
> 184a6564-edab-11e3-b629-0050569ed71c
> fbc7578a-edaa-11e3-b629-0050569ed71c
> e5047089-c911-40ea-8251-4a76810ac1e0
> california allocated (ServiceOffering: 1) (Template: 
> 5)
> 0.068594 Hrs
> 2
> 0.068594
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> california
> 84bbf38f-b582-440b-8a8d-20e63e0dbed7
> fbca2564-edaa-11e3-b629-0050569ed71c
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> XenServer
> 2014-06-08'T'17:05:52+02:00
> 2014-06-08'T'17:15:52+02:00
> 
> 
> 
> Thanks,
> 
> Olivier Lemasle
> 
>

Re: ssh keypair value overridden by template meta data

[Hiroki Ohashi]

Hi Harikrishna

Thank you very much for your reproduction work.
I created a ticket for this issue.

https://issues.apache.org/jira/browse/CLOUDSTACK-6869

Please confirm it.


Best Regards

2014-06-09 15:59 GMT+09:00 Harikrishna Patnala :
> Hi Hiroki,
>
> We should not override the ssh key pair provided in the deployVM API with the 
> ssh key in template metadata.
> I just able to reproduce this. Please create a ticket for this issue.
>
> Thanks
> Harikrishna
>
> On 06-Jun-2014, at 5:18 pm, Hiroki Ohashi  wrote:
>
>> Dear guys
>>
>> I encountered a problem that a ssh public key of ssh_keypairs for a
>> newly created instance is overridden by another ssh key in template
>> meta data. I think this leads to security vulnerability because a
>> template owner can login to other user's instance created from the
>> template. So, could you fix this issue?
>>
>> This behavior is caused by meta data import at commitUserVm method. A
>> ssh key value specified by an instance owner is set to a UserVmVO
>> object at line 2986-2988 of
>> server/src/com/cloud/vm/UserVmManagerImpl.java in 4.3 branch [1], but
>> this value is overridden at line 3035-3038 by template meta data.
>>
>> Please note a database contains meta data entries related to a
>> template which you created from an instance in cloud.template_view
>> like [2]. 2nd row has detail_name and detail_value about ssh key and
>> CloudStack override the user specified ssh key value by the owner's
>> detail value in this situation. It results in delivery of the template
>> owner's ssh key for the instance created from the template to virtual
>> router inspite of specification of instance owner's ssh key.
>>
>> You can reproduce this phenomenon like below.
>>
>>1. Deploy an instance with a ssh key A by specifying 'keypair'
>>   value.
>>2. Create a template from this instance.
>>3. Deploy an instance with another ssh key B by specifying
>>   'keypair' value.
>>
>>
>> [1] server/src/com/cloud/vm/UserVmManagerImpl.java
>>
>>   2971 private UserVmVO commitUserVm(final DataCenter zone, final
>> VirtualMachineTemplate template, final String hostName, final String
>> displayName, final Account owner,
>>   2972 final Long diskOfferingId, final Long diskSize, final
>> String userData, final HypervisorType hypervisor, final Account
>> caller, final Boolean isDisplayVmEnabled,
>>   2973 final String keyboard, final long accountId, final
>> ServiceOfferingVO offering, final boolean isIso, final String
>> sshPublicKey,
>>   2974 final LinkedHashMap networkNicMap,
>> final long id, final String instanceName, final String uuidName, final
>> HypervisorType hypervisorType,
>>   2975 final Map customParameters) throws
>> InsufficientCapacityException {
>>   2976 return Transaction.execute(new
>> TransactionCallbackWithException> InsufficientCapacityException>() {
>>   2977 @Override
>>   2978 public UserVmVO doInTransaction(TransactionStatus
>> status) throws InsufficientCapacityException {
>>   2979 UserVmVO vm = new UserVmVO(id, instanceName,
>> displayName,
>>   2980 template.getId(), hypervisorType,
>> template.getGuestOSId(),
>>   2981 offering.getOfferHA(),
>> offering.getLimitCpuUse(),
>>   2982 owner.getDomainId(), owner.getId(),
>> offering.getId(), userData,
>>   2983 hostName, diskOfferingId);
>>   2984 vm.setUuid(uuidName);
>>   2985
>> vm.setDynamicallyScalable(template.isDynamicallyScalable());
>>   2986 if (sshPublicKey != null) {
>>   2987 vm.setDetail("SSH.PublicKey", sshPublicKey);
>>   2988 }
>>   2989
>>   2990 if (keyboard != null && !keyboard.isEmpty())
>>   2991 vm.setDetail(VmDetailConstants.KEYBOARD, 
>> keyboard);
>>   2992
>>   2993 if (isIso) {
>>   2994 vm.setIsoId(template.getId());
>>   2995 }
>>   2996
>>   2997 if(isDisplayVmEnabled != null){
>>   2998 if(!_accountMgr.isRootAdmin(caller.getType())){
>>   2999 throw new PermissionDeniedException(
>> "Cannot update parameter displayvm, only admin permitted ");
>>   3000 }
>>   3001 vm.setDisplayVm(isDisplayVmEnabled);
>>   3002 }else {
>>   3003 vm.setDisplayVm(true);
>>   3004 }
>>   3005
>>   3006 // If hypervisor is vSphere, check for clone
>> type setting.
>>   3007 if (hypervisorType.equals(HypervisorType.VMware)) {
>>   3008 // retrieve clone flag.
>>   3009 UserVmCloneType cloneType = 
>> UserVmCloneType.linked;
>>   3010 String value =
>> _configDao.getValue(Config.VmwareCreateFullClone.key());
>>   3011 

Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[Koushik Das]

On 09-Jun-2014, at 12:56 PM, sebgoa  wrote:

> 
> On Jun 8, 2014, at 7:25 PM, Mike Tutkowski  
> wrote:
> 
>> Yes, there appears to be at least two lines of thought on x.y-forward
>> branches (specifically using 4.4-forward as an example here).
>> 
>> 1) 4.4-forward and 4.4 should eventually be the same. Once the 4.4 release
>> goes out the door, 4.4-forward should be removed and changes for 4.4.1,
>> should such a release ever be made, should go into 4.4.
> 
> That's what I tried to say. I you commit to 4.4-forward that means that you 
> want it in 4.4.0, otherwise go to master.
> 

[Koushik] What if the fix is not that critical for 4.4 but would be good to 
have in 4.4.1 and above apart from master. I would be concerned if the fix is 
in 4.4-forward but not in master.


>> 
>> 2) 4.4-forward contains changes that might go into 4.4 (if a cherry pick is
>> requested) and changes that would go into 4.4.1, should such a release ever
>> be made.
>> 
>> In both cases: Most all changes that go into 4.4-forward would need to go
>> into master (unless that part of the codebase in master has been modified
>> in such a way as to no longer make this 4.4-forward change relevant for
>> master).
>> 
>> We should gain some consensus on what x.y-forward means.
>> 
>> 
>> On Sun, Jun 8, 2014 at 10:51 AM, Daan Hoogland 
>> wrote:
>> 
>>> As I read your comments you actually don't agree with Sebastien,
>>> Sudha. 4.4-forward was cut of at code freeze. Since then fixes are
>>> committed there and cherry-pick are done by the RM. It seems that this
>>> is exactly what Sebastien meant it to be for.
>>> 
>>> @Sebastien: in Shengs words I read that some of the things in
>>> 4.4-forward would only go in 4.4 after the release (thing with lower
>>> priority then critical)
>>> 
>>> We all don't agree, let's agree on that.;}
>>> 
>>> On Sun, Jun 8, 2014 at 2:27 PM, Sudha Ponnaganti
>>>  wrote:
 Agree with Sebastien. 4.4 is cut too early. By cherry picking and
>>> missing some of the critical fixes in to 4.4, isn't it safe to baseline
>>> with 4.4-forward which has majority of the fixes. It is not like new
>>> features are added to 4.4 forward. Looks like most of them are fixes to
>>> existing functionality on 4.4.
 
 Thanks
 /Sudha
 
 -Original Message-
 From: sebgoa [mailto:run...@gmail.com]
 Sent: Sunday, June 08, 2014 4:52 AM
 To: dev@cloudstack.apache.org
 Subject: Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?
 
 
 On Jun 8, 2014, at 5:31 AM, Sheng Yang  wrote:
 
> To my understanding 4.4-forward is a staging tree for 4.4.1 release
> currently, and only issues critical enough would go for 4.4 branch,
 
 I disagree. That makes things messy, 4.4-forward should only exist till
>>> 4.4 is out, really everything in 4.4-forward should go in 4.4.0.
 then folks commit to 4.4 for future bug fix releases...
 
 -sebastien
 
 
> and
> author would ask for pick up in that case. I think that's what's
> happened with 4.3-forward branch.
> 
> --Sheng
> 
> 
> On Sat, Jun 7, 2014 at 3:05 PM, Sudha Ponnaganti <
> sudha.ponnaga...@citrix.com> wrote:
> 
>> Wondering why not baseline with 4.4-forward branch.
>> 
>> -Original Message-
>> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
>> Sent: Saturday, June 07, 2014 2:53 AM
>> To: dev
>> Subject: [ACS44] 112 unpicked cherries in 4.4-forward. why?
>> 
>> H,
>> 
>> To start with my personal finger pointing: especially test authors
>> and UI devs seem to be to blame for the difference (and me of course).
>> please take a moment to make sure your important work is not going to
>> be lost for future generations.
>> 
>> I am sure I did it way more complicated then needed but here it is:
>> 
>> git config pretty.blame "format:blame %H '%aN <%aE>' - '%cn <%cE>' -
>>> '%s'"
>> git cherry 4.4 4.4-forward | grep -e ^+ | cut -f 2 -d ' ' - | xargs
>> -L
>> 1 git show --pretty=blame | grep -e ^blame
>> 
>> this gave me a nice list of commits that are not going into 4.4 as
>> things are.
>> There is some noise in there that might be filtered out by even more
>> clever scripting on the output with ^- in the first grep command:
>> 
>> 1. things that where picked the other way around for instance
>> fc52e641d8f69d8c0b552119203b0a2bc58e488f 'Daan Hoogland <
>> d...@onecht.net>' - 'Daan Hoogland ' -
>> 'try-with-resource to prevent resource leaks'
>> 2. things that were reverted in the branch (8 occurs so that is a
>> minus 16 dropping the number under 100)
>> 
>> this leaves a lot of things though.
>> 
>> Please have a look in the list below or execute the commands your
>> self and decide what to do about them:
>> 
>> blame 617826d16b4d5220bb3b51ed511b3c065d0e8926 'Koushik Das <
>> kous...@apache.org

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Hieu LE]

Hi guys,

I have updated this proposal wiki[1], included diagram for VM migrate,
volume migrate and snapshot.

Please review and give feedback.

[1]:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage


On Fri, Jun 6, 2014 at 7:14 PM, Todd Pigram  wrote:

> Sorry, thought you were based off the link you provided in this reply.
>
> "In our case, we are using CloudStack integrated in VDI solution to
> provived
> pooled VM type[1]. So may be my approach can bring better UX for user with
> lower bootime ...
>
> A short change in design are followings
> - VM will be deployed with golden primary storage if primary storage is
> marked golden and this VM template is also marked as golden.
> - Choosing the best deploy destionation for both golden primary storage and
> normal root volume primary storage. Chosen host can also access both
> storage pools.
> - New Xen Server plug-in for modifying VHD parent id.
>
> Is there some place for me to submit my design and code. Can I write a new
> proposal in CS wiki ?
>
> [1]:
>
> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html
>  "
>
>
> On Thu, Jun 5, 2014 at 11:55 PM, Hieu LE  wrote:
>
> > Hi Todd,
> >
> >
> > On Fri, Jun 6, 2014 at 9:17 AM, Todd Pigram  wrote:
> >
> > > Hieu,
> > >
> > > I assume you are using MCS for you golden image? What version of XD?
> > Given
> > > you are using pooled desktops, have you thought about using a PVS BDM
> iso
> > > and mount it with in your 1000 VMs? This way you can stagger reboots
> via
> > > PVS console or Studio. This would require a change to your delivery
> > group.
> > >
> > >
> > Sorry but I did not use MCS or XenDesktop in my company :-)
> >
> >
> > >
> > > On Thu, Jun 5, 2014 at 9:28 PM, Mike Tutkowski <
> > > mike.tutkow...@solidfire.com
> > > > wrote:
> > >
> > > > 6) The copy_vhd_from_secondarystorage XenServer plug-in is not used
> > when
> > > > you're using XenServer + XS62ESP1 + XS62ESP1004. In that case, please
> > > refer
> > > > to copyTemplateToPrimaryStorage(CopyCommand) method in the
> > > > Xenserver625StorageProcessor class.
> > > >
> > >
> >
> > Thank Mike, I will take note of that.
> >
> >
> > >  >
> > > > On Thu, Jun 5, 2014 at 1:56 PM, Mike Tutkowski <
> > > > mike.tutkow...@solidfire.com
> > > > > wrote:
> > > >
> > > > > Other than going through a "for" loop and deploying VM after VM, I
> > > don't
> > > > > think CloudStack currently supports a bulk-VM-deploy operation.
> > > > >
> > > > > It would be nice if CS did so at some point in the future; however,
> > > that
> > > > > is probably a separate proposal from Hieu's.
> > > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 12:13 AM, Amit Das 
> > > > wrote:
> > > > >
> > > > >> Hi Hieu,
> > > > >>
> > > > >> Will it be good to include bulk operation of this feature? In
> > > addition,
> > > > >> does Xen support parallel execution of these operations ?
> > > > >>
> > > > >> Regards,
> > > > >> Amit
> > > > >> *CloudByte Inc.* 
> > > > >>
> > > > >>
> > > > >> On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE 
> wrote:
> > > > >>
> > > > >> > Mike, Punith,
> > > > >> >
> > > > >> > Please review "Golden Primary Storage" proposal. [1]
> > > > >> >
> > > > >> > Thank you.
> > > > >> >
> > > > >> > [1]:
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Jun 4, 2014 at 10:32 PM, Mike Tutkowski <
> > > > >> > mike.tutkow...@solidfire.com> wrote:
> > > > >> >
> > > > >> >> Daan helped out with this. You should be good to go now.
> > > > >> >>
> > > > >> >>
> > > > >> >> On Tue, Jun 3, 2014 at 8:50 PM, Hieu LE 
> > > wrote:
> > > > >> >>
> > > > >> >> > Hi Mike,
> > > > >> >> >
> > > > >> >> > Could you please give edit/create permission on ASF Jira/Wiki
> > > > >> >> confluence ?
> > > > >> >> > I can not add a new Wiki page.
> > > > >> >> >
> > > > >> >> > My Jira ID: hieulq
> > > > >> >> > Wiki: hieulq89
> > > > >> >> > Review Board: hieulq
> > > > >> >> >
> > > > >> >> > Thanks !
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Wed, Jun 4, 2014 at 9:17 AM, Mike Tutkowski <
> > > > >> >> > mike.tutkow...@solidfire.com
> > > > >> >> > > wrote:
> > > > >> >> >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > > Yes, please feel free to add a new Wiki page for your
> design.
> > > > >> >> > >
> > > > >> >> > > Here is a link to applicable design info:
> > > > >> >> > >
> > > > >> >> > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design
> > > > >> >> > >
> > > > >> >> > > Also, feel free to ask more questions and have me review
> your
> > > > >> design.
> > > > >> >> > >
> > > > >> >> > > Thanks!
> > > > >> >> > > Mike
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > On Tue, Jun 3, 2014 at 7:29 PM, Hieu LE <
> hieul...@gmail.com>
> > > > >> wrote:
> > > > >> >> > >
> > > > >> >> > > > Hi Mike,
> > > > >> >> > > >
> > > > >> >> > > > Yo

[ACS4.4] cherrypick c282bb3a1293fbbfdb306263ea52464862670fb3

[Rajesh Battala]

Hi Daan,

Cherry pick

c282bb3a1293fbbfdb306263ea52464862670fb3 to resolve the blocker

CLOUDSTACK-6603 [Upgrade]DB Exception while Autoscale monitoring after 
upgrading from 4.3 to 4.4





Thanks

Rajesh Battala

Re: KVM + LXC on the same host

[Nux!]

On 07.06.2014 03:38, John Kinsella wrote:

Hey Ilya -

So, for about a month now we’ve had a system running SmartOS[1],
which gives a combination of containers and KVM, albeit on illumos
instead of linux. In general I’m not impressed by SmartOS’s story, but
we had one customer asking for it (and I sorta expect more, we’re not
officially supporting it yet). It’s a little early for me to even
ponder integrating it with ACS, but might but worth a glance if you’re
looking for that KVM/container mix.

Downsides:
 * Compared to a Linux hypervisor, much less feature support (missing
Ceph is personally bugging me)
 * It’s…Solaris (the OS that won’t go away, c0t0d0s0 gives me
flashbacks from 10 years ago). For us our infrastructure/automation is
fairly tuned for linux, so really embracing this will require…effort.


Check out the Proxmox project which provides both containers and VMs, 
in the form of OpenVZ (not ideal, I know) and KVM.


Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

Re: ssh keypair value overridden by template meta data

[Harikrishna Patnala]

Thanks Hiroki, I have added the fix versions, ’ll fix it asap.

-Harikrishna

On 09-Jun-2014, at 2:35 pm, Hiroki Ohashi  wrote:

> Hi Harikrishna
> 
> Thank you very much for your reproduction work.
> I created a ticket for this issue.
> 
>https://issues.apache.org/jira/browse/CLOUDSTACK-6869
> 
> Please confirm it.
> 
> 
> Best Regards
> 
> 2014-06-09 15:59 GMT+09:00 Harikrishna Patnala 
> :
>> Hi Hiroki,
>> 
>> We should not override the ssh key pair provided in the deployVM API with 
>> the ssh key in template metadata.
>> I just able to reproduce this. Please create a ticket for this issue.
>> 
>> Thanks
>> Harikrishna
>> 
>> On 06-Jun-2014, at 5:18 pm, Hiroki Ohashi  wrote:
>> 
>>> Dear guys
>>> 
>>> I encountered a problem that a ssh public key of ssh_keypairs for a
>>> newly created instance is overridden by another ssh key in template
>>> meta data. I think this leads to security vulnerability because a
>>> template owner can login to other user's instance created from the
>>> template. So, could you fix this issue?
>>> 
>>> This behavior is caused by meta data import at commitUserVm method. A
>>> ssh key value specified by an instance owner is set to a UserVmVO
>>> object at line 2986-2988 of
>>> server/src/com/cloud/vm/UserVmManagerImpl.java in 4.3 branch [1], but
>>> this value is overridden at line 3035-3038 by template meta data.
>>> 
>>> Please note a database contains meta data entries related to a
>>> template which you created from an instance in cloud.template_view
>>> like [2]. 2nd row has detail_name and detail_value about ssh key and
>>> CloudStack override the user specified ssh key value by the owner's
>>> detail value in this situation. It results in delivery of the template
>>> owner's ssh key for the instance created from the template to virtual
>>> router inspite of specification of instance owner's ssh key.
>>> 
>>> You can reproduce this phenomenon like below.
>>> 
>>>   1. Deploy an instance with a ssh key A by specifying 'keypair'
>>>  value.
>>>   2. Create a template from this instance.
>>>   3. Deploy an instance with another ssh key B by specifying
>>>  'keypair' value.
>>> 
>>> 
>>> [1] server/src/com/cloud/vm/UserVmManagerImpl.java
>>> 
>>>  2971 private UserVmVO commitUserVm(final DataCenter zone, final
>>> VirtualMachineTemplate template, final String hostName, final String
>>> displayName, final Account owner,
>>>  2972 final Long diskOfferingId, final Long diskSize, final
>>> String userData, final HypervisorType hypervisor, final Account
>>> caller, final Boolean isDisplayVmEnabled,
>>>  2973 final String keyboard, final long accountId, final
>>> ServiceOfferingVO offering, final boolean isIso, final String
>>> sshPublicKey,
>>>  2974 final LinkedHashMap networkNicMap,
>>> final long id, final String instanceName, final String uuidName, final
>>> HypervisorType hypervisorType,
>>>  2975 final Map customParameters) throws
>>> InsufficientCapacityException {
>>>  2976 return Transaction.execute(new
>>> TransactionCallbackWithException>> InsufficientCapacityException>() {
>>>  2977 @Override
>>>  2978 public UserVmVO doInTransaction(TransactionStatus
>>> status) throws InsufficientCapacityException {
>>>  2979 UserVmVO vm = new UserVmVO(id, instanceName,
>>> displayName,
>>>  2980 template.getId(), hypervisorType,
>>> template.getGuestOSId(),
>>>  2981 offering.getOfferHA(),
>>> offering.getLimitCpuUse(),
>>>  2982 owner.getDomainId(), owner.getId(),
>>> offering.getId(), userData,
>>>  2983 hostName, diskOfferingId);
>>>  2984 vm.setUuid(uuidName);
>>>  2985
>>> vm.setDynamicallyScalable(template.isDynamicallyScalable());
>>>  2986 if (sshPublicKey != null) {
>>>  2987 vm.setDetail("SSH.PublicKey", sshPublicKey);
>>>  2988 }
>>>  2989
>>>  2990 if (keyboard != null && !keyboard.isEmpty())
>>>  2991 vm.setDetail(VmDetailConstants.KEYBOARD, 
>>> keyboard);
>>>  2992
>>>  2993 if (isIso) {
>>>  2994 vm.setIsoId(template.getId());
>>>  2995 }
>>>  2996
>>>  2997 if(isDisplayVmEnabled != null){
>>>  2998 if(!_accountMgr.isRootAdmin(caller.getType())){
>>>  2999 throw new PermissionDeniedException(
>>> "Cannot update parameter displayvm, only admin permitted ");
>>>  3000 }
>>>  3001 vm.setDisplayVm(isDisplayVmEnabled);
>>>  3002 }else {
>>>  3003 vm.setDisplayVm(true);
>>>  3004 }
>>>  3005
>>>  3006 // If hypervisor is vSphere, check for clone
>>> type setting.
>>>  3007 if (hypervisorType.equals(HypervisorType.VMware)) {
>>>  3008 

Issue regarding cloudstack configuration.

[Naval Saini]

Hello There.

Kindly tell me where i can ask from apache support/community the issues
which i am facing during configuration part.

thanks.

-- 
-- 
Warm Regards,
Naval Saini
Technical Support Executive L-2
-
SARV Webs Pvt Ltd.
Global Number : +44 141 4166 
USA/Canada TollFree: 1-888 987 2698
India TollFree: 1-800 102 7278
visit us: sarvmail.com ,tld6.com
--

Re: Issue regarding cloudstack configuration.

[sebgoa]

On Jun 9, 2014, at 8:27 AM, Naval Saini  wrote:

> Hello There.
> 
> Kindly tell me where i can ask from apache support/community the issues
> which i am facing during configuration part.

Hi, better to start on the users lists, us...@cloudstack.apache.org

or on IRC #cloudstack on freenode.

> 
> thanks.
> 
> -- 
> -- 
> Warm Regards,
> Naval Saini
> Technical Support Executive L-2
> -
> SARV Webs Pvt Ltd.
> Global Number : +44 141 4166 
> USA/Canada TollFree: 1-888 987 2698
> India TollFree: 1-800 102 7278
> visit us: sarvmail.com ,tld6.com
> --

Review Request 22376: tagging the test case test_09_delete_detached_volume with the bugId.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22376/
---

Review request for cloudstack and Abhinandan Prateek.


Bugs: CLOUDSTACK-6875
https://issues.apache.org/jira/browse/CLOUDSTACK-6875


Repository: cloudstack-git


Description
---

The test case test_09_delete_detached_volume is failing. we have logged a bug 
CLOUDSTACK-6875 to track this.
this patch tags the test case with the above mentioned bugId.


Diffs
-

  test/integration/smoke/test_volumes.py 73c2722 

Diff: https://reviews.apache.org/r/22376/diff/


Testing
---


Thanks,

bharat kumar

Review Request 22377: tagging the test case test_01_sys_vm_start with the bugId.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22377/
---

Review request for cloudstack and Abhinandan Prateek.


Bugs: CLOUDSTACK-6877
https://issues.apache.org/jira/browse/CLOUDSTACK-6877


Repository: cloudstack-git


Description
---

The test case test_01_sys_vm_start is failing. we have logged a bug 
CLOUDSTACK-6877 to track this.
this patch tags the test case with the above mentioned bugId.


Diffs
-

  test/integration/smoke/test_secondary_storage.py 481d907 

Diff: https://reviews.apache.org/r/22377/diff/


Testing
---


Thanks,

bharat kumar

Review Request 22378: tagging the test case test_01_snapshot_root_disk with the bugId.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22378/
---

Review request for cloudstack and Abhinandan Prateek.


Bugs: CLOUDSTACK-6878
https://issues.apache.org/jira/browse/CLOUDSTACK-6878


Repository: cloudstack-git


Description
---

The test case test_01_snapshot_root_disk is failing. we have logged a bug 
CLOUDSTACK-6876 to track this.
this patch tags the test case with the above mentioned bugId.


Diffs
-

  test/integration/smoke/test_snapshots.py a960e70 

Diff: https://reviews.apache.org/r/22378/diff/


Testing
---


Thanks,

bharat kumar

Review Request 22379: tagging the test case test_vpc_site2site_vpn with the bugId.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22379/
---

Review request for cloudstack and Abhinandan Prateek.


Bugs: CLOUDSTACK-6879
https://issues.apache.org/jira/browse/CLOUDSTACK-6879


Repository: cloudstack-git


Description
---

The test case test_vpc_site2site_vpn is failing. we have logged a bug 
CLOUDSTACK-6876 to track this.
this patch tags the test case with the above mentioned bugId.


Diffs
-

  test/integration/smoke/test_vpc_vpn.py 03826e9 

Diff: https://reviews.apache.org/r/22379/diff/


Testing
---


Thanks,

bharat kumar

Review Request 22375: tagging the test case test_deploy_vgpu_enabled_vm with the bugId.

[bharat kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22375/
---

Review request for cloudstack and Abhinandan Prateek.


Bugs: CLOUDSTACK-6876
https://issues.apache.org/jira/browse/CLOUDSTACK-6876


Repository: cloudstack-git


Description
---

The test case test_deploy_vgpu_enabled_vm is failing. we have logged a bug 
CLOUDSTACK-6876 to track this.
this patch tags the test case with the above mentioned bugId.


Diffs
-

  test/integration/smoke/test_deploy_vgpu_enabled_vm.py 13fad61 

Diff: https://reviews.apache.org/r/22375/diff/


Testing
---


Thanks,

bharat kumar

Re: Review Request 22354: CLOUDSTACK-6850: Return cpu cores, cpu speed and memory in listUsageRecords

[Olivier Lemasle]

> On June 9, 2014, 9:55 a.m., Sebastien Goasguen wrote:
> > Hi, is it a bug or a new feature ? I see that it alters the usage db 
> > schema, will this affects upgrades ?
> > 
> > The jira entry says that it applies to 4.3 as well, does you patch applies 
> > to 4.3 ?

Hi Sebastien,

For me, it's a bug, because a CloudStack feature, usage tracking, is not 
working properly with dynamic compute offerings.
Of course, I might be biased because I work on Amysta development and this bug 
prevents Amysta from tracking consumption with dynamic compute offerings. But 
this is a more general issue: there is currently no way to track VM instance 
usages with dynamic compute offerings using API only.

No, the patch is for 4.4 (because I wrote the DB schema changes in 
schema-430to440.sql It would be great to fix that for ACS 4.3 too, but I'm 
annoyed with the schema update).

Regarding the update process, I tested the patch both with a ACS 4.4 fresh 
install and with a ACS 4.3 update. There was no regression. Usage details are 
returned only for usages processed after the update.


- Olivier


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22354/#review45061
---


On June 8, 2014, 6:40 p.m., Olivier Lemasle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22354/
> ---
> 
> (Updated June 8, 2014, 6:40 p.m.)
> 
> 
> Review request for cloudstack and Kishan Kavala.
> 
> 
> Bugs: CLOUDSTACK-6850
> https://issues.apache.org/jira/browse/CLOUDSTACK-6850
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> This patch updates VMInstanceUsageParser to process usage details (cpu cores, 
> cpu speed and memory) for dynamic compute offering usages, in order to save 
> them in table cloud_usage.cloud_usage and return them with the API 
> (listUsageRecords).
> 
> See CLOUDSTACK-6850.
> 
> A change in database model is required for this patch (three new columns in 
> cloud_usage.cloud_usage).
> 
> The patch should apply on master and 4.4.
> 
> 
> Diffs
> -
> 
>   api/src/org/apache/cloudstack/api/response/UsageRecordResponse.java 5e2e85d 
>   api/src/org/apache/cloudstack/usage/Usage.java 20dc189 
>   engine/schema/src/com/cloud/usage/UsageVMInstanceVO.java 06d4876 
>   engine/schema/src/com/cloud/usage/UsageVO.java 67014ef 
>   engine/schema/src/com/cloud/usage/dao/UsageVMInstanceDaoImpl.java b94e12f 
>   server/src/com/cloud/api/ApiResponseHelper.java 119f56d 
>   setup/db/db/schema-430to440.sql d149a65 
>   usage/src/com/cloud/usage/parser/VMInstanceUsageParser.java ba0d4bf 
> 
> Diff: https://reviews.apache.org/r/22354/diff/
> 
> 
> Testing
> ---
> 
> Packaged in RPMs, installed and manually tested.
> 
> 1. Create a VM with compute offering "Small instance".
> 2. Stop the VM.
> 3. Change service offering to a custom offering, with cpu_number=1, 
> cpu_speed=500 MHz, memory=512 MB
> 4. Start the VM, wait... Stop the VM.
> 6. Change service offering to an other custom offering, with cpu_number=2, 
> cpu_speed=800 MHz, memory=1024 MB
> 7. Start the VM, wait... Stop the VM.
> 8. Change service offering to the first custom offering, with same "details" 
> (cpu_number=2, cpu_speed=800 MHz, memory=1024 MB)
> 9. Start the VM, wait... Stop the VM.
> 10. Change service offering to "Medium instance".
> 
> Usages and details are reflected in database and in API response. There is no 
> regression found for other usage types.
> Details are returned (and saved in database) only for RUNNING_VM and 
> ALLOCATED_VM, only if the service offering is a custom one.
> 
> Example of response:
> 
> 
> admin
> 184a6564-edab-11e3-b629-0050569ed71c
> fbc7578a-edaa-11e3-b629-0050569ed71c
> e5047089-c911-40ea-8251-4a76810ac1e0
> california allocated (ServiceOffering: 12) (Template: 
> 5)
> 0.098078 Hrs
> 2
> 0.098078
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> california
> b05b64e9-8e0f-4335-b0e0-acc5bad81938
> fbca2564-edaa-11e3-b629-0050569ed71c
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> XenServer
> 1
> 500
> 512
> 2014-06-08'T'17:05:52+02:00
> 2014-06-08'T'17:15:52+02:00
> 
> 
> admin
> 184a6564-edab-11e3-b629-0050569ed71c
> fbc7578a-edaa-11e3-b629-0050569ed71c
> e5047089-c911-40ea-8251-4a76810ac1e0
> california allocated (ServiceOffering: 1) (Template: 
> 5)
> 0.068594 Hrs
> 2
> 0.068594
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> california
> 84bbf38f-b582-440b-8a8d-20e63e0dbed7
> fbca2564-edaa-11e3-b629-0050569ed71c
> e89c22af-65cf-42a2-8905-6f4bbd7d59a1
> XenServer
> 2014-06-08'T'17:05:52+02:00
> 2014-06-08'T'17:15:52+02:00
> 
> 
> 
> Thanks,
> 
> Olivier Lemasle
> 

feature : changing volume properties dynamically in 4.5

[Punith S]

hi guys,

since most of the third party storage providers have been implementing 1:1
mapping(managed storage) between a volume(dataset) and a vm disk(vdi/vmdk)
for guaranteeing the Qos, i would like to propose a new feature to
dynamically change the volume properties supported by storage vendors such
as IOPS, Deduplication, Compression, Grace, Syncronization, Latency
etc, depending on properties and features supported by respective storage
vendors. hence providing more flexibility for users.

in case of using default cloudstack storage provider, we can change the
properties of the vdi/vmdk files apart from resizing the volume(vdi/vmdk).

changes in management server include,

new async web api ChangeVolumePropertiesCmd,
new method in VolumeApiService for vo and dao validation implementations.
new method in VolumeServiceManager for supporting callback and calling the
respective storage provider driver's implementation.
new method in PrimaryDataStoreDriver interface for implementing respective
features according to their storage product.

changes in UI include,
new changing volume properties widget in volume section, showing different
properties depending upon listed storage providers.

any suggestions and feedbacks ?

thanks

-- 
regards,

punith s
cloudbyte.com

Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[Mike Tutkowski]

Right - we need to decide as a community what a forward branch really means.

On Monday, June 9, 2014, Koushik Das  wrote:

>
> On 09-Jun-2014, at 12:56 PM, sebgoa >
> wrote:
>
> >
> > On Jun 8, 2014, at 7:25 PM, Mike Tutkowski  > wrote:
> >
> >> Yes, there appears to be at least two lines of thought on x.y-forward
> >> branches (specifically using 4.4-forward as an example here).
> >>
> >> 1) 4.4-forward and 4.4 should eventually be the same. Once the 4.4
> release
> >> goes out the door, 4.4-forward should be removed and changes for 4.4.1,
> >> should such a release ever be made, should go into 4.4.
> >
> > That's what I tried to say. I you commit to 4.4-forward that means that
> you want it in 4.4.0, otherwise go to master.
> >
>
> [Koushik] What if the fix is not that critical for 4.4 but would be good
> to have in 4.4.1 and above apart from master. I would be concerned if the
> fix is in 4.4-forward but not in master.
>
>
> >>
> >> 2) 4.4-forward contains changes that might go into 4.4 (if a cherry
> pick is
> >> requested) and changes that would go into 4.4.1, should such a release
> ever
> >> be made.
> >>
> >> In both cases: Most all changes that go into 4.4-forward would need to
> go
> >> into master (unless that part of the codebase in master has been
> modified
> >> in such a way as to no longer make this 4.4-forward change relevant for
> >> master).
> >>
> >> We should gain some consensus on what x.y-forward means.
> >>
> >>
> >> On Sun, Jun 8, 2014 at 10:51 AM, Daan Hoogland  >
> >> wrote:
> >>
> >>> As I read your comments you actually don't agree with Sebastien,
> >>> Sudha. 4.4-forward was cut of at code freeze. Since then fixes are
> >>> committed there and cherry-pick are done by the RM. It seems that this
> >>> is exactly what Sebastien meant it to be for.
> >>>
> >>> @Sebastien: in Shengs words I read that some of the things in
> >>> 4.4-forward would only go in 4.4 after the release (thing with lower
> >>> priority then critical)
> >>>
> >>> We all don't agree, let's agree on that.;}
> >>>
> >>> On Sun, Jun 8, 2014 at 2:27 PM, Sudha Ponnaganti
> >>>  wrote:
>  Agree with Sebastien. 4.4 is cut too early. By cherry picking and
> >>> missing some of the critical fixes in to 4.4, isn't it safe to baseline
> >>> with 4.4-forward which has majority of the fixes. It is not like new
> >>> features are added to 4.4 forward. Looks like most of them are fixes to
> >>> existing functionality on 4.4.
> 
>  Thanks
>  /Sudha
> 
>  -Original Message-
>  From: sebgoa [mailto:run...@gmail.com]
>  Sent: Sunday, June 08, 2014 4:52 AM
>  To: dev@cloudstack.apache.org
>  Subject: Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?
> 
> 
>  On Jun 8, 2014, at 5:31 AM, Sheng Yang  wrote:
> 
> > To my understanding 4.4-forward is a staging tree for 4.4.1 release
> > currently, and only issues critical enough would go for 4.4 branch,
> 
>  I disagree. That makes things messy, 4.4-forward should only exist
> till
> >>> 4.4 is out, really everything in 4.4-forward should go in 4.4.0.
>  then folks commit to 4.4 for future bug fix releases...
> 
>  -sebastien
> 
> 
> > and
> > author would ask for pick up in that case. I think that's what's
> > happened with 4.3-forward branch.
> >
> > --Sheng
> >
> >
> > On Sat, Jun 7, 2014 at 3:05 PM, Sudha Ponnaganti <
> > sudha.ponnaga...@citrix.com> wrote:
> >
> >> Wondering why not baseline with 4.4-forward branch.
> >>
> >> -Original Message-
> >> From: Daan Hoogland [mailto:daan.hoogl...@gmail.com]
> >> Sent: Saturday, June 07, 2014 2:53 AM
> >> To: dev
> >> Subject: [ACS44] 112 unpicked cherries in 4.4-forward. why?
> >>
> >> H,
> >>
> >> To start with my personal finger pointing: especially test authors
> >> and UI devs seem to be to blame for the difference (and me of
> course).
> >> please take a moment to make sure your important work is not going
> to
> >> be lost for future generations.
> >><



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: Question about guaranteed IOPS feature

[Yoshikazu Nojima]

Mike, Marcus,

Sorry for my belated reply.

Thank you for clarifying the status of current implementation, and
concerns we need to care.
Yes, using hypervisor's rate limiting feature to guarantee IOPS seems
challenging.

Thanks,
Noji


2014-06-02 14:52 GMT-04:00 Marcus :
> I think the current implementation, outside of custom vendors is to simply
> act as a tune-able rate-limiter for the admin to leverage how they see fit.
> I'm not sure it was intended to be a guarantee of service, just a cap,
> therefore the thought of oversubscribing wasn't an issue. It is an exercise
> for the admin, given the tool. We could potentially augment it by adding in
> the idea of 'iops per host' and/or 'iops per primary storage' parameters on
> each host/primary storage, whereby the allocator code can leverage this to
> deploy servers.
>
>
> On Mon, Jun 2, 2014 at 12:18 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com> wrote:
>
>> Hi,
>>
>> 1 and 2) CloudStack now supports managed storage for the following
>> hypervisor types:
>>
>> * Disk Offerings for XenServer and ESX as of CS 4.2
>> * Disk Offerings for KVM as of CS 4.3
>> * Compute Offerings for XenServer and ESX as of CS 4.4
>>
>> The SolidFire plug-in is capable of supporting whatever CloudStack-managed
>> configurations are available in the version of CloudStack that you are
>> using.
>>
>> Rate limiting is supported for the KVM hypervisor only as of CS 4.2.
>>
>> Min knows the details of this KVM rate-limiting feature better than I do
>> (he implemented it), so I have CCed him.
>>
>> I think this rate limiting is only from the hypervisor's point of
>> view...without input into the storage system that actually provides the
>> volume in question. In other words, the hypervisor doesn't actually know
>> how many IOPS the primary storage is capable of delivering...it only knows
>> to limit the IO it sends to the storage system for that volume.
>>
>> Rate limiting is not a guarantee on IOPS, though. It is simply a limit on
>> the rate at which IO can be sent to the storage system.
>>
>> By the way, for the sake of simplicity, when I'm talking in terms of IOPS
>> here, I'm assuming all of your IOPS are the same size...which is probably
>> not the case in the real world, so you'd need to take that into account, as
>> well.
>>
>> In theory, if you have primary storage that has, say, 10,000 IOPS available
>> and you carve out 10 1,000 IOPS (rate limited) volumes on that primary
>> storage, you may be OK.
>>
>> However, this depends. For many storage solutions, you still have to worry
>> about drive groups, LUN assignments, failure scenarios, etc. when it comes
>> to whether or not you'll actually get 1,000 IOPS for each of those 10
>> volumes on the storage side. In these environments, you also need to take
>> into consideration write amplification from data protection (ex. RAID). If
>> the combination of IOPS from all the SSDs on your primary storage is X, you
>> will get less than X from the point of view of the clients of that storage
>> system.
>>
>> All hypervisor rate limiting is doing is saying that at most you can get
>> 1,000 IOPS for each of those 10 volumes...not that you actually will (a
>> guarantee) get those IOPS.
>>
>> If performance can be guaranteed at the storage infrastructure, this
>> becomes much easier.
>>
>> For example, with a SolidFire SAN, you simply specify a minimum
>> (guaranteed) number of IOPS, a maximum number of IOPS, and a burst number
>> of IOPS for each volume (LUN). The SAN will guarantee the minimum number of
>> IOPS (from the point of view of the client) regardless of how the data is
>> physically distributed on its SSDs (you don't have to worry about RAID
>> groups or low-level details like that). This minimum number of IOPS is
>> guaranteed even during failure scenarios.
>>
>> 3) Punith is referring to dynamically changing the number of IOPS available
>> of the SAN volume and making those new IOPS available to the CloudStack
>> volume or volumes that leverage that SAN volume.
>>
>> 4) Hypervisor rate limiting - by itself - does not guarantee network
>> performance, that is true. This is true of storage QoS, as well, but rarely
>> is the network the bottleneck (SolidFire requires a 10G Ethernet network
>> for the storage traffic).
>>
>> I hope that helps,
>> Mike
>>
>> On Mon, Jun 2, 2014 at 1:42 AM, Yoshikazu Nojima  wrote:
>>
>> > Punith, Mike,
>> >
>> > Allow me to ask questions regarding guaranteed IOPS feature.
>> > (Since it may be off-topic to Hieu's original question, I changed the
>> > subject.)
>> >
>> > Q1) Does your plugins(Solidfire, CloudBytes) support KVM? From which
>> > version?
>> >
>> > In my understanding,
>> > Current CloudStack's implementation guarantees IOPS by two regulations:
>> > - Storage appliance (or Hypervisor) limits Volume's max IOPS and
>> > throughput.
>> > - Management Server ensures max IOPS performance of a storage will not
>> > be exceeded by the sum of volumes' IOPS in the storage by controlli

Re: [DISCUSS] extending the libvirt/KVM plugin to also support libvirt/Xen

[Tim Mackey]

Dave,

Thanks for putting this up on the wiki. A few things jumped out at me...

- Please change "Xen" to "XenProject" or "Xen Project" as appropriate.
There's already a ton of confusion out there, and I'd like to see us
get our terms correct from the outset where ever possible.

- It would be good to see a UI mock up for how users would configure
the Xen Project hypervisor option.  I think that would go a long way
to helping with the mixed hypervisor cluster concept and how it could
be blocked.

- It would also be good to see examples of how the APIs might need to
be changed to support this.  Minimally I'd expect to see things like
supported disk/network/os types and that sort of thing.

- I see you have a todo to document the supported Xen Project
hypervisor and libvirt versions, but also dependencies on libxl
changes.  Are these critical dependencies, or if someone doesn't have
latest upstream will things work in a reduced feature set?

- C6.1 talks about exposing a config setting.  Is that really
required?  Couldn't that be set correctly based on hypervisor type?

- Would QCOW2 be used for the Xen Project disk type for all templates
to keep with KVM consistency?  I'm actually thinking  about support
for VMDK, but perhaps that's a different proposal?

- Since we're talking about sharing a libvirt plugin, I'm not clear on
if the shared work is done in a new libvirt plugin which is then
exposed to a KVM and a XenProject plugin or if the existing KVM plugin
is refactored to encompass both.

-tim




On Mon, Jun 9, 2014 at 2:35 AM, Wido den Hollander  wrote:
>
>
> On 06/08/2014 11:14 PM, Dave Scott wrote:
>>
>> Hi Wido,
>>
>> Thanks for your mail!
>>
>> On 8 Jun 2014, at 19:02, Wido den Hollander  wrote:
>>
>>> On 06/08/2014 06:23 PM, Dave Scott wrote:

 Hi,

 Following on from the earlier "[PROPOSAL] Support pure Xen as a
 hypervisor”, I’ve added a design doc to the wiki:


 https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+hosts+running+the+Xen+hypervisor+to+be+managed+via+libvirt

 This design would allow people who want to manage their hypervisors
 purely through the libvirt tools to choose the Xen hypervisor.

  From the code point of view, I want to maximise sharing between the KVM
 and Xen code paths, partly to make QA easier and partly to maximise the
 chance that adding a feature for “Xen” causes it to work for “KVM” and
 vice-versa. In particular this means that, if a genuinely-useful capability
 is currently missing from the libvirt libxl driver, I want to implement it
 rather than work around it.

>>>
>>> Seems like a great route to me! You also want to support Xen+Qemu with
>>> this way?
>>
>>
>> Yes, it should be possible to run fully virtualised VMs with Xen + Qemu. I
>> think we’ll be able to choose whether to run VMs as PV or HVM.
>
>
> Ok, but those will be different code paths at some level.
>
>
>>
>>> We have to be aware that there might be some storage differences between
>>> KVM and Xen like Ceph which is not fully supported yet by Xen.
>>
>>
>> Ceph is an interesting one. Xen itself doesn’t know anything about
>> storage— instead the dom0 takes care of it either via a kernel driver
>> (blkback) or userspace program (qemu or tapdisk). When I tried to make Ceph
>> work about a year ago[1] I hit a bug in libxl (the Xen control library). The
>> good news is the fix made it into Xen 4.4, so with luck we can get it to
>> work.
>>
>
> When Xen runs with Qemu as full HVM it's Qemu which takes care of the Ceph
> storage, so in that case it's fixed.
>
> I haven't got a lot of experience with PV Xen. I heard stories of Ceph being
> integrated in blktap(2), but never tested it.
>
>
>>
>>> If anything is missing in libvirt or the Java bindings we have to fix
>>> that indeed instead of hacking around it.
>>
>>
>> Great :)
>>
>> Cheers,
>> Dave
>>
>> [1]
>> http://xenserver.org/discuss-virtualization/virtualization-blog/entry/tech-preview-of-xenserver-libvirt-ceph.html
>>
>>>
>>> Wido
>>>
 Comments appreciated!

 Cheers,
 Dave

 [1]
 http://mail-archives.apache.org/mod_mbox/cloudstack-users/201403.mbox/%3ccajgxtbnbmqtq81ralgh2kma7v5wjyzkr3xnyasmkc_br+uk...@mail.gmail.com%3e

>>
>

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[David Nalley]

On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
> Hi all,
>
> Seems it's a good timing to bring back the discussion about the gerrit.
>
> We want to do CI, and improve our code quality. One obvious way of doing
> and reduce the workload of devs is introduce a tool to enforce the process.
>
> I've checked out quite a few projects using gerrit, which would force you
> to ask for review, and validation before the code can be committed to the
> repo. Looks it's really a easier way for devs according what I've heard.
>
> Even our competitor laid out a very detail workflow based on the use of
> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it can
> make a good reference.
>
> Well, gerrit has been brought up a few times before. And now the new
> process we want to enforce just fits what gerrit(or other automation
> review/test/commit software) is for.
>
> Maybe it's the time for us to review the possibility of using a tool to
> enforce our commits and improve our code quality(as well as transfer
> knowledge) again?
>
> --Sheng
>

ASF Infra has a very dour view on Gerrit. Don't read that as
impossible; there are many projects at the ASF who are interested in
Gerrit.
That said; what about moving to using github pull requests instead of
RB, and from their, having the jenkins pull request builder
automatically process every pull request and list information.

Here's an example:
https://github.com/jclouds/jclouds-labs/pull/61
You'll see that every time the patch changes, the jenkins plugin
pulled the patch - ran tests against it and reported back.

That said; it almost seems like we have the cart before the horse; we
need to finish figuring out the CI Infrastructure first.

--David

Re: [ANNOUNCE] Demetrius Tsitrelis as committer

[Min Chen]

Congrats, Demetrius.

-min

On 6/6/14 4:17 PM, "John Kinsella"  wrote:

>Folks - this one¹s a little belated - we went through the invite process
>around the
>time of the mail issues, and somehow we didn¹t send the announcement to
>dev@.
>I noticed while doing some housekeeping this week, and wanted to send out
>the
>announcement anyways just to give Demetrius the recognition. :)
>
>The Project Management Committee (PMC) for Apache CloudStack has
>asked Demetrius Tsitrelis to become a committer and we are pleased to
>announce
>that he has accepted.
>
>Being a committer allows many contributors to contribute more
>autonomously. For
>developers, it makes it easier to submit changes and eliminates the need
>to
>have contributions reviewed via the patch submission process. Whether
>contributions are development-related or otherwise, it is a recognition
>of a
>contributor's participation in the project and commitment to the project
>and
>the Apache Way.
>
>Please join me in congratulating Demetrius!
>
>-John, on behalf of the CloudStack PMC

Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[David Nalley]

On Mon, Jun 9, 2014 at 10:39 AM, Mike Tutkowski
 wrote:
> Right - we need to decide as a community what a forward branch really means.
>
> On Monday, June 9, 2014, Koushik Das  wrote:
>

So a bit of background reading:
http://markmail.org/message/aux2yjxpudotu7qu

This is when we started with the -forward branches.

--David

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[Min Chen]

I like github pull request as well from past usage, it is very convenient
for developers and reviewers to perform their tasks, compared to our
current RB. Also agree with David, the pre-requisite for this enforcement
is that we should have CI in place to make this happen.

Thanks
-min

On 6/9/14 8:47 AM, "David Nalley"  wrote:

>On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
>> Hi all,
>>
>> Seems it's a good timing to bring back the discussion about the gerrit.
>>
>> We want to do CI, and improve our code quality. One obvious way of doing
>> and reduce the workload of devs is introduce a tool to enforce the
>>process.
>>
>> I've checked out quite a few projects using gerrit, which would force
>>you
>> to ask for review, and validation before the code can be committed to
>>the
>> repo. Looks it's really a easier way for devs according what I've heard.
>>
>> Even our competitor laid out a very detail workflow based on the use of
>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it
>>can
>> make a good reference.
>>
>> Well, gerrit has been brought up a few times before. And now the new
>> process we want to enforce just fits what gerrit(or other automation
>> review/test/commit software) is for.
>>
>> Maybe it's the time for us to review the possibility of using a tool to
>> enforce our commits and improve our code quality(as well as transfer
>> knowledge) again?
>>
>> --Sheng
>>
>
>ASF Infra has a very dour view on Gerrit. Don't read that as
>impossible; there are many projects at the ASF who are interested in
>Gerrit.
>That said; what about moving to using github pull requests instead of
>RB, and from their, having the jenkins pull request builder
>automatically process every pull request and list information.
>
>Here's an example:
>https://github.com/jclouds/jclouds-labs/pull/61
>You'll see that every time the patch changes, the jenkins plugin
>pulled the patch - ran tests against it and reported back.
>
>That said; it almost seems like we have the cart before the horse; we
>need to finish figuring out the CI Infrastructure first.
>
>--David

Re: [DISCUSS] extending the libvirt/KVM plugin to also support libvirt/Xen

[Dave Scott]

Hi Tim,

Thanks for your feedback!

On 9 Jun 2014, at 16:44, Tim Mackey  wrote:

> Dave,
> 
> Thanks for putting this up on the wiki. A few things jumped out at me...
> 
> - Please change "Xen" to "XenProject" or "Xen Project" as appropriate.
> There's already a ton of confusion out there, and I'd like to see us
> get our terms correct from the outset where ever possible.

Sure — will do.

> - It would be good to see a UI mock up for how users would configure
> the Xen Project hypervisor option.  I think that would go a long way
> to helping with the mixed hypervisor cluster concept and how it could
> be blocked.

My comments about clusters were to emphasise that you shouldn’t expect to live 
migrate a VM between KVM and Xen, even if both are using libvirt underneath. 
Actually perhaps this is a misunderstanding of mine: is it possible today to 
mix hypervisors within a single CloudStack cluster? I’m not trying to change 
anything, but just point out the obvious. Maybe I got that wrong :-)

> - It would also be good to see examples of how the APIs might need to
> be changed to support this.  Minimally I'd expect to see things like
> supported disk/network/os types and that sort of thing.

I can talk about some of this more explicitly in the doc. Since Xen can use 
qemu for disks (for both PV and HVM guests) there should be no difference in 
supported disk formats between this and the existing KVM support. I’m not 
proposing to add anything to the Xen support which isn’t supported by KVM such 
as .vhd via tapdisk. Similarly, networking is handled by the regular Linux 
network stack so that should all work in the same way.

> - I see you have a todo to document the supported Xen Project
> hypervisor and libvirt versions, but also dependencies on libxl
> changes.  Are these critical dependencies, or if someone doesn't have
> latest upstream will things work in a reduced feature set?

In this proposal they would be critical dependencies (I’ll go make that clear). 
It is possible to make transitional arrangements but I didn’t want to 
overburden this proposal with backwards compat.

> - C6.1 talks about exposing a config setting.  Is that really
> required?  Couldn't that be set correctly based on hypervisor type?

You’re right that this would be a hypervisor-specific thing.

I’m still pondering the choice between PV and HVM. Using PV mode is convenient 
because it would allow the VMs to boot under Xen under virtualbox, like current 
devcloud. Using HVM might be more future-proof.

> - Would QCOW2 be used for the Xen Project disk type for all templates
> to keep with KVM consistency?  I'm actually thinking  about support
> for VMDK, but perhaps that's a different proposal?

That sounds a separate proposal, but it shouldn’t conflict with this one 
(assuming the VMDK support is in qemu)

> - Since we're talking about sharing a libvirt plugin, I'm not clear on
> if the shared work is done in a new libvirt plugin which is then
> exposed to a KVM and a XenProject plugin or if the existing KVM plugin
> is refactored to encompass both.

Not totally sure what the best thing to do is — I’ll have to play with the code 
a little more.

Thanks,
Dave


> 
> -tim
> 
> 
> 
> 
> On Mon, Jun 9, 2014 at 2:35 AM, Wido den Hollander  wrote:
>> 
>> 
>> On 06/08/2014 11:14 PM, Dave Scott wrote:
>>> 
>>> Hi Wido,
>>> 
>>> Thanks for your mail!
>>> 
>>> On 8 Jun 2014, at 19:02, Wido den Hollander  wrote:
>>> 
 On 06/08/2014 06:23 PM, Dave Scott wrote:
> 
> Hi,
> 
> Following on from the earlier "[PROPOSAL] Support pure Xen as a
> hypervisor”, I’ve added a design doc to the wiki:
> 
> 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+hosts+running+the+Xen+hypervisor+to+be+managed+via+libvirt
> 
> This design would allow people who want to manage their hypervisors
> purely through the libvirt tools to choose the Xen hypervisor.
> 
> From the code point of view, I want to maximise sharing between the KVM
> and Xen code paths, partly to make QA easier and partly to maximise the
> chance that adding a feature for “Xen” causes it to work for “KVM” and
> vice-versa. In particular this means that, if a genuinely-useful 
> capability
> is currently missing from the libvirt libxl driver, I want to implement it
> rather than work around it.
> 
 
 Seems like a great route to me! You also want to support Xen+Qemu with
 this way?
>>> 
>>> 
>>> Yes, it should be possible to run fully virtualised VMs with Xen + Qemu. I
>>> think we’ll be able to choose whether to run VMs as PV or HVM.
>> 
>> 
>> Ok, but those will be different code paths at some level.
>> 
>> 
>>> 
 We have to be aware that there might be some storage differences between
 KVM and Xen like Ceph which is not fully supported yet by Xen.
>>> 
>>> 
>>> Ceph is an interesting one. Xen itself doesn’t know anything about
>>> storage— instead the dom0 takes care of it either v

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[David Nalley]

On Mon, Jun 9, 2014 at 11:47 AM, David Nalley  wrote:
> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
>> Hi all,
>>
>> Seems it's a good timing to bring back the discussion about the gerrit.
>>
>> We want to do CI, and improve our code quality. One obvious way of doing
>> and reduce the workload of devs is introduce a tool to enforce the process.
>>
>> I've checked out quite a few projects using gerrit, which would force you
>> to ask for review, and validation before the code can be committed to the
>> repo. Looks it's really a easier way for devs according what I've heard.
>>
>> Even our competitor laid out a very detail workflow based on the use of
>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it can
>> make a good reference.
>>
>> Well, gerrit has been brought up a few times before. And now the new
>> process we want to enforce just fits what gerrit(or other automation
>> review/test/commit software) is for.
>>
>> Maybe it's the time for us to review the possibility of using a tool to
>> enforce our commits and improve our code quality(as well as transfer
>> knowledge) again?
>>
>> --Sheng
>>
>
> ASF Infra has a very dour view on Gerrit. Don't read that as
> impossible; there are many projects at the ASF who are interested in
> Gerrit.
> That said; what about moving to using github pull requests instead of
> RB, and from their, having the jenkins pull request builder
> automatically process every pull request and list information.
>
> Here's an example:
> https://github.com/jclouds/jclouds-labs/pull/61
> You'll see that every time the patch changes, the jenkins plugin
> pulled the patch - ran tests against it and reported back.
>
> That said; it almost seems like we have the cart before the horse; we
> need to finish figuring out the CI Infrastructure first.
>
> --David

Just as an additional comment.
CF was using gerrit, and apparently dropped it in favor of GH Pull
Requests plus automated testing with Travis-CI.
I know folks like abayer and jfarrell are working on turning on
automated builds via the github pull request plugin for jenkins. Might
be something to consider.

--David

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[sebgoa]

On Jun 9, 2014, at 7:13 PM, David Nalley  wrote:

> On Mon, Jun 9, 2014 at 11:47 AM, David Nalley  wrote:
>> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
>>> Hi all,
>>> 
>>> Seems it's a good timing to bring back the discussion about the gerrit.
>>> 
>>> We want to do CI, and improve our code quality. One obvious way of doing
>>> and reduce the workload of devs is introduce a tool to enforce the process.
>>> 
>>> I've checked out quite a few projects using gerrit, which would force you
>>> to ask for review, and validation before the code can be committed to the
>>> repo. Looks it's really a easier way for devs according what I've heard.
>>> 
>>> Even our competitor laid out a very detail workflow based on the use of
>>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it can
>>> make a good reference.
>>> 
>>> Well, gerrit has been brought up a few times before. And now the new
>>> process we want to enforce just fits what gerrit(or other automation
>>> review/test/commit software) is for.
>>> 
>>> Maybe it's the time for us to review the possibility of using a tool to
>>> enforce our commits and improve our code quality(as well as transfer
>>> knowledge) again?
>>> 
>>> --Sheng
>>> 
>> 
>> ASF Infra has a very dour view on Gerrit. Don't read that as
>> impossible; there are many projects at the ASF who are interested in
>> Gerrit.
>> That said; what about moving to using github pull requests instead of
>> RB, and from their, having the jenkins pull request builder
>> automatically process every pull request and list information.
>> 
>> Here's an example:
>> https://github.com/jclouds/jclouds-labs/pull/61
>> You'll see that every time the patch changes, the jenkins plugin
>> pulled the patch - ran tests against it and reported back.
>> 
>> That said; it almost seems like we have the cart before the horse; we
>> need to finish figuring out the CI Infrastructure first.
>> 
>> --David
> 
> Just as an additional comment.
> CF was using gerrit, and apparently dropped it in favor of GH Pull
> Requests plus automated testing with Travis-CI.
> I know folks like abayer and jfarrell are working on turning on
> automated builds via the github pull request plugin for jenkins. Might
> be something to consider.

fwiw, I have my personal cloudstack fork on github with a simple travisCI setup.
So when I make a commit it automatically triggers a build.
I was working on running the integration tests on Travis as well but got 
distracted.

If somebody wants to step in:

https://github.com/runseb/cloudstack/blob/master/.travis.yml

help welcome.

:)


> 
> --David

RE: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[Animesh Chaturvedi]

> -Original Message-
> From: David Nalley [mailto:da...@gnsa.us]
> Sent: Monday, June 09, 2014 9:24 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?
> 
> On Mon, Jun 9, 2014 at 10:39 AM, Mike Tutkowski
>  wrote:
> > Right - we need to decide as a community what a forward branch really
> means.
> >
> > On Monday, June 9, 2014, Koushik Das  wrote:
> >
> 
> So a bit of background reading:
> http://markmail.org/message/aux2yjxpudotu7qu
> 
> This is when we started with the -forward branches.
> 
> --David
[Animesh] I had started the forward branches right after first RC because we 
always have tons of fixes until we are ready with RC. IMHO starting it at code 
freeze is too early.

Re: [ACS44] 112 unpicked cherries in 4.4-forward. why?

[Mike Tutkowski]

Thanks for that reminder, David!

OK, so, per David's e-mail reference, it looks like the intent of
x.y-forward is what I referred to as option 2:

4.4-forward contains changes that might go into 4.4 (if a cherry pick is
requested) and changes that would go into 4.4.1, should such a release ever
be made.

We can - of course - debate if this is still what we want to do; however, I
would say - for 4.4 - we should treat 4.4-forward the way we described in
this previous e-mail. In other words, do not assume 4.4-forward will
eventually be equal to 4.4 before 4.4 is released. 4.4-forward should be
used for fixes that need to make their way into 4.4 and for fixes that do
not need to be in 4.4, but rather in 4.4.1, should such a release ever
happen.


On Mon, Jun 9, 2014 at 10:23 AM, David Nalley  wrote:

> On Mon, Jun 9, 2014 at 10:39 AM, Mike Tutkowski
>  wrote:
> > Right - we need to decide as a community what a forward branch really
> means.
> >
> > On Monday, June 9, 2014, Koushik Das  wrote:
> >
>
> So a bit of background reading:
> http://markmail.org/message/aux2yjxpudotu7qu
>
> This is when we started with the -forward branches.
>
> --David
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: redundant virtual routers for VPCs

[Sheng Yang]

Yes, I am in Pacific Time.

--Sheng


On Sat, Jun 7, 2014 at 10:33 PM, Daan Hoogland 
wrote:

> ok, I think your on pacific time, are you?
>
> On Sun, Jun 8, 2014 at 5:33 AM, Sheng Yang  wrote:
> > I would be glad to join.
> >
> > --Sheng
> >
> >
> > On Sat, Jun 7, 2014 at 3:01 AM, Daan Hoogland 
> > wrote:
> >>
> >> At Schuberg Philis, the urge to have virtual routers in a redundant
> >> way is getting to be pressing. It seems Citrix wants to move away from
> >> it entirely and Sungard is working on it but has to little resources
> >> for it. Withing our dev group we are now planning a sprint to
> >> experiment with this subject and then implement or fix the present
> >> routers to support this. However we would very much like input from
> >> the people who have already worked on this and are also very willing
> >> to let them in our sprint-team if they are willing.
> >>
> >> So: who want to join in a irc meeting (or conf call) on the subject
> >> coming week? Dutch people are not in the office on the day after
> >> ascension so tuesday is our intention.
> >>
> >> on behalf of some very impatient operators;)
> >> --
> >> Daan
> >
> >
>
>
>
> --
> Daan
>

Re: redundant virtual routers for VPCs

[Karl Harris]

I will be happy to join!



On Saturday, June 7, 2014, Daan Hoogland  wrote:

> At Schuberg Philis, the urge to have virtual routers in a redundant
> way is getting to be pressing. It seems Citrix wants to move away from
> it entirely and Sungard is working on it but has to little resources
> for it. Withing our dev group we are now planning a sprint to
> experiment with this subject and then implement or fix the present
> routers to support this. However we would very much like input from
> the people who have already worked on this and are also very willing
> to let them in our sprint-team if they are willing.
>
> So: who want to join in a irc meeting (or conf call) on the subject
> coming week? Dutch people are not in the office on the day after
> ascension so tuesday is our intention.
>
> on behalf of some very impatient operators;)
> --
> Daan
>
>

Re: feature : changing volume properties dynamically in 4.5

[Mike Tutkowski]

Hi Punith,

This kind of a feature is something Chris Suich and I discussed a while
back.

We talked about leveraging arbitrary key/value pairs to make this happen
(OpenStack does something similar). The key/value pairs would be vendor
specific.

If we take a key/value approach, we might be able to make this all work the
way things work today when the user wants to change an existing Compute
Offering and/or Disk Offering.

For example, the user would pick a new Compute Offering (with presumably
has different key/value pairs) and CloudStack could inform the applicable
storage provider, who could update the volume in question.

This way we don't need to introduce a new API command and the use model for
the user doesn't really change.

What are you thoughts on this?

Thanks,
Mike


On Mon, Jun 9, 2014 at 8:08 AM, Punith S  wrote:

> hi guys,
>
> since most of the third party storage providers have been implementing 1:1
> mapping(managed storage) between a volume(dataset) and a vm disk(vdi/vmdk)
> for guaranteeing the Qos, i would like to propose a new feature to
> dynamically change the volume properties supported by storage vendors such
> as IOPS, Deduplication, Compression, Grace, Syncronization, Latency
> etc, depending on properties and features supported by respective storage
> vendors. hence providing more flexibility for users.
>
> in case of using default cloudstack storage provider, we can change the
> properties of the vdi/vmdk files apart from resizing the volume(vdi/vmdk).
>
> changes in management server include,
>
> new async web api ChangeVolumePropertiesCmd,
> new method in VolumeApiService for vo and dao validation implementations.
> new method in VolumeServiceManager for supporting callback and calling the
> respective storage provider driver's implementation.
> new method in PrimaryDataStoreDriver interface for implementing respective
> features according to their storage product.
>
> changes in UI include,
> new changing volume properties widget in volume section, showing different
> properties depending upon listed storage providers.
>
> any suggestions and feedbacks ?
>
> thanks
>
> --
> regards,
>
> punith s
> cloudbyte.com
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[Rohit Yadav]

On Mon, Jun 9, 2014 at 11:32 PM, sebgoa  wrote:

>
> On Jun 9, 2014, at 7:13 PM, David Nalley  wrote:
>
> > On Mon, Jun 9, 2014 at 11:47 AM, David Nalley  wrote:
> >> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
> >>> Hi all,
> >>>
> >>> Seems it's a good timing to bring back the discussion about the gerrit.
> >>>
> >>> We want to do CI, and improve our code quality. One obvious way of
> doing
> >>> and reduce the workload of devs is introduce a tool to enforce the
> process.
> >>>
> >>> I've checked out quite a few projects using gerrit, which would force
> you
> >>> to ask for review, and validation before the code can be committed to
> the
> >>> repo. Looks it's really a easier way for devs according what I've
> heard.
> >>>
> >>> Even our competitor laid out a very detail workflow based on the use of
> >>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it
> can
> >>> make a good reference.
> >>>
> >>> Well, gerrit has been brought up a few times before. And now the new
> >>> process we want to enforce just fits what gerrit(or other automation
> >>> review/test/commit software) is for.
> >>>
> >>> Maybe it's the time for us to review the possibility of using a tool to
> >>> enforce our commits and improve our code quality(as well as transfer
> >>> knowledge) again?
> >>>
> >>> --Sheng
> >>>
> >>
> >> ASF Infra has a very dour view on Gerrit. Don't read that as
> >> impossible; there are many projects at the ASF who are interested in
> >> Gerrit.
> >> That said; what about moving to using github pull requests instead of
> >> RB, and from their, having the jenkins pull request builder
> >> automatically process every pull request and list information.
> >>
> >> Here's an example:
> >> https://github.com/jclouds/jclouds-labs/pull/61
> >> You'll see that every time the patch changes, the jenkins plugin
> >> pulled the patch - ran tests against it and reported back.
> >>
> >> That said; it almost seems like we have the cart before the horse; we
> >> need to finish figuring out the CI Infrastructure first.
> >>
> >> --David
> >
> > Just as an additional comment.
> > CF was using gerrit, and apparently dropped it in favor of GH Pull
> > Requests plus automated testing with Travis-CI.
> > I know folks like abayer and jfarrell are working on turning on
> > automated builds via the github pull request plugin for jenkins. Might
> > be something to consider.
>
> fwiw, I have my personal cloudstack fork on github with a simple travisCI
> setup.
> So when I make a commit it automatically triggers a build.
> I was working on running the integration tests on Travis as well but got
> distracted.
>
> If somebody wants to step in:
>
> https://github.com/runseb/cloudstack/blob/master/.travis.yml


This would be great if we could just use Github (pull requests, releases)
and hook up Travis-CI. Both are free for opensource projects.

Regards.


>
>
> help welcome.
>
> :)
>
>
> >
> > --David
>
>

Re: [ACS4.4] cherrypick c282bb3a1293fbbfdb306263ea52464862670fb3

[Daan Hoogland]

On Mon, Jun 9, 2014 at 1:01 PM, Rajesh Battala
 wrote:
> c282bb3a1293fbbfdb306263ea52464862670fb3


is in

-- 
Daan

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[Mike Tutkowski]

That sounds good to me, as well.


On Mon, Jun 9, 2014 at 1:14 PM, Rohit Yadav  wrote:

> On Mon, Jun 9, 2014 at 11:32 PM, sebgoa  wrote:
>
> >
> > On Jun 9, 2014, at 7:13 PM, David Nalley  wrote:
> >
> > > On Mon, Jun 9, 2014 at 11:47 AM, David Nalley  wrote:
> > >> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
> > >>> Hi all,
> > >>>
> > >>> Seems it's a good timing to bring back the discussion about the
> gerrit.
> > >>>
> > >>> We want to do CI, and improve our code quality. One obvious way of
> > doing
> > >>> and reduce the workload of devs is introduce a tool to enforce the
> > process.
> > >>>
> > >>> I've checked out quite a few projects using gerrit, which would force
> > you
> > >>> to ask for review, and validation before the code can be committed to
> > the
> > >>> repo. Looks it's really a easier way for devs according what I've
> > heard.
> > >>>
> > >>> Even our competitor laid out a very detail workflow based on the use
> of
> > >>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess
> it
> > can
> > >>> make a good reference.
> > >>>
> > >>> Well, gerrit has been brought up a few times before. And now the new
> > >>> process we want to enforce just fits what gerrit(or other automation
> > >>> review/test/commit software) is for.
> > >>>
> > >>> Maybe it's the time for us to review the possibility of using a tool
> to
> > >>> enforce our commits and improve our code quality(as well as transfer
> > >>> knowledge) again?
> > >>>
> > >>> --Sheng
> > >>>
> > >>
> > >> ASF Infra has a very dour view on Gerrit. Don't read that as
> > >> impossible; there are many projects at the ASF who are interested in
> > >> Gerrit.
> > >> That said; what about moving to using github pull requests instead of
> > >> RB, and from their, having the jenkins pull request builder
> > >> automatically process every pull request and list information.
> > >>
> > >> Here's an example:
> > >> https://github.com/jclouds/jclouds-labs/pull/61
> > >> You'll see that every time the patch changes, the jenkins plugin
> > >> pulled the patch - ran tests against it and reported back.
> > >>
> > >> That said; it almost seems like we have the cart before the horse; we
> > >> need to finish figuring out the CI Infrastructure first.
> > >>
> > >> --David
> > >
> > > Just as an additional comment.
> > > CF was using gerrit, and apparently dropped it in favor of GH Pull
> > > Requests plus automated testing with Travis-CI.
> > > I know folks like abayer and jfarrell are working on turning on
> > > automated builds via the github pull request plugin for jenkins. Might
> > > be something to consider.
> >
> > fwiw, I have my personal cloudstack fork on github with a simple travisCI
> > setup.
> > So when I make a commit it automatically triggers a build.
> > I was working on running the integration tests on Travis as well but got
> > distracted.
> >
> > If somebody wants to step in:
> >
> > https://github.com/runseb/cloudstack/blob/master/.travis.yml
>
>
> This would be great if we could just use Github (pull requests, releases)
> and hook up Travis-CI. Both are free for opensource projects.
>
> Regards.
>
>
> >
> >
> > help welcome.
> >
> > :)
> >
> >
> > >
> > > --David
> >
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
*™*

Re: redundant virtual routers for VPCs

[Daan Hoogland]

nice,

so given pacific time I can have a meeting at 15:00 utc or at 20:00
utc (8 am or 2 pm if I'm correct). Given office hours in Holland 17:00
(15:00 utc) would be best but if you're not in it could be later. As
the person calling the meeting I would like to participate but if the
highest attendance can be found only between 16 and 22 utc thats fine
as well.

On Mon, Jun 9, 2014 at 8:35 PM, Karl Harris  wrote:
> I will be happy to join!
>
>
>
> On Saturday, June 7, 2014, Daan Hoogland  wrote:
>>
>> At Schuberg Philis, the urge to have virtual routers in a redundant
>> way is getting to be pressing. It seems Citrix wants to move away from
>> it entirely and Sungard is working on it but has to little resources
>> for it. Withing our dev group we are now planning a sprint to
>> experiment with this subject and then implement or fix the present
>> routers to support this. However we would very much like input from
>> the people who have already worked on this and are also very willing
>> to let them in our sprint-team if they are willing.
>>
>> So: who want to join in a irc meeting (or conf call) on the subject
>> coming week? Dutch people are not in the office on the day after
>> ascension so tuesday is our intention.
>>
>> on behalf of some very impatient operators;)
>> --
>> Daan
>>
>



-- 
Daan

Re: feature : changing volume properties dynamically in 4.5

[Mike Tutkowski]

Allow me to follow this up with more detail (with regards to what Chris and
I talked about):

As you are aware, today the way you associate a Compute Offering (CO) or a
Disk Offering (DO) with a Primary Storage (PS) is via storage tagging.

This has some benefits and drawbacks.

One benefit is being able to have some level of vendor independence from
the point of view of the CO or DO. For example, if the storage tag of a DO
is "Fast", then this can be satisfied by PS that describes itself as
"Fast", regardless of vendor.

A major drawback with the storage-tagging approach, however, is that you
are not easily able to leverage vendor-specific features, which is often
why you bought storage from the vendor in question to begin with.

Ideally we do not want to add each vendor's features into the system as
properties that can be seen by the admin regardless of whether or not the
underlying storage he's actually using supports the feature in question.

This coarse approach, however, was sort of business as usual when I started
in with CloudStack 1.5 years ago.

That being the case, when I added QoS options to CS, I did so in a way
where the admin would see Min IOPS and Max IOPS options regardless of
whether or not his storage actually supported those controls (to mitigate
this a bit in the GUI, the admin has to explicitly select "Storage QoS"
from a combobox).

We leverage the same use model with Hypervisor QoS: The admin sees these
options regardless of whether or not they actually apply on the hypervisor
where the VM gets deployed.

Going forward, we want to implement a more fine-grain and generic approach.

We would like to have a storage provider field for the CO and DO windows
(this equates to the name of one and only one storage provider). If the
admin inputs a specific storage provider and does not use the storage tags
field, he can enter in an arbitrary number of key/value pairs in another
text field (perhaps we would provide a nice entry dialog to make this
easier in the GUI). These key value pairs can be passed into the storage
driver when it's asked to create (or update) a volume and the storage
driver can decide what each and every key/value pair means.

What do you think about this approach?

Thanks


On Mon, Jun 9, 2014 at 1:14 PM, Mike Tutkowski  wrote:

> Hi Punith,
>
> This kind of a feature is something Chris Suich and I discussed a while
> back.
>
> We talked about leveraging arbitrary key/value pairs to make this happen
> (OpenStack does something similar). The key/value pairs would be vendor
> specific.
>
> If we take a key/value approach, we might be able to make this all work
> the way things work today when the user wants to change an existing Compute
> Offering and/or Disk Offering.
>
> For example, the user would pick a new Compute Offering (with presumably
> has different key/value pairs) and CloudStack could inform the applicable
> storage provider, who could update the volume in question.
>
> This way we don't need to introduce a new API command and the use model
> for the user doesn't really change.
>
> What are you thoughts on this?
>
> Thanks,
> Mike
>
>
> On Mon, Jun 9, 2014 at 8:08 AM, Punith S  wrote:
>
>> hi guys,
>>
>> since most of the third party storage providers have been implementing
>> 1:1 mapping(managed storage) between a volume(dataset) and a vm
>> disk(vdi/vmdk) for guaranteeing the Qos, i would like to propose a new
>> feature to dynamically change the volume properties supported by storage
>> vendors such as IOPS, Deduplication, Compression, Grace, Syncronization,
>> Latency etc, depending on properties and features supported by respective
>> storage vendors. hence providing more flexibility for users.
>>
>> in case of using default cloudstack storage provider, we can change the
>> properties of the vdi/vmdk files apart from resizing the volume(vdi/vmdk).
>>
>> changes in management server include,
>>
>> new async web api ChangeVolumePropertiesCmd,
>> new method in VolumeApiService for vo and dao validation implementations.
>> new method in VolumeServiceManager for supporting callback and calling
>> the respective storage provider driver's implementation.
>> new method in PrimaryDataStoreDriver interface for implementing
>> respective features according to their storage product.
>>
>> changes in UI include,
>> new changing volume properties widget in volume section, showing
>> different properties depending upon listed storage providers.
>>
>> any suggestions and feedbacks ?
>>
>> thanks
>>
>> --
>> regards,
>>
>> punith s
>> cloudbyte.com
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud

Re: [DISCUSS] extending the libvirt/KVM plugin to also support libvirt/Xen

[Tim Mackey]

On Mon, Jun 9, 2014 at 1:02 PM, Dave Scott  wrote:

>
> > - It would be good to see a UI mock up for how users would configure
> > the Xen Project hypervisor option.  I think that would go a long way
> > to helping with the mixed hypervisor cluster concept and how it could
> > be blocked.
>
> My comments about clusters were to emphasise that you shouldn’t expect to
> live migrate a VM between KVM and Xen, even if both are using libvirt
> underneath. Actually perhaps this is a misunderstanding of mine: is it
> possible today to mix hypervisors within a single CloudStack cluster? I’m
> not trying to change anything, but just point out the obvious. Maybe I got
> that wrong :-)
>

This comes down to your implementation.  Today CS clusters need to have
uniform hypervisor types, but if you're exposing a libvirt plugin, then
those protections won't exist since CS will see things as just libvirt. If
you create a KVM plugin and a separate XenProject one both of which
implement a libvirt base class then you should get the protections for
free.  It should also allow you to define hypervisor specific details in a
cleaner way.


> > - It would also be good to see examples of how the APIs might need to
> > be changed to support this.  Minimally I'd expect to see things like
> > supported disk/network/os types and that sort of thing.
>
> I can talk about some of this more explicitly in the doc. Since Xen can
> use qemu for disks (for both PV and HVM guests) there should be no
> difference in supported disk formats between this and the existing KVM
> support. I’m not proposing to add anything to the Xen support which isn’t
> supported by KVM such as .vhd via tapdisk. Similarly, networking is handled
> by the regular Linux network stack so that should all work in the same way.
>

I wouldn't assume the existing KVM plugin is generic. For example, only
QCOW2 is supported for disk images today.


> > - I see you have a todo to document the supported Xen Project
> > hypervisor and libvirt versions, but also dependencies on libxl
> > changes.  Are these critical dependencies, or if someone doesn't have
> > latest upstream will things work in a reduced feature set?
>
> In this proposal they would be critical dependencies (I’ll go make that
> clear). It is possible to make transitional arrangements but I didn’t want
> to overburden this proposal with backwards compat.
>
> > - C6.1 talks about exposing a config setting.  Is that really
> > required?  Couldn't that be set correctly based on hypervisor type?
>
> You’re right that this would be a hypervisor-specific thing.
>
> I’m still pondering the choice between PV and HVM. Using PV mode is
> convenient because it would allow the VMs to boot under Xen under
> virtualbox, like current devcloud. Using HVM might be more future-proof.
>
> > - Would QCOW2 be used for the Xen Project disk type for all templates
> > to keep with KVM consistency?  I'm actually thinking  about support
> > for VMDK, but perhaps that's a different proposal?
>
> That sounds a separate proposal, but it shouldn’t conflict with this one
> (assuming the VMDK support is in qemu)
>

VMDK support is there, but the existing KVM plugin only supports QCOW2, so
some of this is up to you for what you want to have supported with
XenProject.


>
> > - Since we're talking about sharing a libvirt plugin, I'm not clear on
> > if the shared work is done in a new libvirt plugin which is then
> > exposed to a KVM and a XenProject plugin or if the existing KVM plugin
> > is refactored to encompass both.
>
> Not totally sure what the best thing to do is — I’ll have to play with the
> code a little more.
>

I'd suggest looking at libvirt as a base class for KVM and XenProject and
see if that cleans anything up.  It might not, but worth looking into.


>
> Thanks,
> Dave
>
>
> >
> > -tim
> >
> >
> >
> >
> > On Mon, Jun 9, 2014 at 2:35 AM, Wido den Hollander 
> wrote:
> >>
> >>
> >> On 06/08/2014 11:14 PM, Dave Scott wrote:
> >>>
> >>> Hi Wido,
> >>>
> >>> Thanks for your mail!
> >>>
> >>> On 8 Jun 2014, at 19:02, Wido den Hollander  wrote:
> >>>
>  On 06/08/2014 06:23 PM, Dave Scott wrote:
> >
> > Hi,
> >
> > Following on from the earlier "[PROPOSAL] Support pure Xen as a
> > hypervisor”, I’ve added a design doc to the wiki:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Allow+hosts+running+the+Xen+hypervisor+to+be+managed+via+libvirt
> >
> > This design would allow people who want to manage their hypervisors
> > purely through the libvirt tools to choose the Xen hypervisor.
> >
> > From the code point of view, I want to maximise sharing between the
> KVM
> > and Xen code paths, partly to make QA easier and partly to maximise
> the
> > chance that adding a feature for “Xen” causes it to work for “KVM”
> and
> > vice-versa. In particular this means that, if a genuinely-useful
> capability
> > is currently missing from the libvirt libxl drive

Re: Need help on the CS First class object hiding feature.

[Nitin Mehta]

Answer inline

On 08/06/14 6:45 PM, "Girish Chaudhari" 
wrote:

>Hi Nitin,
>
>Can you please reply back to my previous mail thread.
>
>As well like to update you that I have even tried this feature with
>Networks object. Even in this case, I could set the display flag as
>'0', it get hided from normal users as well admin user similar to what
>I have mentioned above for virtual machine.

Root admin can still list the hidden networks by using the display flag in
the listNetworks.

>
>Thanks,
>Girish
>
>On Thu, Jun 5, 2014 at 9:18 AM, Girish Chaudhari
> wrote:
>> Thanks Nitin for immediate response.
>>
>> "... I see that some more first class entities have gotten added but
>> the underlying concept remains the same."
>>
>> => Can you please name the newly added first class entities if
>> possible, or what could be the best way to figure it out.
>>  Is there any criteria which is used to decided whether particular CS
>> resource is first class candidate or not. Or every resource can be
>> taken as first class object and hiding can be applied to it.
>> Can you please let me know what steps/efforts will be involved in
>> hiding new CS first class object using this feature?
>>
>> "I would think that would be a bug. Please file it and I will try and
>> look into it."
>>
>> => I will double check on it and try the same steps with another
>> mentioned resource. Will file the bug accordingly.
>>
>>>Also like to check whether Admin can update this feature flags or
>>>provide the meta data through UI?
>>
>>
>> "Yes, admin can update the display flag during creation time or using
>> update apis. Check deployvm and updateVm apis."
>>
>> -> You mean to say current CS UI(4.3 or upcoming 4.4) has ability to
>> mark the existing VM as hidden or create new VM with hide flag? I
>> could try it, using API programatically only.
>>
>> As well I have follow up extended questions as:
>>
>> Do this feature only applied to Root Admin vs the users Or even
>> equally applied to "Any Sub Domain Admin User" vs its non admin users?
>>
>>
>> Thanks,
>> Girish
>>
>> On Wed, Jun 4, 2014 at 8:19 PM, Nitin Mehta 
>>wrote:
>>>
>>>
>>> On 04/06/14 4:52 AM, "Girish Chaudhari"
>>>
>>> wrote:
>>>
Hi Team,

I am looking into the CS first class object hiding feature to verify
how the Admin user has better control over the display of CS resources
to end users.

The only reference link I could find is
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ability+to+have+
bet
ter+control+over+first+class+objects+in+CS

Is there any other documentation available around this feature
providing more use cases and examples..etc.
>>>
>>> This is the functional spec and should provide you most of the
>>> information. I see that some more first class entities have gotten
>>>added
>>> but the underlying concept remains the same.
>>>

Whereas I have tried to verify this feature by creating the
resources(VM) by Admin and associated it with the account. In usual
case Account user as well the admin user can retrieve the VM using the
listVMs call. When I am trying to set the display_vm flag as false, as
expected the account user don't see the VM. But even the Admin user
can't see the VM in list.

Not sure whether I am following the right steps or its buggy behavior.
>>>
>>> I would think that would be a bug. Please file it and I will try and
>>>look
>>> into it.
>>>
>>>
>>>

Also like to check whether Admin can update this feature flags or
provide the meta data through UI?
>>>
>>> Yes, admin can update the display flag during creation time or using
>>> update apis. Check deployvm and updateVm apis
>>>

Curious to know whether anyone is using this feature in production and
how?

Thanks,
Girish
>>>

Re: Need help on the CS First class object hiding feature.

[Nitin Mehta]

Answers inline

On 04/06/14 8:48 PM, "Girish Chaudhari" 
wrote:

>Thanks Nitin for immediate response.
>
>"... I see that some more first class entities have gotten added but
>the underlying concept remains the same."
>
>=> Can you please name the newly added first class entities if
>possible, or what could be the best way to figure it out.
> Is there any criteria which is used to decided whether particular CS
>resource is first class candidate or not. Or every resource can be
>taken as first class object and hiding can be applied to it.
>Can you please let me know what steps/efforts will be involved in
>hiding new CS first class object using this feature?

You can check the create/update/list apis and they should have display
flag as a parameter.


>
>"I would think that would be a bug. Please file it and I will try and
>look into it."
>
>=> I will double check on it and try the same steps with another
>mentioned resource. Will file the bug accordingly.
>
>>Also like to check whether Admin can update this feature flags or
>>provide the meta data through UI?
>
>
>"Yes, admin can update the display flag during creation time or using
>update apis. Check deployvm and updateVm apis."
>
>-> You mean to say current CS UI(4.3 or upcoming 4.4) has ability to
>mark the existing VM as hidden or create new VM with hide flag? I
>could try it, using API programatically only.

API doesn't have the ability currently. You can file a bug for that if it
doesn't already exist.

>
>As well I have follow up extended questions as:
>
>Do this feature only applied to Root Admin vs the users Or even
>equally applied to "Any Sub Domain Admin User" vs its non admin users?
>

Only for ROOT admins as far as hiding/exposing the resources are
concerned. The reason being the ROOT admin is the owner of the cloud and
should have this ability.
If you have a use case other than this please bring it forward and we can
discuss.

>
>Thanks,
>Girish
>
>On Wed, Jun 4, 2014 at 8:19 PM, Nitin Mehta 
>wrote:
>>
>>
>> On 04/06/14 4:52 AM, "Girish Chaudhari" 
>> wrote:
>>
>>>Hi Team,
>>>
>>>I am looking into the CS first class object hiding feature to verify
>>>how the Admin user has better control over the display of CS resources
>>>to end users.
>>>
>>>The only reference link I could find is
>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ability+to+have+b
>>>et
>>>ter+control+over+first+class+objects+in+CS
>>>
>>>Is there any other documentation available around this feature
>>>providing more use cases and examples..etc.
>>
>> This is the functional spec and should provide you most of the
>> information. I see that some more first class entities have gotten added
>> but the underlying concept remains the same.
>>
>>>
>>>Whereas I have tried to verify this feature by creating the
>>>resources(VM) by Admin and associated it with the account. In usual
>>>case Account user as well the admin user can retrieve the VM using the
>>>listVMs call. When I am trying to set the display_vm flag as false, as
>>>expected the account user don't see the VM. But even the Admin user
>>>can't see the VM in list.
>>>
>>>Not sure whether I am following the right steps or its buggy behavior.
>>
>> I would think that would be a bug. Please file it and I will try and
>>look
>> into it.
>>
>>
>>
>>>
>>>Also like to check whether Admin can update this feature flags or
>>>provide the meta data through UI?
>>
>> Yes, admin can update the display flag during creation time or using
>> update apis. Check deployvm and updateVm apis
>>
>>>
>>>Curious to know whether anyone is using this feature in production and
>>>how?
>>>
>>>Thanks,
>>>Girish
>>

[GSoC] [CLOUDSTACK-6114] Progress update

[Ian Duffy]

Hi All,

Started making some fast progress on this.

I uploaded the xenserver box[1] to vagrant cloud. This means people can
easily get a xenserver VM by executing vagrant init duffy/xenserver &&
vagrant up.

I adjusted the configuration on the box to allow for multiple xenserver
boxes to be brought up by one vagrant file. In order to allow this I had to
remove the host-only network configuration from the packaged box into an
external script [2]. It can be used as follows:

  config.vm.define "xenserver" do |xenserver|
xenserver.vm.box = "duffy/xenserver"

xenserver.vm.provision "shell" do |s|
  s.path = "xenserver/reset-network.sh"
  s.args = ["eth1", "192.168.56.10", "255.255.255.0"]
end
  end


I created a skeleton for the chef cookbook [3]. At the moment it include
calls to the MySQL, NFS and IPTables gateway recipes with default
attributes specified for the included devcloud.cfg. Along with this I wrote
a systemvm downloading recipe which reads an array of systemvms from the
cookbooks attributes file and downloads/installed them accordingly. It uses
the -t switch to on the cloud-install-sys-tmplt script to avoid querying
the MySQL db for information.

I added some brief usage documentation too [4]


[1] https://github.com/imduffy15/packer-xenserver
[2]
https://github.com/imduffy15/GSoC-2014/blob/master/MySql_NFS_XenServer/xenserver/reset-network.sh
[3] https://github.com/imduffy15/cookbook_cloudstack
[4] https://github.com/imduffy15/GSoC-2014/blob/master/README.md

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Mike Tutkowski]

Thanks, Hieu!

I have reviewed your design (making only minor changes to your Wiki).

Please feel free to have me review your code when you are ready.

Also, do you have a plan for integration testing? It would be great if you
could update your Wiki page to include what your plans on in this regard.

Thanks!
Mike


On Mon, Jun 9, 2014 at 4:24 AM, Hieu LE  wrote:

> Hi guys,
>
> I have updated this proposal wiki[1], included diagram for VM migrate,
> volume migrate and snapshot.
>
> Please review and give feedback.
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
>
>
> On Fri, Jun 6, 2014 at 7:14 PM, Todd Pigram  wrote:
>
> > Sorry, thought you were based off the link you provided in this reply.
> >
> > "In our case, we are using CloudStack integrated in VDI solution to
> > provived
> > pooled VM type[1]. So may be my approach can bring better UX for user
> with
> > lower bootime ...
> >
> > A short change in design are followings
> > - VM will be deployed with golden primary storage if primary storage is
> > marked golden and this VM template is also marked as golden.
> > - Choosing the best deploy destionation for both golden primary storage
> and
> > normal root volume primary storage. Chosen host can also access both
> > storage pools.
> > - New Xen Server plug-in for modifying VHD parent id.
> >
> > Is there some place for me to submit my design and code. Can I write a
> new
> > proposal in CS wiki ?
> >
> > [1]:
> >
> >
> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html
> >  "
> >
> >
> > On Thu, Jun 5, 2014 at 11:55 PM, Hieu LE  wrote:
> >
> > > Hi Todd,
> > >
> > >
> > > On Fri, Jun 6, 2014 at 9:17 AM, Todd Pigram 
> wrote:
> > >
> > > > Hieu,
> > > >
> > > > I assume you are using MCS for you golden image? What version of XD?
> > > Given
> > > > you are using pooled desktops, have you thought about using a PVS BDM
> > iso
> > > > and mount it with in your 1000 VMs? This way you can stagger reboots
> > via
> > > > PVS console or Studio. This would require a change to your delivery
> > > group.
> > > >
> > > >
> > > Sorry but I did not use MCS or XenDesktop in my company :-)
> > >
> > >
> > > >
> > > > On Thu, Jun 5, 2014 at 9:28 PM, Mike Tutkowski <
> > > > mike.tutkow...@solidfire.com
> > > > > wrote:
> > > >
> > > > > 6) The copy_vhd_from_secondarystorage XenServer plug-in is not used
> > > when
> > > > > you're using XenServer + XS62ESP1 + XS62ESP1004. In that case,
> please
> > > > refer
> > > > > to copyTemplateToPrimaryStorage(CopyCommand) method in the
> > > > > Xenserver625StorageProcessor class.
> > > > >
> > > >
> > >
> > > Thank Mike, I will take note of that.
> > >
> > >
> > > >  >
> > > > > On Thu, Jun 5, 2014 at 1:56 PM, Mike Tutkowski <
> > > > > mike.tutkow...@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Other than going through a "for" loop and deploying VM after VM,
> I
> > > > don't
> > > > > > think CloudStack currently supports a bulk-VM-deploy operation.
> > > > > >
> > > > > > It would be nice if CS did so at some point in the future;
> however,
> > > > that
> > > > > > is probably a separate proposal from Hieu's.
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 5, 2014 at 12:13 AM, Amit Das <
> amit@cloudbyte.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Hieu,
> > > > > >>
> > > > > >> Will it be good to include bulk operation of this feature? In
> > > > addition,
> > > > > >> does Xen support parallel execution of these operations ?
> > > > > >>
> > > > > >> Regards,
> > > > > >> Amit
> > > > > >> *CloudByte Inc.* 
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE 
> > wrote:
> > > > > >>
> > > > > >> > Mike, Punith,
> > > > > >> >
> > > > > >> > Please review "Golden Primary Storage" proposal. [1]
> > > > > >> >
> > > > > >> > Thank you.
> > > > > >> >
> > > > > >> > [1]:
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Jun 4, 2014 at 10:32 PM, Mike Tutkowski <
> > > > > >> > mike.tutkow...@solidfire.com> wrote:
> > > > > >> >
> > > > > >> >> Daan helped out with this. You should be good to go now.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Tue, Jun 3, 2014 at 8:50 PM, Hieu LE 
> > > > wrote:
> > > > > >> >>
> > > > > >> >> > Hi Mike,
> > > > > >> >> >
> > > > > >> >> > Could you please give edit/create permission on ASF
> Jira/Wiki
> > > > > >> >> confluence ?
> > > > > >> >> > I can not add a new Wiki page.
> > > > > >> >> >
> > > > > >> >> > My Jira ID: hieulq
> > > > > >> >> > Wiki: hieulq89
> > > > > >> >> > Review Board: hieulq
> > > > > >> >> >
> > > > > >> >> > Thanks !
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Wed, Jun 4, 2014 at 9:17 AM, Mike Tutkowski <
> > > > > >> >> > mike.tutkow...@solidfire.com
> > > > > >> >> > > wrote:
> > > > > >> >> >
> > > > 

deleting or cancelling broken ACS jobs

[Andrei Mikhailovsky]

Hello guys, 

was wondering if anyone have come across an issue where acs would get stuck on 
several jobs and keeps trying to do them over and over again? 

I've come across an issue a few days ago. For some reason I have about 5 or 6 
XenServer cluster jobs which have gone crazy. These jobs are of different 
nature, like template creation, vm start and enable host maintenance. 
They keep on repeating in the logs about 20-30 times a second, causing 
overfilling of logs. I get about 20GB of management server logs each day and it 
seems that these stuck jobs are causing the overflow. I am also not able to 
perform any activity on the XenServer cluster which has those stuck jobs. I am 
unable to start or stop jobs or pretty much do anything with it. 

I've tried restarting both the management server and the xenserver hosts, but 
that didn't help. After a short while following a restart the same thing starts 
to happen. 

Is there a way for ACS to cancel / remove these jobs? I've looked at the 
async_job and async_job_view db tables and I can see 28 entries there amongst 
which are these stuck jobs gone crazy. Is it safe for me to simply remove them 
from the database and restart the management server? Are there any other db 
tables that I should look at? 

Many thanks 

Andrei 

Re: deleting or cancelling broken ACS jobs

[Rafael Weingartner]

have you seem this thread?
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%3cblu176-w37d2e5fc141028f7d1937db4...@phx.gbl%3E
I think this might work to delete the broken tasks.


On Mon, Jun 9, 2014 at 5:35 PM, Andrei Mikhailovsky 
wrote:

> Hello guys,
>
> was wondering if anyone have come across an issue where acs would get
> stuck on several jobs and keeps trying to do them over and over again?
>
> I've come across an issue a few days ago. For some reason I have about 5
> or 6 XenServer cluster jobs which have gone crazy. These jobs are of
> different nature, like template creation, vm start and enable host
> maintenance.
> They keep on repeating in the logs about 20-30 times a second, causing
> overfilling of logs. I get about 20GB of management server logs each day
> and it seems that these stuck jobs are causing the overflow. I am also not
> able to perform any activity on the XenServer cluster which has those stuck
> jobs. I am unable to start or stop jobs or pretty much do anything with it.
>
> I've tried restarting both the management server and the xenserver hosts,
> but that didn't help. After a short while following a restart the same
> thing starts to happen.
>
> Is there a way for ACS to cancel / remove these jobs? I've looked at the
> async_job and async_job_view db tables and I can see 28 entries there
> amongst which are these stuck jobs gone crazy. Is it safe for me to simply
> remove them from the database and restart the management server? Are there
> any other db tables that I should look at?
>
> Many thanks
>
> Andrei
>
>
>


-- 
Rafael Weingärtner

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Tim Mackey]

Hieu,

I made a couple of minor edits to your design to ensure everything is
"XenServer" based.  If you haven't done so already, please also fetch the
most recent master and base off of that.  I refactored the old Xen plugin
into a XenServer specific one since Xen Project isn't currently supported,
and files have moved.  Also please ensure you don't use the term "Xen" in
your code/docs to avoid any future confusion when the Xen Project work
starts to materialize.

Looking forward to seeing your work!!

-tim


On Mon, Jun 9, 2014 at 4:31 PM, Mike Tutkowski  wrote:

> Thanks, Hieu!
>
> I have reviewed your design (making only minor changes to your Wiki).
>
> Please feel free to have me review your code when you are ready.
>
> Also, do you have a plan for integration testing? It would be great if you
> could update your Wiki page to include what your plans on in this regard.
>
> Thanks!
> Mike
>
>
> On Mon, Jun 9, 2014 at 4:24 AM, Hieu LE  wrote:
>
> > Hi guys,
> >
> > I have updated this proposal wiki[1], included diagram for VM migrate,
> > volume migrate and snapshot.
> >
> > Please review and give feedback.
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> >
> >
> > On Fri, Jun 6, 2014 at 7:14 PM, Todd Pigram  wrote:
> >
> > > Sorry, thought you were based off the link you provided in this reply.
> > >
> > > "In our case, we are using CloudStack integrated in VDI solution to
> > > provived
> > > pooled VM type[1]. So may be my approach can bring better UX for user
> > with
> > > lower bootime ...
> > >
> > > A short change in design are followings
> > > - VM will be deployed with golden primary storage if primary storage is
> > > marked golden and this VM template is also marked as golden.
> > > - Choosing the best deploy destionation for both golden primary storage
> > and
> > > normal root volume primary storage. Chosen host can also access both
> > > storage pools.
> > > - New Xen Server plug-in for modifying VHD parent id.
> > >
> > > Is there some place for me to submit my design and code. Can I write a
> > new
> > > proposal in CS wiki ?
> > >
> > > [1]:
> > >
> > >
> >
> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html
> > >  "
> > >
> > >
> > > On Thu, Jun 5, 2014 at 11:55 PM, Hieu LE  wrote:
> > >
> > > > Hi Todd,
> > > >
> > > >
> > > > On Fri, Jun 6, 2014 at 9:17 AM, Todd Pigram 
> > wrote:
> > > >
> > > > > Hieu,
> > > > >
> > > > > I assume you are using MCS for you golden image? What version of
> XD?
> > > > Given
> > > > > you are using pooled desktops, have you thought about using a PVS
> BDM
> > > iso
> > > > > and mount it with in your 1000 VMs? This way you can stagger
> reboots
> > > via
> > > > > PVS console or Studio. This would require a change to your delivery
> > > > group.
> > > > >
> > > > >
> > > > Sorry but I did not use MCS or XenDesktop in my company :-)
> > > >
> > > >
> > > > >
> > > > > On Thu, Jun 5, 2014 at 9:28 PM, Mike Tutkowski <
> > > > > mike.tutkow...@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > 6) The copy_vhd_from_secondarystorage XenServer plug-in is not
> used
> > > > when
> > > > > > you're using XenServer + XS62ESP1 + XS62ESP1004. In that case,
> > please
> > > > > refer
> > > > > > to copyTemplateToPrimaryStorage(CopyCommand) method in the
> > > > > > Xenserver625StorageProcessor class.
> > > > > >
> > > > >
> > > >
> > > > Thank Mike, I will take note of that.
> > > >
> > > >
> > > > >  >
> > > > > > On Thu, Jun 5, 2014 at 1:56 PM, Mike Tutkowski <
> > > > > > mike.tutkow...@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Other than going through a "for" loop and deploying VM after
> VM,
> > I
> > > > > don't
> > > > > > > think CloudStack currently supports a bulk-VM-deploy operation.
> > > > > > >
> > > > > > > It would be nice if CS did so at some point in the future;
> > however,
> > > > > that
> > > > > > > is probably a separate proposal from Hieu's.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 5, 2014 at 12:13 AM, Amit Das <
> > amit@cloudbyte.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Hieu,
> > > > > > >>
> > > > > > >> Will it be good to include bulk operation of this feature? In
> > > > > addition,
> > > > > > >> does Xen support parallel execution of these operations ?
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Amit
> > > > > > >> *CloudByte Inc.* 
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Jun 5, 2014 at 8:59 AM, Hieu LE 
> > > wrote:
> > > > > > >>
> > > > > > >> > Mike, Punith,
> > > > > > >> >
> > > > > > >> > Please review "Golden Primary Storage" proposal. [1]
> > > > > > >> >
> > > > > > >> > Thank you.
> > > > > > >> >
> > > > > > >> > [1]:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Jun 

Re: [DISCUSS] Increasing VM IOPS by separating golden image in high IOPS partition in Xen Server ?

[Mike Tutkowski]

Yes, I was going to mention what Tim said about using the term "XenServer"
instead of "Xen" as Tim has done a bunch of work recently to separate the
two.

I made a few changes in your Wiki when I saw a reference to "Xen" instead
of to "XenServer."


On Mon, Jun 9, 2014 at 2:53 PM, Tim Mackey  wrote:

> Hieu,
>
> I made a couple of minor edits to your design to ensure everything is
> "XenServer" based.  If you haven't done so already, please also fetch the
> most recent master and base off of that.  I refactored the old Xen plugin
> into a XenServer specific one since Xen Project isn't currently supported,
> and files have moved.  Also please ensure you don't use the term "Xen" in
> your code/docs to avoid any future confusion when the Xen Project work
> starts to materialize.
>
> Looking forward to seeing your work!!
>
> -tim
>
>
> On Mon, Jun 9, 2014 at 4:31 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com
> > wrote:
>
> > Thanks, Hieu!
> >
> > I have reviewed your design (making only minor changes to your Wiki).
> >
> > Please feel free to have me review your code when you are ready.
> >
> > Also, do you have a plan for integration testing? It would be great if
> you
> > could update your Wiki page to include what your plans on in this regard.
> >
> > Thanks!
> > Mike
> >
> >
> > On Mon, Jun 9, 2014 at 4:24 AM, Hieu LE  wrote:
> >
> > > Hi guys,
> > >
> > > I have updated this proposal wiki[1], included diagram for VM migrate,
> > > volume migrate and snapshot.
> > >
> > > Please review and give feedback.
> > >
> > > [1]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Golden+Primary+Storage
> > >
> > >
> > > On Fri, Jun 6, 2014 at 7:14 PM, Todd Pigram 
> wrote:
> > >
> > > > Sorry, thought you were based off the link you provided in this
> reply.
> > > >
> > > > "In our case, we are using CloudStack integrated in VDI solution to
> > > > provived
> > > > pooled VM type[1]. So may be my approach can bring better UX for user
> > > with
> > > > lower bootime ...
> > > >
> > > > A short change in design are followings
> > > > - VM will be deployed with golden primary storage if primary storage
> is
> > > > marked golden and this VM template is also marked as golden.
> > > > - Choosing the best deploy destionation for both golden primary
> storage
> > > and
> > > > normal root volume primary storage. Chosen host can also access both
> > > > storage pools.
> > > > - New Xen Server plug-in for modifying VHD parent id.
> > > >
> > > > Is there some place for me to submit my design and code. Can I write
> a
> > > new
> > > > proposal in CS wiki ?
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-choose-scheme-type-rho.html
> > > >  "
> > > >
> > > >
> > > > On Thu, Jun 5, 2014 at 11:55 PM, Hieu LE  wrote:
> > > >
> > > > > Hi Todd,
> > > > >
> > > > >
> > > > > On Fri, Jun 6, 2014 at 9:17 AM, Todd Pigram 
> > > wrote:
> > > > >
> > > > > > Hieu,
> > > > > >
> > > > > > I assume you are using MCS for you golden image? What version of
> > XD?
> > > > > Given
> > > > > > you are using pooled desktops, have you thought about using a PVS
> > BDM
> > > > iso
> > > > > > and mount it with in your 1000 VMs? This way you can stagger
> > reboots
> > > > via
> > > > > > PVS console or Studio. This would require a change to your
> delivery
> > > > > group.
> > > > > >
> > > > > >
> > > > > Sorry but I did not use MCS or XenDesktop in my company :-)
> > > > >
> > > > >
> > > > > >
> > > > > > On Thu, Jun 5, 2014 at 9:28 PM, Mike Tutkowski <
> > > > > > mike.tutkow...@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > 6) The copy_vhd_from_secondarystorage XenServer plug-in is not
> > used
> > > > > when
> > > > > > > you're using XenServer + XS62ESP1 + XS62ESP1004. In that case,
> > > please
> > > > > > refer
> > > > > > > to copyTemplateToPrimaryStorage(CopyCommand) method in the
> > > > > > > Xenserver625StorageProcessor class.
> > > > > > >
> > > > > >
> > > > >
> > > > > Thank Mike, I will take note of that.
> > > > >
> > > > >
> > > > > >  >
> > > > > > > On Thu, Jun 5, 2014 at 1:56 PM, Mike Tutkowski <
> > > > > > > mike.tutkow...@solidfire.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Other than going through a "for" loop and deploying VM after
> > VM,
> > > I
> > > > > > don't
> > > > > > > > think CloudStack currently supports a bulk-VM-deploy
> operation.
> > > > > > > >
> > > > > > > > It would be nice if CS did so at some point in the future;
> > > however,
> > > > > > that
> > > > > > > > is probably a separate proposal from Hieu's.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 5, 2014 at 12:13 AM, Amit Das <
> > > amit@cloudbyte.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Hieu,
> > > > > > > >>
> > > > > > > >> Will it be good to include bulk operation of this feature?
> In
> > > > > > addition,
> > > > > > > >> does Xen support parallel execution of these operations ?
> 

Re: deleting or cancelling broken ACS jobs

[Andrei Mikhailovsky]

Raf, 

Not sure if this thread is related to my problem. I have a single management 
server with a few messed up tasks. 

Is it safe to remove everything from the async_job and async_job_view tables 
(apart from any current jobs)? 

Thanks 

Andrei 

- Original Message -

From: "Rafael Weingartner"  
To: dev@cloudstack.apache.org 
Sent: Monday, 9 June, 2014 9:42:24 PM 
Subject: Re: deleting or cancelling broken ACS jobs 

have you seem this thread? 
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%3cblu176-w37d2e5fc141028f7d1937db4...@phx.gbl%3E
 
I think this might work to delete the broken tasks. 


On Mon, Jun 9, 2014 at 5:35 PM, Andrei Mikhailovsky  
wrote: 

> Hello guys, 
> 
> was wondering if anyone have come across an issue where acs would get 
> stuck on several jobs and keeps trying to do them over and over again? 
> 
> I've come across an issue a few days ago. For some reason I have about 5 
> or 6 XenServer cluster jobs which have gone crazy. These jobs are of 
> different nature, like template creation, vm start and enable host 
> maintenance. 
> They keep on repeating in the logs about 20-30 times a second, causing 
> overfilling of logs. I get about 20GB of management server logs each day 
> and it seems that these stuck jobs are causing the overflow. I am also not 
> able to perform any activity on the XenServer cluster which has those stuck 
> jobs. I am unable to start or stop jobs or pretty much do anything with it. 
> 
> I've tried restarting both the management server and the xenserver hosts, 
> but that didn't help. After a short while following a restart the same 
> thing starts to happen. 
> 
> Is there a way for ACS to cancel / remove these jobs? I've looked at the 
> async_job and async_job_view db tables and I can see 28 entries there 
> amongst which are these stuck jobs gone crazy. Is it safe for me to simply 
> remove them from the database and restart the management server? Are there 
> any other db tables that I should look at? 
> 
> Many thanks 
> 
> Andrei 
> 
> 
> 


-- 
Rafael Weingärtner 

Re: redundant virtual routers for VPCs

[Sander Botman]

Hi all

I can make 15:00.

Cannon join at 20:00.

Cheers Sander

Verstuurd vanaf mijn iPhone

> Op 9 jun. 2014 om 21:31 heeft "Daan Hoogland"  het 
> volgende geschreven:
> 
> nice,
> 
> so given pacific time I can have a meeting at 15:00 utc or at 20:00
> utc (8 am or 2 pm if I'm correct). Given office hours in Holland 17:00
> (15:00 utc) would be best but if you're not in it could be later. As
> the person calling the meeting I would like to participate but if the
> highest attendance can be found only between 16 and 22 utc thats fine
> as well.
> 
>> On Mon, Jun 9, 2014 at 8:35 PM, Karl Harris  
>> wrote:
>> I will be happy to join!
>> 
>> 
>> 
>>> On Saturday, June 7, 2014, Daan Hoogland  wrote:
>>> 
>>> At Schuberg Philis, the urge to have virtual routers in a redundant
>>> way is getting to be pressing. It seems Citrix wants to move away from
>>> it entirely and Sungard is working on it but has to little resources
>>> for it. Withing our dev group we are now planning a sprint to
>>> experiment with this subject and then implement or fix the present
>>> routers to support this. However we would very much like input from
>>> the people who have already worked on this and are also very willing
>>> to let them in our sprint-team if they are willing.
>>> 
>>> So: who want to join in a irc meeting (or conf call) on the subject
>>> coming week? Dutch people are not in the office on the day after
>>> ascension so tuesday is our intention.
>>> 
>>> on behalf of some very impatient operators;)
>>> --
>>> Daan
> 
> 
> 
> -- 
> Daan

Re: deleting or cancelling broken ACS jobs

[Rafael Weingartner]

Well, I think it is safe, but before you do that, put disabe your zones,
shutdown the MS and then remove the tasks.
Or, you could simple look for some task ids that are giving you trouble and
then delete them.


On Mon, Jun 9, 2014 at 6:37 PM, Andrei Mikhailovsky 
wrote:

> Raf,
>
> Not sure if this thread is related to my problem. I have a single
> management server with a few messed up tasks.
>
> Is it safe to remove everything from the async_job and async_job_view
> tables (apart from any current jobs)?
>
> Thanks
>
> Andrei
>
> - Original Message -
>
> From: "Rafael Weingartner" 
> To: dev@cloudstack.apache.org
> Sent: Monday, 9 June, 2014 9:42:24 PM
> Subject: Re: deleting or cancelling broken ACS jobs
>
> have you seem this thread?
>
> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/%3cblu176-w37d2e5fc141028f7d1937db4...@phx.gbl%3E
> I think this might work to delete the broken tasks.
>
>
> On Mon, Jun 9, 2014 at 5:35 PM, Andrei Mikhailovsky 
> wrote:
>
> > Hello guys,
> >
> > was wondering if anyone have come across an issue where acs would get
> > stuck on several jobs and keeps trying to do them over and over again?
> >
> > I've come across an issue a few days ago. For some reason I have about 5
> > or 6 XenServer cluster jobs which have gone crazy. These jobs are of
> > different nature, like template creation, vm start and enable host
> > maintenance.
> > They keep on repeating in the logs about 20-30 times a second, causing
> > overfilling of logs. I get about 20GB of management server logs each day
> > and it seems that these stuck jobs are causing the overflow. I am also
> not
> > able to perform any activity on the XenServer cluster which has those
> stuck
> > jobs. I am unable to start or stop jobs or pretty much do anything with
> it.
> >
> > I've tried restarting both the management server and the xenserver hosts,
> > but that didn't help. After a short while following a restart the same
> > thing starts to happen.
> >
> > Is there a way for ACS to cancel / remove these jobs? I've looked at the
> > async_job and async_job_view db tables and I can see 28 entries there
> > amongst which are these stuck jobs gone crazy. Is it safe for me to
> simply
> > remove them from the database and restart the management server? Are
> there
> > any other db tables that I should look at?
> >
> > Many thanks
> >
> > Andrei
> >
> >
> >
>
>
> --
> Rafael Weingärtner
>
>


-- 
Rafael Weingärtner

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[David Nalley]

On Mon, Jun 9, 2014 at 3:14 PM, Rohit Yadav  wrote:
> On Mon, Jun 9, 2014 at 11:32 PM, sebgoa  wrote:
>
>>
>> On Jun 9, 2014, at 7:13 PM, David Nalley  wrote:
>>
>> > On Mon, Jun 9, 2014 at 11:47 AM, David Nalley  wrote:
>> >> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang  wrote:
>> >>> Hi all,
>> >>>
>> >>> Seems it's a good timing to bring back the discussion about the gerrit.
>> >>>
>> >>> We want to do CI, and improve our code quality. One obvious way of
>> doing
>> >>> and reduce the workload of devs is introduce a tool to enforce the
>> process.
>> >>>
>> >>> I've checked out quite a few projects using gerrit, which would force
>> you
>> >>> to ask for review, and validation before the code can be committed to
>> the
>> >>> repo. Looks it's really a easier way for devs according what I've
>> heard.
>> >>>
>> >>> Even our competitor laid out a very detail workflow based on the use of
>> >>> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it
>> can
>> >>> make a good reference.
>> >>>
>> >>> Well, gerrit has been brought up a few times before. And now the new
>> >>> process we want to enforce just fits what gerrit(or other automation
>> >>> review/test/commit software) is for.
>> >>>
>> >>> Maybe it's the time for us to review the possibility of using a tool to
>> >>> enforce our commits and improve our code quality(as well as transfer
>> >>> knowledge) again?
>> >>>
>> >>> --Sheng
>> >>>
>> >>
>> >> ASF Infra has a very dour view on Gerrit. Don't read that as
>> >> impossible; there are many projects at the ASF who are interested in
>> >> Gerrit.
>> >> That said; what about moving to using github pull requests instead of
>> >> RB, and from their, having the jenkins pull request builder
>> >> automatically process every pull request and list information.
>> >>
>> >> Here's an example:
>> >> https://github.com/jclouds/jclouds-labs/pull/61
>> >> You'll see that every time the patch changes, the jenkins plugin
>> >> pulled the patch - ran tests against it and reported back.
>> >>
>> >> That said; it almost seems like we have the cart before the horse; we
>> >> need to finish figuring out the CI Infrastructure first.
>> >>
>> >> --David
>> >
>> > Just as an additional comment.
>> > CF was using gerrit, and apparently dropped it in favor of GH Pull
>> > Requests plus automated testing with Travis-CI.
>> > I know folks like abayer and jfarrell are working on turning on
>> > automated builds via the github pull request plugin for jenkins. Might
>> > be something to consider.
>>
>> fwiw, I have my personal cloudstack fork on github with a simple travisCI
>> setup.
>> So when I make a commit it automatically triggers a build.
>> I was working on running the integration tests on Travis as well but got
>> distracted.
>>
>> If somebody wants to step in:
>>
>> https://github.com/runseb/cloudstack/blob/master/.travis.yml
>
>
> This would be great if we could just use Github (pull requests, releases)
> and hook up Travis-CI. Both are free for opensource projects.
>

To stop us from going down a bad road.
We have to use the ASF repo as the canonical repo. We can use the
github mirrors.

Also - I am not sure if travis-ci would give us free services when the
CI stuff we are talking about needing is well into 6 figures.

--David

Re: SNAT iptable entry on VirtualRouter

[Sheng Yang]

correctVif is not null would means the network you want to is already
created.

Then ip would get device id later from:

ip.setNicDevId(Integer.valueOf(correctVif.getDevice(conn)));

So seems your correctVif is wrong with public network.

I am not very familiar with hypervisor part of find the correct network.

Anthony, could you help to take a look?

And yes, the command would also executed when you have any additional
public IPs to be associate with VR, like static nat and port forwarding.
The new interface would be only created if the public IPs are in the
different vlans.

--Sheng


On Sun, Jun 8, 2014 at 6:47 AM, Sachchidanand Vaidya 
wrote:

>  Hi Sheng,
>Thanks. We are using XenServer 6.2 SP1. After adding debug in the code,
> I see that addVif never gets called  in my case.
> That's why nic deviceId is zero. Code falls thru the case where vif is
> already present.If I dump "correctVif.getDevice(conn)"
> it returns zero.
>
>  Does this command handler also gets called when Public-ip is associated
> with VM's private ip (StaticNAT) ?
> Do we create a new interface in DomainRouter when staticNAT entry is
> created?
>
>  Thanks,
> Sachin
>
>   From: Sheng Yang 
> Date: Friday, June 6, 2014 4:12 PM
> To: "" ,
> Sachchidanand Vaidya 
> Subject: Re: SNAT iptable entry on VirtualRouter
>
>   Hi Sachin,
>
>  The nicDevId() you see is coming from
> prepareNetworkElementCommand(IpAssocCommand cmd) in CitrixResourceBase in
> case of Xen.
>
>  You would see this:
>
>  if (addVif) {
> // Add a new VIF to DomR
> String vifDeviceNum =
> getLowestAvailableVIFDeviceNum(conn, router);
>
>  if (vifDeviceNum == null) {
> throw new InternalErrorException("There were no
> more available slots for a new VIF on router: " +
> router.getNameLabel(conn));
> }
>
>  nic.setDeviceId(Integer.valueOf(vifDeviceNum));
>
>  correctVif = createVif(conn, routerName, router,
> null, nic);
> correctVif.plug(conn);
> // Add iptables rule for network usage
> networkUsage(conn, routerIp, "addVif", "eth" +
> correctVif.getDevice(conn));
> }
>
>  And nic.setDeviceId() should set the public nic id(which should be 2 in
> your case) to it.
>
>  And what's the XenServer version you're using? Could you help to debug
> it further more? Sadly we cannot reproduce it in our lab...
>
>  Thanks!
>
>  --Sheng
>
>
> On Fri, Jun 6, 2014 at 12:29 AM, Sachchidanand Vaidya <
> vaidy...@juniper.net> wrote:
>
>> Hi,
>>I'm seeing the same issue with 4.4 code.  After further debug, I see
>> that CS mgmt server is sending
>> following command to XenHost,
>> xensource.log:  /opt/cloud/bin/ipassoc.sh -A -s -f -l
>> 10.84.59.131/24 -c eth0 -g 10.84.59.254 
>> VirtualRouter's public interface is eth2. Also as per dump of VIF list on
>> XenHost, deviceid for public interface
>> of domainRouter is 2.
>> As part of VirtualRoutingResource.java:generateConfig(), CS mgmt server
>> generates this command.
>> It generates publicNic = "eth" + ip.getNicDevId()?
>> Which deviceId does it refer to? Shouldn't it be the deviceid as per the
>> XenHost dump?
>> Does anyone have input on what could he happening here ?
>>
>> Thanks,
>> Sachin
>>
>>
>>
>> >Hi,
>> >I have an isolated network (192.168.3.x/24) being served by
>> >VirtualRouter, where 10.84.59.131 is SourceNAT address
>> >and eth0 is VN interface of VirtualRouter & eth2 is  the public interface
>> >of VirtualRouter.
>> >
>> > When I look at the nat table entries on the VirtualRouter, it shows
>> >following :
>> >
>> >root@r-6-VM:~# iptables -L -t nat -n -v
>> >..
>> >..
>> >Chain POSTROUTING (policy ACCEPT 330 packets, 22113 bytes)
>> > pkts bytes target prot opt in out source
>> >destination
>> >0 0 SNAT   all  --  *  eth00.0.0.0/0
>> >0.0.0.0/0to:10.84.59.131
>> >
>> >--> Why the "out" interface for the SNAT entry is VN interface (eth0)
>> >instead of Public interface (eth2) ?
>> >
>> >I'm using "Cloudstack Release 4.3.0 (64-bit) Thu Apr 10 20:27:11 UTC
>> >2014" cloudstack-release template.
>> >
>> >Thanks,
>> >Sachin
>> >
>> >---
>> >root@r-6-VM:~# ifconfig
>> >eth0  Link encap:Ethernet  HWaddr 02:13:87:88:e6:dd
>> >  inet addr:192.168.3.226  Bcast:192.168.3.255
>>  Mask:255.255.255.0
>> >  inet6 addr: fe80::13:87ff:fe88:e6dd/64 Scope:Link
>> >  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>> >  RX packets:350 errors:0 dropped:0 overruns:0 frame:0
>> >  TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
>> >  collisions:0 txqueuelen:1000
>> >  RX bytes:29400 (28.7 KiB)  TX bytes:602 (602.0 B)
>> >  Interrupt:25
>> >
>> >eth1  Link encap:Ethernet  HWaddr 0e:00:a9:fe:02:6b
>> >  inet addr:169.254.2.107  Bcast:169.254.255.255
>>  Mask

Re: redundant virtual routers for VPCs

[Sheng Yang]

BTW, do we have an agenda or some topics we want to discuss for the
meeting? I just want to make sure it's efficient.

--Sheng


On Mon, Jun 9, 2014 at 1:14 PM, Sander Botman 
wrote:

> Hi all
>
> I can make 15:00.
>
> Cannon join at 20:00.
>
> Cheers Sander
>
> Verstuurd vanaf mijn iPhone
>
> > Op 9 jun. 2014 om 21:31 heeft "Daan Hoogland" 
> het volgende geschreven:
> >
> > nice,
> >
> > so given pacific time I can have a meeting at 15:00 utc or at 20:00
> > utc (8 am or 2 pm if I'm correct). Given office hours in Holland 17:00
> > (15:00 utc) would be best but if you're not in it could be later. As
> > the person calling the meeting I would like to participate but if the
> > highest attendance can be found only between 16 and 22 utc thats fine
> > as well.
> >
> >> On Mon, Jun 9, 2014 at 8:35 PM, Karl Harris 
> wrote:
> >> I will be happy to join!
> >>
> >>
> >>
> >>> On Saturday, June 7, 2014, Daan Hoogland 
> wrote:
> >>>
> >>> At Schuberg Philis, the urge to have virtual routers in a redundant
> >>> way is getting to be pressing. It seems Citrix wants to move away from
> >>> it entirely and Sungard is working on it but has to little resources
> >>> for it. Withing our dev group we are now planning a sprint to
> >>> experiment with this subject and then implement or fix the present
> >>> routers to support this. However we would very much like input from
> >>> the people who have already worked on this and are also very willing
> >>> to let them in our sprint-team if they are willing.
> >>>
> >>> So: who want to join in a irc meeting (or conf call) on the subject
> >>> coming week? Dutch people are not in the office on the day after
> >>> ascension so tuesday is our intention.
> >>>
> >>> on behalf of some very impatient operators;)
> >>> --
> >>> Daan
> >
> >
> >
> > --
> > Daan
>

[ACS44] Cherry pick request

[Amogh Vasekar]

Hi Daan,

Request you to please cherry-pick the following two commits to 4.4 branch :

1. ac92b3690304ff224e7e2530ea7d8e39f28a05c3 for CLOUDSTACK-6710
2. a4b401f29f83f2f0b467a9d05b509f951b5a3bca for CLOUDSTACK-6358

Thanks,
Amogh

Re: Review Request 22364: Fixed 26 issues reported by coverity

[Rajani Karuturi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22364/
---

(Updated June 10, 2014, 4:06 a.m.)


Review request for cloudstack, daan Hoogland, Kelven Yang, Koushik Das, and 
Santhosh Edukulla.


Repository: cloudstack-git


Description
---

NPEs, unused code or dead code, unwritten field access and self assignment


Diffs
-

  engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java 25c67db 

Diff: https://reviews.apache.org/r/22364/diff/


Testing
---


Thanks,

Rajani Karuturi

FW: New Defects reported by Coverity Scan for cloudstack

[Animesh Chaturvedi]

Folks please check on the issues reported.

> -Original Message-
> From: scan-ad...@coverity.com [mailto:scan-ad...@coverity.com]
> Sent: Monday, June 09, 2014 7:37 PM
> Subject: New Defects reported by Coverity Scan for cloudstack
> 
> 
> Hi,
> 
> 
> Please find the latest report on new defect(s) introduced to cloudstack
> found with Coverity Scan.
> 
> Defect(s) Reported-by: Coverity Scan
> Showing 20 of 67 defect(s)
> 
> 
> ** CID 1222156:  Value not atomically updated  (ATOMICITY)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 5272 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(com.cl
> oud.agent.api.OvsDestroyBridgeCommand)()
> 
> ** CID 1222157:  Dereference after null check  (FORWARD_NULL)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 1097 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetwork(co
> m.xensource.xenapi.Connection, com.cloud.agent.api.to.NicTO)()
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 1088 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetwork(co
> m.xensource.xenapi.Connection, com.cloud.agent.api.to.NicTO)()
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 1088 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetwork(co
> m.xensource.xenapi.Connection, com.cloud.agent.api.to.NicTO)()
> 
> ** CID 1222175:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 3025 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(com.cl
> oud.agent.api.MigrateCommand)()
> 
> ** CID 1222174:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 6347 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(com.cl
> oud.agent.api.AttachVolumeCommand)()
> 
> ** CID 1222173:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 6285 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(com.cl
> oud.agent.api.AttachVolumeCommand)()
> 
> ** CID 1222172:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 7200 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.clusterVMMet
> aDataSync(com.xensource.xenapi.Connection)()
> 
> ** CID 1222171:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 6131 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNfsSR(com.
> xensource.xenapi.Connection, com.cloud.agent.api.to.StorageFilerTO)()
> 
> ** CID 1222170:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 6412 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getVMSnapsh
> otChainSize(com.xensource.xenapi.Connection,
> org.apache.cloudstack.storage.to.VolumeObjectTO, java.lang.String)()
> 
> ** CID 1222169:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 6098 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getIscsiSR(com
> .xensource.xenapi.Connection, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, boolean)()
> 
> ** CID 1222168:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 1944 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.prepareManag
> edStorage(com.xensource.xenapi.Connection, java.util.Map, java.lang.String,
> java.lang.String)()
> 
> ** CID 1222167:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 7519 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.prepareNetwo
> rkElementCommand(com.cloud.agent.api.routing.SetNetworkACLCommand
> )()
> 
> ** CID 1222166:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixResourceBase.java: 7499 in
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.prepareNetwo
> rkElementCommand(com.cloud.agent.api.routing.SetSourceNatCommand)()
> 
> ** CID 1222165:  Dereference null return value  (NULL_RETURNS)
> /plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resou
> rce/CitrixRes

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[Rajani Karuturi]

+1 for github pull requests. They are much better and cleaner than review board.

~Rajani



On 09-Jun-2014, at 9:17 pm, David Nalley mailto:da...@gnsa.us>> 
wrote:

On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang 
mailto:sh...@yasker.org>> wrote:
Hi all,

Seems it's a good timing to bring back the discussion about the gerrit.

We want to do CI, and improve our code quality. One obvious way of doing
and reduce the workload of devs is introduce a tool to enforce the process.

I've checked out quite a few projects using gerrit, which would force you
to ask for review, and validation before the code can be committed to the
repo. Looks it's really a easier way for devs according what I've heard.

Even our competitor laid out a very detail workflow based on the use of
gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it can
make a good reference.

Well, gerrit has been brought up a few times before. And now the new
process we want to enforce just fits what gerrit(or other automation
review/test/commit software) is for.

Maybe it's the time for us to review the possibility of using a tool to
enforce our commits and improve our code quality(as well as transfer
knowledge) again?

--Sheng


ASF Infra has a very dour view on Gerrit. Don't read that as
impossible; there are many projects at the ASF who are interested in
Gerrit.
That said; what about moving to using github pull requests instead of
RB, and from their, having the jenkins pull request builder
automatically process every pull request and list information.

Here's an example:
https://github.com/jclouds/jclouds-labs/pull/61
You'll see that every time the patch changes, the jenkins plugin
pulled the patch - ran tests against it and reported back.

That said; it almost seems like we have the cart before the horse; we
need to finish figuring out the CI Infrastructure first.

--David

Re: Review Request 22356: Fixed few coverity issues reported

[Santhosh Edukulla]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22356/
---

(Updated June 10, 2014, 4:43 a.m.)


Review request for cloudstack and daan Hoogland.


Changes
---

Added Latest one


Repository: cloudstack-git


Description
---

Fixed few coverity issues reported for resource leak, value comparison, invalid 
loop check for result set.


Diffs (updated)
-

  engine/schema/src/com/cloud/upgrade/DatabaseCreator.java 91ef318 
  engine/schema/src/com/cloud/upgrade/DatabaseIntegrityChecker.java c20a418 
  engine/schema/src/com/cloud/upgrade/DatabaseUpgradeChecker.java 0761c9f 
  framework/db/src/com/cloud/utils/crypt/EncryptionSecretKeyChanger.java 
58584f9 
  framework/db/src/com/cloud/utils/db/Merovingian2.java 6eeea9f 
  framework/db/src/com/cloud/utils/db/ScriptRunner.java 6614527 
  framework/db/src/com/cloud/utils/db/TransactionLegacy.java ac0ea21 
  server/src/com/cloud/test/IPRangeConfig.java 1d56471 
  usage/src/com/cloud/usage/UsageSanityChecker.java 5e6123b 
  utils/src/com/cloud/utils/crypt/EncryptionSecretKeySender.java 086e8a8 

Diff: https://reviews.apache.org/r/22356/diff/


Testing
---

1.Built the code and found no issues.
2.Built the simulator and ran a deploy datacenter with the changes.


Thanks,

Santhosh Edukulla

Re: Review Request 19270: Added service-instance delete command to Juniper Contrail Plugin and related fixes

[Rajesh Battala]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19270/#review45197
---


Looks good to me.
I didn't fine unittests for the new commands and functionality added.
community wont push new features without UnitTests/Marvin tests.
what about documentation of these features?


- Rajesh Battala


On May 7, 2014, 10:44 p.m., Sachchidanand Vaidya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19270/
> ---
> 
> (Updated May 7, 2014, 10:44 p.m.)
> 
> 
> Review request for cloudstack and Rajesh Battala.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> This patch has following changes: 
>  - Added service-instance delete command to Juniper Contrail Plugin.
>  - Fixes to plugin code to handle instance-delete operation properly.
>  - Fix related to service-instance persistence and re-creation during Mgmt 
> server reboot.
> 
> 
> Diffs
> -
> 
>   api/src/com/cloud/event/EventTypes.java 075b1c8 
>   client/tomcatconf/commands.properties.in d3a24f6 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/command/CreateServiceInstanceCmd.java
>  59c4e61 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/api/command/DeleteServiceInstanceCmd.java
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailElementImpl.java
>  05723b0 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ContrailGuru.java
>  f81c0d0 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServerDBSyncImpl.java
>  b9a3ed3 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManager.java
>  e4ef26d 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/ServiceManagerImpl.java
>  f34eacc 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/model/ServiceInstanceModel.java
>  e79053c 
> 
> Diff: https://reviews.apache.org/r/19270/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed and additional testing done locally.
> 
> 
> Thanks,
> 
> Sachchidanand Vaidya
> 
>

Re: [DISCUSS] Introducing Gerrit for quality? was: [PROPOSAL] Using continuous integration to maintain our code quality...

[Hugo Trippaers]

Hey,

I’m all for automated solutions. I’m a happy gerrit user on some other projects 
and quite fond of working with Github pull requests as well. However there is 
one important point that makes working with those tools work and that is a 
willingness by the committers to review requests. Both systems rely on either a 
well functioning and fast CI system or committers that consistently and rapidly 
review requests. Where the latter is actually the most important one.

Both gerrit and pull requests do not improve quality. They are just tools to 
facilitate a certain way of working. If we want to improve quality we have to 
do it ourselves, no amount of automated tooling is going to solve it for us. As 
committers it is our job to review commits and make sure that quality is 
maintained. It is also our job to make sure that automated tests exist that 
will catch problems.

At the moment we have a 433,412 line codebase with on average 3.91 potential 
defects per line of code (according to coverity). We have a very small amount 
of unit test coverage on our core code and no real idea how much or what code 
is covered by functional testing. If we want to improve quality i think that is 
the place to start. 

Of course it is also wise to see if we can improve the quality of the incoming 
commits, but that is easily done by taking a few moments during the day to 
review everything that was pushed to master and fix, revert and add unit tests 
where required. Coach committers/contributors that consistently have trouble 
with adding testing cases on how to do it. That part is the responsibility of 
being a committer, not just the bit that allows access to the repo.

If we are able to get this bit going, i’ll happily jump on any barricade and 
start a revolution to get whatever automated tooling we need to support this 
process.

Cheers,

Hugo



On 10 jun. 2014, at 06:15, Rajani Karuturi  wrote:

> +1 for github pull requests. They are much better and cleaner than review 
> board.
> 
> ~Rajani
> 
> 
> 
> On 09-Jun-2014, at 9:17 pm, David Nalley 
> mailto:da...@gnsa.us>> wrote:
> 
> On Fri, Jun 6, 2014 at 7:26 PM, Sheng Yang 
> mailto:sh...@yasker.org>> wrote:
> Hi all,
> 
> Seems it's a good timing to bring back the discussion about the gerrit.
> 
> We want to do CI, and improve our code quality. One obvious way of doing
> and reduce the workload of devs is introduce a tool to enforce the process.
> 
> I've checked out quite a few projects using gerrit, which would force you
> to ask for review, and validation before the code can be committed to the
> repo. Looks it's really a easier way for devs according what I've heard.
> 
> Even our competitor laid out a very detail workflow based on the use of
> gerrit( https://wiki.openstack.org/wiki/Gerrit_Workflow ). I guess it can
> make a good reference.
> 
> Well, gerrit has been brought up a few times before. And now the new
> process we want to enforce just fits what gerrit(or other automation
> review/test/commit software) is for.
> 
> Maybe it's the time for us to review the possibility of using a tool to
> enforce our commits and improve our code quality(as well as transfer
> knowledge) again?
> 
> --Sheng
> 
> 
> ASF Infra has a very dour view on Gerrit. Don't read that as
> impossible; there are many projects at the ASF who are interested in
> Gerrit.
> That said; what about moving to using github pull requests instead of
> RB, and from their, having the jenkins pull request builder
> automatically process every pull request and list information.
> 
> Here's an example:
> https://github.com/jclouds/jclouds-labs/pull/61
> You'll see that every time the patch changes, the jenkins plugin
> pulled the patch - ran tests against it and reported back.
> 
> That said; it almost seems like we have the cart before the horse; we
> need to finish figuring out the CI Infrastructure first.
> 
> --David
>