RE: KVM CloudStack Agent Hacking proposal

[Paul Angus]

Cool. Thanks for the explanation Ivan,  I'll watch with interest.

paul.an...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 


-Original Message-
From: Ivan Kudryavtsev  
Sent: 23 October 2018 14:55
To: dev 
Subject: Re: KVM CloudStack Agent Hacking proposal

Hello, Paul. You have implemented the second part of the proposal, which is 
related to Qemu hook. Unfortunately, Qemu hooks are not always the right place 
to implement features. I would like it to be, but it's not because of:
https://www.libvirt.org/hooks.html#recursive

In my case, I even implemented a standalone unix-socket server which I have 
called from the hook without awaiting and which, in turn, forked the process to 
interact with libvirt, but it still causes the deadlock because of CS KVM agent 
and security_groups.py still does their interaction and it leads to a deadlock. 
So, hooks are not enough, but important to be. It's cool that current agent 
implementation already includes hooks - less work to do)

So, my proposal is to inject the capability into CS KVM agent. If the design I 
introduced in the first e-mail is OK, we can implement it.

> I would like is to introduce a more generic approach, so the
administrator can specify additional scripts in the
> agent.properties, which will be called the same way "security_groups.py"
called.
> custom.vm.start=/path/to/script1,path/to.script2
> custom.vm.stop=/path/to/script3,path/to.script4

Thank you for your time and opinions.


вт, 23 окт. 2018 г. в 2:56, Paul Angus :

> Hi Ivan,
>
> I think that this may already have been added in 4.12 by ShapeBlue
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+hook+script
> +include
>
> if nothing else it sounds like you want to build upon this rather than 
> rewrite it.
>
>
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK @shapeblue
>
>
>
>
> -Original Message-
> From: Wido den Hollander 
> Sent: 23 October 2018 07:46
> To: dev@cloudstack.apache.org
> Subject: Re: KVM CloudStack Agent Hacking proposal
>
>
>
> On 10/22/18 8:02 PM, Ivan Kudryavtsev wrote:
> > Hello, Devs.
> >
> > I would like to introduce a feature and decided to consult with you
> > about its design before implementation. The feature is connected with
> > KVM CloudStack agent. We have found it beneficial to be able to launch
> > custom scripts upon VM start/stop. It can be done using Qemu hook but
> > it has several drawbacks:
> > - the hook is deployed by CS and adding additional lines into it leads
> > to extra efforts when ACS package is updated.
> > - it leads to deadlocks as you cannot effectively and easy to
> > communicate with libvirt from hook even with "fork & exec" because
> > security_groups.py and agent also participate and as a result it causes
> deadlocks.
> >
> > Now, in the code, we have a call for "security_groups.py":
> >
> > Start:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L103
> >
> > Stop:
> > https://github.com/apache/cloudstack/blob/65f31f1a9fbc1c20cd752d80a7e1
> > 117efc0248a5/plugins/hypervisors/kvm/src/main/java/com/cloud/hyperviso
> > r/kvm/resource/wrapper/LibvirtStopCommandWrapper.java#L88
> >
> > I would like is to introduce a more generic approach, so the
> > administrator can specify additional scripts in the agent.properties,
> > which will be called the same way "security_groups.py" called.
> >
> > custom.vm.start=/path/to/script1,path/to.script2
> > custom.vm.stop=/path/to/script3,path/to.script4
> >
> > So, this feature will help users to do custom hotplug mechanisms. E.g.
> > we have such implementation which adds per-account VXLAN as a hotplug
> > ethernet device. So, even for a Basic Zone, every VM gets automatic
> > second NIC which helps to build a private network for an account.
> >
> > Currently, we do the job thru adding lines into security_groups.py,
> > which is not a good approach, especially for end users who don't want
> > to hack the system.
> >
> > Also, I'm thinking about changing /etc/libvirt/hooks/qemu the same
> > way, so it was just an entry point to  /etc/libvirt/hooks/qemu.d/*
> located scripts.
> >
> > Let me know about this feature proposal and if its design is good, we
> > start developing it.
> >
>
> Seems like a good thing! It adds flexibility to the VM.
>
> How are you planning on getting things like the VM name and other details
> to the scripts?
>
> Wido
>
> > Have a good day.
> >
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks LLC
Cell RU: +7-923-414-1515
Cell USA: +1-201-257-1512
WWW: http://bitworks.software/ 

[GitHub] andrijapanic commented on issue #13: Update Quick Installation Guide

[GitBox]

andrijapanic commented on issue #13: Update Quick Installation Guide
URL: 
https://github.com/apache/cloudstack-documentation/pull/13#issuecomment-432659098
 
 
   Hi Alex,
   
   Let me begin :)
   
   First thanks for the update, I have followed "blindly" the updated tutorial 
(while still doing sanity checks and frequent reboots...) and I find it very 
straight-forward, so big thanks for the update !
   
   I do have a couple of small fixes / missing points, so can you please fix it 
- but I would also wait for @rhtyd to give his feedback.
   
   ## Line break for eth0 / enp3s0 config
   
   Next section seems printed in one line, instead of multiple lines (at least 
while viewing file on github)
   ```
   TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes 
IPV6INIT=no NAME=enp5s0 UUID=26f024e6-1113-416e-b319-58ebec347886 DEVICE=enp3s0 
ONBOOT=yes BRIDGE=cloudbr0
   ```
   
   Further more, UUID defined in "eth0" equivalent interface is not needed and 
bare minimum is enough, as following:
   
   ( ifcfg-eth0 # or other name which you assume in tutorial, ensp30 (there is 
type above also!)
   
   ```
   TYPE=Ethernet
   BOOTPROTO=none
   DEFROUTE=yes
   IPV6INIT=no
   NAME=eth0 
   DEVICE=eth0
   ONBOOT=yes
   BRIDGE=cloudbr0
   ```
   If you choose to accept changes above, then also remove the following 
sentence:
   
   ```
   You should not use the Hardware Address (aka the MAC address, or UUID) from 
our example for your configuration. It is network interface specific, so you 
should keep the address already provided in the UUID directive...
   ```
   
   ## Firewall script issues - many, many issues here, read carefully please !
   
   In general I don't like this whole firewall sections, because it caused tons 
of problem for me, when following this modified guide, during test setup (last 
3 h):
   - ssvm/cpvm agents could not connect to mgmt server - so whole zone was 
broken
   - ssmv.sh script reports bad IP address of the Secondary Storage - since 
agent didn't run and no additional configuration was done inside SSVM.
   - since SSVM was not functional, Dashboard showed ZERO capacity for 
Secondary Storage..
   - I could not access UI on 8080 from my laptop over VPN
   - etc.
   
   @rhtyd  I propose, for simple purpose of Quick Installation Guide, to either 
COMPLETELY DISABLE FIREWALL ! (we anyway "disable" selinux, we don't set it up 
"properly"...) or at least drop/remove the "DENY" rules/lines and add more 
"allow" lines/ports, because I could not access UI from remote laptop (over 
VPN, etc), also SSVM/CPVM could not connect to 8250 on mgmt server, since this 
is not covered in the firewall configuration, etc.
   
   So either disable firewall completely ("cloudstack-setup-management" command 
seems to have done this for me, until I rebooted host ...)  or at least do 
following changes to the firewalldnfs.sh script
   
   * replace CIDR "204.168.1.0/24" with the one from tutorial "172.16.10.0/24" 
( @AlexBeez this is copy/paste leftover I assume)
   * Remove both "deny" lines from current places (50, 60), they are duplicated 
and also have to come after all ACCEPT rules
   * Add lines that will enable remote access to 53 (cpvm DNS resolution) 8080 
(UI access), 8250(cpvm/ssvm to mgmt) ,3306 (for sake of DB management) ,80 
(forward chain, for CPVM access), 5900-6100 (vnc, for CPVM) - and some of these 
have soruce set to 0.0.0.0/0 (common sense)
   
   So the final script looks like following, with modification from above, 
looks as following (and yet, we have not covered ports for VM live migration 
etc..)
   
   ```
   #!/bin/bash
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 10 -m state 
--state ESTABLISHED,RELATED -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 20 -p icmp -j 
ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 30 -i lo -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 40 -m state 
--state NEW -m tcp -p tcp --dport 22 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 70 -s 
172.16.10.0/24 -m state --state NEW -p udp --dport 111 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 80 -s 
172.16.10.0/24 -m state --state NEW -p tcp --dport 111 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 90 -s 
172.16.10.0/24 -m state --state NEW -p tcp --dport 2049 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 100 -s 
172.16.10.0/24 -m state --state NEW -p tcp --dport 32803 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 110 -s 
172.16.10.0/24 -m state --state NEW -p udp --dport 32769 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 120 -s 
172.16.10.0/24 -m state --state NEW -p tcp --dport 892 -j ACCEPT
   firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 130 -s 
172.16.10.0/24 -m state --state NEW -p udp --dport 892 -j ACCEPT
   firewall-cmd --di

[GitHub] AlexBeez commented on issue #13: Update Quick Installation Guide

[GitBox]

AlexBeez commented on issue #13: Update Quick Installation Guide
URL: 
https://github.com/apache/cloudstack-documentation/pull/13#issuecomment-432660503
 
 
   Ah see I see where I went wrong.  
   
   During my specific install, I specifically went into the Security Policy 
settings, and told it to NOT apply a security policy - hence my list was 
already empty from the get go. My mistake.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: CloudStack Collab in Brazil

[Rafael Weingärtner]

Yes, they already have a date set. It should be 23 -  27 April, 2019.
I should be talking with them again this week to check what we need to move
thing forward.

What do you guys think about these dates?

On Mon, Oct 22, 2018 at 5:07 PM Tutkowski, Mike 
wrote:

> Hi Rafael,
>
> Do you have a specific date in mind for CCC Brazil? It sounds like, in
> general, we are looking at April.
>
> Thanks!
> Mike
>
> On 10/1/18, 12:51 PM, "Rafael Weingärtner" 
> wrote:
>
> NetApp Security WARNING: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> Yes, that is what I also believe. From the feedback, I think we can
> easily
> use 10 presentations. I will move on with the organization. I think it
> is
> feasible to get more room space in case we receive more presentation
> and
> people. I will try to not overlap presentations though (like we did in
> ApacheCon).
>
> On Mon, Oct 1, 2018 at 3:36 PM Tutkowski, Mike <
> mike.tutkow...@netapp.com>
> wrote:
>
> > I guess it depends on how many people expect to be able to attend.
> >
> > Ten presentation slots is probably a good starting point.
> >
> > Get Outlook for iOS
> > 
> > From: Rafael Weingärtner 
> > Sent: Monday, October 1, 2018 10:10:55 AM
> > To: users
> > Cc: dev
> > Subject: Re: CloudStack Collab in Brazil
> >
> > NetApp Security WARNING: This is an external email. Do not click
> links or
> > open attachments unless you recognize the sender and know the
> content is
> > safe.
> >
> >
> >
> >
> > Thank you guys for the feedback!
> >
> > I will reach out the organizers to discuss our requirements. What do
> you
> > guys think that we need?
> > Would 10 presentation slots (50min. each) be enough? Or, do you guys
> think
> > that we need more?
> >
> > Also, I think that we should also do a Hackathon. Therefore, I will
> also be
> > asking for a room such as the one we used in Montreal.
> >
> > On Mon, Oct 1, 2018 at 12:03 PM Nicolas Vazquez <
> > nicolas.vazq...@shapeblue.com> wrote:
> >
> > > I would be interested in an event in Brazil as well.
> > >
> > >
> > > Regards,
> > >
> > > Nicolas Vazquez
> > >
> > > 
> > > From: Gabriel Beims Bräscher 
> > > Sent: Monday, October 1, 2018 11:58:07 AM
> > > To: users
> > > Cc: dev
> > > Subject: Re: CloudStack Collab in Brazil
> > >
> > > As a Brazilian, that lives in Florianópolis, I cannot pass this
> > opportunity
> > > ;)
> > > Count on me!
> > >
> > > Em seg, 1 de out de 2018 às 11:27, Tutkowski, Mike <
> > > mike.tutkow...@netapp.com> escreveu:
> > >
> > > > I would be really interested in an event in Brazil.
> > > >
> > > > 
> > > > From: Rafael Weingärtner 
> > > > Sent: Monday, October 1, 2018 5:38 AM
> > > > To: users
> > > > Cc: dev
> > > > Subject: Re: CloudStack Collab in Brazil
> > > >
> > > > NetApp Security WARNING: This is an external email. Do not click
> links
> > or
> > > > open attachments unless you recognize the sender and know the
> content
> > is
> > > > safe.
> > > >
> > > >
> > > >
> > > >
> > > > Hey Marco,
> > > > Yes, they run a very successful conference every year. I have
> just got
> > > back
> > > > from Montreal, and I talked with people there regarding the
> conference.
> > > >
> > > > Now, for all CloudStackers (users and devs); I will repeat what
> I said
> > in
> > > > Montreal. The TDC conference will happen with or without us.
> Therefore,
> > > we
> > > > only need to decide if we will join them in their Cloud tracks.
> We did
> > > not
> > > > hear much feedback here, but I will try again.
> > > >
> > > > If you are part of the CloudStack community (as a contributor,
> > committer,
> > > > user, operator, and so on), please do provide your feedback.
> Would you
> > > like
> > > > to see a CloudStack Collab Conference in Florianopolis, Brazil,
> 2019? I
> > > am
> > > > only asking you guys, what you think. I do understand the
> logistics
> > > > problems for some folks to attend a conference this far.
> > > >
> > > > Now, about the city; the island has an airport (airport code =
> FLN).
> > > > However, most flights to FLN will have a connection either on
> GRU (Sao
> > > > Paulo airport) or GIG (Rio de Janeiro airport); KLM, AA, Delta,
> > > AirFrance,
> > > > Tap, and others have flights to FLN. I have also found some
> useful
> > links
> > > in
> > > > English that can be used by your guys to check the city. In this
> link
> > [1]
>

Re: CloudStack Collab in Brazil

[Tutkowski, Mike]

Thanks, Rafael!

The dates work for me.

Get Outlook for iOS

From: Rafael Weingärtner 
Sent: Wednesday, October 24, 2018 5:02:14 PM
To: users
Cc: dev
Subject: Re: CloudStack Collab in Brazil

NetApp Security WARNING: This is an external email. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.




Yes, they already have a date set. It should be 23 -  27 April, 2019.
I should be talking with them again this week to check what we need to move
thing forward.

What do you guys think about these dates?

On Mon, Oct 22, 2018 at 5:07 PM Tutkowski, Mike 
wrote:

> Hi Rafael,
>
> Do you have a specific date in mind for CCC Brazil? It sounds like, in
> general, we are looking at April.
>
> Thanks!
> Mike
>
> On 10/1/18, 12:51 PM, "Rafael Weingärtner" 
> wrote:
>
> NetApp Security WARNING: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> Yes, that is what I also believe. From the feedback, I think we can
> easily
> use 10 presentations. I will move on with the organization. I think it
> is
> feasible to get more room space in case we receive more presentation
> and
> people. I will try to not overlap presentations though (like we did in
> ApacheCon).
>
> On Mon, Oct 1, 2018 at 3:36 PM Tutkowski, Mike <
> mike.tutkow...@netapp.com>
> wrote:
>
> > I guess it depends on how many people expect to be able to attend.
> >
> > Ten presentation slots is probably a good starting point.
> >
> > Get Outlook for iOS
> > 
> > From: Rafael Weingärtner 
> > Sent: Monday, October 1, 2018 10:10:55 AM
> > To: users
> > Cc: dev
> > Subject: Re: CloudStack Collab in Brazil
> >
> > NetApp Security WARNING: This is an external email. Do not click
> links or
> > open attachments unless you recognize the sender and know the
> content is
> > safe.
> >
> >
> >
> >
> > Thank you guys for the feedback!
> >
> > I will reach out the organizers to discuss our requirements. What do
> you
> > guys think that we need?
> > Would 10 presentation slots (50min. each) be enough? Or, do you guys
> think
> > that we need more?
> >
> > Also, I think that we should also do a Hackathon. Therefore, I will
> also be
> > asking for a room such as the one we used in Montreal.
> >
> > On Mon, Oct 1, 2018 at 12:03 PM Nicolas Vazquez <
> > nicolas.vazq...@shapeblue.com> wrote:
> >
> > > I would be interested in an event in Brazil as well.
> > >
> > >
> > > Regards,
> > >
> > > Nicolas Vazquez
> > >
> > > 
> > > From: Gabriel Beims Bräscher 
> > > Sent: Monday, October 1, 2018 11:58:07 AM
> > > To: users
> > > Cc: dev
> > > Subject: Re: CloudStack Collab in Brazil
> > >
> > > As a Brazilian, that lives in Florianópolis, I cannot pass this
> > opportunity
> > > ;)
> > > Count on me!
> > >
> > > Em seg, 1 de out de 2018 às 11:27, Tutkowski, Mike <
> > > mike.tutkow...@netapp.com> escreveu:
> > >
> > > > I would be really interested in an event in Brazil.
> > > >
> > > > 
> > > > From: Rafael Weingärtner 
> > > > Sent: Monday, October 1, 2018 5:38 AM
> > > > To: users
> > > > Cc: dev
> > > > Subject: Re: CloudStack Collab in Brazil
> > > >
> > > > NetApp Security WARNING: This is an external email. Do not click
> links
> > or
> > > > open attachments unless you recognize the sender and know the
> content
> > is
> > > > safe.
> > > >
> > > >
> > > >
> > > >
> > > > Hey Marco,
> > > > Yes, they run a very successful conference every year. I have
> just got
> > > back
> > > > from Montreal, and I talked with people there regarding the
> conference.
> > > >
> > > > Now, for all CloudStackers (users and devs); I will repeat what
> I said
> > in
> > > > Montreal. The TDC conference will happen with or without us.
> Therefore,
> > > we
> > > > only need to decide if we will join them in their Cloud tracks.
> We did
> > > not
> > > > hear much feedback here, but I will try again.
> > > >
> > > > If you are part of the CloudStack community (as a contributor,
> > committer,
> > > > user, operator, and so on), please do provide your feedback.
> Would you
> > > like
> > > > to see a CloudStack Collab Conference in Florianopolis, Brazil,
> 2019? I
> > > am
> > > > only asking you guys, what you think. I do understand the
> logistics
> > > > problems for some folks to attend a conference this far.
> > > >
> > > > Now, about the city; the island has an airport (airport 

Re: [VOTE] Apache CloudStack 4.11.2.0 RC3

[Rohit Yadav]

-1 (binding)


A VMware blocker issue was discovered due to which I would like to request a 
RC4. The blocker has been fixed, reviewed: 
https://github.com/apache/cloudstack/pull/2916


With the above fix, smoketests also pass 100% on vmware/kvm/xenserver (results 
on the PR).


A RC4 may be cut as soon as next week to allow others to continue testing and 
report any more blockers. The following critical PRs are currently under review 
and testing and against the 4.11.2.0 milestone:


https://github.com/apache/cloudstack/pull/2926 (force stop old VR during 
network restart with cleanup=true)
https://github.com/apache/cloudstack/issues/2880 (unable to reproduce the 
issue, likely to do with concurrent start/stop of VRs triggered by vm power 
sync on VMware, PR 2926 may potentially fix it)

https://github.com/apache/cloudstack/pull/2927 (this fixes a flood of unwanted 
log messages, non-critical though)


Please continue to test RC3 and report any blockers you may find, thanks.


- Rohit


From: Paul Angus 
Sent: Thursday, October 18, 2018 4:24:12 PM
To: dev@cloudstack.apache.org; us...@cloudstack.apache.org
Subject: RE: [VOTE] Apache CloudStack 4.11.2.0 RC3

Hi All,

The issues in RC2 have been fixed so we're ready to go with RC3
I've created a 4.11.2.0 release (RC3), with the following artefacts up for 
testing and a vote:

Git Branch and Commit SH:
https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/
Commit: a8e53d0e9674028973c8f3a98a5a8ff15b24f5da

Source release (checksums and signatures are available at the same location):
https://dist.apache.org/repos/dist/dev/cloudstack/4.11.2.0/
PGP release keys (signed using 8B309F7251EE0BC8):
https://dist.apache.org/repos/dist/release/cloudstack/KEYS

The vote will be open until the middle of next week, 26th September 2018.

For sanity in tallying the vote, can PMC members please be sure to indicate 
"(binding)" with their vote

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Additional information:

For users' convenience, I've built packages from 
a8e53d0e9674028973c8f3a98a5a8ff15b24f5da and published RC3 repository here:
http://packages.shapeblue.com/testing/4112rc3/

4.11.2 systemvm templates are available from here:
http://packages.shapeblue.com/testing/systemvm/4112rc3





paul.an...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue