Upgrade CloudStack from 4.9.2.0 to 4.11.0

[Marc Poll Garcia]

Hello,

My current infrastructure is Apache Cloudstack 4.9.2 with VMware hosts and
the management server on CentOS.


I'm planning to perform an upgrade from the actual 4.9.2 versión to the
latest one.

I found this tutorial from Cloudstack website:

http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.0.0/upgrade/upgrade-4.9.html

But i don't know if any of you already did it, and had upgraded the system?
I was wondering if anyone had any issues during the execution of the
process.

And also if someone can send more info, or another guide to follow or best
practice?

We've check it out and found that we need to compile our own cloudstack
software because we're using vmware hosts, is it true? any suggestions?

Thanks in advance.

Kind regards.


-- 
Marc Poll Garcia
Technology Infrastructure . Àrea de Serveis TIC
Telèfon:  93.405.43.57

[image: UPCnet]

--
Aquest correu electrònic pot contenir informació confidencial o legalment
protegida i està exclusivament dirigit a la persona o entitat destinatària.
Si vostè no és el destinatari final o persona encarregada de recollir-lo,
no està autoritzat a llegir-lo, retenir-lo, modificar-lo, distribuir-lo,
copiar-lo ni a revelar el seu contingut. Si ha rebut aquest correu
electrònic per error, li preguem que informi al remitent i elimini del seu
sistema el missatge i el material annex que pugui contenir.
Gràcies per la seva col.laboració.
--

*** Si us plau, no m'imprimeixis. Vull seguir sent digital ***
*** Por favor, no me imprimas. Quiero seguir siendo digital ***
*** Please, don't print me. I want to remain digital ***
--

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Rafael Weingärtner]

Khosrow thanks for the interesting feature. You mention two possible
methods to manage certificates; one using the CA framework, and other using
third party such as Vault and Let’s Encrypt.

Have you considered using the sshKeyPair API methods (is it part of the CA
framework?)? I mean, users already can generate key pairs via ACS, and then
they are presented with the private key. You could simply list these
certificates for the user when they want to configure a new certificate for
a VPN or generate one in runtime using this feature. Reading your feature
proposal I did not understand how you are binding certificated with a VPN
(are you always generating new ones and simply returning the private key to
users?).

Moreover, as the sshKeyPair methods, I do believe you should only return
the private key once. Therefore, you should not store it in ACS.

On Mon, Apr 2, 2018 at 4:36 PM, Khosrow Moossavi 
wrote:

> Hi Community
>
> I want to open up a discussion around the new Remote Access VPN
> implementation on VRs. Currently
> we have only L2TP implementation, which lacks different features (such as
> verbos logging), so we
> decided to start developing new implementation based on IKEv2 (on top of
> the existing strongSwan).
>
> We have this feature working locally for over a week now, and seems to be
> ready for opening up a
> PR on official repo. But before doing so we agreed to open up a discussion
> here first.
>
> The current implementation we use EAP + Public Key for authentication, so
> we need to have a PKI
> Engine somewhere. Rather than start re-inventing the wheel (and start
> extending the current CA Framework
> which was done by Rohit) we decided to delegate this functionality to
> HashiCorp Vault, which will act as
> a PKI backend engine for Cloudstack.
>
> The way I implemented this specific part of the code, is that it can easily
> be extended/implemented with other
> concrete classes or designs (such as going forward with in-house PKI
> engine, or even use external services
> such as Let's Encrypt), but at the end of the day we strongly suggest to
> use Vault, as it is really easy to use.
>
>
> Please find the design document here[1], and share your feedback. I will
> open up a PR -as is- soon to be able
> to have a source code to discuss around it as well.
>
> [1]:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> VPN+Implementation+based+on+IKEv2+backed+by+Vault+as+PKI+Engine
>
>
> Thanks
>
> Khosrow Moossavi
>
> Cloud Infrastructure Developer
>
> t 514.447.3456
>
> 
>



-- 
Rafael Weingärtner

Re: System VM Template

[Rafael Weingärtner]

Hey Mike,

This week I have been using ACS 4.12 to do some testing. VRs and system VMs
are deploying just fine with the system VM template of 4.11. Of course, by
using this template (the 4.11) I am not receiving the changes already made
to it in both 4.11 and current master branch.

During my testes, I allocated a public IP, created some NAT rules,
allocated directly attach IPs. Everything was working as expected.


The hypervisor I am using is XenServer both 6.5 and 7.2.

On Tue, Apr 3, 2018 at 1:56 AM, Tutkowski, Mike 
wrote:

> Hi,
>
> I may have missed an e-mail about this recently.
>
> Can someone provide me with the current URL I can use to download system
> VM templates for 4.12?
>
> I’ve tried 4.11 from here:
>
> http://cloudstack.apt-get.eu/systemvm/4.11/
>
> and master from here:
>
> https://builds.cloudstack.org/job/build-master-systemvm/
>
> However, in neither case can I get the VR up and running on 4.12.
>
> Thanks!
> Mike
>



-- 
Rafael Weingärtner

Re: Upgrade CloudStack from 4.9.2.0 to 4.11.0

[Dag Sonstebo]

Hi Stephan,

Thanks for the summary – can you log these as new issues in the new issues 
tracker https://github.com/apache/cloudstack/issues  please (note not Jira).

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 04/04/2018, 10:39, "Stephan Seitz"  wrote:

Hi!

We're currently using XenServer instead of VMware, so I just don't know
if you really need to build your own packages. Afaik shapeblue's public
repository has been built with noredist.

Here's short list (sorry, we didn't report everything to the bugtracker
by now) of caveats:

* There's a more precise dashboard (XX.XX% instead of XX%)
-> Nice, but only works with locale set to EN or C or anything with
decimalpoints instead of commas :) ... consequently the default
language of the GUI will also be selected identical to your locale.

-> Ldap Authentication doesn't work. You need to apply https://github.c
om/apache/cloudstack/pull/2517 to get this working.

-> Adding a NIC to a running VM (only tested in Advanced Zone,
Xenserver, Shared Network to add) fails with an "duplicate MAC-address" 
error. See my post on the ML yesterday.

-> cloudstack-usage doesn't start since (at least Ubuntu, deb packages)
the update doesn't clean old libs from /usr/share/cloudstack-
usage/libs. For us cleanup and reinstall fixed that.

-> SSVM's java keystore doesn't contain Let's Encrypt Root-CA (maybe
others are also missing) so don't expect working downloads from
cdimage.debian.org via https :)

-> A few nasty popups occur (but can be ignored) in the GUI e.g.
selecting a project and viewing networks.

-> A minor documentation bug in the upgrade document: The apt-get.eu
Repository doesn't contain 4.11 right now. download.cloudstack.org
does.


To us none of that problems was a stopper, but your mileage may vary.

cheers,

- Stephan


Am Mittwoch, den 04.04.2018, 11:08 +0200 schrieb Marc Poll Garcia:
> Hello,
> 
> My current infrastructure is Apache Cloudstack 4.9.2 with VMware
> hosts and
> the management server on CentOS.
> 
> 
> I'm planning to perform an upgrade from the actual 4.9.2 versión to
> the
> latest one.
> 
> I found this tutorial from Cloudstack website:
> 
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/e
> n/4.11.0.0/upgrade/upgrade-4.9.html
> 
> But i don't know if any of you already did it, and had upgraded the
> system?
> I was wondering if anyone had any issues during the execution of the
> process.
> 
> And also if someone can send more info, or another guide to follow or
> best
> practice?
> 
> We've check it out and found that we need to compile our own
> cloudstack
> software because we're using vmware hosts, is it true? any
> suggestions?
> 
> Thanks in advance.
> 
> Kind regards.
> 
> 
-- 

Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-44
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht
Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin




dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Khosrow Moossavi]

Rafael,

We cannot use SshKeyPair functionality because the proposed VPN
implementation
does need a signed certificate and not a ssh key pair. The process is as
follow:

1) generate root CA (if doesn't exist)
2) generate bunch of intermediate steps (config urls, CRLs, role name, ...)
[I'm not going
in detail now, here, for simplicity]
3) self sign a certificate against the root CA (regenerate every time start
VPN command
executed)

This will produce:

1) Root CA cert (one per domain in cloudstack)
2) Server cert (one per VR)
3) Server private key (one per VR)

Then all the above will be pushed to the said VR we want to start VPN on,
and start
ipsec service on it (with extra configuration - which will be available in
codebase) and
finally present Root CA for user to download and install on their local
machine to be
able to "trust" VR they are VPNing to.



On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Khosrow thanks for the interesting feature. You mention two possible
> methods to manage certificates; one using the CA framework, and other using
> third party such as Vault and Let’s Encrypt.
>
> Have you considered using the sshKeyPair API methods (is it part of the CA
> framework?)? I mean, users already can generate key pairs via ACS, and then
> they are presented with the private key. You could simply list these
> certificates for the user when they want to configure a new certificate for
> a VPN or generate one in runtime using this feature. Reading your feature
> proposal I did not understand how you are binding certificated with a VPN
> (are you always generating new ones and simply returning the private key to
> users?).
>
> Moreover, as the sshKeyPair methods, I do believe you should only return
> the private key once. Therefore, you should not store it in ACS.
>
> On Mon, Apr 2, 2018 at 4:36 PM, Khosrow Moossavi 
> wrote:
>
> > Hi Community
> >
> > I want to open up a discussion around the new Remote Access VPN
> > implementation on VRs. Currently
> > we have only L2TP implementation, which lacks different features (such as
> > verbos logging), so we
> > decided to start developing new implementation based on IKEv2 (on top of
> > the existing strongSwan).
> >
> > We have this feature working locally for over a week now, and seems to be
> > ready for opening up a
> > PR on official repo. But before doing so we agreed to open up a
> discussion
> > here first.
> >
> > The current implementation we use EAP + Public Key for authentication, so
> > we need to have a PKI
> > Engine somewhere. Rather than start re-inventing the wheel (and start
> > extending the current CA Framework
> > which was done by Rohit) we decided to delegate this functionality to
> > HashiCorp Vault, which will act as
> > a PKI backend engine for Cloudstack.
> >
> > The way I implemented this specific part of the code, is that it can
> easily
> > be extended/implemented with other
> > concrete classes or designs (such as going forward with in-house PKI
> > engine, or even use external services
> > such as Let's Encrypt), but at the end of the day we strongly suggest to
> > use Vault, as it is really easy to use.
> >
> >
> > Please find the design document here[1], and share your feedback. I will
> > open up a PR -as is- soon to be able
> > to have a source code to discuss around it as well.
> >
> > [1]:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > VPN+Implementation+based+on+IKEv2+backed+by+Vault+as+PKI+Engine
> >
> >
> > Thanks
> >
> > Khosrow Moossavi
> >
> > Cloud Infrastructure Developer
> >
> > t 514.447.3456
> >
> > 
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Rafael Weingärtner]

So, you need a certificate that is signed by the CA that is used by the VPN
service. Is that it?



It has been a while that I do not configure these VPN systems; do you need
access to the private key of the CA? Or, does the program simply validate
the user (VPN client) certificate to see if it is issued by a specific CA?
I believe it also needs the public key of the user to execute the handshake
and create the connection.




On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi 
wrote:

> Rafael,
>
> We cannot use SshKeyPair functionality because the proposed VPN
> implementation
> does need a signed certificate and not a ssh key pair. The process is as
> follow:
>
> 1) generate root CA (if doesn't exist)
> 2) generate bunch of intermediate steps (config urls, CRLs, role name, ...)
> [I'm not going
> in detail now, here, for simplicity]
> 3) self sign a certificate against the root CA (regenerate every time start
> VPN command
> executed)
>
> This will produce:
>
> 1) Root CA cert (one per domain in cloudstack)
> 2) Server cert (one per VR)
> 3) Server private key (one per VR)
>
> Then all the above will be pushed to the said VR we want to start VPN on,
> and start
> ipsec service on it (with extra configuration - which will be available in
> codebase) and
> finally present Root CA for user to download and install on their local
> machine to be
> able to "trust" VR they are VPNing to.
>
>
>
> On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Khosrow thanks for the interesting feature. You mention two possible
> > methods to manage certificates; one using the CA framework, and other
> using
> > third party such as Vault and Let’s Encrypt.
> >
> > Have you considered using the sshKeyPair API methods (is it part of the
> CA
> > framework?)? I mean, users already can generate key pairs via ACS, and
> then
> > they are presented with the private key. You could simply list these
> > certificates for the user when they want to configure a new certificate
> for
> > a VPN or generate one in runtime using this feature. Reading your feature
> > proposal I did not understand how you are binding certificated with a VPN
> > (are you always generating new ones and simply returning the private key
> to
> > users?).
> >
> > Moreover, as the sshKeyPair methods, I do believe you should only return
> > the private key once. Therefore, you should not store it in ACS.
> >
> > On Mon, Apr 2, 2018 at 4:36 PM, Khosrow Moossavi  >
> > wrote:
> >
> > > Hi Community
> > >
> > > I want to open up a discussion around the new Remote Access VPN
> > > implementation on VRs. Currently
> > > we have only L2TP implementation, which lacks different features (such
> as
> > > verbos logging), so we
> > > decided to start developing new implementation based on IKEv2 (on top
> of
> > > the existing strongSwan).
> > >
> > > We have this feature working locally for over a week now, and seems to
> be
> > > ready for opening up a
> > > PR on official repo. But before doing so we agreed to open up a
> > discussion
> > > here first.
> > >
> > > The current implementation we use EAP + Public Key for authentication,
> so
> > > we need to have a PKI
> > > Engine somewhere. Rather than start re-inventing the wheel (and start
> > > extending the current CA Framework
> > > which was done by Rohit) we decided to delegate this functionality to
> > > HashiCorp Vault, which will act as
> > > a PKI backend engine for Cloudstack.
> > >
> > > The way I implemented this specific part of the code, is that it can
> > easily
> > > be extended/implemented with other
> > > concrete classes or designs (such as going forward with in-house PKI
> > > engine, or even use external services
> > > such as Let's Encrypt), but at the end of the day we strongly suggest
> to
> > > use Vault, as it is really easy to use.
> > >
> > >
> > > Please find the design document here[1], and share your feedback. I
> will
> > > open up a PR -as is- soon to be able
> > > to have a source code to discuss around it as well.
> > >
> > > [1]:
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > VPN+Implementation+based+on+IKEv2+backed+by+Vault+as+PKI+Engine
> > >
> > >
> > > Thanks
> > >
> > > Khosrow Moossavi
> > >
> > > Cloud Infrastructure Developer
> > >
> > > t 514.447.3456
> > >
> > > 
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Khosrow Moossavi]

On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> So, you need a certificate that is signed by the CA that is used by the VPN
> service. Is that it?
>
>
Correct, a self signed "server certificate" against CA, to be installed
directly on VR.


>
> It has been a while that I do not configure these VPN systems; do you need
> access to the private key of the CA? Or, does the program simply validate
> the user (VPN client) certificate to see if it is issued by a specific CA?
> I believe it also needs the public key of the user to execute the handshake
> and create the connection.
>
>
>
No, end user only needs to have Root CA at hand, to *trust* it. Both the
"Server
Certificate" and "Server Private Key" are sensitive information and only
exist  on
VR.

User then can go ahead and install the Root CA on their local machine and
open
up VPN connection with strongSwan client of the correspondning OS they're on
import the Root CA, and their credential (EAP on VPN side), and that's it.


>
>
> On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi 
> wrote:
>
> > Rafael,
> >
> > We cannot use SshKeyPair functionality because the proposed VPN
> > implementation
> > does need a signed certificate and not a ssh key pair. The process is as
> > follow:
> >
> > 1) generate root CA (if doesn't exist)
> > 2) generate bunch of intermediate steps (config urls, CRLs, role name,
> ...)
> > [I'm not going
> > in detail now, here, for simplicity]
> > 3) self sign a certificate against the root CA (regenerate every time
> start
> > VPN command
> > executed)
> >
> > This will produce:
> >
> > 1) Root CA cert (one per domain in cloudstack)
> > 2) Server cert (one per VR)
> > 3) Server private key (one per VR)
> >
> > Then all the above will be pushed to the said VR we want to start VPN on,
> > and start
> > ipsec service on it (with extra configuration - which will be available
> in
> > codebase) and
> > finally present Root CA for user to download and install on their local
> > machine to be
> > able to "trust" VR they are VPNing to.
> >
> >
> >
> > On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > Khosrow thanks for the interesting feature. You mention two possible
> > > methods to manage certificates; one using the CA framework, and other
> > using
> > > third party such as Vault and Let’s Encrypt.
> > >
> > > Have you considered using the sshKeyPair API methods (is it part of the
> > CA
> > > framework?)? I mean, users already can generate key pairs via ACS, and
> > then
> > > they are presented with the private key. You could simply list these
> > > certificates for the user when they want to configure a new certificate
> > for
> > > a VPN or generate one in runtime using this feature. Reading your
> feature
> > > proposal I did not understand how you are binding certificated with a
> VPN
> > > (are you always generating new ones and simply returning the private
> key
> > to
> > > users?).
> > >
> > > Moreover, as the sshKeyPair methods, I do believe you should only
> return
> > > the private key once. Therefore, you should not store it in ACS.
> > >
> > > On Mon, Apr 2, 2018 at 4:36 PM, Khosrow Moossavi <
> kmooss...@cloudops.com
> > >
> > > wrote:
> > >
> > > > Hi Community
> > > >
> > > > I want to open up a discussion around the new Remote Access VPN
> > > > implementation on VRs. Currently
> > > > we have only L2TP implementation, which lacks different features
> (such
> > as
> > > > verbos logging), so we
> > > > decided to start developing new implementation based on IKEv2 (on top
> > of
> > > > the existing strongSwan).
> > > >
> > > > We have this feature working locally for over a week now, and seems
> to
> > be
> > > > ready for opening up a
> > > > PR on official repo. But before doing so we agreed to open up a
> > > discussion
> > > > here first.
> > > >
> > > > The current implementation we use EAP + Public Key for
> authentication,
> > so
> > > > we need to have a PKI
> > > > Engine somewhere. Rather than start re-inventing the wheel (and start
> > > > extending the current CA Framework
> > > > which was done by Rohit) we decided to delegate this functionality to
> > > > HashiCorp Vault, which will act as
> > > > a PKI backend engine for Cloudstack.
> > > >
> > > > The way I implemented this specific part of the code, is that it can
> > > easily
> > > > be extended/implemented with other
> > > > concrete classes or designs (such as going forward with in-house PKI
> > > > engine, or even use external services
> > > > such as Let's Encrypt), but at the end of the day we strongly suggest
> > to
> > > > use Vault, as it is really easy to use.
> > > >
> > > >
> > > > Please find the design document here[1], and share your feedback. I
> > will
> > > > open up a PR -as is- soon to be able
> > > > to have a source code to discuss around it as well.
> > > >
> > > > [1]:
> > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> 

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Rafael Weingärtner]

Got it. Thanks for the explanations.
There is one other thing I do not understand. This Vault thing that you
mention, how does it work? Is it similar to let's encrypt?

On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi 
wrote:

> On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > So, you need a certificate that is signed by the CA that is used by the
> VPN
> > service. Is that it?
> >
> >
> Correct, a self signed "server certificate" against CA, to be installed
> directly on VR.
>
>
> >
> > It has been a while that I do not configure these VPN systems; do you
> need
> > access to the private key of the CA? Or, does the program simply validate
> > the user (VPN client) certificate to see if it is issued by a specific
> CA?
> > I believe it also needs the public key of the user to execute the
> handshake
> > and create the connection.
> >
> >
> >
> No, end user only needs to have Root CA at hand, to *trust* it. Both the
> "Server
> Certificate" and "Server Private Key" are sensitive information and only
> exist  on
> VR.
>
> User then can go ahead and install the Root CA on their local machine and
> open
> up VPN connection with strongSwan client of the correspondning OS they're
> on
> import the Root CA, and their credential (EAP on VPN side), and that's it.
>
>
> >
> >
> > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> kmooss...@cloudops.com>
> > wrote:
> >
> > > Rafael,
> > >
> > > We cannot use SshKeyPair functionality because the proposed VPN
> > > implementation
> > > does need a signed certificate and not a ssh key pair. The process is
> as
> > > follow:
> > >
> > > 1) generate root CA (if doesn't exist)
> > > 2) generate bunch of intermediate steps (config urls, CRLs, role name,
> > ...)
> > > [I'm not going
> > > in detail now, here, for simplicity]
> > > 3) self sign a certificate against the root CA (regenerate every time
> > start
> > > VPN command
> > > executed)
> > >
> > > This will produce:
> > >
> > > 1) Root CA cert (one per domain in cloudstack)
> > > 2) Server cert (one per VR)
> > > 3) Server private key (one per VR)
> > >
> > > Then all the above will be pushed to the said VR we want to start VPN
> on,
> > > and start
> > > ipsec service on it (with extra configuration - which will be available
> > in
> > > codebase) and
> > > finally present Root CA for user to download and install on their local
> > > machine to be
> > > able to "trust" VR they are VPNing to.
> > >
> > >
> > >
> > > On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > Khosrow thanks for the interesting feature. You mention two possible
> > > > methods to manage certificates; one using the CA framework, and other
> > > using
> > > > third party such as Vault and Let’s Encrypt.
> > > >
> > > > Have you considered using the sshKeyPair API methods (is it part of
> the
> > > CA
> > > > framework?)? I mean, users already can generate key pairs via ACS,
> and
> > > then
> > > > they are presented with the private key. You could simply list these
> > > > certificates for the user when they want to configure a new
> certificate
> > > for
> > > > a VPN or generate one in runtime using this feature. Reading your
> > feature
> > > > proposal I did not understand how you are binding certificated with a
> > VPN
> > > > (are you always generating new ones and simply returning the private
> > key
> > > to
> > > > users?).
> > > >
> > > > Moreover, as the sshKeyPair methods, I do believe you should only
> > return
> > > > the private key once. Therefore, you should not store it in ACS.
> > > >
> > > > On Mon, Apr 2, 2018 at 4:36 PM, Khosrow Moossavi <
> > kmooss...@cloudops.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Community
> > > > >
> > > > > I want to open up a discussion around the new Remote Access VPN
> > > > > implementation on VRs. Currently
> > > > > we have only L2TP implementation, which lacks different features
> > (such
> > > as
> > > > > verbos logging), so we
> > > > > decided to start developing new implementation based on IKEv2 (on
> top
> > > of
> > > > > the existing strongSwan).
> > > > >
> > > > > We have this feature working locally for over a week now, and seems
> > to
> > > be
> > > > > ready for opening up a
> > > > > PR on official repo. But before doing so we agreed to open up a
> > > > discussion
> > > > > here first.
> > > > >
> > > > > The current implementation we use EAP + Public Key for
> > authentication,
> > > so
> > > > > we need to have a PKI
> > > > > Engine somewhere. Rather than start re-inventing the wheel (and
> start
> > > > > extending the current CA Framework
> > > > > which was done by Rohit) we decided to delegate this functionality
> to
> > > > > HashiCorp Vault, which will act as
> > > > > a PKI backend engine for Cloudstack.
> > > > >
> > > > > The way I implemented this specific part of the code, is that it
> can
> > > > easily
> > > > > be extended/implemented with othe

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Khosrow Moossavi]

One of the things Vault does is essentially one of the thing Let's Encrypt
does,
acting as CA and generating/signing certificates.

>From the Vault website itself:

"HashiCorp Vault secures, stores, and tightly controls access to tokens,
passwords,
certificates, API keys, and other secrets in modern computing. Vault
handles leasing,
key revocation, key rolling, and auditing. Through a unified API, users can
access an
encrypted Key/Value store and network encryption-as-a-service, or generate
AWS
IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
credentials,
and more."

In our case we are going to use Vault as PKI backend engine, to act as Root
CA,
sign certificates, handle CRL (Certificate Revocation List), etc.
Technically we can
do these with Let's Encrypt, but I haven't started exploring the
possibilities or potential
limitation. Using external services (such as Let's Encrypt) or going
forward with
Bring You Own Certificate model would be for future, it they ever made
sense to do.



On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Got it. Thanks for the explanations.
> There is one other thing I do not understand. This Vault thing that you
> mention, how does it work? Is it similar to let's encrypt?
>
> On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi 
> wrote:
>
> > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > So, you need a certificate that is signed by the CA that is used by the
> > VPN
> > > service. Is that it?
> > >
> > >
> > Correct, a self signed "server certificate" against CA, to be installed
> > directly on VR.
> >
> >
> > >
> > > It has been a while that I do not configure these VPN systems; do you
> > need
> > > access to the private key of the CA? Or, does the program simply
> validate
> > > the user (VPN client) certificate to see if it is issued by a specific
> > CA?
> > > I believe it also needs the public key of the user to execute the
> > handshake
> > > and create the connection.
> > >
> > >
> > >
> > No, end user only needs to have Root CA at hand, to *trust* it. Both the
> > "Server
> > Certificate" and "Server Private Key" are sensitive information and only
> > exist  on
> > VR.
> >
> > User then can go ahead and install the Root CA on their local machine and
> > open
> > up VPN connection with strongSwan client of the correspondning OS they're
> > on
> > import the Root CA, and their credential (EAP on VPN side), and that's
> it.
> >
> >
> > >
> > >
> > > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> > kmooss...@cloudops.com>
> > > wrote:
> > >
> > > > Rafael,
> > > >
> > > > We cannot use SshKeyPair functionality because the proposed VPN
> > > > implementation
> > > > does need a signed certificate and not a ssh key pair. The process is
> > as
> > > > follow:
> > > >
> > > > 1) generate root CA (if doesn't exist)
> > > > 2) generate bunch of intermediate steps (config urls, CRLs, role
> name,
> > > ...)
> > > > [I'm not going
> > > > in detail now, here, for simplicity]
> > > > 3) self sign a certificate against the root CA (regenerate every time
> > > start
> > > > VPN command
> > > > executed)
> > > >
> > > > This will produce:
> > > >
> > > > 1) Root CA cert (one per domain in cloudstack)
> > > > 2) Server cert (one per VR)
> > > > 3) Server private key (one per VR)
> > > >
> > > > Then all the above will be pushed to the said VR we want to start VPN
> > on,
> > > > and start
> > > > ipsec service on it (with extra configuration - which will be
> available
> > > in
> > > > codebase) and
> > > > finally present Root CA for user to download and install on their
> local
> > > > machine to be
> > > > able to "trust" VR they are VPNing to.
> > > >
> > > >
> > > >
> > > > On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
> > > > rafaelweingart...@gmail.com> wrote:
> > > >
> > > > > Khosrow thanks for the interesting feature. You mention two
> possible
> > > > > methods to manage certificates; one using the CA framework, and
> other
> > > > using
> > > > > third party such as Vault and Let’s Encrypt.
> > > > >
> > > > > Have you considered using the sshKeyPair API methods (is it part of
> > the
> > > > CA
> > > > > framework?)? I mean, users already can generate key pairs via ACS,
> > and
> > > > then
> > > > > they are presented with the private key. You could simply list
> these
> > > > > certificates for the user when they want to configure a new
> > certificate
> > > > for
> > > > > a VPN or generate one in runtime using this feature. Reading your
> > > feature
> > > > > proposal I did not understand how you are binding certificated
> with a
> > > VPN
> > > > > (are you always generating new ones and simply returning the
> private
> > > key
> > > > to
> > > > > users?).
> > > > >
> > > > > Moreover, as the sshKeyPair methods, I do believe you should only
> > > return
> > > > > the private key once. Therefore, you should not store it in ACS.
> > > > >

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Rafael Weingärtner]

Thanks for sharing the details. Now I have a better perspective of the
proposal.It is an interesting integration of CloudStack VPN service with
Vault PKI feature.

On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi 
wrote:

> One of the things Vault does is essentially one of the thing Let's Encrypt
> does,
> acting as CA and generating/signing certificates.
>
> From the Vault website itself:
>
> "HashiCorp Vault secures, stores, and tightly controls access to tokens,
> passwords,
> certificates, API keys, and other secrets in modern computing. Vault
> handles leasing,
> key revocation, key rolling, and auditing. Through a unified API, users can
> access an
> encrypted Key/Value store and network encryption-as-a-service, or generate
> AWS
> IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
> credentials,
> and more."
>
> In our case we are going to use Vault as PKI backend engine, to act as Root
> CA,
> sign certificates, handle CRL (Certificate Revocation List), etc.
> Technically we can
> do these with Let's Encrypt, but I haven't started exploring the
> possibilities or potential
> limitation. Using external services (such as Let's Encrypt) or going
> forward with
> Bring You Own Certificate model would be for future, it they ever made
> sense to do.
>
>
>
> On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Got it. Thanks for the explanations.
> > There is one other thing I do not understand. This Vault thing that you
> > mention, how does it work? Is it similar to let's encrypt?
> >
> > On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi <
> kmooss...@cloudops.com>
> > wrote:
> >
> > > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > So, you need a certificate that is signed by the CA that is used by
> the
> > > VPN
> > > > service. Is that it?
> > > >
> > > >
> > > Correct, a self signed "server certificate" against CA, to be installed
> > > directly on VR.
> > >
> > >
> > > >
> > > > It has been a while that I do not configure these VPN systems; do you
> > > need
> > > > access to the private key of the CA? Or, does the program simply
> > validate
> > > > the user (VPN client) certificate to see if it is issued by a
> specific
> > > CA?
> > > > I believe it also needs the public key of the user to execute the
> > > handshake
> > > > and create the connection.
> > > >
> > > >
> > > >
> > > No, end user only needs to have Root CA at hand, to *trust* it. Both
> the
> > > "Server
> > > Certificate" and "Server Private Key" are sensitive information and
> only
> > > exist  on
> > > VR.
> > >
> > > User then can go ahead and install the Root CA on their local machine
> and
> > > open
> > > up VPN connection with strongSwan client of the correspondning OS
> they're
> > > on
> > > import the Root CA, and their credential (EAP on VPN side), and that's
> > it.
> > >
> > >
> > > >
> > > >
> > > > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> > > kmooss...@cloudops.com>
> > > > wrote:
> > > >
> > > > > Rafael,
> > > > >
> > > > > We cannot use SshKeyPair functionality because the proposed VPN
> > > > > implementation
> > > > > does need a signed certificate and not a ssh key pair. The process
> is
> > > as
> > > > > follow:
> > > > >
> > > > > 1) generate root CA (if doesn't exist)
> > > > > 2) generate bunch of intermediate steps (config urls, CRLs, role
> > name,
> > > > ...)
> > > > > [I'm not going
> > > > > in detail now, here, for simplicity]
> > > > > 3) self sign a certificate against the root CA (regenerate every
> time
> > > > start
> > > > > VPN command
> > > > > executed)
> > > > >
> > > > > This will produce:
> > > > >
> > > > > 1) Root CA cert (one per domain in cloudstack)
> > > > > 2) Server cert (one per VR)
> > > > > 3) Server private key (one per VR)
> > > > >
> > > > > Then all the above will be pushed to the said VR we want to start
> VPN
> > > on,
> > > > > and start
> > > > > ipsec service on it (with extra configuration - which will be
> > available
> > > > in
> > > > > codebase) and
> > > > > finally present Root CA for user to download and install on their
> > local
> > > > > machine to be
> > > > > able to "trust" VR they are VPNing to.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner <
> > > > > rafaelweingart...@gmail.com> wrote:
> > > > >
> > > > > > Khosrow thanks for the interesting feature. You mention two
> > possible
> > > > > > methods to manage certificates; one using the CA framework, and
> > other
> > > > > using
> > > > > > third party such as Vault and Let’s Encrypt.
> > > > > >
> > > > > > Have you considered using the sshKeyPair API methods (is it part
> of
> > > the
> > > > > CA
> > > > > > framework?)? I mean, users already can generate key pairs via
> ACS,
> > > and
> > > > > then
> > > > > > they are presented with the private key. You could simply list
> > these
> > > > > > certificates for the u

RE: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Paul Angus]

You guys should speak to Rohit about the CA framework.  CloudStack can manage 
certificates now, including creating them itself and acting as a root CA.




Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rafael Weingärtner  
Sent: 04 April 2018 16:51
To: dev 
Subject: Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

Thanks for sharing the details. Now I have a better perspective of the 
proposal.It is an interesting integration of CloudStack VPN service with Vault 
PKI feature.

On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi 
wrote:

> One of the things Vault does is essentially one of the thing Let's 
> Encrypt does, acting as CA and generating/signing certificates.
>
> From the Vault website itself:
>
> "HashiCorp Vault secures, stores, and tightly controls access to 
> tokens, passwords, certificates, API keys, and other secrets in modern 
> computing. Vault handles leasing, key revocation, key rolling, and 
> auditing. Through a unified API, users can access an encrypted 
> Key/Value store and network encryption-as-a-service, or generate AWS 
> IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH 
> credentials, and more."
>
> In our case we are going to use Vault as PKI backend engine, to act as 
> Root CA, sign certificates, handle CRL (Certificate Revocation List), 
> etc.
> Technically we can
> do these with Let's Encrypt, but I haven't started exploring the 
> possibilities or potential limitation. Using external services (such 
> as Let's Encrypt) or going forward with Bring You Own Certificate 
> model would be for future, it they ever made sense to do.
>
>
>
> On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner < 
> rafaelweingart...@gmail.com> wrote:
>
> > Got it. Thanks for the explanations.
> > There is one other thing I do not understand. This Vault thing that 
> > you mention, how does it work? Is it similar to let's encrypt?
> >
> > On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi <
> kmooss...@cloudops.com>
> > wrote:
> >
> > > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner < 
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > So, you need a certificate that is signed by the CA that is used 
> > > > by
> the
> > > VPN
> > > > service. Is that it?
> > > >
> > > >
> > > Correct, a self signed "server certificate" against CA, to be 
> > > installed directly on VR.
> > >
> > >
> > > >
> > > > It has been a while that I do not configure these VPN systems; 
> > > > do you
> > > need
> > > > access to the private key of the CA? Or, does the program simply
> > validate
> > > > the user (VPN client) certificate to see if it is issued by a
> specific
> > > CA?
> > > > I believe it also needs the public key of the user to execute 
> > > > the
> > > handshake
> > > > and create the connection.
> > > >
> > > >
> > > >
> > > No, end user only needs to have Root CA at hand, to *trust* it. 
> > > Both
> the
> > > "Server
> > > Certificate" and "Server Private Key" are sensitive information 
> > > and
> only
> > > exist  on
> > > VR.
> > >
> > > User then can go ahead and install the Root CA on their local 
> > > machine
> and
> > > open
> > > up VPN connection with strongSwan client of the correspondning OS
> they're
> > > on
> > > import the Root CA, and their credential (EAP on VPN side), and 
> > > that's
> > it.
> > >
> > >
> > > >
> > > >
> > > > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> > > kmooss...@cloudops.com>
> > > > wrote:
> > > >
> > > > > Rafael,
> > > > >
> > > > > We cannot use SshKeyPair functionality because the proposed 
> > > > > VPN implementation does need a signed certificate and not a 
> > > > > ssh key pair. The process
> is
> > > as
> > > > > follow:
> > > > >
> > > > > 1) generate root CA (if doesn't exist)
> > > > > 2) generate bunch of intermediate steps (config urls, CRLs, 
> > > > > role
> > name,
> > > > ...)
> > > > > [I'm not going
> > > > > in detail now, here, for simplicity]
> > > > > 3) self sign a certificate against the root CA (regenerate 
> > > > > every
> time
> > > > start
> > > > > VPN command
> > > > > executed)
> > > > >
> > > > > This will produce:
> > > > >
> > > > > 1) Root CA cert (one per domain in cloudstack)
> > > > > 2) Server cert (one per VR)
> > > > > 3) Server private key (one per VR)
> > > > >
> > > > > Then all the above will be pushed to the said VR we want to 
> > > > > start
> VPN
> > > on,
> > > > > and start
> > > > > ipsec service on it (with extra configuration - which will be
> > available
> > > > in
> > > > > codebase) and
> > > > > finally present Root CA for user to download and install on 
> > > > > their
> > local
> > > > > machine to be
> > > > > able to "trust" VR they are VPNing to.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 4, 2018 at 6:19 AM, Rafael Weingärtner < 
> > > > > rafaelweingart...@gmail.com> wrote:
> > > > >
> > > > > > Kho

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Khosrow Moossavi]

Thanks Paul, the proposed feature will enable the functionality to use
Vault to
act as CA if enabled in ACS, otherwise will fall back to "default"
implementation
which Rohit has already done.


On Wed, Apr 4, 2018 at 12:29 PM, Paul Angus 
wrote:

> You guys should speak to Rohit about the CA framework.  CloudStack can
> manage certificates now, including creating them itself and acting as a
> root CA.
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rafael Weingärtner 
> Sent: 04 April 2018 16:51
> To: dev 
> Subject: Re: [DISCUSS] New VPN implementation based on IKEv2 backed by
> Vault
>
> Thanks for sharing the details. Now I have a better perspective of the
> proposal.It is an interesting integration of CloudStack VPN service with
> Vault PKI feature.
>
> On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi 
> wrote:
>
> > One of the things Vault does is essentially one of the thing Let's
> > Encrypt does, acting as CA and generating/signing certificates.
> >
> > From the Vault website itself:
> >
> > "HashiCorp Vault secures, stores, and tightly controls access to
> > tokens, passwords, certificates, API keys, and other secrets in modern
> > computing. Vault handles leasing, key revocation, key rolling, and
> > auditing. Through a unified API, users can access an encrypted
> > Key/Value store and network encryption-as-a-service, or generate AWS
> > IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
> > credentials, and more."
> >
> > In our case we are going to use Vault as PKI backend engine, to act as
> > Root CA, sign certificates, handle CRL (Certificate Revocation List),
> > etc.
> > Technically we can
> > do these with Let's Encrypt, but I haven't started exploring the
> > possibilities or potential limitation. Using external services (such
> > as Let's Encrypt) or going forward with Bring You Own Certificate
> > model would be for future, it they ever made sense to do.
> >
> >
> >
> > On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > Got it. Thanks for the explanations.
> > > There is one other thing I do not understand. This Vault thing that
> > > you mention, how does it work? Is it similar to let's encrypt?
> > >
> > > On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi <
> > kmooss...@cloudops.com>
> > > wrote:
> > >
> > > > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> > > > rafaelweingart...@gmail.com> wrote:
> > > >
> > > > > So, you need a certificate that is signed by the CA that is used
> > > > > by
> > the
> > > > VPN
> > > > > service. Is that it?
> > > > >
> > > > >
> > > > Correct, a self signed "server certificate" against CA, to be
> > > > installed directly on VR.
> > > >
> > > >
> > > > >
> > > > > It has been a while that I do not configure these VPN systems;
> > > > > do you
> > > > need
> > > > > access to the private key of the CA? Or, does the program simply
> > > validate
> > > > > the user (VPN client) certificate to see if it is issued by a
> > specific
> > > > CA?
> > > > > I believe it also needs the public key of the user to execute
> > > > > the
> > > > handshake
> > > > > and create the connection.
> > > > >
> > > > >
> > > > >
> > > > No, end user only needs to have Root CA at hand, to *trust* it.
> > > > Both
> > the
> > > > "Server
> > > > Certificate" and "Server Private Key" are sensitive information
> > > > and
> > only
> > > > exist  on
> > > > VR.
> > > >
> > > > User then can go ahead and install the Root CA on their local
> > > > machine
> > and
> > > > open
> > > > up VPN connection with strongSwan client of the correspondning OS
> > they're
> > > > on
> > > > import the Root CA, and their credential (EAP on VPN side), and
> > > > that's
> > > it.
> > > >
> > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> > > > kmooss...@cloudops.com>
> > > > > wrote:
> > > > >
> > > > > > Rafael,
> > > > > >
> > > > > > We cannot use SshKeyPair functionality because the proposed
> > > > > > VPN implementation does need a signed certificate and not a
> > > > > > ssh key pair. The process
> > is
> > > > as
> > > > > > follow:
> > > > > >
> > > > > > 1) generate root CA (if doesn't exist)
> > > > > > 2) generate bunch of intermediate steps (config urls, CRLs,
> > > > > > role
> > > name,
> > > > > ...)
> > > > > > [I'm not going
> > > > > > in detail now, here, for simplicity]
> > > > > > 3) self sign a certificate against the root CA (regenerate
> > > > > > every
> > time
> > > > > start
> > > > > > VPN command
> > > > > > executed)
> > > > > >
> > > > > > This will produce:
> > > > > >
> > > > > > 1) Root CA cert (one per domain in cloudstack)
> > > > > > 2) Server cert (one per VR)
> > > > > > 3) Server private key (one per VR)
> > > > > >
> > > > > > Then all the above will be pushed to the said VR we want t

[DISCUSS] CloudStack graceful shutdown

[ilya musayev]

Use case:
In any environment - time to time - administrator needs to perform a
maintenance. Current stop sequence of cloudstack management server will
ignore the fact that there may be long running async jobs - and terminate
the process. This in turn can create a poor user experience and occasional
inconsistency  in cloudstack db.

This is especially painful in large environments where the user has
thousands of nodes and there is a continuous patching that happens around
the clock - that requires migration of workload from one node to another.

With that said - i've created a script that monitors the async job queue
for given MS and waits for it complete all jobs. More details are posted
below.

I'd like to introduce "graceful-shutdown" into the systemctl/service of
cloudstack-management service.

The details of how it will work is below:

Workflow for graceful shutdown:
  Using iptables/firewalld - block any connection attempts on 8080/8443 (we
can identify the ports dynamically)
  Identify the MSID for the node, using the proper msid - query async_job
table for
1) any jobs that are still running (or job_status=“0”)
2) job_dispatcher not like “pseudoJobDispatcher"
3) job_init_msid=$my_ms_id

Monitor this async_job table for 60 minutes - until all async jobs for MSID
are done, then proceed with shutdown
If failed for any reason or terminated, catch the exit via trap command
and unblock the 8080/8443

Comments are welcome

Regards,
ilya

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[ilya musayev]

Khosrow

My 2c, little less than ideal to manage yet another external end point
like.

While i understand that it makes it easier to manage certificates - it also
means going forward - Vault implementation will become a requirement to
validate future ACS release.

With that said - i do like the proposal and not against it, but:
1) Please consider decoupling it from cloudstack-management server - and
release it as server plugin
2) Test coverage must be sufficient enough to validate the functionality
(perhaps mock vault endpoints and response)

Regards,
ilya

On Wed, Apr 4, 2018 at 10:49 AM, Khosrow Moossavi 
wrote:

> Thanks Paul, the proposed feature will enable the functionality to use
> Vault to
> act as CA if enabled in ACS, otherwise will fall back to "default"
> implementation
> which Rohit has already done.
>
>
> On Wed, Apr 4, 2018 at 12:29 PM, Paul Angus 
> wrote:
>
> > You guys should speak to Rohit about the CA framework.  CloudStack can
> > manage certificates now, including creating them itself and acting as a
> > root CA.
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Rafael Weingärtner 
> > Sent: 04 April 2018 16:51
> > To: dev 
> > Subject: Re: [DISCUSS] New VPN implementation based on IKEv2 backed by
> > Vault
> >
> > Thanks for sharing the details. Now I have a better perspective of the
> > proposal.It is an interesting integration of CloudStack VPN service with
> > Vault PKI feature.
> >
> > On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi <
> kmooss...@cloudops.com>
> > wrote:
> >
> > > One of the things Vault does is essentially one of the thing Let's
> > > Encrypt does, acting as CA and generating/signing certificates.
> > >
> > > From the Vault website itself:
> > >
> > > "HashiCorp Vault secures, stores, and tightly controls access to
> > > tokens, passwords, certificates, API keys, and other secrets in modern
> > > computing. Vault handles leasing, key revocation, key rolling, and
> > > auditing. Through a unified API, users can access an encrypted
> > > Key/Value store and network encryption-as-a-service, or generate AWS
> > > IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
> > > credentials, and more."
> > >
> > > In our case we are going to use Vault as PKI backend engine, to act as
> > > Root CA, sign certificates, handle CRL (Certificate Revocation List),
> > > etc.
> > > Technically we can
> > > do these with Let's Encrypt, but I haven't started exploring the
> > > possibilities or potential limitation. Using external services (such
> > > as Let's Encrypt) or going forward with Bring You Own Certificate
> > > model would be for future, it they ever made sense to do.
> > >
> > >
> > >
> > > On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > Got it. Thanks for the explanations.
> > > > There is one other thing I do not understand. This Vault thing that
> > > > you mention, how does it work? Is it similar to let's encrypt?
> > > >
> > > > On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi <
> > > kmooss...@cloudops.com>
> > > > wrote:
> > > >
> > > > > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> > > > > rafaelweingart...@gmail.com> wrote:
> > > > >
> > > > > > So, you need a certificate that is signed by the CA that is used
> > > > > > by
> > > the
> > > > > VPN
> > > > > > service. Is that it?
> > > > > >
> > > > > >
> > > > > Correct, a self signed "server certificate" against CA, to be
> > > > > installed directly on VR.
> > > > >
> > > > >
> > > > > >
> > > > > > It has been a while that I do not configure these VPN systems;
> > > > > > do you
> > > > > need
> > > > > > access to the private key of the CA? Or, does the program simply
> > > > validate
> > > > > > the user (VPN client) certificate to see if it is issued by a
> > > specific
> > > > > CA?
> > > > > > I believe it also needs the public key of the user to execute
> > > > > > the
> > > > > handshake
> > > > > > and create the connection.
> > > > > >
> > > > > >
> > > > > >
> > > > > No, end user only needs to have Root CA at hand, to *trust* it.
> > > > > Both
> > > the
> > > > > "Server
> > > > > Certificate" and "Server Private Key" are sensitive information
> > > > > and
> > > only
> > > > > exist  on
> > > > > VR.
> > > > >
> > > > > User then can go ahead and install the Root CA on their local
> > > > > machine
> > > and
> > > > > open
> > > > > up VPN connection with strongSwan client of the correspondning OS
> > > they're
> > > > > on
> > > > > import the Root CA, and their credential (EAP on VPN side), and
> > > > > that's
> > > > it.
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 4, 2018 at 11:22 AM, Khosrow Moossavi <
> > > > > kmooss...@cloudops.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Rafael,
> > > > >

Re: [DISCUSS] CloudStack graceful shutdown

[Rafael Weingärtner]

Big +1 for this feature; I only have a few doubts.

* Regarding the tasks/jobs that management servers (MSs) execute; are these
tasks originate from requests that come to the MS, or is it possible that
requests received by one management server to be executed by other? I mean,
if I execute a request against MS1, will this request always be
executed/threated by MS1, or is it possible that this request is executed
by another MS (e.g. MS2)?

* I would suggest that after we block traffic coming from 8080/8443/8250(we
will need to block this as well right?), we can log the execution of tasks.
I mean, something saying, there are XXX tasks (enumerate tasks) still being
executed, we will wait for them to finish before shutting down.

* The timeout (60 minutes suggested) could be global settings that we can
load before executing the graceful-shutdown.

On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev 
wrote:

> Use case:
> In any environment - time to time - administrator needs to perform a
> maintenance. Current stop sequence of cloudstack management server will
> ignore the fact that there may be long running async jobs - and terminate
> the process. This in turn can create a poor user experience and occasional
> inconsistency  in cloudstack db.
>
> This is especially painful in large environments where the user has
> thousands of nodes and there is a continuous patching that happens around
> the clock - that requires migration of workload from one node to another.
>
> With that said - i've created a script that monitors the async job queue
> for given MS and waits for it complete all jobs. More details are posted
> below.
>
> I'd like to introduce "graceful-shutdown" into the systemctl/service of
> cloudstack-management service.
>
> The details of how it will work is below:
>
> Workflow for graceful shutdown:
>   Using iptables/firewalld - block any connection attempts on 8080/8443 (we
> can identify the ports dynamically)
>   Identify the MSID for the node, using the proper msid - query async_job
> table for
> 1) any jobs that are still running (or job_status=“0”)
> 2) job_dispatcher not like “pseudoJobDispatcher"
> 3) job_init_msid=$my_ms_id
>
> Monitor this async_job table for 60 minutes - until all async jobs for MSID
> are done, then proceed with shutdown
> If failed for any reason or terminated, catch the exit via trap command
> and unblock the 8080/8443
>
> Comments are welcome
>
> Regards,
> ilya
>



-- 
Rafael Weingärtner

Re: [DISCUSS] CloudStack graceful shutdown

[Tutkowski, Mike]

I may be remembering this incorrectly, but from what I recall, if a resource is 
owned by one MS and a request related to that resource comes in to another MS, 
the MS that received the request passes it on to the other MS.

> On Apr 4, 2018, at 2:36 PM, Rafael Weingärtner  
> wrote:
> 
> Big +1 for this feature; I only have a few doubts.
> 
> * Regarding the tasks/jobs that management servers (MSs) execute; are these
> tasks originate from requests that come to the MS, or is it possible that
> requests received by one management server to be executed by other? I mean,
> if I execute a request against MS1, will this request always be
> executed/threated by MS1, or is it possible that this request is executed
> by another MS (e.g. MS2)?
> 
> * I would suggest that after we block traffic coming from 8080/8443/8250(we
> will need to block this as well right?), we can log the execution of tasks.
> I mean, something saying, there are XXX tasks (enumerate tasks) still being
> executed, we will wait for them to finish before shutting down.
> 
> * The timeout (60 minutes suggested) could be global settings that we can
> load before executing the graceful-shutdown.
> 
> On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev 
> wrote:
> 
>> Use case:
>> In any environment - time to time - administrator needs to perform a
>> maintenance. Current stop sequence of cloudstack management server will
>> ignore the fact that there may be long running async jobs - and terminate
>> the process. This in turn can create a poor user experience and occasional
>> inconsistency  in cloudstack db.
>> 
>> This is especially painful in large environments where the user has
>> thousands of nodes and there is a continuous patching that happens around
>> the clock - that requires migration of workload from one node to another.
>> 
>> With that said - i've created a script that monitors the async job queue
>> for given MS and waits for it complete all jobs. More details are posted
>> below.
>> 
>> I'd like to introduce "graceful-shutdown" into the systemctl/service of
>> cloudstack-management service.
>> 
>> The details of how it will work is below:
>> 
>> Workflow for graceful shutdown:
>>  Using iptables/firewalld - block any connection attempts on 8080/8443 (we
>> can identify the ports dynamically)
>>  Identify the MSID for the node, using the proper msid - query async_job
>> table for
>> 1) any jobs that are still running (or job_status=“0”)
>> 2) job_dispatcher not like “pseudoJobDispatcher"
>> 3) job_init_msid=$my_ms_id
>> 
>> Monitor this async_job table for 60 minutes - until all async jobs for MSID
>> are done, then proceed with shutdown
>>If failed for any reason or terminated, catch the exit via trap command
>> and unblock the 8080/8443
>> 
>> Comments are welcome
>> 
>> Regards,
>> ilya
>> 
> 
> 
> 
> -- 
> Rafael Weingärtner

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Rafael Weingärtner]

To complement one thing that Ilya mentioned here. I do not worry much about
the “requirement” for Vault systems to test ACS. This would be the case if
Khosrow, when developing, only created tests using what the community calls
integration tests.

However, it is an implementation from scratch and as such it can and should
use a very high bar for code quality, which enables proper unit testing for
all methods. This means, that we can check all of the code in our domain
(code base) without requiring third-party software. It does not mean that
we do not need “integration tests” checking the system integration with
Vault, but we could then restrict this execution to RCs.

On Wed, Apr 4, 2018 at 5:29 PM, ilya musayev 
wrote:

> Khosrow
>
> My 2c, little less than ideal to manage yet another external end point
> like.
>
> While i understand that it makes it easier to manage certificates - it also
> means going forward - Vault implementation will become a requirement to
> validate future ACS release.
>
> With that said - i do like the proposal and not against it, but:
> 1) Please consider decoupling it from cloudstack-management server - and
> release it as server plugin
> 2) Test coverage must be sufficient enough to validate the functionality
> (perhaps mock vault endpoints and response)
>
> Regards,
> ilya
>
> On Wed, Apr 4, 2018 at 10:49 AM, Khosrow Moossavi 
> wrote:
>
> > Thanks Paul, the proposed feature will enable the functionality to use
> > Vault to
> > act as CA if enabled in ACS, otherwise will fall back to "default"
> > implementation
> > which Rohit has already done.
> >
> >
> > On Wed, Apr 4, 2018 at 12:29 PM, Paul Angus 
> > wrote:
> >
> > > You guys should speak to Rohit about the CA framework.  CloudStack can
> > > manage certificates now, including creating them itself and acting as a
> > > root CA.
> > >
> > >
> > >
> > >
> > > Kind regards,
> > >
> > > Paul Angus
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Rafael Weingärtner 
> > > Sent: 04 April 2018 16:51
> > > To: dev 
> > > Subject: Re: [DISCUSS] New VPN implementation based on IKEv2 backed by
> > > Vault
> > >
> > > Thanks for sharing the details. Now I have a better perspective of the
> > > proposal.It is an interesting integration of CloudStack VPN service
> with
> > > Vault PKI feature.
> > >
> > > On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi <
> > kmooss...@cloudops.com>
> > > wrote:
> > >
> > > > One of the things Vault does is essentially one of the thing Let's
> > > > Encrypt does, acting as CA and generating/signing certificates.
> > > >
> > > > From the Vault website itself:
> > > >
> > > > "HashiCorp Vault secures, stores, and tightly controls access to
> > > > tokens, passwords, certificates, API keys, and other secrets in
> modern
> > > > computing. Vault handles leasing, key revocation, key rolling, and
> > > > auditing. Through a unified API, users can access an encrypted
> > > > Key/Value store and network encryption-as-a-service, or generate AWS
> > > > IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
> > > > credentials, and more."
> > > >
> > > > In our case we are going to use Vault as PKI backend engine, to act
> as
> > > > Root CA, sign certificates, handle CRL (Certificate Revocation List),
> > > > etc.
> > > > Technically we can
> > > > do these with Let's Encrypt, but I haven't started exploring the
> > > > possibilities or potential limitation. Using external services (such
> > > > as Let's Encrypt) or going forward with Bring You Own Certificate
> > > > model would be for future, it they ever made sense to do.
> > > >
> > > >
> > > >
> > > > On Wed, Apr 4, 2018 at 11:20 AM, Rafael Weingärtner <
> > > > rafaelweingart...@gmail.com> wrote:
> > > >
> > > > > Got it. Thanks for the explanations.
> > > > > There is one other thing I do not understand. This Vault thing that
> > > > > you mention, how does it work? Is it similar to let's encrypt?
> > > > >
> > > > > On Wed, Apr 4, 2018 at 12:15 PM, Khosrow Moossavi <
> > > > kmooss...@cloudops.com>
> > > > > wrote:
> > > > >
> > > > > > On Wed, Apr 4, 2018 at 10:36 AM, Rafael Weingärtner <
> > > > > > rafaelweingart...@gmail.com> wrote:
> > > > > >
> > > > > > > So, you need a certificate that is signed by the CA that is
> used
> > > > > > > by
> > > > the
> > > > > > VPN
> > > > > > > service. Is that it?
> > > > > > >
> > > > > > >
> > > > > > Correct, a self signed "server certificate" against CA, to be
> > > > > > installed directly on VR.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > It has been a while that I do not configure these VPN systems;
> > > > > > > do you
> > > > > > need
> > > > > > > access to the private key of the CA? Or, does the program
> simply
> > > > > validate
> > > > > > > the user (VPN client) certificate to see if it is issued by a
> > > > specific
> > > 

Re: Committee to Sort through CCC Presentation Submissions

[Rafael Weingärtner]

I think everybody that “raised their hands here” already signed up to
review.

Mike, what about if we only gathered the reviews from Apache main review
system, and then we use that to decide which presentations will get in
CloudStack tracks? Then, we reduce the work on our side (we also remove
bias…). I do believe that the review from other peers from Apache community
(even the one outside from our small community) will be fair and technical
(meaning, without passion and or favoritism).

Having said that, I think we only need a small group of PMCs to gather the
results and out of the best ranked proposals, we pick the ones to our
tracks.

What do you (Mike) and others think?


On Tue, Apr 3, 2018 at 5:07 PM, Tutkowski, Mike 
wrote:

> Hi Ron,
>
> I don’t actually have insight into how many people have currently signed
> up online to be CFP reviewers for ApacheCon. At present, I’m only aware of
> those who have responded to this e-mail chain.
>
> We should be able to find out more in the coming weeks. We’re still quite
> early in the process.
>
> Thanks for your feedback,
> Mike
>
> On 4/1/18, 9:18 AM, "Ron Wheeler"  wrote:
>
> How many people have signed up to be reviewers?
>
> I don't think that scheduling is part of the review process and that
> can
> be done by the person/team "organizing" ApacheCon on behalf of the PMC.
>
> To me review is looking at content for
> - relevance
> - quality of the presentations (suggest fixes to content, English,
> graphics, etc.)
> This should result in a consensus score
> - Perfect - ready for prime time
> - Needs minor changes as documented by the reviewers
> - Great topic but needs more work - perhaps a reviewer could volunteer
> to work with the presenter to get it ready if chosen
> - Not recommended for topic or content reasons
>
> The reviewers could also make non-binding recommendations about the
> balance between topics - marketing(why Cloudstack),
> Operations/implementation, Technical details, Roadmap, etc. based on
> what they have seen.
>
> This should be used by the organizers to make the choices and organize
> the program.
> The organizers have the final say on the choice of presentations and
> schedule
>
> Reviewers are there to help the process not control it.
>
> I would be worried that you do not have enough reviewers rather than
> too
> many.
> Then the work falls on the PMC and organizers.
>
> When planning meetings, I would recommend that you clearly separate the
> roles and only invite the reviewers to the meetings about review. Get
> the list of presentation to present to the reviewers and decide if
> there
> are any instructions that you want to give to reviewers.
> I would recommend that you keep the organizing group small. Membership
> should be set by the PMC and should be people that are committed to the
> ApacheCon project and have the time. The committee can request help for
> specific tasks from others in the community who are not on the
> committee.
>
> I would also recommend that organizers do not do reviews. They should
> read the finalists but if they do reviews, there may be a suggestion of
> favouring presentations that they reviewed. It also ensures that the
> organizers are not getting heat from rejected presenters - "it is the
> reviewers fault you did not get selected".
>
> My advice is to get as many reviewers as you can so that no one is
> essential and each reviewer has a limited number of presentations to
> review but each presentation gets reviewed by multiple people. Also
> bear
> in mind that not all reviewers have the same ability to review each
> presentation.
> Reviews should be anonymous and only the summary comments given to the
> presenter. Reviewers of a presentation should be able to discuss the
> presentation during the review to make sure that reviewers do not feel
> isolated or get lost when they hit content that they don't understand
> fully.
>
>
>
> Ron
>
>
> On 01/04/2018 12:20 AM, Tutkowski, Mike wrote:
> > Thanks for the feedback, Will!
> >
> > I agree with the approach you outlined.
> >
> > Thanks for being so involved in the process! Let’s chat with Giles
> once he’s back to see if we can get your questions answered.
> >
> >> On Mar 31, 2018, at 10:14 PM, Will Stevens 
> wrote:
> >>
> >> In the past the committee was chosen as a relatively small group in
> order
> >> to make it easier to manage feedback.  In order to make it fair to
> everyone
> >> in the community, I would suggest that instead of doing it with a
> small
> >> group, we do it out in the open on a scheduled call.
> >>
> >> We will have to get a list of the talks that are CloudStack
> specific from
> >> ApacheCon, but that should be possible.
> >>
> >> Once we have the talks selected

Re: [DISCUSS] CloudStack graceful shutdown

[Andrija Panic]

One comment here (I had to shutdown whole DC for few hours recently),
please make sure to perhaps at least consider snapshoting process as the
special case - it can take few hours for snapshot to complete really (copy
process from Primary to Secondary Storage)

I did (in my recent unfortunate DC shutdown), actually stop MS (we also
have script to identify running async jobs), so we stop it once safe, but
any running qemu-img processes (we use kVM) need to be killed manually
(ansbile) after MS is stopped, etc,etc...

I can assume most jobs can take reasonable long time to complete, but
snapshots are probably the biggest exceptions as can take extremely long
time to complete...

Cheers

On 4 April 2018 at 22:46, Tutkowski, Mike  wrote:

> I may be remembering this incorrectly, but from what I recall, if a
> resource is owned by one MS and a request related to that resource comes in
> to another MS, the MS that received the request passes it on to the other
> MS.
>
> > On Apr 4, 2018, at 2:36 PM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
> >
> > Big +1 for this feature; I only have a few doubts.
> >
> > * Regarding the tasks/jobs that management servers (MSs) execute; are
> these
> > tasks originate from requests that come to the MS, or is it possible that
> > requests received by one management server to be executed by other? I
> mean,
> > if I execute a request against MS1, will this request always be
> > executed/threated by MS1, or is it possible that this request is executed
> > by another MS (e.g. MS2)?
> >
> > * I would suggest that after we block traffic coming from
> 8080/8443/8250(we
> > will need to block this as well right?), we can log the execution of
> tasks.
> > I mean, something saying, there are XXX tasks (enumerate tasks) still
> being
> > executed, we will wait for them to finish before shutting down.
> >
> > * The timeout (60 minutes suggested) could be global settings that we can
> > load before executing the graceful-shutdown.
> >
> > On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev <
> ilya.mailing.li...@gmail.com>
> > wrote:
> >
> >> Use case:
> >> In any environment - time to time - administrator needs to perform a
> >> maintenance. Current stop sequence of cloudstack management server will
> >> ignore the fact that there may be long running async jobs - and
> terminate
> >> the process. This in turn can create a poor user experience and
> occasional
> >> inconsistency  in cloudstack db.
> >>
> >> This is especially painful in large environments where the user has
> >> thousands of nodes and there is a continuous patching that happens
> around
> >> the clock - that requires migration of workload from one node to
> another.
> >>
> >> With that said - i've created a script that monitors the async job queue
> >> for given MS and waits for it complete all jobs. More details are posted
> >> below.
> >>
> >> I'd like to introduce "graceful-shutdown" into the systemctl/service of
> >> cloudstack-management service.
> >>
> >> The details of how it will work is below:
> >>
> >> Workflow for graceful shutdown:
> >>  Using iptables/firewalld - block any connection attempts on 8080/8443
> (we
> >> can identify the ports dynamically)
> >>  Identify the MSID for the node, using the proper msid - query async_job
> >> table for
> >> 1) any jobs that are still running (or job_status=“0”)
> >> 2) job_dispatcher not like “pseudoJobDispatcher"
> >> 3) job_init_msid=$my_ms_id
> >>
> >> Monitor this async_job table for 60 minutes - until all async jobs for
> MSID
> >> are done, then proceed with shutdown
> >>If failed for any reason or terminated, catch the exit via trap
> command
> >> and unblock the 8080/8443
> >>
> >> Comments are welcome
> >>
> >> Regards,
> >> ilya
> >>
> >
> >
> >
> > --
> > Rafael Weingärtner
>



-- 

Andrija Panić

Re: [DISCUSS] New VPN implementation based on IKEv2 backed by Vault

[Khosrow Moossavi]

Thanks Ilya for the feedback.

The way I currently implemented it, two items need to be set in global
settings beforehand:

- you need to specify the VPN implementation (either L2TP or IKEv2)
- then select the PKI engine backend (Vault or Default)

so there won't be any immediate and blocking coupling between
management-server and Vault.

But...

I would argue that we should leverage these new *tools* in our own infra
where Cloudstack runs, such as ELK, RabbitMQ, Kafka, Redis, MongoDB,
or Vault in this particular case. I would assume everyone of us does use
these tools one way or another to keep the infrastructure up an running.
Why not provide an easy, OOB way to do so from ACS itself?

On the other hand, I totally agree that ACS must not fully depend on these
so if any of these were not available ACS won't work. But at the end of the
day ACS is a *webapp* -which does something special of its own- and it
should get help from all the *cool kids*.

On code quality, test coverage and integrations tests I completely 100%
agree.





On Wed, Apr 4, 2018 at 4:53 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> To complement one thing that Ilya mentioned here. I do not worry much about
> the “requirement” for Vault systems to test ACS. This would be the case if
> Khosrow, when developing, only created tests using what the community calls
> integration tests.
>
> However, it is an implementation from scratch and as such it can and should
> use a very high bar for code quality, which enables proper unit testing for
> all methods. This means, that we can check all of the code in our domain
> (code base) without requiring third-party software. It does not mean that
> we do not need “integration tests” checking the system integration with
> Vault, but we could then restrict this execution to RCs.
>
> On Wed, Apr 4, 2018 at 5:29 PM, ilya musayev  >
> wrote:
>
> > Khosrow
> >
> > My 2c, little less than ideal to manage yet another external end point
> > like.
> >
> > While i understand that it makes it easier to manage certificates - it
> also
> > means going forward - Vault implementation will become a requirement to
> > validate future ACS release.
> >
> > With that said - i do like the proposal and not against it, but:
> > 1) Please consider decoupling it from cloudstack-management server - and
> > release it as server plugin
> > 2) Test coverage must be sufficient enough to validate the functionality
> > (perhaps mock vault endpoints and response)
> >
> > Regards,
> > ilya
> >
> > On Wed, Apr 4, 2018 at 10:49 AM, Khosrow Moossavi <
> kmooss...@cloudops.com>
> > wrote:
> >
> > > Thanks Paul, the proposed feature will enable the functionality to use
> > > Vault to
> > > act as CA if enabled in ACS, otherwise will fall back to "default"
> > > implementation
> > > which Rohit has already done.
> > >
> > >
> > > On Wed, Apr 4, 2018 at 12:29 PM, Paul Angus 
> > > wrote:
> > >
> > > > You guys should speak to Rohit about the CA framework.  CloudStack
> can
> > > > manage certificates now, including creating them itself and acting
> as a
> > > > root CA.
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Paul Angus
> > > >
> > > > paul.an...@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Rafael Weingärtner 
> > > > Sent: 04 April 2018 16:51
> > > > To: dev 
> > > > Subject: Re: [DISCUSS] New VPN implementation based on IKEv2 backed
> by
> > > > Vault
> > > >
> > > > Thanks for sharing the details. Now I have a better perspective of
> the
> > > > proposal.It is an interesting integration of CloudStack VPN service
> > with
> > > > Vault PKI feature.
> > > >
> > > > On Wed, Apr 4, 2018 at 12:38 PM, Khosrow Moossavi <
> > > kmooss...@cloudops.com>
> > > > wrote:
> > > >
> > > > > One of the things Vault does is essentially one of the thing Let's
> > > > > Encrypt does, acting as CA and generating/signing certificates.
> > > > >
> > > > > From the Vault website itself:
> > > > >
> > > > > "HashiCorp Vault secures, stores, and tightly controls access to
> > > > > tokens, passwords, certificates, API keys, and other secrets in
> > modern
> > > > > computing. Vault handles leasing, key revocation, key rolling, and
> > > > > auditing. Through a unified API, users can access an encrypted
> > > > > Key/Value store and network encryption-as-a-service, or generate
> AWS
> > > > > IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH
> > > > > credentials, and more."
> > > > >
> > > > > In our case we are going to use Vault as PKI backend engine, to act
> > as
> > > > > Root CA, sign certificates, handle CRL (Certificate Revocation
> List),
> > > > > etc.
> > > > > Technically we can
> > > > > do these with Let's Encrypt, but I haven't started exploring the
> > > > > possibilities or potential limitation. Using external services
> (such
> > 

Re: [DISCUSS] CloudStack graceful shutdown

[ilya musayev]

Andrija

This is the reason for this enhancement, snapshot, migration and others -
are all async jobs - and therefore should be tracked in async_job table
under specific MS.It is known they may take a while to complete and last
thing we want is to interrupt it.

Depending on what value you have set in Configurations - it may time out -
but continue working on the background.. meaning cloudstack will stop
tracking the async job beyond specific interval - but cloudstack agent will
push forward.

I dont see a harm of taking the server offline - if there are no jobs that
are being tracked.

However - we should not stop the server - if we identify any jobs that are
still active. The user can decide to append the forceful shutdown after the
graceful one if he feels like it. For example

[shell] # service cloudstack-management graceful-shutdown; service
cloudstack-management shutdown

For your issue,

Please check the value for "job.cancel.threshold.minutes"

  "category": "Advanced",

  "description": "Time (in minutes) for async-jobs to be forcely
cancelled if it has been in process for long",

  "name": "job.cancel.threshold.minutes",

  "value": "60"


I propose for the graceful shutdown command to source
"job.cancel.threshold.minutes"
as a max value - before giving up on the endeavor.


The only issue i'm on the fence about - is blocking access to 8080/8443 -
if you have a single node setup.


There is a chance you may block the access to cloudstack for over an hour -
and that may not be what you intended.


Perhaps we add a parameter in db.properties for
"graceful.shutdown.block.api.server = true/false"


Regards,

ilya

On Wed, Apr 4, 2018 at 2:22 PM, Andrija Panic 
wrote:

> One comment here (I had to shutdown whole DC for few hours recently),
> please make sure to perhaps at least consider snapshoting process as the
> special case - it can take few hours for snapshot to complete really (copy
> process from Primary to Secondary Storage)
>
> I did (in my recent unfortunate DC shutdown), actually stop MS (we also
> have script to identify running async jobs), so we stop it once safe, but
> any running qemu-img processes (we use kVM) need to be killed manually
> (ansbile) after MS is stopped, etc,etc...
>
> I can assume most jobs can take reasonable long time to complete, but
> snapshots are probably the biggest exceptions as can take extremely long
> time to complete...
>
> Cheers
>
> On 4 April 2018 at 22:46, Tutkowski, Mike 
> wrote:
>
> > I may be remembering this incorrectly, but from what I recall, if a
> > resource is owned by one MS and a request related to that resource comes
> in
> > to another MS, the MS that received the request passes it on to the other
> > MS.
> >
> > > On Apr 4, 2018, at 2:36 PM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> > >
> > > Big +1 for this feature; I only have a few doubts.
> > >
> > > * Regarding the tasks/jobs that management servers (MSs) execute; are
> > these
> > > tasks originate from requests that come to the MS, or is it possible
> that
> > > requests received by one management server to be executed by other? I
> > mean,
> > > if I execute a request against MS1, will this request always be
> > > executed/threated by MS1, or is it possible that this request is
> executed
> > > by another MS (e.g. MS2)?
> > >
> > > * I would suggest that after we block traffic coming from
> > 8080/8443/8250(we
> > > will need to block this as well right?), we can log the execution of
> > tasks.
> > > I mean, something saying, there are XXX tasks (enumerate tasks) still
> > being
> > > executed, we will wait for them to finish before shutting down.
> > >
> > > * The timeout (60 minutes suggested) could be global settings that we
> can
> > > load before executing the graceful-shutdown.
> > >
> > > On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev <
> > ilya.mailing.li...@gmail.com>
> > > wrote:
> > >
> > >> Use case:
> > >> In any environment - time to time - administrator needs to perform a
> > >> maintenance. Current stop sequence of cloudstack management server
> will
> > >> ignore the fact that there may be long running async jobs - and
> > terminate
> > >> the process. This in turn can create a poor user experience and
> > occasional
> > >> inconsistency  in cloudstack db.
> > >>
> > >> This is especially painful in large environments where the user has
> > >> thousands of nodes and there is a continuous patching that happens
> > around
> > >> the clock - that requires migration of workload from one node to
> > another.
> > >>
> > >> With that said - i've created a script that monitors the async job
> queue
> > >> for given MS and waits for it complete all jobs. More details are
> posted
> > >> below.
> > >>
> > >> I'd like to introduce "graceful-shutdown" into the systemctl/service
> of
> > >> cloudstack-management service.
> > >>
> > >> The details of how it will work is below:
> > >>
> > >> Workflow for graceful shutdown:
> > >>  Using ipt

Re: [DISCUSS] CloudStack graceful shutdown

[ilya musayev]

Rafael

> * Regarding the tasks/jobs that management servers (MSs) execute; are
these
tasks originate from requests that come to the MS, or is it possible that
requests received by one management server to be executed by other? I mean,
if I execute a request against MS1, will this request always be
executed/threated by MS1, or is it possible that this request is executed
by another MS (e.g. MS2)?

Yes its possible, but it will be tracked under async_job with proper MS
that is responsible for this task.

My initial goal was to prevent user from creating more async jobs on the
node thats about to go down for maintenance - but as i'm thinking about it
- i dont know if it matters - since async job will be executed on the MS
node that tracks a specific hypervisor/agent - as defined in cloud.host
table.

Maybe i'll leave off the blocking off 8080/8443 and just focus on tracking
async_jobs instead. Assuming you are managing your MS with Load Balancer -
it should be smart enough to shift the user traffic to MS that is up.

> * I would suggest that after we block traffic coming from
8080/8443/8250(we
will need to block this as well right?), we can log the execution of tasks.
I mean, something saying, there are XXX tasks (enumerate tasks) still being
executed, we will wait for them to finish before shutting down

8250 - is a bit too aggressive in my opinion andwe dont want to do that. If
you block 8250 and you have a long running tasks - you are waiting on to
complete - then it may fail - because you block agent communication on 8250.

Thanks
ilya


On Wed, Apr 4, 2018 at 1:36 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Big +1 for this feature; I only have a few doubts.
>
> * Regarding the tasks/jobs that management servers (MSs) execute; are these
> tasks originate from requests that come to the MS, or is it possible that
> requests received by one management server to be executed by other? I mean,
> if I execute a request against MS1, will this request always be
> executed/threated by MS1, or is it possible that this request is executed
> by another MS (e.g. MS2)?
>
> * I would suggest that after we block traffic coming from 8080/8443/8250(we
> will need to block this as well right?), we can log the execution of tasks.
> I mean, something saying, there are XXX tasks (enumerate tasks) still being
> executed, we will wait for them to finish before shutting down.
>
> * The timeout (60 minutes suggested) could be global settings that we can
> load before executing the graceful-shutdown.
>
> On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev  >
> wrote:
>
> > Use case:
> > In any environment - time to time - administrator needs to perform a
> > maintenance. Current stop sequence of cloudstack management server will
> > ignore the fact that there may be long running async jobs - and terminate
> > the process. This in turn can create a poor user experience and
> occasional
> > inconsistency  in cloudstack db.
> >
> > This is especially painful in large environments where the user has
> > thousands of nodes and there is a continuous patching that happens around
> > the clock - that requires migration of workload from one node to another.
> >
> > With that said - i've created a script that monitors the async job queue
> > for given MS and waits for it complete all jobs. More details are posted
> > below.
> >
> > I'd like to introduce "graceful-shutdown" into the systemctl/service of
> > cloudstack-management service.
> >
> > The details of how it will work is below:
> >
> > Workflow for graceful shutdown:
> >   Using iptables/firewalld - block any connection attempts on 8080/8443
> (we
> > can identify the ports dynamically)
> >   Identify the MSID for the node, using the proper msid - query async_job
> > table for
> > 1) any jobs that are still running (or job_status=“0”)
> > 2) job_dispatcher not like “pseudoJobDispatcher"
> > 3) job_init_msid=$my_ms_id
> >
> > Monitor this async_job table for 60 minutes - until all async jobs for
> MSID
> > are done, then proceed with shutdown
> > If failed for any reason or terminated, catch the exit via trap
> command
> > and unblock the 8080/8443
> >
> > Comments are welcome
> >
> > Regards,
> > ilya
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: [DISCUSS] CloudStack graceful shutdown

[ilya musayev]

I'm thinking of using a configuration from "job.cancel.threshold.minutes" -
it will be the longest

  "category": "Advanced",

  "description": "Time (in minutes) for async-jobs to be forcely
cancelled if it has been in process for long",

  "name": "job.cancel.threshold.minutes",

  "value": "60"




On Wed, Apr 4, 2018 at 1:36 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Big +1 for this feature; I only have a few doubts.
>
> * Regarding the tasks/jobs that management servers (MSs) execute; are these
> tasks originate from requests that come to the MS, or is it possible that
> requests received by one management server to be executed by other? I mean,
> if I execute a request against MS1, will this request always be
> executed/threated by MS1, or is it possible that this request is executed
> by another MS (e.g. MS2)?
>
> * I would suggest that after we block traffic coming from 8080/8443/8250(we
> will need to block this as well right?), we can log the execution of tasks.
> I mean, something saying, there are XXX tasks (enumerate tasks) still being
> executed, we will wait for them to finish before shutting down.
>
> * The timeout (60 minutes suggested) could be global settings that we can
> load before executing the graceful-shutdown.
>
> On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev  >
> wrote:
>
> > Use case:
> > In any environment - time to time - administrator needs to perform a
> > maintenance. Current stop sequence of cloudstack management server will
> > ignore the fact that there may be long running async jobs - and terminate
> > the process. This in turn can create a poor user experience and
> occasional
> > inconsistency  in cloudstack db.
> >
> > This is especially painful in large environments where the user has
> > thousands of nodes and there is a continuous patching that happens around
> > the clock - that requires migration of workload from one node to another.
> >
> > With that said - i've created a script that monitors the async job queue
> > for given MS and waits for it complete all jobs. More details are posted
> > below.
> >
> > I'd like to introduce "graceful-shutdown" into the systemctl/service of
> > cloudstack-management service.
> >
> > The details of how it will work is below:
> >
> > Workflow for graceful shutdown:
> >   Using iptables/firewalld - block any connection attempts on 8080/8443
> (we
> > can identify the ports dynamically)
> >   Identify the MSID for the node, using the proper msid - query async_job
> > table for
> > 1) any jobs that are still running (or job_status=“0”)
> > 2) job_dispatcher not like “pseudoJobDispatcher"
> > 3) job_init_msid=$my_ms_id
> >
> > Monitor this async_job table for 60 minutes - until all async jobs for
> MSID
> > are done, then proceed with shutdown
> > If failed for any reason or terminated, catch the exit via trap
> command
> > and unblock the 8080/8443
> >
> > Comments are welcome
> >
> > Regards,
> > ilya
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: [DISCUSS] CloudStack graceful shutdown

[Rafael Weingärtner]

Ilya, still regarding the management server that is being shut down issue;
if other MSs/or maybe system VMs (I am not sure to know if they are able to
do such tasks) can direct/redirect/send new jobs to this management server
(the one being shut down), the process might never end because new tasks
are always being created for the management server that we want to shut
down. Is this scenario possible?

That is why I mentioned blocking the port 8250 for the “graceful-shutdown”.

If this scenario is not possible, then everything s fine.


On Wed, Apr 4, 2018 at 7:14 PM, ilya musayev 
wrote:

> I'm thinking of using a configuration from "job.cancel.threshold.minutes" -
> it will be the longest
>
>   "category": "Advanced",
>
>   "description": "Time (in minutes) for async-jobs to be forcely
> cancelled if it has been in process for long",
>
>   "name": "job.cancel.threshold.minutes",
>
>   "value": "60"
>
>
>
>
> On Wed, Apr 4, 2018 at 1:36 PM, Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Big +1 for this feature; I only have a few doubts.
> >
> > * Regarding the tasks/jobs that management servers (MSs) execute; are
> these
> > tasks originate from requests that come to the MS, or is it possible that
> > requests received by one management server to be executed by other? I
> mean,
> > if I execute a request against MS1, will this request always be
> > executed/threated by MS1, or is it possible that this request is executed
> > by another MS (e.g. MS2)?
> >
> > * I would suggest that after we block traffic coming from
> 8080/8443/8250(we
> > will need to block this as well right?), we can log the execution of
> tasks.
> > I mean, something saying, there are XXX tasks (enumerate tasks) still
> being
> > executed, we will wait for them to finish before shutting down.
> >
> > * The timeout (60 minutes suggested) could be global settings that we can
> > load before executing the graceful-shutdown.
> >
> > On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev <
> ilya.mailing.li...@gmail.com
> > >
> > wrote:
> >
> > > Use case:
> > > In any environment - time to time - administrator needs to perform a
> > > maintenance. Current stop sequence of cloudstack management server will
> > > ignore the fact that there may be long running async jobs - and
> terminate
> > > the process. This in turn can create a poor user experience and
> > occasional
> > > inconsistency  in cloudstack db.
> > >
> > > This is especially painful in large environments where the user has
> > > thousands of nodes and there is a continuous patching that happens
> around
> > > the clock - that requires migration of workload from one node to
> another.
> > >
> > > With that said - i've created a script that monitors the async job
> queue
> > > for given MS and waits for it complete all jobs. More details are
> posted
> > > below.
> > >
> > > I'd like to introduce "graceful-shutdown" into the systemctl/service of
> > > cloudstack-management service.
> > >
> > > The details of how it will work is below:
> > >
> > > Workflow for graceful shutdown:
> > >   Using iptables/firewalld - block any connection attempts on 8080/8443
> > (we
> > > can identify the ports dynamically)
> > >   Identify the MSID for the node, using the proper msid - query
> async_job
> > > table for
> > > 1) any jobs that are still running (or job_status=“0”)
> > > 2) job_dispatcher not like “pseudoJobDispatcher"
> > > 3) job_init_msid=$my_ms_id
> > >
> > > Monitor this async_job table for 60 minutes - until all async jobs for
> > MSID
> > > are done, then proceed with shutdown
> > > If failed for any reason or terminated, catch the exit via trap
> > command
> > > and unblock the 8080/8443
> > >
> > > Comments are welcome
> > >
> > > Regards,
> > > ilya
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner

Re: [DISCUSS] CloudStack graceful shutdown

[Sergey Levitskiy]

This is not simple e.g. for VMware. Each management server also acts as an 
agent proxy so tasks against a particular ESX host will be always forwarded. 
That right answer will be to a native support for “maintenance mode” for 
management server. When entered to such mode the management server should 
release all agents including save, block/redirect API calls and login request 
and finish all a sync job it originated.

Sent from my iPhone

> On Apr 4, 2018, at 3:31 PM, Rafael Weingärtner  
> wrote:
> 
> Ilya, still regarding the management server that is being shut down issue;
> if other MSs/or maybe system VMs (I am not sure to know if they are able to
> do such tasks) can direct/redirect/send new jobs to this management server
> (the one being shut down), the process might never end because new tasks
> are always being created for the management server that we want to shut
> down. Is this scenario possible?
> 
> That is why I mentioned blocking the port 8250 for the “graceful-shutdown”.
> 
> If this scenario is not possible, then everything s fine.
> 
> 
> On Wed, Apr 4, 2018 at 7:14 PM, ilya musayev 
> wrote:
> 
>> I'm thinking of using a configuration from "job.cancel.threshold.minutes" -
>> it will be the longest
>> 
>>  "category": "Advanced",
>> 
>>  "description": "Time (in minutes) for async-jobs to be forcely
>> cancelled if it has been in process for long",
>> 
>>  "name": "job.cancel.threshold.minutes",
>> 
>>  "value": "60"
>> 
>> 
>> 
>> 
>> On Wed, Apr 4, 2018 at 1:36 PM, Rafael Weingärtner <
>> rafaelweingart...@gmail.com> wrote:
>> 
>>> Big +1 for this feature; I only have a few doubts.
>>> 
>>> * Regarding the tasks/jobs that management servers (MSs) execute; are
>> these
>>> tasks originate from requests that come to the MS, or is it possible that
>>> requests received by one management server to be executed by other? I
>> mean,
>>> if I execute a request against MS1, will this request always be
>>> executed/threated by MS1, or is it possible that this request is executed
>>> by another MS (e.g. MS2)?
>>> 
>>> * I would suggest that after we block traffic coming from
>> 8080/8443/8250(we
>>> will need to block this as well right?), we can log the execution of
>> tasks.
>>> I mean, something saying, there are XXX tasks (enumerate tasks) still
>> being
>>> executed, we will wait for them to finish before shutting down.
>>> 
>>> * The timeout (60 minutes suggested) could be global settings that we can
>>> load before executing the graceful-shutdown.
>>> 
>>> On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev <
>> ilya.mailing.li...@gmail.com
 
>>> wrote:
>>> 
 Use case:
 In any environment - time to time - administrator needs to perform a
 maintenance. Current stop sequence of cloudstack management server will
 ignore the fact that there may be long running async jobs - and
>> terminate
 the process. This in turn can create a poor user experience and
>>> occasional
 inconsistency  in cloudstack db.
 
 This is especially painful in large environments where the user has
 thousands of nodes and there is a continuous patching that happens
>> around
 the clock - that requires migration of workload from one node to
>> another.
 
 With that said - i've created a script that monitors the async job
>> queue
 for given MS and waits for it complete all jobs. More details are
>> posted
 below.
 
 I'd like to introduce "graceful-shutdown" into the systemctl/service of
 cloudstack-management service.
 
 The details of how it will work is below:
 
 Workflow for graceful shutdown:
  Using iptables/firewalld - block any connection attempts on 8080/8443
>>> (we
 can identify the ports dynamically)
  Identify the MSID for the node, using the proper msid - query
>> async_job
 table for
 1) any jobs that are still running (or job_status=“0”)
 2) job_dispatcher not like “pseudoJobDispatcher"
 3) job_init_msid=$my_ms_id
 
 Monitor this async_job table for 60 minutes - until all async jobs for
>>> MSID
 are done, then proceed with shutdown
If failed for any reason or terminated, catch the exit via trap
>>> command
 and unblock the 8080/8443
 
 Comments are welcome
 
 Regards,
 ilya
 
>>> 
>>> 
>>> 
>>> --
>>> Rafael Weingärtner
>>> 
>> 
> 
> 
> 
> -- 
> Rafael Weingärtner

Re: [DISCUSS] CloudStack graceful shutdown

[Sergey Levitskiy]

Now without spellchecking :)

This is not simple e.g. for VMware. Each management server also acts as an 
agent proxy so tasks against a particular ESX host will be always forwarded. 
That right answer will be to support a native “maintenance mode” for management 
server. When entered to such mode the management server should release all 
agents including SSVM, block/redirect API calls and login request and finish 
all async job it originated.



On Apr 4, 2018, at 5:15 PM, Sergey Levitskiy 
mailto:serg...@hotmail.com>> wrote:

This is not simple e.g. for VMware. Each management server also acts as an 
agent proxy so tasks against a particular ESX host will be always forwarded. 
That right answer will be to a native support for “maintenance mode” for 
management server. When entered to such mode the management server should 
release all agents including save, block/redirect API calls and login request 
and finish all a sync job it originated.

Sent from my iPhone

On Apr 4, 2018, at 3:31 PM, Rafael Weingärtner 
mailto:rafaelweingart...@gmail.com>> wrote:

Ilya, still regarding the management server that is being shut down issue;
if other MSs/or maybe system VMs (I am not sure to know if they are able to
do such tasks) can direct/redirect/send new jobs to this management server
(the one being shut down), the process might never end because new tasks
are always being created for the management server that we want to shut
down. Is this scenario possible?

That is why I mentioned blocking the port 8250 for the “graceful-shutdown”.

If this scenario is not possible, then everything s fine.


On Wed, Apr 4, 2018 at 7:14 PM, ilya musayev 
mailto:ilya.mailing.li...@gmail.com>>
wrote:

I'm thinking of using a configuration from "job.cancel.threshold.minutes" -
it will be the longest

"category": "Advanced",

"description": "Time (in minutes) for async-jobs to be forcely
cancelled if it has been in process for long",

"name": "job.cancel.threshold.minutes",

"value": "60"




On Wed, Apr 4, 2018 at 1:36 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

Big +1 for this feature; I only have a few doubts.

* Regarding the tasks/jobs that management servers (MSs) execute; are
these
tasks originate from requests that come to the MS, or is it possible that
requests received by one management server to be executed by other? I
mean,
if I execute a request against MS1, will this request always be
executed/threated by MS1, or is it possible that this request is executed
by another MS (e.g. MS2)?

* I would suggest that after we block traffic coming from
8080/8443/8250(we
will need to block this as well right?), we can log the execution of
tasks.
I mean, something saying, there are XXX tasks (enumerate tasks) still
being
executed, we will wait for them to finish before shutting down.

* The timeout (60 minutes suggested) could be global settings that we can
load before executing the graceful-shutdown.

On Wed, Apr 4, 2018 at 5:15 PM, ilya musayev <
ilya.mailing.li...@gmail.com

wrote:

Use case:
In any environment - time to time - administrator needs to perform a
maintenance. Current stop sequence of cloudstack management server will
ignore the fact that there may be long running async jobs - and
terminate
the process. This in turn can create a poor user experience and
occasional
inconsistency  in cloudstack db.

This is especially painful in large environments where the user has
thousands of nodes and there is a continuous patching that happens
around
the clock - that requires migration of workload from one node to
another.

With that said - i've created a script that monitors the async job
queue
for given MS and waits for it complete all jobs. More details are
posted
below.

I'd like to introduce "graceful-shutdown" into the systemctl/service of
cloudstack-management service.

The details of how it will work is below:

Workflow for graceful shutdown:
Using iptables/firewalld - block any connection attempts on 8080/8443
(we
can identify the ports dynamically)
Identify the MSID for the node, using the proper msid - query
async_job
table for
1) any jobs that are still running (or job_status=“0”)
2) job_dispatcher not like “pseudoJobDispatcher"
3) job_init_msid=$my_ms_id

Monitor this async_job table for 60 minutes - until all async jobs for
MSID
are done, then proceed with shutdown
  If failed for any reason or terminated, catch the exit via trap
command
and unblock the 8080/8443

Comments are welcome

Regards,
ilya




--
Rafael Weingärtner





--
Rafael Weingärtner