Re: IPV6 in Isolated/VPC networks
Thanks Hean, Kristaps, Wido for your feedback. I think we've some quorum and consensus on how we should proceed with IPv6 support with static routing (phase1). Based on my proof-of-concept and discussions, I believe we may target this feature as early as 4.17 and I welcome offer by Kristaps and others who may want to be involved in testing the feature as and when we'll develop it. As the next step I'll write a short design doc on cwiki including some of the new ideas/suggestions and share it on this thread for another iteration. Regards. From: Wido den Hollander Sent: Friday, August 13, 2021 15:48 To: dev@cloudstack.apache.org Subject: Re: IPV6 in Isolated/VPC networks Hi, See my inline responses: Op 11-08-2021 om 14:26 schreef Rohit Yadav: > Hi all, > > Thanks for your feedback and ideas, I've gone ahead with discussing them with > Alex and came up with a PoC/design which can be implemented in the following > phases: > >* Phase1: implement ipv6 support in isolated networks and VPC with > static routing >* Phase2: discuss and implement support for dynamic routing (TBD) > > For Phase1 here's the high-level proposal: > >* IPv6 address management: > * At the zone level root-admin specifies a /64 public range that will > be used for VRs, then they can add a /48, or /56 IPv6 range for guest > networks (to be used by isolated networks and VPC tiers) > * On creation of any IPv6 enabled isolated network or VPC tier, from > the /48 or /56 block a /64 network is allocated/used > * We assume SLAAC and autoconfiguration, no DHCPv6 in the zone > (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be > explored?) Privacy Extensions are only a concern for client devices which roam between different IPv6 networks. If you IPv6 address of a client keeps the same suffix (MAC based) and switches network then only the prefix (/64) will change. This way a network like Google, Facebook, etc could track your device moving from network to network if they only look at the last 64-bits of the IPv6 address. For servers this is not a problem as you already know in which network they are. >* Network offerings: root-admin can create new network offerings (with > VPC too) that specifies a network stack option: > * ipv4 only (default, for backward compatibility all > networks/offerings post-upgrade migrate to this option) > * ipv4-and-ipv6 > * ipv6-only (this can be phase 1.b) > * A new routing option: static (phase1), dynamic (phase2, with > multiple sub-options such as ospf/bgp etc...) This means that the network admin will need to statically route the IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router: set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop 2001:db8:100::50 I'm assuming that 2001:db8:100::50 is the address of the VR on the outside (/64) network. In reality this will probably be a longer address, but this is for just the example. >* VR changes: > * VR gets its guest and public nics set to inet6 auto > * For each /64 allocated to guest network and VPC tiers, radvd is > configured to do RA radvd is fine, but looking at phase 2 with dynamic routing you might already want to look into FRRouting. FRR can also advertise RAs while not doing any routing. interface ens4 no ipv6 nd suppress-ra ipv6 nd prefix 2001:db8:500::/64 ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 See: http://docs.frrouting.org/en/latest/ipv6.html > * Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 > firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC > tiers are managed/implemented by ipv6 firewall on VR Please take a look at the existing security_group.py script which implements RFC4890 https://datatracker.ietf.org/doc/html/rfc4890 ICMPv6 is a vital part of IPv6 and certain packets should always be allowed. > * It is assumed that static routes are created on the core/main > router by the admin or automated using some scripts/tools; for this > CloudStack will announce events with details of /64 networks and VR's public > IPv6 address that can be consumed by a rabbitmq/message bus client (for > example), or a custom cron job or script as part of orchestration. (this > wouldn't be necessary for dynamic routing bgp with phase2)\\ You would only need to announce the /48 or /56 allocated to the VR, that's all. You don't need to inform the upstream router about the /64 subnets created within that larger subnet. >* Guest Networking: With SLAAC, it's easy for CloudStack to calculate > allocate and use a /64 and determine the IPv6 address of VR nics and guest VM > nics > * A user create an isolated network/VPC with an offering that is ipv6 > enabled > * A user can manage firewall for the IPv6 address/guest nics; > there'll be no po
Re: [DISCUSS] NextGen VR?
Hi PL, I tested OpenWRT on KVM, and I don't see any performance loss in terms of bandwidth/IO and memory, in fact it seems to be faster than the Debian based systemvmtemplate due to: * Smaller kernel size (the Debian kernel includes all sorts of device drivers for sound and what not which is not necessary for VR/systemvm) * Faster ssh (the dropbear ssh kernel is on-par with openssh-server but has smaller footprint and missing features such as sftp that we don't need in VR/systemvm; the ssh and programming seemed really fast compared to Debian's openssh-server) * Really low memory consumption (Debian10+ eats up some RAM due to non-optimised services, esp systemd compared to OpenWRT) * Lua based really-fast APIs and bus (openwrt has https://openwrt.org/docs/techref/ubus, I'm looking into how we can build something similar for our Debian based systemvmtemplate if migrating to OpenWRT is going to be a pain or unacceptable by the community) Based on your comment I found that VyOS rolling release has API support (https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however that is probably not supported for their lts/legacy release, we'll also face issue with installing/configuring custom packages (java etc) that we need for systemvm/VR. Regards. From: Pierre-Luc Dion Sent: Friday, August 13, 2021 21:51 To: dev Subject: Re: [DISCUSS] NextGen VR? >From a certain point of view, OpenWRT is most likely not suitable for our needs, this project is highly oriented for router appliance platforms and kernel is most likely highly optimised for a small memory footprint and might not have performance and virtualization in mind. It's a shame that there is not some sort of API driven open source VyOS :-S Anything useful that would come out of the CloudNative projects? On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav wrote: > Hi Pierre-Luc, > > Thanks for replying. > > We've another thread on-going to discuss IPv6 support where we've indeed > discussed idea of introducing dynamic routing capabilities in the VR in a > future phase with FRR in the VR. > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > iptables, software packages/services such as dnsmasq etc. Right now it's > possible to configure some of the services in network offering (for ex. if > you don't want LB etc). > > I've indeed looked at some alternatives: > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > Alpine and the gains weren't significant so we decided to drop the idea of > moving from Debian to Alpine. > * VyOS: I've concerns about project and community (I could be wrong) > to have our VR depend on it as the "default" provider; though I see an old > PR > * OpenWRT: Among available router distributions, I really liked it - > it has (a) a good health and active community and commit activity ( > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > configuration system (UCI) and UI (LUCI) which can be used to manage > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > packaging system, most software packages or equivalent are available that > our Debian based systemvmtemplate needs/uses (java jre workaround from > Alpine though), (e) good documentation on extending features/writing > plugins etc, (f) they tend to do regular releases and use Linux/LTS kernel. > * pfSense/opnSense: the major cons are FreeBSD-based kernel and skills > that CloudStack community may not have in terms of debugging, investigation > etc like they have for Linux > > Among the above, OpenWRT is probably worth exploring in my opinion; other > some firewall/iptables and iproute2 (interfaces and routes) API/library > that can be used by the VR agent on Debian to program the firewall (acls, > firewall, pf), IP/interface management and routing features in the VR > (currently it's all done in a manual sense which is untested and often > source of bugs and issues). > > Lastly, we already have this PR which aims to do automatic > systemvmtemplate registeration/installation/seeding and handle upgrades > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The > PR also unifies the systemvm template use to be extended for CKS use-case > (i.e. no separate template installation required for CKS). > > > Regards. > > > From: Pierre-Luc Dion > Sent: Thursday, August 12, 2021 16:45 > To: dev > Subject: Re: [DISCUSS] NextGen VR? > > Hi Rohit, > > Like it! Our VR system is due for some rethinking! > I don't have much points to add to issues you highlight, it seems pretty > complete. > > Here are some more features or ideas that would be interesting to see
Re: [DISCUSS] NextGen VR?
Hi Rohit, So look like OpenWRT could be a good alternative then, but how would you handle other services such as load-balancing, vpn,...? would that be from another Service VMs, kind of doing service mesh ? Cheers, On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav wrote: > Hi PL, > > I tested OpenWRT on KVM, and I don't see any performance loss in terms of > bandwidth/IO and memory, in fact it seems to be faster than the Debian > based systemvmtemplate due to: > > * Smaller kernel size (the Debian kernel includes all sorts of device > drivers for sound and what not which is not necessary for VR/systemvm) > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server > but has smaller footprint and missing features such as sftp that we don't > need in VR/systemvm; the ssh and programming seemed really fast compared to > Debian's openssh-server) > * Really low memory consumption (Debian10+ eats up some RAM due to > non-optimised services, esp systemd compared to OpenWRT) > * Lua based really-fast APIs and bus (openwrt has > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build > something similar for our Debian based systemvmtemplate if migrating to > OpenWRT is going to be a pain or unacceptable by the community) > > Based on your comment I found that VyOS rolling release has API support ( > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however > that is probably not supported for their lts/legacy release, we'll also > face issue with installing/configuring custom packages (java etc) that we > need for systemvm/VR. > > > Regards. > > > From: Pierre-Luc Dion > Sent: Friday, August 13, 2021 21:51 > To: dev > Subject: Re: [DISCUSS] NextGen VR? > > From a certain point of view, OpenWRT is most likely not suitable for our > needs, this project is highly oriented for router appliance platforms and > kernel is most likely highly optimised for a small memory footprint and > might not have performance and virtualization in mind. > > It's a shame that there is not some sort of API driven open source VyOS :-S > > Anything useful that would come out of the CloudNative projects? > > > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav > wrote: > > > Hi Pierre-Luc, > > > > Thanks for replying. > > > > We've another thread on-going to discuss IPv6 support where we've indeed > > discussed idea of introducing dynamic routing capabilities in the VR in a > > future phase with FRR in the VR. > > > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > > iptables, software packages/services such as dnsmasq etc. Right now it's > > possible to configure some of the services in network offering (for ex. > if > > you don't want LB etc). > > > > I've indeed looked at some alternatives: > > > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > > Alpine and the gains weren't significant so we decided to drop the idea > of > > moving from Debian to Alpine. > > * VyOS: I've concerns about project and community (I could be wrong) > > to have our VR depend on it as the "default" provider; though I see an > old > > PR > > * OpenWRT: Among available router distributions, I really liked it - > > it has (a) a good health and active community and commit activity ( > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > > configuration system (UCI) and UI (LUCI) which can be used to manage > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > > packaging system, most software packages or equivalent are available that > > our Debian based systemvmtemplate needs/uses (java jre workaround from > > Alpine though), (e) good documentation on extending features/writing > > plugins etc, (f) they tend to do regular releases and use Linux/LTS > kernel. > > * pfSense/opnSense: the major cons are FreeBSD-based kernel and > skills > > that CloudStack community may not have in terms of debugging, > investigation > > etc like they have for Linux > > > > Among the above, OpenWRT is probably worth exploring in my opinion; other > > some firewall/iptables and iproute2 (interfaces and routes) API/library > > that can be used by the VR agent on Debian to program the firewall (acls, > > firewall, pf), IP/interface management and routing features in the VR > > (currently it's all done in a manual sense which is untested and often > > source of bugs and issues). > > > > Lastly, we already have this PR which aims to do automatic > > systemvmtemplate registeration/installation/seeding and handle upgrades > > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The > > PR also unifies the systemvm template use to be extended for CKS use-case > > (i
Re: [DISCUSS] NextGen VR?
Hi All, Perhaps you also want to have a look at OPNSense. I think others also mentioned it before. It is mature and stable. It is based on hardened free-bsd. I have worked with Ansible Modules to automate its deployment and I can see a lot of benefits that would make integration easy. For example its configuration is robust and comes from a single file that can recreate the whole firewall configuration: Routes, VPNs, Certificate Authorities, NAT Rules, DHCP Servers, Reservations, Resolvers, Optional Packages, Intrusion Detection, etc. Some functions, like Rules, are available via API. I know the project team and they are working on improving API so that all the firewall options are available also via API. Perhaps there is a natural fit for ACS. Rafael On Mon, 2021-08-16 04:54 PM, Pierre-Luc Dion wrote: > Hi Rohit, > > So look like OpenWRT could be a good alternative then, but how would you > handle other services such as load-balancing, vpn,...? would that be from > another Service VMs, kind of doing service mesh ? > > Cheers, > > > > On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav " > target="_blank"> > wrote: > > > Hi PL, > > > > I tested OpenWRT on KVM, and I don't see any performance loss in terms of > > bandwidth/IO and memory, in fact it seems to be faster than the Debian > > based systemvmtemplate due to: > > > > * Smaller kernel size (the Debian kernel includes all sorts of device > > drivers for sound and what not which is not necessary for VR/systemvm) > > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server > > but has smaller footprint and missing features such as sftp that we don't > > need in VR/systemvm; the ssh and programming seemed really fast compared to > > Debian's openssh-server) > > * Really low memory consumption (Debian10+ eats up some RAM due to > > non-optimised services, esp systemd compared to OpenWRT) > > * Lua based really-fast APIs and bus (openwrt has > > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build > > something similar for our Debian based systemvmtemplate if migrating to > > OpenWRT is going to be a pain or unacceptable by the community) > > > > Based on your comment I found that VyOS rolling release has API support ( > > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however > > that is probably not supported for their lts/legacy release, we'll also > > face issue with installing/configuring custom packages (java etc) that we > > need for systemvm/VR. > > > > > > Regards. > > > > > > From: Pierre-Luc Dion " target="_blank"> > > Sent: Friday, August 13, 2021 21:51 > > To: dev " target="_blank"> > > Subject: Re: [DISCUSS] NextGen VR? > > > > From a certain point of view, OpenWRT is most likely not suitable for our > > needs, this project is highly oriented for router appliance platforms and > > kernel is most likely highly optimised for a small memory footprint and > > might not have performance and virtualization in mind. > > > > It's a shame that there is not some sort of API driven open source VyOS :-S > > > > Anything useful that would come out of the CloudNative projects? > > > > > > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav " > > target="_blank"> > > wrote: > > > > > Hi Pierre-Luc, > > > > > > Thanks for replying. > > > > > > We've another thread on-going to discuss IPv6 support where we've indeed > > > discussed idea of introducing dynamic routing capabilities in the VR in a > > > future phase with FRR in the VR. > > > > > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > > > iptables, software packages/services such as dnsmasq etc. Right now it's > > > possible to configure some of the services in network offering (for ex. > > if > > > you don't want LB etc). > > > > > > I've indeed looked at some alternatives: > > > > > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > > > Alpine and the gains weren't significant so we decided to drop the idea > > of > > > moving from Debian to Alpine. > > > * VyOS: I've concerns about project and community (I could be wrong) > > > to have our VR depend on it as the "default" provider; though I see an > > old > > > PR > > > * OpenWRT: Among available router distributions, I really liked it - > > > it has (a) a good health and active community and commit activity ( > > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > > > configuration system (UCI) and UI (LUCI) which can be used to manage > > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > > > packaging system, most software packages or equivalent are available that > > > our Debian based systemvmtemplate needs/uses
Re: [DISCUSS] NextGen VR?
Hi PL, I think Openwrt can do all of them, similar to Debian' apt-get Openwrt ships with a opkg package management tool that supports thousands of pkgs and using opkg you can install any of the packages such as strongswan/ipsec, haproxy, java/jre for systemvm agents etc, customise the rc.d startups scripts etc. In the meanwhile I'm also exploring Debian and some IPC and rpc mechanisms for the VR agent with some success using named pipes. One issue why VR programming is slow is because for every operation, currently we first create and scp a json file and then call a python script to process unprocessed json databags, this two part process can be combined into a single process where we scp the json databag (or similar rpc payload) directly to a named pipe path which a VR agent listens to and reads and processes (much like a http server), this would make two scp/ssh sessions unnecessary. Still exploring and doing some PoC in this direction. Regards. From: Pierre-Luc Dion Sent: Monday, August 16, 2021 8:24:20 PM To: dev Subject: Re: [DISCUSS] NextGen VR? Hi Rohit, So look like OpenWRT could be a good alternative then, but how would you handle other services such as load-balancing, vpn,...? would that be from another Service VMs, kind of doing service mesh ? Cheers, On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav wrote: > Hi PL, > > I tested OpenWRT on KVM, and I don't see any performance loss in terms of > bandwidth/IO and memory, in fact it seems to be faster than the Debian > based systemvmtemplate due to: > > * Smaller kernel size (the Debian kernel includes all sorts of device > drivers for sound and what not which is not necessary for VR/systemvm) > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server > but has smaller footprint and missing features such as sftp that we don't > need in VR/systemvm; the ssh and programming seemed really fast compared to > Debian's openssh-server) > * Really low memory consumption (Debian10+ eats up some RAM due to > non-optimised services, esp systemd compared to OpenWRT) > * Lua based really-fast APIs and bus (openwrt has > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build > something similar for our Debian based systemvmtemplate if migrating to > OpenWRT is going to be a pain or unacceptable by the community) > > Based on your comment I found that VyOS rolling release has API support ( > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however > that is probably not supported for their lts/legacy release, we'll also > face issue with installing/configuring custom packages (java etc) that we > need for systemvm/VR. > > > Regards. > > > From: Pierre-Luc Dion > Sent: Friday, August 13, 2021 21:51 > To: dev > Subject: Re: [DISCUSS] NextGen VR? > > From a certain point of view, OpenWRT is most likely not suitable for our > needs, this project is highly oriented for router appliance platforms and > kernel is most likely highly optimised for a small memory footprint and > might not have performance and virtualization in mind. > > It's a shame that there is not some sort of API driven open source VyOS :-S > > Anything useful that would come out of the CloudNative projects? > > > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav > wrote: > > > Hi Pierre-Luc, > > > > Thanks for replying. > > > > We've another thread on-going to discuss IPv6 support where we've indeed > > discussed idea of introducing dynamic routing capabilities in the VR in a > > future phase with FRR in the VR. > > > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > > iptables, software packages/services such as dnsmasq etc. Right now it's > > possible to configure some of the services in network offering (for ex. > if > > you don't want LB etc). > > > > I've indeed looked at some alternatives: > > > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > > Alpine and the gains weren't significant so we decided to drop the idea > of > > moving from Debian to Alpine. > > * VyOS: I've concerns about project and community (I could be wrong) > > to have our VR depend on it as the "default" provider; though I see an > old > > PR > > * OpenWRT: Among available router distributions, I really liked it - > > it has (a) a good health and active community and commit activity ( > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > > configuration system (UCI) and UI (LUCI) which can be used to manage > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > > packaging system, most software packages or equivalent are available that > > our Debian based systemvm
[GitHub] [cloudstack-documentation] davidjumani opened a new pull request #237: Adding a note on xen boot mode
davidjumani opened a new pull request #237: URL: https://github.com/apache/cloudstack-documentation/pull/237 Adding a note / warning about boot mode changes when migrating VMs between Xen / XCP-ng versions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [cloudstack-documentation] davidjumani commented on pull request #237: Adding a note on xen boot mode
davidjumani commented on pull request #237: URL: https://github.com/apache/cloudstack-documentation/pull/237#issuecomment-900035477 @blueorangutan docbuild -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [cloudstack-documentation] blueorangutan commented on pull request #237: Adding a note on xen boot mode
blueorangutan commented on pull request #237: URL: https://github.com/apache/cloudstack-documentation/pull/237#issuecomment-900035962 @davidjumani a Jenkins job has been kicked to build the document. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [cloudstack-documentation] blueorangutan commented on pull request #237: Adding a note on xen boot mode
blueorangutan commented on pull request #237: URL: https://github.com/apache/cloudstack-documentation/pull/237#issuecomment-900036399 Doc build preview: http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/237. (SL-JID 131) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
