Hi all, IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ? Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing Regards, Abhishek ________________________________ From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 13 September 2021 14:30 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17). Regards. ________________________________ From: Wei ZHOU <ustcweiz...@gmail.com> Sent: Friday, September 10, 2021 20:22 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Agree with Alex. We only need to know how many /64 are allocated. We do not care how many ipv6 addresses are used by VMs. -Wei On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <alex.matti...@shapeblue.com> wrote: > Hi Rohit, > > I'd go for option 2, don't see a point tracking anything smaller than a > /64 tbh. > > Cheers > Alex > > > > > -----Original Message----- > From: Rohit Yadav <rohit.ya...@shapeblue.com> > Sent: 09 September 2021 12:44 > To: dev@cloudstack.apache.org > Subject: Re: IPV6 in Isolated/VPC networks > > Thanks Alex, Kristaps. I've updated the design doc to reflect two > agreements: > > * Allocate /64 for both isolated network and VPC tiers, no large > allocation of prefixes to VPC (cons: more static routing rules for upstream > router/admins) > * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a > dedicated /64 prefix/block for systemvms > > The only outstanding question now is: > > * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage > for shared network (design, implementation and use-cases?) > Option1: We don't do it, all user VMs nics have ipv4 address which whose > usage we don't track. For public VR/nics/networks, we can simply add the > IPv6 details for a related IPv4 address. > Option2: Implement a separate, first-class IPv6 address or /64 prefix > tracking/management and usage for all VMs and systemvms nic (this means > account/domain level limits and new billing/records) > Option3: other thoughts? > > > Regards. > > ________________________________ > From: Alex Mattioli <alex.matti...@shapeblue.com> > Sent: Wednesday, September 8, 2021 23:24 > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: RE: IPV6 in Isolated/VPC networks > > Hi Rohit, Kristaps, > > I'd say option 1 as well, it does create a bit more overhead with static > routes but if that's automated for a VPC it can also be easily automated > for several tiers of a VPC. We also don't constrain the amount of tiers in > a VPC. > It has the added advantage to be closer to the desired behaviour with > dynamic routing in the future, where a VPC VR can announce several subnets > upstream. > > Cheers > Alex > > > > > > > > > > > -----Original Message----- > From: Rohit Yadav <rohit.ya...@shapeblue.com> > Sent: 08 September 2021 19:04 > To: dev@cloudstack.apache.org > Subject: Re: IPV6 in Isolated/VPC networks > > Hi Kristaps, > > Thanks for sharing, I suppose that means individual tiers should be > allocated /64 instead of larger ipv6 blocks to the whole VPC which could > cause wastage. > > Any objection from anybody? > > Regards. > ________________________________ > From: Kristaps Cudars <kristaps.cud...@gmail.com> > Sent: Wednesday, September 8, 2021 9:24:01 PM > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: Re: IPV6 in Isolated/VPC networks > > Hello, > > I asked networking team to comment on "How should the IPv6 > block/allocation work in VPCs?" > Option1: They haven't seen lately devices with limits on how many static > routes can be created. > Option2: With /60 and /62 assignments and big quantity of routers IPv6 > assignment from RIPE NNC can be drained fast. > > /48 contains 64000 /64 > /60 contains 16 /64 > 64000 / 16 = 4000 routers > > > On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > > All, > > > > After another iteration with Alex, I've updated the design doc. Kindly > review: > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in > > +Isolated+Network+and+VPC+with+Static+Routing > > > > > > ... and advise on some outstanding questions: > > > > * How should the IPv6 block/allocation work in VPCs? > > Option1: Should this be simply /64 allocation on any new tier, the > > cons of this option is one static route/rule per VPC tier. (many > > upstream routers may have limit on no. of static routes?) > > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 > (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks > global setting (3 is default). The pros of this option are less no. of > static route/rule and easy programming, but potential wastage of multiple > /64 prefix blocks for unused/uncreated tiers. > > * How do we manage IPv6 usage? Can anyone advise how we do IPv6 > usage for shared network (design, implementation and use-cases?) > > Option1: We don't do it, all user VMs nics have ipv4 address which whose > usage we don't track. For public VR/nics/networks, we can simply add the > IPv6 details for a related IPv4 address. > > Option2: Implement a separate, first-class IPv6 address or /64 prefix > tracking/management and usage for all VMs and systemvms nic (this means > account/domain level limits and new billing/records) > > * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default > if zone has a IPv6 address block allocated/assigned for use for systemvms > (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons > of this?) > > * > > > > Regards. > > > > ________________________________ > > From: Rohit Yadav <rohit.ya...@shapeblue.com> > > Sent: Thursday, August 19, 2021 15:45 > > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > > Subject: Re: IPV6 in Isolated/VPC networks > > > > Hi all, > > > > I've taken feedback from this thread and wrote this design doc: > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in > > +Isolated+Network+and+VPC+with+Static+Routing > > > > Kindly review and advise if I missed anything or anything that needs to > be changed/updated. You may comment on the wiki directly too. > > > > Kindly suggest your views on the following (also in the design doc > above): > > > > Outstanding Questions: > > > > * Should admin or user be able to specify how VPC super CIDRs are > created/needed; for example a user can ask for VPC with /60 super CIDR? Or > should CloudStack automatically find/allocate a /64 for a new VPC tier from > the root-admin configured /64-/48 block? > > * Should we explore FRR and iBGP or other strategies to do dynamic > routing which may not require advance/complex configuration in the VR or > for the users/admin? > > * With SLAAC and no dhcpv6, is there a way to support secondary IPv6 > addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's > nics? > > * Any thoughts on UI/UX for firewall/routing management? > > * Any other feature/support for isolated network or VPC feature that > must be explored or supported such as PF, VPN, LB, vpc static routes, vpc > gateway etc. > > * For usage - should we have any consideration, or should we assume > that IPv4 and IPv6 address will go together for every nic; so IPv6 usage > for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new > biling/usage needed? > > * For smoketests, local dev-test should we explore ULA? Unique Local > Address - in the range fc00::/7. Typically only within the 'local' half > fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. > This prefix can be randomly generated at first install by CloudStack in a > zone using zoneid etc? > > * Should we expand support for VR diagnostic feature to include > support for ipv6, traceroute6? > > * Discuss - expand support/ability to allocate a /60, or /56 etc > prefix to an individual VM in shared network (Wido's suggestion) > > > > > > Regards. > > > > ________________________________ > > From: Wei ZHOU <ustcweiz...@gmail.com> > > Sent: Tuesday, August 17, 2021 21:16 > > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > > Subject: Re: IPV6 in Isolated/VPC networks > > > > Thanks Kristaps, Wido, Rohit and Alex for your replies. > > > > I am fine with not having stateful dhcpv6 and privacy > > extension/temporary address in phase 1. If community decides not to do > > eventually , it is also ok to me. > > > > We could explore how to better use secondary ipv6 addresses as Wido > > advised. It would be great if anyone share some user experience. > > > > -Wei > > > > > > On Tuesday, 17 August 2021, Kristaps Cudars > > <kristaps.cud...@gmail.com> > > wrote: > > > > > Hi Wei, > > > > > > Published this month's RFC 9099 and explains in different > > > words/perspective what has been written by Alex, Rohit and Wido. > > > https://www.rfc-editor.org/rfc/rfc9099.html > > > > > > > > > On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote: > > > > Hi Wido, > > > > > > > > (cc to Rohit and Alex) > > > > > > > > It is a good suggestion to use FRR for ipv6. The configuration is > > > > quite simple and the VMs can get SLAAC, routes, etc. > > > > > > > > Privacy extension looks not the same as what you mentioned. see > > > > https://datatracker.ietf.org/doc/html/rfc4941 > > > > > > > > You are right. To use static routing, the admins need to configure > > > > the routes in the upstream router, and add some ipv6 ranges (eg > > > > /56 for VPCs and /64 for isolated networks) and their next-hop > > > > (which will be configured in VRs) in CloudStack. CloudStack will > > > > pick up an IPv6 range > > > and > > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm > wrong. > > > > > > > > I have a question, it looks stateless dhcpv6 (SLAAC from > > > > router/VR, router/dns etc via RA messages) will be the only option > > > > for now (related > > > to > > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it > > > > be > > > good > > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) > > > > as an option in cloudstack ? The advantages are > > > > (1) support other ipv6 cidr sizes than /64. > > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 > > > > addresses can be changed > > > > (4) an Ipv6 addresses can be re-used by multiple vms. > > > > The problem is, stateful dhcpv6 does not support > > > > routers,nameservers, > > > etc. > > > > we need to figure it out (probably use radvd/frr and dnsmasq both). > > > > > > > > -Wei > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote: > > > > > > > > > Hi, > > > > > > > > > > See my inline responses: > > > > > > > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav: > > > > > > Hi all, > > > > > > > > > > > > Thanks for your feedback and ideas, I've gone ahead with > > > > > > discussing > > > them > > > > > with Alex and came up with a PoC/design which can be implemented > > > > > in the following phases: > > > > > > > > > > > > * Phase1: implement ipv6 support in isolated networks and > VPC > > > with > > > > > static routing > > > > > > * Phase2: discuss and implement support for dynamic routing > > > (TBD) > > > > > > > > > > > > For Phase1 here's the high-level proposal: > > > > > > > > > > > > * IPv6 address management: > > > > > > * At the zone level root-admin specifies a /64 public > range > > > that > > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range > > > > > for > > > guest > > > > > networks (to be used by isolated networks and VPC tiers) > > > > > > * On creation of any IPv6 enabled isolated network or VPC > > > tier, > > > > > from the /48 or /56 block a /64 network is allocated/used > > > > > > * We assume SLAAC and autoconfiguration, no DHCPv6 in the > > > zone > > > > > (discuss: is privacy a concern, can privacy extensions rfc4941 > > > > > of > > > slaac be > > > > > explored?) > > > > > > > > > > Privacy Extensions are only a concern for client devices which > > > > > roam between different IPv6 networks. > > > > > > > > > > If you IPv6 address of a client keeps the same suffix (MAC > > > > > based) and switches network then only the prefix (/64) will change. > > > > > > > > > > This way a network like Google, Facebook, etc could track your > > > > > device moving from network to network if they only look at the > > > > > last 64-bits of the IPv6 address. > > > > > > > > > > For servers this is not a problem as you already know in which > > > > > network they are. > > > > > > > > > > > * Network offerings: root-admin can create new network > offerings > > > > > (with VPC too) that specifies a network stack option: > > > > > > * ipv4 only (default, for backward compatibility all > > > > > networks/offerings post-upgrade migrate to this option) > > > > > > * ipv4-and-ipv6 > > > > > > * ipv6-only (this can be phase 1.b) > > > > > > * A new routing option: static (phase1), dynamic (phase2, > > > with > > > > > multiple sub-options such as ospf/bgp etc...) > > > > > > > > > > This means that the network admin will need to statically route > > > > > the > > > IPv6 > > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS > > > router: > > > > > > > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 > > > > > next-hop > > > > > 2001:db8:100::50 > > > > > > > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on > > > > > the outside (/64) network. In reality this will probably be a > > > > > longer address, but this is for just the example. > > > > > > > > > > > * VR changes: > > > > > > * VR gets its guest and public nics set to inet6 auto > > > > > > * For each /64 allocated to guest network and VPC tiers, > > > radvd > > > > > is configured to do RA > > > > > > > > > > radvd is fine, but looking at phase 2 with dynamic routing you > > > > > might already want to look into FRRouting. FRR can also > > > > > advertise RAs while not doing any routing. > > > > > > > > > > interface ens4 > > > > > no ipv6 nd suppress-ra > > > > > ipv6 nd prefix 2001:db8:500::/64 > > > > > ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 > > > > > > > > > > See: http://docs.frrouting.org/en/latest/ipv6.html > > > > > > > > > > > * Firewall: a new ipv6 zone/chain is created for ipv6 > where > > > ipv6 > > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs > > > > > between > > > VPC > > > > > tiers are managed/implemented by ipv6 firewall on VR > > > > > > > > > > Please take a look at the existing security_group.py script > > > > > which implements RFC4890 > > > > > > > > > > https://datatracker.ietf.org/doc/html/rfc4890 > > > > > > > > > > ICMPv6 is a vital part of IPv6 and certain packets should always > > > > > be allowed. > > > > > > > > > > > * It is assumed that static routes are created on the > > > core/main > > > > > router by the admin or automated using some scripts/tools; for > > > > > this CloudStack will announce events with details of /64 > > > > > networks and VR's public IPv6 address that can be consumed by a > > > > > rabbitmq/message bus > > > client > > > > > (for example), or a custom cron job or script as part of > orchestration. > > > > > (this wouldn't be necessary for dynamic routing bgp with > > > > > phase2)\\ > > > > > > > > > > You would only need to announce the /48 or /56 allocated to the > > > > > VR, that's all. You don't need to inform the upstream router > > > > > about the /64 subnets created within that larger subnet. > > > > > > > > > > > * Guest Networking: With SLAAC, it's easy for CloudStack to > > > > > calculate allocate and use a /64 and determine the IPv6 address > > > > > of VR > > > nics > > > > > and guest VM nics > > > > > > * A user create an isolated network/VPC with an offering > > > that is > > > > > ipv6 enabled > > > > > > * A user can manage firewall for the IPv6 address/guest > nics; > > > > > there'll be no port forward and LB feature though for IPv6 > > > > > > * A users can run workloads in the guest VMs that listen > on > > > > > publically routable ipv6 addresses > > > > > > * Usage/billing etc continue to work, no change needed > > > > > > > > > > > > Network layout: > > > > > > > > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a > > > > > > VLAN] -> > > > > > [guest VMs/nics] > > > > > > *core/ISP router needs static routes to be added (manually or > > > > > automated), assumes a /48 or /56 configured for the zone > > > > > > > > > > > > Thoughts, feedback? > > > > > > > > > > Looks doable! > > > > > > > > > > Side-note: It would be very cool if you could use parts of this > > > > > implementation to also route /48, /56, or /60 subnets to > > > > > individual VMs in Shared networks. > > > > > > > > > > Why? This allows for running Docker containers with native IPv6 > > > > > inside the VM or running a (Open)VPN server inside a VM which > > > > > then assigns public IPv6 addresses to clients connected. > > > > > > > > > > Instead of routing the subnet to a VR we route the subnet to a > > > > > single instance in a shared network. > > > > > > > > > > If we could then also move these subnets between Instances > > > > > easily one can quickly migrate to a different instance while > > > > > keeping the same IPv6 subnet. > > > > > > > > > > Wido > > > > > > > > > > > > > > > > > Proof-of-concept commentary: here's what I did to test the idea: > > > > > > > > > > > > * Created an isolated network and deployed a VM in my home > lab > > > > > > The VR running on KVM has following nics > > > > > > eth0 - guest network > > > > > > eth1 - link local > > > > > > eth2 - public network > > > > > > > > > > > > * I setup a custom openwrt router on a RPi4 to serve as a > > > toy-core > > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and > > > > > I got > > > a > > > > > /48 allocated. My configuration looks like: > > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker) > > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by) > > > > > > > > > > > > I create a LAN ipv6 (public network for CloudStack VR): at > > > subnet/prefix > > > > > 0: > > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: > > > > > > SLAAC+stateless DHCP (no dhcpv6) > > > > > > * > > > > > > * > > > > > > In the isolated VR, I enabled ipv6 as: > > > > > > net.ipv6.conf.all.disable_ipv6 = 0 > > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra = > > > > > > 1 net.ipv6.conf.all.accept_redirects = 1 > > > > > > net.ipv6.conf.all.autoconf = 1 > > > > > > > > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And > > > > > > configured the nics: > > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo > > > > > > iface eth2 inet6 auto >> /etc/network/interfaces > > > > > > /etc/init.d/networking restart Next, restart ACS isolated > > > > > > network without cleanup to have it > > > > > reconfigure IPv4 nics, firewall, NAT etc > > > > > > > > > > > > * > > > > > > Next, I created a /64 network for the isolated guest network > > > > > > on eth0 > > > of > > > > > VR using radvd: > > > > > > > > > > > > # cat /etc/radvd.conf > > > > > > interface eth0 > > > > > > { > > > > > > AdvSendAdvert on; > > > > > > MinRtrAdvInterval 5; > > > > > > MaxRtrAdvInterval 15; > > > > > > prefix 2001:470:ed36:1::/64 > > > > > > { > > > > > > AdvOnLink on; > > > > > > AdvAutonomous on; > > > > > > }; > > > > > > }; > > > > > > systemctl restart radvd > > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in > > > > > > this > > > > > ...:1::/64 network > > > > > > * Finally I added a static route in toy core-router for the > new > > > /64 > > > > > IPv6 range in the isolated network > > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on > > > > > > eth2> dev > > > > > <local lan nic> > > > > > > * > > > > > > ... and I enabled firewall rules to allow any traffic to pass > > > > > > for the > > > > > new /64 network > > > > > > > > > > > > And voila all done! I create a domain AAAA record that points > > > > > > to my > > > > > guest VM IPv6 address a test webserver on > > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/ > > > > > > > > > > > > (Note: I'll get rid of the tunnel and request a new /48 block > > > > > > after a > > > > > few days, sharing this solely for testing purposes) > > > > > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
