Thanks Alex, Kristaps. I've updated the design doc to reflect two agreements:
* Allocate /64 for both isolated network and VPC tiers, no large allocation of prefixes to VPC (cons: more static routing rules for upstream router/admins) * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a dedicated /64 prefix/block for systemvms The only outstanding question now is: * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?) Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address. Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records) Option3: other thoughts? Regards. ________________________________ From: Alex Mattioli <alex.matti...@shapeblue.com> Sent: Wednesday, September 8, 2021 23:24 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: RE: IPV6 in Isolated/VPC networks Hi Rohit, Kristaps, I'd say option 1 as well, it does create a bit more overhead with static routes but if that's automated for a VPC it can also be easily automated for several tiers of a VPC. We also don't constrain the amount of tiers in a VPC. It has the added advantage to be closer to the desired behaviour with dynamic routing in the future, where a VPC VR can announce several subnets upstream. Cheers Alex -----Original Message----- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 08 September 2021 19:04 To: dev@cloudstack.apache.org Subject: Re: IPV6 in Isolated/VPC networks Hi Kristaps, Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage. Any objection from anybody? Regards. ________________________________ From: Kristaps Cudars <kristaps.cud...@gmail.com> Sent: Wednesday, September 8, 2021 9:24:01 PM To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Hello, I asked networking team to comment on "How should the IPv6 block/allocation work in VPCs?" Option1: They haven't seen lately devices with limits on how many static routes can be created. Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast. /48 contains 64000 /64 /60 contains 16 /64 64000 / 16 = 4000 routers On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > All, > > After another iteration with Alex, I've updated the design doc. Kindly review: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in > +Isolated+Network+and+VPC+with+Static+Routing > > > ... and advise on some outstanding questions: > > * How should the IPv6 block/allocation work in VPCs? > Option1: Should this be simply /64 allocation on any new tier, the > cons of this option is one static route/rule per VPC tier. (many > upstream routers may have limit on no. of static routes?) > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 > tiers) for the VPC, this can be filtered based on the vpc.max.networks global > setting (3 is default). The pros of this option are less no. of static > route/rule and easy programming, but potential wastage of multiple /64 prefix > blocks for unused/uncreated tiers. > * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for > shared network (design, implementation and use-cases?) > Option1: We don't do it, all user VMs nics have ipv4 address which whose > usage we don't track. For public VR/nics/networks, we can simply add the IPv6 > details for a related IPv4 address. > Option2: Implement a separate, first-class IPv6 address or /64 prefix > tracking/management and usage for all VMs and systemvms nic (this means > account/domain level limits and new billing/records) > * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if > zone has a IPv6 address block allocated/assigned for use for systemvms (this > was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?) > * > > Regards. > > ________________________________ > From: Rohit Yadav <rohit.ya...@shapeblue.com> > Sent: Thursday, August 19, 2021 15:45 > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: Re: IPV6 in Isolated/VPC networks > > Hi all, > > I've taken feedback from this thread and wrote this design doc: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in > +Isolated+Network+and+VPC+with+Static+Routing > > Kindly review and advise if I missed anything or anything that needs to be > changed/updated. You may comment on the wiki directly too. > > Kindly suggest your views on the following (also in the design doc above): > > Outstanding Questions: > > * Should admin or user be able to specify how VPC super CIDRs are > created/needed; for example a user can ask for VPC with /60 super CIDR? Or > should CloudStack automatically find/allocate a /64 for a new VPC tier from > the root-admin configured /64-/48 block? > * Should we explore FRR and iBGP or other strategies to do dynamic > routing which may not require advance/complex configuration in the VR or for > the users/admin? > * With SLAAC and no dhcpv6, is there a way to support secondary IPv6 > addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's > nics? > * Any thoughts on UI/UX for firewall/routing management? > * Any other feature/support for isolated network or VPC feature that must > be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway > etc. > * For usage - should we have any consideration, or should we assume that > IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic > is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage > needed? > * For smoketests, local dev-test should we explore ULA? Unique Local > Address - in the range fc00::/7. Typically only within the 'local' half > fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This > prefix can be randomly generated at first install by CloudStack in a zone > using zoneid etc? > * Should we expand support for VR diagnostic feature to include support > for ipv6, traceroute6? > * Discuss - expand support/ability to allocate a /60, or /56 etc prefix > to an individual VM in shared network (Wido's suggestion) > > > Regards. > > ________________________________ > From: Wei ZHOU <ustcweiz...@gmail.com> > Sent: Tuesday, August 17, 2021 21:16 > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: Re: IPV6 in Isolated/VPC networks > > Thanks Kristaps, Wido, Rohit and Alex for your replies. > > I am fine with not having stateful dhcpv6 and privacy > extension/temporary address in phase 1. If community decides not to do > eventually , it is also ok to me. > > We could explore how to better use secondary ipv6 addresses as Wido > advised. It would be great if anyone share some user experience. > > -Wei > > > On Tuesday, 17 August 2021, Kristaps Cudars > <kristaps.cud...@gmail.com> > wrote: > > > Hi Wei, > > > > Published this month's RFC 9099 and explains in different > > words/perspective what has been written by Alex, Rohit and Wido. > > https://www.rfc-editor.org/rfc/rfc9099.html > > > > > > On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote: > > > Hi Wido, > > > > > > (cc to Rohit and Alex) > > > > > > It is a good suggestion to use FRR for ipv6. The configuration is > > > quite simple and the VMs can get SLAAC, routes, etc. > > > > > > Privacy extension looks not the same as what you mentioned. see > > > https://datatracker.ietf.org/doc/html/rfc4941 > > > > > > You are right. To use static routing, the admins need to configure > > > the routes in the upstream router, and add some ipv6 ranges (eg > > > /56 for VPCs and /64 for isolated networks) and their next-hop > > > (which will be configured in VRs) in CloudStack. CloudStack will > > > pick up an IPv6 range > > and > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong. > > > > > > I have a question, it looks stateless dhcpv6 (SLAAC from > > > router/VR, router/dns etc via RA messages) will be the only option > > > for now (related > > to > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it > > > be > > good > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) > > > as an option in cloudstack ? The advantages are > > > (1) support other ipv6 cidr sizes than /64. > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 > > > addresses can be changed > > > (4) an Ipv6 addresses can be re-used by multiple vms. > > > The problem is, stateful dhcpv6 does not support > > > routers,nameservers, > > etc. > > > we need to figure it out (probably use radvd/frr and dnsmasq both). > > > > > > -Wei > > > > > > > > > > > > > > > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote: > > > > > > > Hi, > > > > > > > > See my inline responses: > > > > > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav: > > > > > Hi all, > > > > > > > > > > Thanks for your feedback and ideas, I've gone ahead with > > > > > discussing > > them > > > > with Alex and came up with a PoC/design which can be implemented > > > > in the following phases: > > > > > > > > > > * Phase1: implement ipv6 support in isolated networks and VPC > > with > > > > static routing > > > > > * Phase2: discuss and implement support for dynamic routing > > (TBD) > > > > > > > > > > For Phase1 here's the high-level proposal: > > > > > > > > > > * IPv6 address management: > > > > > * At the zone level root-admin specifies a /64 public range > > that > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range > > > > for > > guest > > > > networks (to be used by isolated networks and VPC tiers) > > > > > * On creation of any IPv6 enabled isolated network or VPC > > tier, > > > > from the /48 or /56 block a /64 network is allocated/used > > > > > * We assume SLAAC and autoconfiguration, no DHCPv6 in the > > zone > > > > (discuss: is privacy a concern, can privacy extensions rfc4941 > > > > of > > slaac be > > > > explored?) > > > > > > > > Privacy Extensions are only a concern for client devices which > > > > roam between different IPv6 networks. > > > > > > > > If you IPv6 address of a client keeps the same suffix (MAC > > > > based) and switches network then only the prefix (/64) will change. > > > > > > > > This way a network like Google, Facebook, etc could track your > > > > device moving from network to network if they only look at the > > > > last 64-bits of the IPv6 address. > > > > > > > > For servers this is not a problem as you already know in which > > > > network they are. > > > > > > > > > * Network offerings: root-admin can create new network offerings > > > > (with VPC too) that specifies a network stack option: > > > > > * ipv4 only (default, for backward compatibility all > > > > networks/offerings post-upgrade migrate to this option) > > > > > * ipv4-and-ipv6 > > > > > * ipv6-only (this can be phase 1.b) > > > > > * A new routing option: static (phase1), dynamic (phase2, > > with > > > > multiple sub-options such as ospf/bgp etc...) > > > > > > > > This means that the network admin will need to statically route > > > > the > > IPv6 > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS > > router: > > > > > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 > > > > next-hop > > > > 2001:db8:100::50 > > > > > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on > > > > the outside (/64) network. In reality this will probably be a > > > > longer address, but this is for just the example. > > > > > > > > > * VR changes: > > > > > * VR gets its guest and public nics set to inet6 auto > > > > > * For each /64 allocated to guest network and VPC tiers, > > radvd > > > > is configured to do RA > > > > > > > > radvd is fine, but looking at phase 2 with dynamic routing you > > > > might already want to look into FRRouting. FRR can also > > > > advertise RAs while not doing any routing. > > > > > > > > interface ens4 > > > > no ipv6 nd suppress-ra > > > > ipv6 nd prefix 2001:db8:500::/64 > > > > ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 > > > > > > > > See: http://docs.frrouting.org/en/latest/ipv6.html > > > > > > > > > * Firewall: a new ipv6 zone/chain is created for ipv6 where > > ipv6 > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs > > > > between > > VPC > > > > tiers are managed/implemented by ipv6 firewall on VR > > > > > > > > Please take a look at the existing security_group.py script > > > > which implements RFC4890 > > > > > > > > https://datatracker.ietf.org/doc/html/rfc4890 > > > > > > > > ICMPv6 is a vital part of IPv6 and certain packets should always > > > > be allowed. > > > > > > > > > * It is assumed that static routes are created on the > > core/main > > > > router by the admin or automated using some scripts/tools; for > > > > this CloudStack will announce events with details of /64 > > > > networks and VR's public IPv6 address that can be consumed by a > > > > rabbitmq/message bus > > client > > > > (for example), or a custom cron job or script as part of orchestration. > > > > (this wouldn't be necessary for dynamic routing bgp with > > > > phase2)\\ > > > > > > > > You would only need to announce the /48 or /56 allocated to the > > > > VR, that's all. You don't need to inform the upstream router > > > > about the /64 subnets created within that larger subnet. > > > > > > > > > * Guest Networking: With SLAAC, it's easy for CloudStack to > > > > calculate allocate and use a /64 and determine the IPv6 address > > > > of VR > > nics > > > > and guest VM nics > > > > > * A user create an isolated network/VPC with an offering > > that is > > > > ipv6 enabled > > > > > * A user can manage firewall for the IPv6 address/guest nics; > > > > there'll be no port forward and LB feature though for IPv6 > > > > > * A users can run workloads in the guest VMs that listen on > > > > publically routable ipv6 addresses > > > > > * Usage/billing etc continue to work, no change needed > > > > > > > > > > Network layout: > > > > > > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a > > > > > VLAN] -> > > > > [guest VMs/nics] > > > > > *core/ISP router needs static routes to be added (manually or > > > > automated), assumes a /48 or /56 configured for the zone > > > > > > > > > > Thoughts, feedback? > > > > > > > > Looks doable! > > > > > > > > Side-note: It would be very cool if you could use parts of this > > > > implementation to also route /48, /56, or /60 subnets to > > > > individual VMs in Shared networks. > > > > > > > > Why? This allows for running Docker containers with native IPv6 > > > > inside the VM or running a (Open)VPN server inside a VM which > > > > then assigns public IPv6 addresses to clients connected. > > > > > > > > Instead of routing the subnet to a VR we route the subnet to a > > > > single instance in a shared network. > > > > > > > > If we could then also move these subnets between Instances > > > > easily one can quickly migrate to a different instance while > > > > keeping the same IPv6 subnet. > > > > > > > > Wido > > > > > > > > > > > > > > Proof-of-concept commentary: here's what I did to test the idea: > > > > > > > > > > * Created an isolated network and deployed a VM in my home lab > > > > > The VR running on KVM has following nics > > > > > eth0 - guest network > > > > > eth1 - link local > > > > > eth2 - public network > > > > > > > > > > * I setup a custom openwrt router on a RPi4 to serve as a > > toy-core > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and > > > > I got > > a > > > > /48 allocated. My configuration looks like: > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker) > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by) > > > > > > > > > > I create a LAN ipv6 (public network for CloudStack VR): at > > subnet/prefix > > > > 0: > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: > > > > > SLAAC+stateless DHCP (no dhcpv6) > > > > > * > > > > > * > > > > > In the isolated VR, I enabled ipv6 as: > > > > > net.ipv6.conf.all.disable_ipv6 = 0 > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra = > > > > > 1 net.ipv6.conf.all.accept_redirects = 1 > > > > > net.ipv6.conf.all.autoconf = 1 > > > > > > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And > > > > > configured the nics: > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo > > > > > iface eth2 inet6 auto >> /etc/network/interfaces > > > > > /etc/init.d/networking restart Next, restart ACS isolated > > > > > network without cleanup to have it > > > > reconfigure IPv4 nics, firewall, NAT etc > > > > > > > > > > * > > > > > Next, I created a /64 network for the isolated guest network > > > > > on eth0 > > of > > > > VR using radvd: > > > > > > > > > > # cat /etc/radvd.conf > > > > > interface eth0 > > > > > { > > > > > AdvSendAdvert on; > > > > > MinRtrAdvInterval 5; > > > > > MaxRtrAdvInterval 15; > > > > > prefix 2001:470:ed36:1::/64 > > > > > { > > > > > AdvOnLink on; > > > > > AdvAutonomous on; > > > > > }; > > > > > }; > > > > > systemctl restart radvd > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in > > > > > this > > > > ...:1::/64 network > > > > > * Finally I added a static route in toy core-router for the new > > /64 > > > > IPv6 range in the isolated network > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on > > > > > eth2> dev > > > > <local lan nic> > > > > > * > > > > > ... and I enabled firewall rules to allow any traffic to pass > > > > > for the > > > > new /64 network > > > > > > > > > > And voila all done! I create a domain AAAA record that points > > > > > to my > > > > guest VM IPv6 address a test webserver on > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/ > > > > > > > > > > (Note: I'll get rid of the tunnel and request a new /48 block > > > > > after a > > > > few days, sharing this solely for testing purposes) > > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > >
