Hi all, I've taken feedback from this thread and wrote this design doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too. Kindly suggest your views on the following (also in the design doc above): Outstanding Questions: * Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block? * Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin? * With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics? * Any thoughts on UI/UX for firewall/routing management? * Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc. * For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed? * For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the ‘local’ half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc? * Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6? * Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion) Regards. ________________________________ From: Wei ZHOU <ustcweiz...@gmail.com> Sent: Tuesday, August 17, 2021 21:16 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Thanks Kristaps, Wido, Rohit and Alex for your replies. I am fine with not having stateful dhcpv6 and privacy extension/temporary address in phase 1. If community decides not to do eventually , it is also ok to me. We could explore how to better use secondary ipv6 addresses as Wido advised. It would be great if anyone share some user experience. -Wei On Tuesday, 17 August 2021, Kristaps Cudars <kristaps.cud...@gmail.com> wrote: > Hi Wei, > > Published this month’s RFC 9099 and explains in different > words/perspective what has been written by Alex, Rohit and Wido. > https://www.rfc-editor.org/rfc/rfc9099.html > > > On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote: > > Hi Wido, > > > > (cc to Rohit and Alex) > > > > It is a good suggestion to use FRR for ipv6. The configuration is quite > > simple and the VMs can get SLAAC, routes, etc. > > > > Privacy extension looks not the same as what you mentioned. see > > https://datatracker.ietf.org/doc/html/rfc4941 > > > > You are right. To use static routing, the admins need to configure the > > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs > > and /64 for isolated networks) and their next-hop (which will be > > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range > and > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong. > > > > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, > > router/dns etc via RA messages) will be the only option for now (related > to > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be > good > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an > > option in cloudstack ? The advantages are > > (1) support other ipv6 cidr sizes than /64. > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can > > be changed > > (4) an Ipv6 addresses can be re-used by multiple vms. > > The problem is, stateful dhcpv6 does not support routers,nameservers, > etc. > > we need to figure it out (probably use radvd/frr and dnsmasq both). > > > > -Wei > > > > > > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote: > > > > > Hi, > > > > > > See my inline responses: > > > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav: > > > > Hi all, > > > > > > > > Thanks for your feedback and ideas, I've gone ahead with discussing > them > > > with Alex and came up with a PoC/design which can be implemented in the > > > following phases: > > > > > > > > * Phase1: implement ipv6 support in isolated networks and VPC > with > > > static routing > > > > * Phase2: discuss and implement support for dynamic routing > (TBD) > > > > > > > > For Phase1 here's the high-level proposal: > > > > > > > > * IPv6 address management: > > > > * At the zone level root-admin specifies a /64 public range > that > > > will be used for VRs, then they can add a /48, or /56 IPv6 range for > guest > > > networks (to be used by isolated networks and VPC tiers) > > > > * On creation of any IPv6 enabled isolated network or VPC > tier, > > > from the /48 or /56 block a /64 network is allocated/used > > > > * We assume SLAAC and autoconfiguration, no DHCPv6 in the > zone > > > (discuss: is privacy a concern, can privacy extensions rfc4941 of > slaac be > > > explored?) > > > > > > Privacy Extensions are only a concern for client devices which roam > > > between different IPv6 networks. > > > > > > If you IPv6 address of a client keeps the same suffix (MAC based) and > > > switches network then only the prefix (/64) will change. > > > > > > This way a network like Google, Facebook, etc could track your device > > > moving from network to network if they only look at the last 64-bits of > > > the IPv6 address. > > > > > > For servers this is not a problem as you already know in which network > > > they are. > > > > > > > * Network offerings: root-admin can create new network offerings > > > (with VPC too) that specifies a network stack option: > > > > * ipv4 only (default, for backward compatibility all > > > networks/offerings post-upgrade migrate to this option) > > > > * ipv4-and-ipv6 > > > > * ipv6-only (this can be phase 1.b) > > > > * A new routing option: static (phase1), dynamic (phase2, > with > > > multiple sub-options such as ospf/bgp etc...) > > > > > > This means that the network admin will need to statically route the > IPv6 > > > subnet to the VR's outside IPv6 address, for example, on a JunOS > router: > > > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop > > > 2001:db8:100::50 > > > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on the > > > outside (/64) network. In reality this will probably be a longer > > > address, but this is for just the example. > > > > > > > * VR changes: > > > > * VR gets its guest and public nics set to inet6 auto > > > > * For each /64 allocated to guest network and VPC tiers, > radvd > > > is configured to do RA > > > > > > radvd is fine, but looking at phase 2 with dynamic routing you might > > > already want to look into FRRouting. FRR can also advertise RAs while > > > not doing any routing. > > > > > > interface ens4 > > > no ipv6 nd suppress-ra > > > ipv6 nd prefix 2001:db8:500::/64 > > > ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 > > > > > > See: http://docs.frrouting.org/en/latest/ipv6.html > > > > > > > * Firewall: a new ipv6 zone/chain is created for ipv6 where > ipv6 > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between > VPC > > > tiers are managed/implemented by ipv6 firewall on VR > > > > > > Please take a look at the existing security_group.py script which > > > implements RFC4890 > > > > > > https://datatracker.ietf.org/doc/html/rfc4890 > > > > > > ICMPv6 is a vital part of IPv6 and certain packets should always be > > > allowed. > > > > > > > * It is assumed that static routes are created on the > core/main > > > router by the admin or automated using some scripts/tools; for this > > > CloudStack will announce events with details of /64 networks and VR's > > > public IPv6 address that can be consumed by a rabbitmq/message bus > client > > > (for example), or a custom cron job or script as part of orchestration. > > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\ > > > > > > You would only need to announce the /48 or /56 allocated to the VR, > > > that's all. You don't need to inform the upstream router about the /64 > > > subnets created within that larger subnet. > > > > > > > * Guest Networking: With SLAAC, it's easy for CloudStack to > > > calculate allocate and use a /64 and determine the IPv6 address of VR > nics > > > and guest VM nics > > > > * A user create an isolated network/VPC with an offering > that is > > > ipv6 enabled > > > > * A user can manage firewall for the IPv6 address/guest nics; > > > there'll be no port forward and LB feature though for IPv6 > > > > * A users can run workloads in the guest VMs that listen on > > > publically routable ipv6 addresses > > > > * Usage/billing etc continue to work, no change needed > > > > > > > > Network layout: > > > > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> > > > [guest VMs/nics] > > > > *core/ISP router needs static routes to be added (manually or > > > automated), assumes a /48 or /56 configured for the zone > > > > > > > > Thoughts, feedback? > > > > > > Looks doable! > > > > > > Side-note: It would be very cool if you could use parts of this > > > implementation to also route /48, /56, or /60 subnets to individual VMs > > > in Shared networks. > > > > > > Why? This allows for running Docker containers with native IPv6 inside > > > the VM or running a (Open)VPN server inside a VM which then assigns > > > public IPv6 addresses to clients connected. > > > > > > Instead of routing the subnet to a VR we route the subnet to a single > > > instance in a shared network. > > > > > > If we could then also move these subnets between Instances easily one > > > can quickly migrate to a different instance while keeping the same IPv6 > > > subnet. > > > > > > Wido > > > > > > > > > > > Proof-of-concept commentary: here's what I did to test the idea: > > > > > > > > * Created an isolated network and deployed a VM in my home lab > > > > The VR running on KVM has following nics > > > > eth0 - guest network > > > > eth1 - link local > > > > eth2 - public network > > > > > > > > * I setup a custom openwrt router on a RPi4 to serve as a > toy-core > > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got > a > > > /48 allocated. My configuration looks like: > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker) > > > > /64 - 2001:470:36:3e2::/64 (default allocated by) > > > > > > > > I create a LAN ipv6 (public network for CloudStack VR): at > subnet/prefix > > > 0: > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 > > > > Address mode: SLAAC+stateless DHCP (no dhcpv6) > > > > * > > > > * > > > > In the isolated VR, I enabled ipv6 as: > > > > net.ipv6.conf.all.disable_ipv6 = 0 > > > > net.ipv6.conf.all.forwarding = 1 > > > > net.ipv6.conf.all.accept_ra = 1 > > > > net.ipv6.conf.all.accept_redirects = 1 > > > > net.ipv6.conf.all.autoconf = 1 > > > > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf > > > > And configured the nics: > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces > > > > echo iface eth2 inet6 auto >> /etc/network/interfaces > > > > /etc/init.d/networking restart > > > > Next, restart ACS isolated network without cleanup to have it > > > reconfigure IPv4 nics, firewall, NAT etc > > > > > > > > * > > > > Next, I created a /64 network for the isolated guest network on eth0 > of > > > VR using radvd: > > > > > > > > # cat /etc/radvd.conf > > > > interface eth0 > > > > { > > > > AdvSendAdvert on; > > > > MinRtrAdvInterval 5; > > > > MaxRtrAdvInterval 15; > > > > prefix 2001:470:ed36:1::/64 > > > > { > > > > AdvOnLink on; > > > > AdvAutonomous on; > > > > }; > > > > }; > > > > systemctl restart radvd > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this > > > ...:1::/64 network > > > > * Finally I added a static route in toy core-router for the new > /64 > > > IPv6 range in the isolated network > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev > > > <local lan nic> > > > > * > > > > ... and I enabled firewall rules to allow any traffic to pass for the > > > new /64 network > > > > > > > > And voila all done! I create a domain AAAA record that points to my > > > guest VM IPv6 address a test webserver on > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/ > > > > > > > > (Note: I'll get rid of the tunnel and request a new /48 block after a > > > few days, sharing this solely for testing purposes) > > > > > > > > > > > > Regards. > > > > > > > > > > > > > >
