Hi Ahbishek,
Yes, the message should include the subnet.
The message should be sent when the IPv6 subnets are allocated or released,
right ? If so, there will not be multiple subnets/routes in the message.
-Wei
On Thu, 12 May 2022 at 09:56, Abhishek Kumar <abhishek.ku...@shapeblue.com>
wrote:
> Hi Wido,
>
> I do not understand what you mean by WAB address but
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network (IPv6
> of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
> Currently, the message on event bus does not include subnet. Should that
> be included?
> In case of VPCs, there could be multiple tiers which will need multiple
> routes to be added. Will that be an issue if we include current
> network/tier subnet in the event message?
>
> Regards,
> Abhishek
> ________________________________
> From: Wido den Hollander <w...@widodh.nl>
> Sent: 10 May 2022 19:01
> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek Kumar
> <abhishek.ku...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> > Yes. When a public IPv6 is assigned or released, CloudStack will publish
> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> > These event notifications can be tracked. And with improvements in
> events framework, these event messages will have network uuid as entityuuid
> and Network as entitytype. Using this network can be queried using to list
> IPv6 routes that need to be added.
> >
> > Sample event message,
> >
> > {"eventDateTime":"2022-05-10 09:32:12
> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
> public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> >
> >
> > ?Sample API call,
> >> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
> filter=id,name,ip6routes
> > {
> > "count": 1,
> > "network": [
> > {
> > "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> > "ip6routes": [
> > {
> > "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> > "subnet": "fd23:313a:2f53:3cbf::/64"
> > }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
> > ],
> > "name": "routing_test"
> > }
> > ]
> > }
> >
> >
> >
> >
> > ________________________________
> > From: Wido den Hollander <w...@widodh.nl>
> > Sent: 10 May 2022 13:59
> > To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
> Kumar <abhishek.ku...@shapeblue.com>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> >> Hi all,
> >>
> >> IPv6 Support in Isolated Network and VPC with Static Routing based on
> the design doc [1] has been implemented and is available in 4.17.0 RC2. I
> hope while testing 4.17.0 RC2 you will also try to test it ?
> >> Documentation for it is available at
> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> (will be available in the official docs once 4.17.0 version of docs is
> built).
> >>
> >
> > Great work!
> >
> > I see only static routing is supported. But do we publish something on
> > the message bus once a new VR/VPC is created?
> >
> > This way you could pick up these messages and have the network create a
> > (static) route based on those.
> >
> > ExaBGP for example could be used to inject such routes.
> >
> > Wido
> >
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> Regards,
> >> Abhishek
> >>
> >> ________________________________
> >> From: Rohit Yadav <rohit.ya...@shapeblue.com>
> >> Sent: 13 September 2021 14:30
> >> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Thanks Alex, Wei. I've updated the docs here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> I'll leave the thread open for futher discussion/ideas/feedback. I
> think we've completed the phase1 design doc including all feedback comments
> for adding IPv6 support in CloudStack and some initial poc/work can be
> started. My colleagues and I will keep everyone posted on this thread
> and/or on a Github PR as and when we're able to start our work on the same
> (after 4.16, potentially towards 4.17).
> >>
> >>
> >> Regards.
> >>
> >> ________________________________
> >> From: Wei ZHOU <ustcweiz...@gmail.com>
> >> Sent: Friday, September 10, 2021 20:22
> >> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Agree with Alex.
> >> We only need to know how many /64 are allocated. We do not care how many
> >> ipv6 addresses are used by VMs.
> >>
> >> -Wei
> >>
> >> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <
> alex.matti...@shapeblue.com>
> >> wrote:
> >>
> >>> Hi Rohit,
> >>>
> >>> I'd go for option 2, don't see a point tracking anything smaller than a
> >>> /64 tbh.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
> >>> Sent: 09 September 2021 12:44
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> >>> agreements:
> >>>
> >>> * Allocate /64 for both isolated network and VPC tiers, no large
> >>> allocation of prefixes to VPC (cons: more static routing rules for
> upstream
> >>> router/admins)
> >>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone
> has a
> >>> dedicated /64 prefix/block for systemvms
> >>>
> >>> The only outstanding question now is:
> >>>
> >>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage
> >>> for shared network (design, implementation and use-cases?)
> >>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>> Option3: other thoughts?
> >>>
> >>>
> >>> Regards.
> >>>
> >>> ________________________________
> >>> From: Alex Mattioli <alex.matti...@shapeblue.com>
> >>> Sent: Wednesday, September 8, 2021 23:24
> >>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >>> Subject: RE: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Rohit, Kristaps,
> >>>
> >>> I'd say option 1 as well, it does create a bit more overhead with
> static
> >>> routes but if that's automated for a VPC it can also be easily
> automated
> >>> for several tiers of a VPC. We also don't constrain the amount of
> tiers in
> >>> a VPC.
> >>> It has the added advantage to be closer to the desired behaviour with
> >>> dynamic routing in the future, where a VPC VR can announce several
> subnets
> >>> upstream.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
> >>> Sent: 08 September 2021 19:04
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Kristaps,
> >>>
> >>> Thanks for sharing, I suppose that means individual tiers should be
> >>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
> could
> >>> cause wastage.
> >>>
> >>> Any objection from anybody?
> >>>
> >>> Regards.
> >>> ________________________________
> >>> From: Kristaps Cudars <kristaps.cud...@gmail.com>
> >>> Sent: Wednesday, September 8, 2021 9:24:01 PM
> >>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hello,
> >>>
> >>> I asked networking team to comment on "How should the IPv6
> >>> block/allocation work in VPCs?"
> >>> Option1: They haven't seen lately devices with limits on how many
> static
> >>> routes can be created.
> >>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> >>> assignment from RIPE NNC can be drained fast.
> >>>
> >>> /48 contains 64000 /64
> >>> /60 contains 16 /64
> >>> 64000 / 16 = 4000 routers
> >>>
> >>>
> >>> On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
> >>>> All,
> >>>>
> >>>> After another iteration with Alex, I've updated the design doc. Kindly
> >>> review:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>>
> >>>> ... and advise on some outstanding questions:
> >>>>
> >>>> * How should the IPv6 block/allocation work in VPCs?
> >>>> Option1: Should this be simply /64 allocation on any new tier, the
> >>>> cons of this option is one static route/rule per VPC tier. (many
> >>>> upstream routers may have limit on no. of static routes?)
> >>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> >>> (4 tiers) for the VPC, this can be filtered based on the
> vpc.max.networks
> >>> global setting (3 is default). The pros of this option are less no. of
> >>> static route/rule and easy programming, but potential wastage of
> multiple
> >>> /64 prefix blocks for unused/uncreated tiers.
> >>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> >>> usage for shared network (design, implementation and use-cases?)
> >>>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
> default
> >>> if zone has a IPv6 address block allocated/assigned for use for
> systemvms
> >>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any
> cons
> >>> of this?)
> >>>> *
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
> >>>> Sent: Thursday, August 19, 2021 15:45
> >>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've taken feedback from this thread and wrote this design doc:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>> Kindly review and advise if I missed anything or anything that needs
> to
> >>> be changed/updated. You may comment on the wiki directly too.
> >>>>
> >>>> Kindly suggest your views on the following (also in the design doc
> >>> above):
> >>>>
> >>>> Outstanding Questions:
> >>>>
> >>>> * Should admin or user be able to specify how VPC super CIDRs
> are
> >>> created/needed; for example a user can ask for VPC with /60 super
> CIDR? Or
> >>> should CloudStack automatically find/allocate a /64 for a new VPC tier
> from
> >>> the root-admin configured /64-/48 block?
> >>>> * Should we explore FRR and iBGP or other strategies to do
> dynamic
> >>> routing which may not require advance/complex configuration in the VR
> or
> >>> for the users/admin?
> >>>> * With SLAAC and no dhcpv6, is there a way to support secondary
> IPv6
> >>> addresses (or floating IPv6 addresses for VR/public traffic) for guest
> VM's
> >>> nics?
> >>>> * Any thoughts on UI/UX for firewall/routing management?
> >>>> * Any other feature/support for isolated network or VPC feature
> that
> >>> must be explored or supported such as PF, VPN, LB, vpc static routes,
> vpc
> >>> gateway etc.
> >>>> * For usage - should we have any consideration, or should we
> assume
> >>> that IPv4 and IPv6 address will go together for every nic; so IPv6
> usage
> >>> for a nic is in tandem with Ipv4 address for a nic, i.e. no
> explicit/new
> >>> biling/usage needed?
> >>>> * For smoketests, local dev-test should we explore ULA? Unique
> Local
> >>> Address - in the range fc00::/7. Typically only within the 'local' half
> >>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> >>> This prefix can be randomly generated at first install by CloudStack
> in a
> >>> zone using zoneid etc?
> >>>> * Should we expand support for VR diagnostic feature to include
> >>> support for ipv6, traceroute6?
> >>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
> >>> prefix to an individual VM in shared network (Wido's suggestion)
> >>>>
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Wei ZHOU <ustcweiz...@gmail.com>
> >>>> Sent: Tuesday, August 17, 2021 21:16
> >>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >>>>
> >>>> I am fine with not having stateful dhcpv6 and privacy
> >>>> extension/temporary address in phase 1. If community decides not to do
> >>>> eventually , it is also ok to me.
> >>>>
> >>>> We could explore how to better use secondary ipv6 addresses as Wido
> >>>> advised. It would be great if anyone share some user experience.
> >>>>
> >>>> -Wei
> >>>>
> >>>>
> >>>> On Tuesday, 17 August 2021, Kristaps Cudars
> >>>> <kristaps.cud...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Wei,
> >>>>>
> >>>>> Published this month's RFC 9099 and explains in different
> >>>>> words/perspective what has been written by Alex, Rohit and Wido.
> >>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> >>>>>
> >>>>>
> >>>>> On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote:
> >>>>>> Hi Wido,
> >>>>>>
> >>>>>> (cc to Rohit and Alex)
> >>>>>>
> >>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
> >>>>>> quite simple and the VMs can get SLAAC, routes, etc.
> >>>>>>
> >>>>>> Privacy extension looks not the same as what you mentioned. see
> >>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> >>>>>>
> >>>>>> You are right. To use static routing, the admins need to configure
> >>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
> >>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
> >>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
> >>>>>> pick up an IPv6 range
> >>>>> and
> >>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
> >>> wrong.
> >>>>>>
> >>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
> >>>>>> router/VR, router/dns etc via RA messages) will be the only option
> >>>>>> for now (related
> >>>>> to
> >>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> >>>>>> be
> >>>>> good
> >>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> >>>>>> as an option in cloudstack ? The advantages are
> >>>>>> (1) support other ipv6 cidr sizes than /64.
> >>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> >>>>>> addresses can be changed
> >>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
> >>>>>> The problem is, stateful dhcpv6 does not support
> >>>>>> routers,nameservers,
> >>>>> etc.
> >>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
> >>>>>>
> >>>>>> -Wei
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> See my inline responses:
> >>>>>>>
> >>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
> >>>>>>>> discussing
> >>>>> them
> >>>>>>> with Alex and came up with a PoC/design which can be implemented
> >>>>>>> in the following phases:
> >>>>>>>>
> >>>>>>>> * Phase1: implement ipv6 support in isolated networks and
> >>> VPC
> >>>>> with
> >>>>>>> static routing
> >>>>>>>> * Phase2: discuss and implement support for dynamic routing
> >>>>> (TBD)
> >>>>>>>>
> >>>>>>>> For Phase1 here's the high-level proposal:
> >>>>>>>>
> >>>>>>>> * IPv6 address management:
> >>>>>>>> * At the zone level root-admin specifies a /64 public
> >>> range
> >>>>> that
> >>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
> >>>>>>> for
> >>>>> guest
> >>>>>>> networks (to be used by isolated networks and VPC tiers)
> >>>>>>>> * On creation of any IPv6 enabled isolated network or
> VPC
> >>>>> tier,
> >>>>>>> from the /48 or /56 block a /64 network is allocated/used
> >>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in
> the
> >>>>> zone
> >>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
> >>>>>>> of
> >>>>> slaac be
> >>>>>>> explored?)
> >>>>>>>
> >>>>>>> Privacy Extensions are only a concern for client devices which
> >>>>>>> roam between different IPv6 networks.
> >>>>>>>
> >>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
> >>>>>>> based) and switches network then only the prefix (/64) will change.
> >>>>>>>
> >>>>>>> This way a network like Google, Facebook, etc could track your
> >>>>>>> device moving from network to network if they only look at the
> >>>>>>> last 64-bits of the IPv6 address.
> >>>>>>>
> >>>>>>> For servers this is not a problem as you already know in which
> >>>>>>> network they are.
> >>>>>>>
> >>>>>>>> * Network offerings: root-admin can create new network
> >>> offerings
> >>>>>>> (with VPC too) that specifies a network stack option:
> >>>>>>>> * ipv4 only (default, for backward compatibility all
> >>>>>>> networks/offerings post-upgrade migrate to this option)
> >>>>>>>> * ipv4-and-ipv6
> >>>>>>>> * ipv6-only (this can be phase 1.b)
> >>>>>>>> * A new routing option: static (phase1), dynamic
> (phase2,
> >>>>> with
> >>>>>>> multiple sub-options such as ospf/bgp etc...)
> >>>>>>>
> >>>>>>> This means that the network admin will need to statically route
> >>>>>>> the
> >>>>> IPv6
> >>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
> >>>>> router:
> >>>>>>>
> >>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
> >>>>>>> next-hop
> >>>>>>> 2001:db8:100::50
> >>>>>>>
> >>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
> >>>>>>> the outside (/64) network. In reality this will probably be a
> >>>>>>> longer address, but this is for just the example.
> >>>>>>>
> >>>>>>>> * VR changes:
> >>>>>>>> * VR gets its guest and public nics set to inet6 auto
> >>>>>>>> * For each /64 allocated to guest network and VPC tiers,
> >>>>> radvd
> >>>>>>> is configured to do RA
> >>>>>>>
> >>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
> >>>>>>> might already want to look into FRRouting. FRR can also
> >>>>>>> advertise RAs while not doing any routing.
> >>>>>>>
> >>>>>>> interface ens4
> >>>>>>> no ipv6 nd suppress-ra
> >>>>>>> ipv6 nd prefix 2001:db8:500::/64
> >>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >>>>>>>
> >>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> >>>>>>>
> >>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
> >>> where
> >>>>> ipv6
> >>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
> >>>>>>> between
> >>>>> VPC
> >>>>>>> tiers are managed/implemented by ipv6 firewall on VR
> >>>>>>>
> >>>>>>> Please take a look at the existing security_group.py script
> >>>>>>> which implements RFC4890
> >>>>>>>
> >>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> >>>>>>>
> >>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
> >>>>>>> be allowed.
> >>>>>>>
> >>>>>>>> * It is assumed that static routes are created on the
> >>>>> core/main
> >>>>>>> router by the admin or automated using some scripts/tools; for
> >>>>>>> this CloudStack will announce events with details of /64
> >>>>>>> networks and VR's public IPv6 address that can be consumed by a
> >>>>>>> rabbitmq/message bus
> >>>>> client
> >>>>>>> (for example), or a custom cron job or script as part of
> >>> orchestration.
> >>>>>>> (this wouldn't be necessary for dynamic routing bgp with
> >>>>>>> phase2)\\
> >>>>>>>
> >>>>>>> You would only need to announce the /48 or /56 allocated to the
> >>>>>>> VR, that's all. You don't need to inform the upstream router
> >>>>>>> about the /64 subnets created within that larger subnet.
> >>>>>>>
> >>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
> >>>>>>> calculate allocate and use a /64 and determine the IPv6 address
> >>>>>>> of VR
> >>>>> nics
> >>>>>>> and guest VM nics
> >>>>>>>> * A user create an isolated network/VPC with an offering
> >>>>> that is
> >>>>>>> ipv6 enabled
> >>>>>>>> * A user can manage firewall for the IPv6 address/guest
> >>> nics;
> >>>>>>> there'll be no port forward and LB feature though for IPv6
> >>>>>>>> * A users can run workloads in the guest VMs that listen
> >>> on
> >>>>>>> publically routable ipv6 addresses
> >>>>>>>> * Usage/billing etc continue to work, no change needed
> >>>>>>>>
> >>>>>>>> Network layout:
> >>>>>>>>
> >>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> >>>>>>>> VLAN] ->
> >>>>>>> [guest VMs/nics]
> >>>>>>>> *core/ISP router needs static routes to be added (manually or
> >>>>>>> automated), assumes a /48 or /56 configured for the zone
> >>>>>>>>
> >>>>>>>> Thoughts, feedback?
> >>>>>>>
> >>>>>>> Looks doable!
> >>>>>>>
> >>>>>>> Side-note: It would be very cool if you could use parts of this
> >>>>>>> implementation to also route /48, /56, or /60 subnets to
> >>>>>>> individual VMs in Shared networks.
> >>>>>>>
> >>>>>>> Why? This allows for running Docker containers with native IPv6
> >>>>>>> inside the VM or running a (Open)VPN server inside a VM which
> >>>>>>> then assigns public IPv6 addresses to clients connected.
> >>>>>>>
> >>>>>>> Instead of routing the subnet to a VR we route the subnet to a
> >>>>>>> single instance in a shared network.
> >>>>>>>
> >>>>>>> If we could then also move these subnets between Instances
> >>>>>>> easily one can quickly migrate to a different instance while
> >>>>>>> keeping the same IPv6 subnet.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
> >>>>>>>>
> >>>>>>>> * Created an isolated network and deployed a VM in my home
> >>> lab
> >>>>>>>> The VR running on KVM has following nics
> >>>>>>>> eth0 - guest network
> >>>>>>>> eth1 - link local
> >>>>>>>> eth2 - public network
> >>>>>>>>
> >>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
> >>>>> toy-core
> >>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
> >>>>>>> I got
> >>>>> a
> >>>>>>> /48 allocated. My configuration looks like:
> >>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> >>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
> >>>>>>>>
> >>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
> >>>>> subnet/prefix
> >>>>>>> 0:
> >>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> >>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
> >>>>>>>> *
> >>>>>>>> *
> >>>>>>>> In the isolated VR, I enabled ipv6 as:
> >>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
> >>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> >>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
> >>>>>>>> net.ipv6.conf.all.autoconf = 1
> >>>>>>>>
> >>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> >>>>>>>> configured the nics:
> >>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> >>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
> >>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
> >>>>>>>> network without cleanup to have it
> >>>>>>> reconfigure IPv4 nics, firewall, NAT etc
> >>>>>>>>
> >>>>>>>> *
> >>>>>>>> Next, I created a /64 network for the isolated guest network
> >>>>>>>> on eth0
> >>>>> of
> >>>>>>> VR using radvd:
> >>>>>>>>
> >>>>>>>> # cat /etc/radvd.conf
> >>>>>>>> interface eth0
> >>>>>>>> {
> >>>>>>>> AdvSendAdvert on;
> >>>>>>>> MinRtrAdvInterval 5;
> >>>>>>>> MaxRtrAdvInterval 15;
> >>>>>>>> prefix 2001:470:ed36:1::/64
> >>>>>>>> {
> >>>>>>>> AdvOnLink on;
> >>>>>>>> AdvAutonomous on;
> >>>>>>>> };
> >>>>>>>> };
> >>>>>>>> systemctl restart radvd
> >>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> >>>>>>>> this
> >>>>>>> ...:1::/64 network
> >>>>>>>> * Finally I added a static route in toy core-router for the
> >>> new
> >>>>> /64
> >>>>>>> IPv6 range in the isolated network
> >>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> >>>>>>>> eth2> dev
> >>>>>>> <local lan nic>
> >>>>>>>> *
> >>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
> >>>>>>>> for the
> >>>>>>> new /64 network
> >>>>>>>>
> >>>>>>>> And voila all done! I create a domain AAAA record that points
> >>>>>>>> to my
> >>>>>>> guest VM IPv6 address a test webserver on
> >>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >>>>>>>>
> >>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
> >>>>>>>> after a
> >>>>>>> few days, sharing this solely for testing purposes)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >
>
