> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
That's correct
> Or a larger subnet:
> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
Not really, the subnets for isolated/VPC networks are always /64. Which means
also no real need to include subnets as well.
Cheers
Alex
-----Original Message-----
From: Wido den Hollander <w...@widodh.nl>
Sent: 12 May 2022 16:04
To: Abhishek Kumar <abhishek.ku...@shapeblue.com>; dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
On 5/12/22 09:55, Abhishek Kumar wrote:
> Hi Wido,
>
> I do not understand what you mean by WAB address but
WAB was a type. I meant WAN.
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
> (IPv6 of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
So that means that I would need to run this command on my upstream router:
ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
Or a larger subnet:
ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> Currently, the message on event bus does not include subnet. Should
> that be included?
Yes, because then you can pickup those messages and inject the route via ExaBGP
into a routing table right away.
> In case of VPCs, there could be multiple tiers which will need
> multiple routes to be added. Will that be an issue if we include
> current network/tier subnet in the event message?
No, as long as it points to the same VR you simply have multiple subnets being
routed to the same VR.
I do wonder what happens if you destroy the VR and create a new one. The WAN
address then changes (due to SLAAC) and thus the routes need to be
re-programmed.
Wido
>
> Regards,
> Abhishek
>
>
>
>
>
>
> ----------------------------------------------------------------------
> --
> *From:* Wido den Hollander <w...@widodh.nl>
> *Sent:* 10 May 2022 19:01
> *To:* dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
> Kumar <abhishek.ku...@shapeblue.com>
> *Subject:* Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>> Yes. When a public IPv6 is assigned or released, CloudStack will publish
>> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>> These event notifications can be tracked. And with improvements in events
>> framework, these event messages will have network uuid as entityuuid and
>> Network as entitytype. Using this network can be queried using to list IPv6
>> routes that need to be added.
>>
>> Sample event message,
>>
>> {"eventDateTime":"2022-05-10 09:32:12
>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public
>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
>> af19-1e00320001f3","entity":"Network","status":"Completed"}
>>
>>
>> ?Sample API call,
>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>> filter=id,name,ip6routes
>> {
>> "count": 1,
>> "network": [
>> {
>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>> "ip6routes": [
>> {
>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>> "subnet": "fd23:313a:2f53:3cbf::/64"
>> }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
> to that IP?
>
> The event message does not include the subnet, that makes it a bit
> more difficult as you would then also need to do a API-call to gather
> that information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
>> ],
>> "name": "routing_test"
>> }
>> ]
>> }
>>
>>
>>
>>
>> ________________________________
>> From: Wido den Hollander <w...@widodh.nl>
>> Sent: 10 May 2022 13:59
>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
>> Kumar <abhishek.ku...@shapeblue.com>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>> Hi all,
>>>
>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the
>>> design doc [1] has been implemented and is available in 4.17.0 RC2. I hope
>>> while testing 4.17.0 RC2 you will also try to test it ?
>>> Documentation for it is available at
>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>>> l#isolated-network-and-vpc-tier
> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
> #isolated-network-and-vpc-tier> (will be available in the official
> docs once 4.17.0 version of docs is built).
>>>
>>
>> Great work!
>>
>> I see only static routing is supported. But do we publish something
>> on the message bus once a new VR/VPC is created?
>>
>> This way you could pick up these messages and have the network create
>> a
>> (static) route based on those.
>>
>> ExaBGP for example could be used to inject such routes.
>>
>> Wido
>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> ________________________________
>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>> Sent: 13 September 2021 14:30
>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Wei. I've updated the docs here:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>> think we've completed the phase1 design doc including all feedback
>>> comments for adding IPv6 support in CloudStack and some initial
>>> poc/work can be started. My colleagues and I will keep everyone
>>> posted on this thread and/or on a Github PR as and when we're able
>>> to
> start our work on the same (after 4.16, potentially towards 4.17).
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <ustcweiz...@gmail.com>
>>> Sent: Friday, September 10, 2021 20:22
>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Agree with Alex.
>>> We only need to know how many /64 are allocated. We do not care how
>>> many
>>> ipv6 addresses are used by VMs.
>>>
>>> -Wei
>>>
>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>> <alex.matti...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>> than a
>>>> /64 tbh.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>> Sent: 09 September 2021 12:44
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>> agreements:
>>>>
>>>> * Allocate /64 for both isolated network and VPC tiers, no
>>>>large allocation of prefixes to VPC (cons: more static routing
>>>>rules for upstream
>>>> router/admins)
>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if
>>>>zone has a dedicated /64 prefix/block for systemvms
>>>>
>>>> The only outstanding question now is:
>>>>
>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>IPv6 usage for shared network (design, implementation and
>>>>use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>whose usage we don't track. For public VR/nics/networks, we can
>>>>simply add the
>>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>prefix tracking/management and usage for all VMs and systemvms nic
>>>>(this means account/domain level limits and new billing/records)
>>>> Option3: other thoughts?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Alex Mattioli <alex.matti...@shapeblue.com>
>>>> Sent: Wednesday, September 8, 2021 23:24
>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Rohit, Kristaps,
>>>>
>>>> I'd say option 1 as well, it does create a bit more overhead with
>>>> static routes but if that's automated for a VPC it can also be
>>>> easily automated for several tiers of a VPC. We also don't
>>>> constrain the amount of tiers in a VPC.
>>>> It has the added advantage to be closer to the desired behaviour
>>>> with dynamic routing in the future, where a VPC VR can announce
>>>> several subnets upstream.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>> Sent: 08 September 2021 19:04
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Kristaps,
>>>>
>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
>>>> could cause wastage.
>>>>
>>>> Any objection from anybody?
>>>>
>>>> Regards.
>>>> ________________________________
>>>> From: Kristaps Cudars <kristaps.cud...@gmail.com>
>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hello,
>>>>
>>>> I asked networking team to comment on "How should the IPv6
>>>> block/allocation work in VPCs?"
>>>> Option1: They haven't seen lately devices with limits on how many
>>>> static routes can be created.
>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>
>>>> /48 contains 64000 /64
>>>> /60 contains 16 /64
>>>> 64000 / 16 = 4000 routers
>>>>
>>>>
>>>> On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
>>>>> All,
>>>>>
>>>>> After another iteration with Alex, I've updated the design doc.
>>>>> Kindly
>>>> review:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>>
>>>>> ... and advise on some outstanding questions:
>>>>>
>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>>cons of this option is one static route/rule per VPC tier. (many
>>>>>upstream routers may have limit on no. of static routes?)
>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or
>>>>>/62
>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>> option are less no. of static route/rule and easy programming, but
>>>> potential wastage of multiple
>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>>IPv6
>>>> usage for shared network (design, implementation and use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose
>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>> add the
>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>> means account/domain level limits and new billing/records)
>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
>>>>>default
>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>> cpvms too - any cons of this?)
>>>>> *
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>> Kindly review and advise if I missed anything or anything that
>>>>> needs to
>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>
>>>>> Kindly suggest your views on the following (also in the design doc
>>>> above):
>>>>>
>>>>> Outstanding Questions:
>>>>>
>>>>> * Should admin or user be able to specify how VPC super
>>>>>CIDRs are
>>>> created/needed; for example a user can ask for VPC with /60 super
>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>> * Should we explore FRR and iBGP or other strategies to do
>>>>>dynamic
>>>> routing which may not require advance/complex configuration in the
>>>> VR or for the users/admin?
>>>>> * With SLAAC and no dhcpv6, is there a way to support
>>>>>secondary IPv6
>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>> guest VM's nics?
>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>> * Any other feature/support for isolated network or VPC
>>>>>feature that
>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>> routes, vpc gateway etc.
>>>>> * For usage - should we have any consideration, or should we
>>>>>assume
>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>> explicit/new biling/usage needed?
>>>>> * For smoketests, local dev-test should we explore ULA?
>>>>>Unique Local
>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network
>>>> addressing.
>>>> This prefix can be randomly generated at first install by
>>>> CloudStack in a zone using zoneid etc?
>>>>> * Should we expand support for VR diagnostic feature to
>>>>>include
>>>> support for ipv6, traceroute6?
>>>>> * Discuss - expand support/ability to allocate a /60, or /56
>>>>>etc
>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Wei ZHOU <ustcweiz...@gmail.com>
>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>
>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>> extension/temporary address in phase 1. If community decides not
>>>>> to do eventually , it is also ok to me.
>>>>>
>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>> <kristaps.cud...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> Published this month's RFC 9099 and explains in different
>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>
>>>>>>
>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> (cc to Rohit and Alex)
>>>>>>>
>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>
>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>
>>>>>>> You are right. To use static routing, the admins need to
>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>> ranges (eg
>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>> pick up an IPv6 range
>>>>>> and
>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>> I'm
>>>> wrong.
>>>>>>>
>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>> option for now (related
>>>>>> to
>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>> be
>>>>>> good
>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>> addresses can be changed
>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>> routers,nameservers,
>>>>>> etc.
>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> See my inline responses:
>>>>>>>>
>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>> discussing
>>>>>> them
>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>> implemented in the following phases:
>>>>>>>>>
>>>>>>>>> * Phase1: implement ipv6 support in isolated networks
>>>>>>>>>and
>>>> VPC
>>>>>> with
>>>>>>>> static routing
>>>>>>>>> * Phase2: discuss and implement support for dynamic
>>>>>>>>>routing
>>>>>> (TBD)
>>>>>>>>>
>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>
>>>>>>>>> * IPv6 address management:
>>>>>>>>> * At the zone level root-admin specifies a /64
>>>>>>>>>public
>>>> range
>>>>>> that
>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>> range for
>>>>>> guest
>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>> * On creation of any IPv6 enabled isolated network
>>>>>>>>>or VPC
>>>>>> tier,
>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6
>>>>>>>>>in the
>>>>>> zone
>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>> of
>>>>>> slaac be
>>>>>>>> explored?)
>>>>>>>>
>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>> roam between different IPv6 networks.
>>>>>>>>
>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>
>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>> device moving from network to network if they only look at the
>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>
>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>> network they are.
>>>>>>>>
>>>>>>>>> * Network offerings: root-admin can create new network
>>>> offerings
>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>> * A new routing option: static (phase1), dynamic
>>>>>>>>>(phase2,
>>>>>> with
>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>
>>>>>>>> This means that the network admin will need to statically route
>>>>>>>> the
>>>>>> IPv6
>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>> JunOS
>>>>>> router:
>>>>>>>>
>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>> next-hop
>>>>>>>> 2001:db8:100::50
>>>>>>>>
>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>> longer address, but this is for just the example.
>>>>>>>>
>>>>>>>>> * VR changes:
>>>>>>>>> * VR gets its guest and public nics set to inet6
>>>>>>>>>auto
>>>>>>>>> * For each /64 allocated to guest network and VPC
>>>>>>>>>tiers,
>>>>>> radvd
>>>>>>>> is configured to do RA
>>>>>>>>
>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>
>>>>>>>> interface ens4
>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>
>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>
>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>ipv6
>>>> where
>>>>>> ipv6
>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>> between
>>>>>> VPC
>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>
>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>> which implements RFC4890
>>>>>>>>
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>
>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>> always be allowed.
>>>>>>>>
>>>>>>>>> * It is assumed that static routes are created on
>>>>>>>>>the
>>>>>> core/main
>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>> rabbitmq/message bus
>>>>>> client
>>>>>>>> (for example), or a custom cron job or script as part of
>>>> orchestration.
>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>> phase2)\\
>>>>>>>>
>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>
>>>>>>>>> * Guest Networking: With SLAAC, it's easy for
>>>>>>>>>CloudStack to
>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>> of VR
>>>>>> nics
>>>>>>>> and guest VM nics
>>>>>>>>> * A user create an isolated network/VPC with an
>>>>>>>>>offering
>>>>>> that is
>>>>>>>> ipv6 enabled
>>>>>>>>> * A user can manage firewall for the IPv6
>>>>>>>>>address/guest
>>>> nics;
>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>> * A users can run workloads in the guest VMs that
>>>>>>>>>listen
>>>> on
>>>>>>>> publically routable ipv6 addresses
>>>>>>>>> * Usage/billing etc continue to work, no change
>>>>>>>>>needed
>>>>>>>>>
>>>>>>>>> Network layout:
>>>>>>>>>
>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>> VLAN] ->
>>>>>>>> [guest VMs/nics]
>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>
>>>>>>>>> Thoughts, feedback?
>>>>>>>>
>>>>>>>> Looks doable!
>>>>>>>>
>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>> individual VMs in Shared networks.
>>>>>>>>
>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>
>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>> single instance in a shared network.
>>>>>>>>
>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>
>>>>>>>>> * Created an isolated network and deployed a VM in my
>>>>>>>>>home
>>>> lab
>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>> eth0 - guest network
>>>>>>>>> eth1 - link local
>>>>>>>>> eth2 - public network
>>>>>>>>>
>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as
>>>>>>>>>a
>>>>>> toy-core
>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>> and I got
>>>>>> a
>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>
>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>> subnet/prefix
>>>>>>>> 0:
>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>net.ipv6.conf.all.autoconf = 1
>>>>>>>>>
>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>> configured the nics:
>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>> network without cleanup to have it
>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>on eth0
>>>>>> of
>>>>>>>> VR using radvd:
>>>>>>>>>
>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>> interface eth0
>>>>>>>>> {
>>>>>>>>> AdvSendAdvert on;
>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>> {
>>>>>>>>> AdvOnLink on;
>>>>>>>>> AdvAutonomous on;
>>>>>>>>> };
>>>>>>>>> };
>>>>>>>>> systemctl restart radvd
>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>this
>>>>>>>> ...:1::/64 network
>>>>>>>>> * Finally I added a static route in toy core-router for
>>>>>>>>>the
>>>> new
>>>>>> /64
>>>>>>>> IPv6 range in the isolated network
>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>> eth2> dev
>>>>>>>> <local lan nic>
>>>>>>>>> *
>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>for the
>>>>>>>> new /64 network
>>>>>>>>>
>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>> to my
>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>
>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>> after a
>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
