> But there can be multiple networks behind the VPC router or not? If there are
> multiple networks you need
>/64 as you can then allocate /64s from that larger subnet.
Yes, there will be multiple /64s in a VPC, but they are very likely to be
non-contiguous, so in general we should treat as completely separate networks.
One could in theory dedicate a range to a domain/account and statically route
to that, but I'm not sure if that will happen often.
Cheers
Alex
-----Original Message-----
From: Wido den Hollander <w...@widodh.nl>
Sent: 13 May 2022 11:30
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>> ipv6 route fd23:313a:2f53:3cbf::/64
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> That's correct
Ok. So in that case the subnet would be very welcome to be present in the
message on the bus.
>
>> Or a larger subnet:
>> ipv6 route fd23:313a:2f53:3c00::/56
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Not really, the subnets for isolated/VPC networks are always /64. Which
> means also no real need to include subnets as well.
>
But there can be multiple networks behind the VPC router or not? If there are
multiple networks you need >/64 as you can then allocate /64s from that larger
subnet.
Wido
> Cheers
> Alex
>
>
>
>
>
> -----Original Message-----
> From: Wido den Hollander <w...@widodh.nl>
> Sent: 12 May 2022 16:04
> To: Abhishek Kumar <abhishek.ku...@shapeblue.com>;
> dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> On 5/12/22 09:55, Abhishek Kumar wrote:
>> Hi Wido,
>>
>> I do not understand what you mean by WAB address but
>
> WAB was a type. I meant WAN.
>
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
>> (IPv6 of the public NIC of the network VR) in the sample.
>> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
>> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>>
>
> So that means that I would need to run this command on my upstream router:
>
> ipv6 route fd23:313a:2f53:3cbf::/64
> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Or a larger subnet:
>
> ipv6 route fd23:313a:2f53:3c00::/56
> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
>> Currently, the message on event bus does not include subnet. Should
>> that be included?
>
> Yes, because then you can pickup those messages and inject the route via
> ExaBGP into a routing table right away.
>
>> In case of VPCs, there could be multiple tiers which will need
>> multiple routes to be added. Will that be an issue if we include
>> current network/tier subnet in the event message?
>
> No, as long as it points to the same VR you simply have multiple subnets
> being routed to the same VR.
>
> I do wonder what happens if you destroy the VR and create a new one. The WAN
> address then changes (due to SLAAC) and thus the routes need to be
> re-programmed.
>
> Wido
>
>>
>> Regards,
>> Abhishek
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> -
>> --
>> *From:* Wido den Hollander <w...@widodh.nl>
>> *Sent:* 10 May 2022 19:01
>> *To:* dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
>> Kumar <abhishek.ku...@shapeblue.com>
>> *Subject:* Re: IPV6 in Isolated/VPC networks
>>
>> Hi,
>>
>> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>>> Yes. When a public IPv6 is assigned or released, CloudStack will publish
>>> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>>> These event notifications can be tracked. And with improvements in events
>>> framework, these event messages will have network uuid as entityuuid and
>>> Network as entitytype. Using this network can be queried using to list IPv6
>>> routes that need to be added.
>>>
>>> Sample event message,
>>>
>>> {"eventDateTime":"2022-05-10 09:32:12
>>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
>>> public
>>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec
>>> - af19-1e00320001f3","entity":"Network","status":"Completed"}
>>>
>>>
>>> ?Sample API call,
>>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>>> filter=id,name,ip6routes
>>> {
>>> "count": 1,
>>> "network": [
>>> {
>>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>> "ip6routes": [
>>> {
>>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>> "subnet": "fd23:313a:2f53:3cbf::/64"
>>> }
>>
>> Looking at this JSON, does this mean that
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>>
>> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
>> to that IP?
>>
>> The event message does not include the subnet, that makes it a bit
>> more difficult as you would then also need to do a API-call to gather
>> that information.
>>
>> Wido
>>
>> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
>> AAAA-record for IPv6!
>>
>>> ],
>>> "name": "routing_test"
>>> }
>>> ]
>>> }
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Wido den Hollander <w...@widodh.nl>
>>> Sent: 10 May 2022 13:59
>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
>>> Kumar <abhishek.ku...@shapeblue.com>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>>> Hi all,
>>>>
>>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the
>>>> design doc [1] has been implemented and is available in 4.17.0 RC2. I hope
>>>> while testing 4.17.0 RC2 you will also try to test it ?
>>>> Documentation for it is available at
>>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.ht
>>>> m
>>>> l#isolated-network-and-vpc-tier
>> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>> l #isolated-network-and-vpc-tier> (will be available in the official
>> docs once 4.17.0 version of docs is built).
>>>>
>>>
>>> Great work!
>>>
>>> I see only static routing is supported. But do we publish something
>>> on the message bus once a new VR/VPC is created?
>>>
>>> This way you could pick up these messages and have the network
>>> create a
>>> (static) route based on those.
>>>
>>> ExaBGP for example could be used to inject such routes.
>>>
>>> Wido
>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>> Sent: 13 September 2021 14:30
>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Wei. I've updated the docs here:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>>> think we've completed the phase1 design doc including all feedback
>>>> comments for adding IPv6 support in CloudStack and some initial
>>>> poc/work can be started. My colleagues and I will keep everyone
>>>> posted on this thread and/or on a Github PR as and when we're able
>>>> to
>> start our work on the same (after 4.16, potentially towards 4.17).
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <ustcweiz...@gmail.com>
>>>> Sent: Friday, September 10, 2021 20:22
>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Agree with Alex.
>>>> We only need to know how many /64 are allocated. We do not care how
>>>> many
>>>> ipv6 addresses are used by VMs.
>>>>
>>>> -Wei
>>>>
>>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>>> <alex.matti...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Rohit,
>>>>>
>>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>>> than a
>>>>> /64 tbh.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>>> Sent: 09 September 2021 12:44
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>>> agreements:
>>>>>
>>>>> * Allocate /64 for both isolated network and VPC tiers, no
>>>>> large allocation of prefixes to VPC (cons: more static routing
>>>>> rules for upstream
>>>>> router/admins)
>>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address
>>>>> if zone has a dedicated /64 prefix/block for systemvms
>>>>>
>>>>> The only outstanding question now is:
>>>>>
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>> IPv6 usage for shared network (design, implementation and
>>>>> use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose usage we don't track. For public VR/nics/networks, we can
>>>>> simply add the
>>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix tracking/management and usage for all VMs and systemvms
>>>>> nic (this means account/domain level limits and new
>>>>> billing/records)
>>>>> Option3: other thoughts?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Alex Mattioli <alex.matti...@shapeblue.com>
>>>>> Sent: Wednesday, September 8, 2021 23:24
>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Rohit, Kristaps,
>>>>>
>>>>> I'd say option 1 as well, it does create a bit more overhead with
>>>>> static routes but if that's automated for a VPC it can also be
>>>>> easily automated for several tiers of a VPC. We also don't
>>>>> constrain the amount of tiers in a VPC.
>>>>> It has the added advantage to be closer to the desired behaviour
>>>>> with dynamic routing in the future, where a VPC VR can announce
>>>>> several subnets upstream.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>>> Sent: 08 September 2021 19:04
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Kristaps,
>>>>>
>>>>> Thanks for sharing, I suppose that means individual tiers should
>>>>> be allocated /64 instead of larger ipv6 blocks to the whole VPC
>>>>> which could cause wastage.
>>>>>
>>>>> Any objection from anybody?
>>>>>
>>>>> Regards.
>>>>> ________________________________
>>>>> From: Kristaps Cudars <kristaps.cud...@gmail.com>
>>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hello,
>>>>>
>>>>> I asked networking team to comment on "How should the IPv6
>>>>> block/allocation work in VPCs?"
>>>>> Option1: They haven't seen lately devices with limits on how many
>>>>> static routes can be created.
>>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>>
>>>>> /48 contains 64000 /64
>>>>> /60 contains 16 /64
>>>>> 64000 / 16 = 4000 routers
>>>>>
>>>>>
>>>>> On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> After another iteration with Alex, I've updated the design doc.
>>>>>> Kindly
>>>>> review:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>>
>>>>>> ... and advise on some outstanding questions:
>>>>>>
>>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>>> Option1: Should this be simply /64 allocation on any new tier,
>>>>>> the cons of this option is one static route/rule per VPC tier.
>>>>>> (many upstream routers may have limit on no. of static routes?)
>>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers)
>>>>>> or
>>>>>> /62
>>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>>> option are less no. of static route/rule and easy programming, but
>>>>> potential wastage of multiple
>>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>> * How do we manage IPv6 usage? Can anyone advise how we
>>>>>> do
>>>>>> IPv6
>>>>> usage for shared network (design, implementation and use-cases?)
>>>>>> Option1: We don't do it, all user VMs nics have ipv4 address
>>>>>> which whose
>>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>>> add the
>>>>> IPv6 details for a related IPv4 address.
>>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>>> prefix
>>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>>> means account/domain level limits and new billing/records)
>>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM)
>>>>>> by default
>>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>>> cpvms too - any cons of this?)
>>>>>> *
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Rohit Yadav <rohit.ya...@shapeblue.com>
>>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>> Kindly review and advise if I missed anything or anything that
>>>>>> needs to
>>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>>
>>>>>> Kindly suggest your views on the following (also in the design
>>>>>> doc
>>>>> above):
>>>>>>
>>>>>> Outstanding Questions:
>>>>>>
>>>>>> * Should admin or user be able to specify how VPC super
>>>>>> CIDRs are
>>>>> created/needed; for example a user can ask for VPC with /60 super
>>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>> * Should we explore FRR and iBGP or other strategies to
>>>>>> do dynamic
>>>>> routing which may not require advance/complex configuration in the
>>>>> VR or for the users/admin?
>>>>>> * With SLAAC and no dhcpv6, is there a way to support
>>>>>> secondary IPv6
>>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>>> guest VM's nics?
>>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>>> * Any other feature/support for isolated network or VPC
>>>>>> feature that
>>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>>> routes, vpc gateway etc.
>>>>>> * For usage - should we have any consideration, or should
>>>>>> we assume
>>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>>> explicit/new biling/usage needed?
>>>>>> * For smoketests, local dev-test should we explore ULA?
>>>>>> Unique Local
>>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network
>>>>> addressing.
>>>>> This prefix can be randomly generated at first install by
>>>>> CloudStack in a zone using zoneid etc?
>>>>>> * Should we expand support for VR diagnostic feature to
>>>>>> include
>>>>> support for ipv6, traceroute6?
>>>>>> * Discuss - expand support/ability to allocate a /60, or
>>>>>> /56 etc
>>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>>
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Wei ZHOU <ustcweiz...@gmail.com>
>>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>>> To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>>
>>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>>> extension/temporary address in phase 1. If community decides not
>>>>>> to do eventually , it is also ok to me.
>>>>>>
>>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>>> <kristaps.cud...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wei,
>>>>>>>
>>>>>>> Published this month's RFC 9099 and explains in different
>>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> (cc to Rohit and Alex)
>>>>>>>>
>>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>>
>>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>>
>>>>>>>> You are right. To use static routing, the admins need to
>>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>>> ranges (eg
>>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack
>>>>>>>> will pick up an IPv6 range
>>>>>>> and
>>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>>> I'm
>>>>> wrong.
>>>>>>>>
>>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>>> option for now (related
>>>>>>> to
>>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
>> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>>> be
>>>>>>> good
>>>>>>>> to provide stateful dhcpv6 (which can be implemented by
>>>>>>>> dnsmasq) as an option in cloudstack ? The advantages are
>>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>>> addresses can be changed
>>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>>> routers,nameservers,
>>>>>>> etc.
>>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> See my inline responses:
>>>>>>>>>
>>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>>> discussing
>>>>>>> them
>>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>>> implemented in the following phases:
>>>>>>>>>>
>>>>>>>>>> * Phase1: implement ipv6 support in isolated
>>>>>>>>>> networks and
>>>>> VPC
>>>>>>> with
>>>>>>>>> static routing
>>>>>>>>>> * Phase2: discuss and implement support for dynamic
>>>>>>>>>> routing
>>>>>>> (TBD)
>>>>>>>>>>
>>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>>
>>>>>>>>>> * IPv6 address management:
>>>>>>>>>> * At the zone level root-admin specifies a /64
>>>>>>>>>> public
>>>>> range
>>>>>>> that
>>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>>> range for
>>>>>>> guest
>>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>> * On creation of any IPv6 enabled isolated
>>>>>>>>>> network or VPC
>>>>>>> tier,
>>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>> * We assume SLAAC and autoconfiguration, no
>>>>>>>>>> DHCPv6 in the
>>>>>>> zone
>>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>>> of
>>>>>>> slaac be
>>>>>>>>> explored?)
>>>>>>>>>
>>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>>> roam between different IPv6 networks.
>>>>>>>>>
>>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>>
>>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>>> device moving from network to network if they only look at the
>>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>>
>>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>>> network they are.
>>>>>>>>>
>>>>>>>>>> * Network offerings: root-admin can create new
>>>>>>>>>> network
>>>>> offerings
>>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>> * ipv4 only (default, for backward compatibility
>>>>>>>>>> all
>>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>>> * A new routing option: static (phase1), dynamic
>>>>>>>>>> (phase2,
>>>>>>> with
>>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>>
>>>>>>>>> This means that the network admin will need to statically
>>>>>>>>> route the
>>>>>>> IPv6
>>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>>> JunOS
>>>>>>> router:
>>>>>>>>>
>>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>>> next-hop
>>>>>>>>> 2001:db8:100::50
>>>>>>>>>
>>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>>> longer address, but this is for just the example.
>>>>>>>>>
>>>>>>>>>> * VR changes:
>>>>>>>>>> * VR gets its guest and public nics set to inet6
>>>>>>>>>> auto
>>>>>>>>>> * For each /64 allocated to guest network and VPC
>>>>>>>>>> tiers,
>>>>>>> radvd
>>>>>>>>> is configured to do RA
>>>>>>>>>
>>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>>
>>>>>>>>> interface ens4
>>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>>
>>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>>
>>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>> ipv6
>>>>> where
>>>>>>> ipv6
>>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>>> between
>>>>>>> VPC
>>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>>
>>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>>> which implements RFC4890
>>>>>>>>>
>>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>>
>>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>>> always be allowed.
>>>>>>>>>
>>>>>>>>>> * It is assumed that static routes are created on
>>>>>>>>>> the
>>>>>>> core/main
>>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>>> networks and VR's public IPv6 address that can be consumed by
>>>>>>>>> a rabbitmq/message bus
>>>>>>> client
>>>>>>>>> (for example), or a custom cron job or script as part of
>>>>> orchestration.
>>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>>> phase2)\\
>>>>>>>>>
>>>>>>>>> You would only need to announce the /48 or /56 allocated to
>>>>>>>>> the VR, that's all. You don't need to inform the upstream
>>>>>>>>> router about the /64 subnets created within that larger subnet.
>>>>>>>>>
>>>>>>>>>> * Guest Networking: With SLAAC, it's easy for
>>>>>>>>>> CloudStack to
>>>>>>>>> calculate allocate and use a /64 and determine the IPv6
>>>>>>>>> address of VR
>>>>>>> nics
>>>>>>>>> and guest VM nics
>>>>>>>>>> * A user create an isolated network/VPC with an
>>>>>>>>>> offering
>>>>>>> that is
>>>>>>>>> ipv6 enabled
>>>>>>>>>> * A user can manage firewall for the IPv6
>>>>>>>>>> address/guest
>>>>> nics;
>>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>> * A users can run workloads in the guest VMs that
>>>>>>>>>> listen
>>>>> on
>>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>> * Usage/billing etc continue to work, no change
>>>>>>>>>> needed
>>>>>>>>>>
>>>>>>>>>> Network layout:
>>>>>>>>>>
>>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>>> VLAN] ->
>>>>>>>>> [guest VMs/nics]
>>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>>
>>>>>>>>>> Thoughts, feedback?
>>>>>>>>>
>>>>>>>>> Looks doable!
>>>>>>>>>
>>>>>>>>> Side-note: It would be very cool if you could use parts of
>>>>>>>>> this implementation to also route /48, /56, or /60 subnets to
>>>>>>>>> individual VMs in Shared networks.
>>>>>>>>>
>>>>>>>>> Why? This allows for running Docker containers with native
>>>>>>>>> IPv6 inside the VM or running a (Open)VPN server inside a VM
>>>>>>>>> which then assigns public IPv6 addresses to clients connected.
>>>>>>>>>
>>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>>> single instance in a shared network.
>>>>>>>>>
>>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>>
>>>>>>>>>> * Created an isolated network and deployed a VM in
>>>>>>>>>> my home
>>>>> lab
>>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>>> eth0 - guest network
>>>>>>>>>> eth1 - link local
>>>>>>>>>> eth2 - public network
>>>>>>>>>>
>>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve
>>>>>>>>>> as a
>>>>>>> toy-core
>>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>>> and I got
>>>>>>> a
>>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>>
>>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>>> subnet/prefix
>>>>>>>>> 0:
>>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>> *
>>>>>>>>>> *
>>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra
>>>>>>>>>> =
>>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>>
>>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>>> configured the nics:
>>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>>> network without cleanup to have it
>>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>>
>>>>>>>>>> *
>>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>> on eth0
>>>>>>> of
>>>>>>>>> VR using radvd:
>>>>>>>>>>
>>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>>> interface eth0
>>>>>>>>>> {
>>>>>>>>>> AdvSendAdvert on;
>>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>>> {
>>>>>>>>>> AdvOnLink on;
>>>>>>>>>> AdvAutonomous on;
>>>>>>>>>> };
>>>>>>>>>> };
>>>>>>>>>> systemctl restart radvd
>>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>> this
>>>>>>>>> ...:1::/64 network
>>>>>>>>>> * Finally I added a static route in toy core-router
>>>>>>>>>> for the
>>>>> new
>>>>>>> /64
>>>>>>>>> IPv6 range in the isolated network
>>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>>> eth2> dev
>>>>>>>>> <local lan nic>
>>>>>>>>>> *
>>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>> for the
>>>>>>>>> new /64 network
>>>>>>>>>>
>>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>>> to my
>>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>>
>>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>>> after a
>>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
