Regards,
Abhishek
----------------------------------------------------------------------
--
*From:* Wido den Hollander <w...@widodh.nl>
*Sent:* 10 May 2022 19:01
*To:* dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
Kumar <abhishek.ku...@shapeblue.com>
*Subject:* Re: IPV6 in Isolated/VPC networks
Hi,
Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
Yes. When a public IPv6 is assigned or released, CloudStack will publish event
with type NET.IP6ASSIGN, NET.IP6RELEASE.
These event notifications can be tracked. And with improvements in events
framework, these event messages will have network uuid as entityuuid and
Network as entitytype. Using this network can be queried using to list IPv6
routes that need to be added.
Sample event message,
{"eventDateTime":"2022-05-10 09:32:12
+0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public
IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
af19-1e00320001f3","entity":"Network","status":"Completed"}
?Sample API call,
list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
filter=id,name,ip6routes
{
"count": 1,
"network": [
{
"id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
"ip6routes": [
{
"gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
"subnet": "fd23:313a:2f53:3cbf::/64"
}
Looking at this JSON, does this mean that
fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
to that IP?
The event message does not include the subnet, that makes it a bit
more difficult as you would then also need to do a API-call to gather
that information.
Wido
P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
AAAA-record for IPv6!
],
"name": "routing_test"
}
]
}
________________________________
From: Wido den Hollander <w...@widodh.nl>
Sent: 10 May 2022 13:59
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek
Kumar <abhishek.ku...@shapeblue.com>
Subject: Re: IPV6 in Isolated/VPC networks
Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
Hi all,
IPv6 Support in Isolated Network and VPC with Static Routing based on the
design doc [1] has been implemented and is available in 4.17.0 RC2. I hope
while testing 4.17.0 RC2 you will also try to test it ?
Documentation for it is available at
http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
l#isolated-network-and-vpc-tier
<http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
#isolated-network-and-vpc-tier> (will be available in the official
docs once 4.17.0 version of docs is built).
Great work!
I see only static routing is supported. But do we publish something
on the message bus once a new VR/VPC is created?
This way you could pick up these messages and have the network create
a
(static) route based on those.
ExaBGP for example could be used to inject such routes.
Wido
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
in+Isolated+Network+and+VPC+with+Static+Routing
<https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
n+Isolated+Network+and+VPC+with+Static+Routing>
Regards,
Abhishek
________________________________
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: 13 September 2021 14:30
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Thanks Alex, Wei. I've updated the docs here:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
in+Isolated+Network+and+VPC+with+Static+Routing
<https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
n+Isolated+Network+and+VPC+with+Static+Routing>
I'll leave the thread open for futher discussion/ideas/feedback. I
think we've completed the phase1 design doc including all feedback
comments for adding IPv6 support in CloudStack and some initial
poc/work can be started. My colleagues and I will keep everyone
posted on this thread and/or on a Github PR as and when we're able
to
start our work on the same (after 4.16, potentially towards 4.17).
Regards.
________________________________
From: Wei ZHOU <ustcweiz...@gmail.com>
Sent: Friday, September 10, 2021 20:22
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Agree with Alex.
We only need to know how many /64 are allocated. We do not care how
many
ipv6 addresses are used by VMs.
-Wei
On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
<alex.matti...@shapeblue.com>
wrote:
Hi Rohit,
I'd go for option 2, don't see a point tracking anything smaller
than a
/64 tbh.
Cheers
Alex
-----Original Message-----
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: 09 September 2021 12:44
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
Thanks Alex, Kristaps. I've updated the design doc to reflect two
agreements:
* Allocate /64 for both isolated network and VPC tiers, no
large allocation of prefixes to VPC (cons: more static routing
rules for upstream
router/admins)
* All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if
zone has a dedicated /64 prefix/block for systemvms
The only outstanding question now is:
* How do we manage IPv6 usage? Can anyone advise how we do
IPv6 usage for shared network (design, implementation and
use-cases?)
Option1: We don't do it, all user VMs nics have ipv4 address which
whose usage we don't track. For public VR/nics/networks, we can
simply add the
IPv6 details for a related IPv4 address.
Option2: Implement a separate, first-class IPv6 address or /64
prefix tracking/management and usage for all VMs and systemvms nic
(this means account/domain level limits and new billing/records)
Option3: other thoughts?
Regards.
________________________________
From: Alex Mattioli <alex.matti...@shapeblue.com>
Sent: Wednesday, September 8, 2021 23:24
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: RE: IPV6 in Isolated/VPC networks
Hi Rohit, Kristaps,
I'd say option 1 as well, it does create a bit more overhead with
static routes but if that's automated for a VPC it can also be
easily automated for several tiers of a VPC. We also don't
constrain the amount of tiers in a VPC.
It has the added advantage to be closer to the desired behaviour
with dynamic routing in the future, where a VPC VR can announce
several subnets upstream.
Cheers
Alex
-----Original Message-----
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: 08 September 2021 19:04
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
Hi Kristaps,
Thanks for sharing, I suppose that means individual tiers should be
allocated /64 instead of larger ipv6 blocks to the whole VPC which
could cause wastage.
Any objection from anybody?
Regards.
________________________________
From: Kristaps Cudars <kristaps.cud...@gmail.com>
Sent: Wednesday, September 8, 2021 9:24:01 PM
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Hello,
I asked networking team to comment on "How should the IPv6
block/allocation work in VPCs?"
Option1: They haven't seen lately devices with limits on how many
static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers
IPv6 assignment from RIPE NNC can be drained fast.
/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers
On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
All,
After another iteration with Alex, I've updated the design doc.
Kindly
review:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
t+in
<https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
n>
+Isolated+Network+and+VPC+with+Static+Routing
... and advise on some outstanding questions:
* How should the IPv6 block/allocation work in VPCs?
Option1: Should this be simply /64 allocation on any new tier, the
cons of this option is one static route/rule per VPC tier. (many
upstream routers may have limit on no. of static routes?)
Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or
/62
(4 tiers) for the VPC, this can be filtered based on the
vpc.max.networks global setting (3 is default). The pros of this
option are less no. of static route/rule and easy programming, but
potential wastage of multiple
/64 prefix blocks for unused/uncreated tiers.
* How do we manage IPv6 usage? Can anyone advise how we do
IPv6
usage for shared network (design, implementation and use-cases?)
Option1: We don't do it, all user VMs nics have ipv4 address which
whose
usage we don't track. For public VR/nics/networks, we can simply
add the
IPv6 details for a related IPv4 address.
Option2: Implement a separate, first-class IPv6 address or /64
prefix
tracking/management and usage for all VMs and systemvms nic (this
means account/domain level limits and new billing/records)
* Enable IPv6 on systemvms (specifically SSVM and CPVM) by
default
if zone has a IPv6 address block allocated/assigned for use for
systemvms (this was mainly thought for VRs, but why no ssvm and
cpvms too - any cons of this?)
*
Regards.
________________________________
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: Thursday, August 19, 2021 15:45
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Hi all,
I've taken feedback from this thread and wrote this design doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
t+in
<https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
n>
+Isolated+Network+and+VPC+with+Static+Routing
Kindly review and advise if I missed anything or anything that
needs to
be changed/updated. You may comment on the wiki directly too.
Kindly suggest your views on the following (also in the design doc
above):
Outstanding Questions:
* Should admin or user be able to specify how VPC super
CIDRs are
created/needed; for example a user can ask for VPC with /60 super
CIDR? Or should CloudStack automatically find/allocate a /64 for a
new VPC tier from the root-admin configured /64-/48 block?
* Should we explore FRR and iBGP or other strategies to do
dynamic
routing which may not require advance/complex configuration in the
VR or for the users/admin?
* With SLAAC and no dhcpv6, is there a way to support
secondary IPv6
addresses (or floating IPv6 addresses for VR/public traffic) for
guest VM's nics?
* Any thoughts on UI/UX for firewall/routing management?
* Any other feature/support for isolated network or VPC
feature that
must be explored or supported such as PF, VPN, LB, vpc static
routes, vpc gateway etc.
* For usage - should we have any consideration, or should we
assume
that IPv4 and IPv6 address will go together for every nic; so IPv6
usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
explicit/new biling/usage needed?
* For smoketests, local dev-test should we explore ULA?
Unique Local
Address - in the range fc00::/7. Typically only within the 'local'
half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
This prefix can be randomly generated at first install by
CloudStack in a zone using zoneid etc?
* Should we expand support for VR diagnostic feature to
include
support for ipv6, traceroute6?
* Discuss - expand support/ability to allocate a /60, or /56
etc
prefix to an individual VM in shared network (Wido's suggestion)
Regards.
________________________________
From: Wei ZHOU <ustcweiz...@gmail.com>
Sent: Tuesday, August 17, 2021 21:16
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Thanks Kristaps, Wido, Rohit and Alex for your replies.
I am fine with not having stateful dhcpv6 and privacy
extension/temporary address in phase 1. If community decides not
to do eventually , it is also ok to me.
We could explore how to better use secondary ipv6 addresses as
Wido advised. It would be great if anyone share some user experience.
-Wei
On Tuesday, 17 August 2021, Kristaps Cudars
<kristaps.cud...@gmail.com>
wrote:
Hi Wei,
Published this month's RFC 9099 and explains in different
words/perspective what has been written by Alex, Rohit and Wido.
https://www.rfc-editor.org/rfc/rfc9099.html
<https://www.rfc-editor.org/rfc/rfc9099.html>
On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote:
Hi Wido,
(cc to Rohit and Alex)
It is a good suggestion to use FRR for ipv6. The configuration
is quite simple and the VMs can get SLAAC, routes, etc.
Privacy extension looks not the same as what you mentioned. see
https://datatracker.ietf.org/doc/html/rfc4941
<https://datatracker.ietf.org/doc/html/rfc4941>
You are right. To use static routing, the admins need to
configure the routes in the upstream router, and add some ipv6
ranges (eg
/56 for VPCs and /64 for isolated networks) and their next-hop
(which will be configured in VRs) in CloudStack. CloudStack will
pick up an IPv6 range
and
assign it to an isolated network or vpc. @Rohit, correct me if
I'm
wrong.
I have a question, it looks stateless dhcpv6 (SLAAC from
router/VR, router/dns etc via RA messages) will be the only
option for now (related
to
your pr https://github.com/apache/cloudstack/pull/3077)
<https://github.com/apache/cloudstack/pull/3077)> . Would it
be
good
to provide stateful dhcpv6 (which can be implemented by dnsmasq)
as an option in cloudstack ? The advantages are
(1) support other ipv6 cidr sizes than /64.
(2) we can assign a specified Ipv6 address to a vm. vm Ipv6
addresses can be changed
(4) an Ipv6 addresses can be re-used by multiple vms.
The problem is, stateful dhcpv6 does not support
routers,nameservers,
etc.
we need to figure it out (probably use radvd/frr and dnsmasq both).
-Wei
On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote:
Hi,
See my inline responses:
Op 11-08-2021 om 14:26 schreef Rohit Yadav:
Hi all,
Thanks for your feedback and ideas, I've gone ahead with
discussing
them
with Alex and came up with a PoC/design which can be
implemented in the following phases:
* Phase1: implement ipv6 support in isolated networks
and
VPC
with
static routing
* Phase2: discuss and implement support for dynamic
routing
(TBD)
For Phase1 here's the high-level proposal:
* IPv6 address management:
* At the zone level root-admin specifies a /64
public
range
that
will be used for VRs, then they can add a /48, or /56 IPv6
range for
guest
networks (to be used by isolated networks and VPC tiers)
* On creation of any IPv6 enabled isolated network
or VPC
tier,
from the /48 or /56 block a /64 network is allocated/used
* We assume SLAAC and autoconfiguration, no DHCPv6
in the
zone
(discuss: is privacy a concern, can privacy extensions rfc4941
of
slaac be
explored?)
Privacy Extensions are only a concern for client devices which
roam between different IPv6 networks.
If you IPv6 address of a client keeps the same suffix (MAC
based) and switches network then only the prefix (/64) will change.
This way a network like Google, Facebook, etc could track your
device moving from network to network if they only look at the
last 64-bits of the IPv6 address.
For servers this is not a problem as you already know in which
network they are.
* Network offerings: root-admin can create new network
offerings
(with VPC too) that specifies a network stack option:
* ipv4 only (default, for backward compatibility all
networks/offerings post-upgrade migrate to this option)
* ipv4-and-ipv6
* ipv6-only (this can be phase 1.b)
* A new routing option: static (phase1), dynamic
(phase2,
with
multiple sub-options such as ospf/bgp etc...)
This means that the network admin will need to statically route
the
IPv6
subnet to the VR's outside IPv6 address, for example, on a
JunOS
router:
set routing-options rib inet6.0 static route 2001:db8:500::/48
next-hop
2001:db8:100::50
I'm assuming that 2001:db8:100::50 is the address of the VR on
the outside (/64) network. In reality this will probably be a
longer address, but this is for just the example.
* VR changes:
* VR gets its guest and public nics set to inet6
auto
* For each /64 allocated to guest network and VPC
tiers,
radvd
is configured to do RA
radvd is fine, but looking at phase 2 with dynamic routing you
might already want to look into FRRouting. FRR can also
advertise RAs while not doing any routing.
interface ens4
no ipv6 nd suppress-ra
ipv6 nd prefix 2001:db8:500::/64
ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
See: http://docs.frrouting.org/en/latest/ipv6.html
<http://docs.frrouting.org/en/latest/ipv6.html>
* Firewall: a new ipv6 zone/chain is created for
ipv6
where
ipv6
firewall rules (ACLs, ingress, egress) are implemented; ACLs
between
VPC
tiers are managed/implemented by ipv6 firewall on VR
Please take a look at the existing security_group.py script
which implements RFC4890
https://datatracker.ietf.org/doc/html/rfc4890
<https://datatracker.ietf.org/doc/html/rfc4890>
ICMPv6 is a vital part of IPv6 and certain packets should
always be allowed.
* It is assumed that static routes are created on
the
core/main
router by the admin or automated using some scripts/tools; for
this CloudStack will announce events with details of /64
networks and VR's public IPv6 address that can be consumed by a
rabbitmq/message bus
client
(for example), or a custom cron job or script as part of
orchestration.
(this wouldn't be necessary for dynamic routing bgp with
phase2)\\
You would only need to announce the /48 or /56 allocated to the
VR, that's all. You don't need to inform the upstream router
about the /64 subnets created within that larger subnet.
* Guest Networking: With SLAAC, it's easy for
CloudStack to
calculate allocate and use a /64 and determine the IPv6 address
of VR
nics
and guest VM nics
* A user create an isolated network/VPC with an
offering
that is
ipv6 enabled
* A user can manage firewall for the IPv6
address/guest
nics;
there'll be no port forward and LB feature though for IPv6
* A users can run workloads in the guest VMs that
listen
on
publically routable ipv6 addresses
* Usage/billing etc continue to work, no change
needed
Network layout:
[core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
VLAN] ->
[guest VMs/nics]
*core/ISP router needs static routes to be added (manually or
automated), assumes a /48 or /56 configured for the zone
Thoughts, feedback?
Looks doable!
Side-note: It would be very cool if you could use parts of this
implementation to also route /48, /56, or /60 subnets to
individual VMs in Shared networks.
Why? This allows for running Docker containers with native IPv6
inside the VM or running a (Open)VPN server inside a VM which
then assigns public IPv6 addresses to clients connected.
Instead of routing the subnet to a VR we route the subnet to a
single instance in a shared network.
If we could then also move these subnets between Instances
easily one can quickly migrate to a different instance while
keeping the same IPv6 subnet.
Wido
Proof-of-concept commentary: here's what I did to test the idea:
* Created an isolated network and deployed a VM in my
home
lab
The VR running on KVM has following nics
eth0 - guest network
eth1 - link local
eth2 - public network
* I setup a custom openwrt router on a RPi4 to serve as
a
toy-core
router where I create a wan6 IPv6 tunnel using tunnel broker
and I got
a
/48 allocated. My configuration looks like:
/48 - 2001:470:ed36::/48 (allocated by tunnel broker)
/64 - 2001:470:36:3e2::/64 (default allocated by)
I create a LAN ipv6 (public network for CloudStack VR): at
subnet/prefix
0:
LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
SLAAC+stateless DHCP (no dhcpv6)
*
*
In the isolated VR, I enabled ipv6 as:
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
1 net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.autoconf = 1
Set up a IPv6 nameserver/dns in /etc/resolve.conf And
configured the nics:
echo iface eth0 inet6 auto >> /etc/network/interfaces echo
iface eth2 inet6 auto >> /etc/network/interfaces
/etc/init.d/networking restart Next, restart ACS isolated
network without cleanup to have it
reconfigure IPv4 nics, firewall, NAT etc
*
Next, I created a /64 network for the isolated guest network
on eth0
of
VR using radvd:
# cat /etc/radvd.conf
interface eth0
{
AdvSendAdvert on;
MinRtrAdvInterval 5;
MaxRtrAdvInterval 15;
prefix 2001:470:ed36:1::/64
{
AdvOnLink on;
AdvAutonomous on;
};
};
systemctl restart radvd
All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
this
...:1::/64 network
* Finally I added a static route in toy core-router for
the
new
/64
IPv6 range in the isolated network
2001:470:ed36:1::/64 via <public IPv6 address of the VR on
eth2> dev
<local lan nic>
*
... and I enabled firewall rules to allow any traffic to pass
for the
new /64 network
And voila all done! I create a domain AAAA record that points
to my
guest VM IPv6 address a test webserver on
http://ipv6-isolated-ntwk-demo.yadav.cloud/
<http://ipv6-isolated-ntwk-demo.yadav.cloud/>
(Note: I'll get rid of the tunnel and request a new /48 block
after a
few days, sharing this solely for testing purposes)
Regards.
