Hi Wido, In current setup, each Cloudstack have own VR, so in this new IPv6 subnet allocation , each VR (which have Frr) will need to have peering with ISP router (and either BGP or Static Route) , and there is 1000 Acocunts, it will 1000 BGP session with ISP router , Am I right for this ? or I understand wrong .
I understand IPv6 is different then IPv4, and in IPv6 it suppose each devices have own IP. It just how to realize in easy way. On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <w...@widodh.nl> wrote: > > > Op 16-07-2021 om 05:54 schreef Hean Seng: > > Hi Wido, > > > > My initial thought is not like this, it is the /48 at ISP router, and > /64 > > subnet assign to AdvanceZoneVR, AdvanceZoneVR responsible is > > distribule IPv6 ip (from the assigned /64 sunet) to VM, and not routing > > the traffic, in the VM that get the IPv6 IP will default route to ISP > > router as gw. It can may be a bridge over via Advancezone-VR. > > > > How would you bridge this? That sounds like NAT? > > IPv6 is meant to be routed. Not to be translated or bridged in any way. > > The way a made the drawing is exactly how IPv6 should work in a VPC > environment. > > Traffic flows through the VR where it can do firewalling of the traffic. > > > However, If do as the way described in the drawing, then i suppose will > be > > another kind of virtual router going to introduce , to get hold the /48 > in > > this virtual router right ? > > > > It can be the same VR. But keep in mind that IPv6 != IPv4. > > The VR will get Frr as a new daemon which can talk BGP with the upper > network to route traffic. > > > After this, The Advance Zone, NAT's VR will peer with this new IPv6 VR > > for getting the IPv6 /64 prefix ? > > > > IPv4 will be behind NAT, but IPv6 will not be behind NAT. > > > If do in this way, then I guess you just only need Static route, with > > peering ip both end as one /48 can have a lot of /64 on it. And > hardware > > budgeting for new IPv6-VR will become very important, as all traffic will > > need to pass over it . > > > > Routing or NAT is the same for the VR. You don't need a very beefy VR > for this. > > > It will be like > > > > ISP Router ------ > (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM > > > > Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on > > OSPF instead of BGP , otherwise few thousand of AdvanceZone-VR wil have > > few thousand of BGP session. on new-IPv6-VR > > > > Also, I suppose we cannot do ISP router. -->. Advancezone VR direct, , > > otherwise ISP router will be full of /64 prefix route either on BGP( Many > > BGP Session) , or Many Static route . If few thousand account, ti will > > be few thousand of BGP session with ISP router or few thousand static > route > > which is not possible . > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <w...@widodh.nl> > wrote: > > > >> But you still need routing. See the attached PNG (and draw.io XML). > >> > >> You need to route the /48 subnet TO the VR which can then route it to > >> the Virtual Networks behind the VR. > >> > >> There is no other way then routing with either BGP or a Static route. > >> > >> Wido > >> > >> Op 15-07-2021 om 12:39 schreef Hean Seng: > >>> Or explain like this : > >>> > >>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin > >>> assigned to Cloudstack > >>> 2) Cloudsack allocated the subnet (that generated from step1) to > Virtual > >>> Router, one Virtual Router have one subniet /64 > >>> 3) Virtual Router allocate single IPv6 (within the range of /64 > >>> allocated to VR) to VM > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heans...@gmail.com > >>> <mailto:heans...@gmail.com>> wrote: > >>> > >>> Hi Wido, > >>> > >>> I think the /48 is at physical router as gateway , and subnet of > /64 > >>> at VR of Cloudstack. Cloudstack only keep which /48 prefix and > >>> vlan information of this /48 to be later split the /64. to VR. > >>> > >>> And the instances is getting singe IPv6 of /64 IP. The VR is > >>> getting /64. The default gateway shall goes to /48 of physical > >>> router ip . In this case ,does not need any BGP router . > >>> > >>> > >>> Similar concept as IPv4 : > >>> > >>> /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 > that > >>> created in Network. > >>> and /64 of IPv6 is equivalent to single IP of IPv4 assign to VM. > >>> > >>> > >>> > >>> > >>> On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander < > w...@widodh.nl > >>> <mailto:w...@widodh.nl>> wrote: > >>> > >>> > >>> > >>> Op 14-07-2021 om 16:44 schreef Hean Seng: > >>> > Hi > >>> > > >>> > I replied in another thread, i think do not need implement > >>> BGP or OSPF, > >>> > that would be complicated . > >>> > > >>> > We only need assign IPv6 's /64 prefix to Virtual Router > >>> (VR) in NAT > >>> > zone, and the VR responsible to deliver single IPv6 to VM > via > >>> DHCP6. > >>> > > >>> > In VR, you need to have Default IPv6 route to Physical > >>> Router's /48. IP > >>> > as IPv6 Gateway. Thens should be done . > >>> > > >>> > Example : > >>> > Physical Router Interface > >>> > IPv6 IP : 2000:aaaa::1/48 > >>> > > >>> > Cloudstack virtual router : 2000:aaaa:200:201::1/64 with > >>> default ipv6 > >>> > route to router ip 2000:aaaa::1 > >>> > and Clodustack Virtual router dhcp allocate IP to VM , and > >>> VM will have > >>> > default route to VR. IPv6 2000:aaaa:200:201::1 > >>> > > >>> > So in cloudstack need to allow user to enter , IPv6 > >>> gwateway , and > >>> > the /48 Ipv6 prefix , then it will self allocate the /64 > ip > >>> to the VR , > >>> > and maintain make sure not ovelap allocation > >>> > > >>> > > >>> > >>> But NAT is truly not the solution with IPv6. IPv6 is supposed > to > >> be > >>> routable. In addition you should avoid DHCPv6 as much as > >>> possible as > >>> that's not really the intended use-case for address allocation > >>> with IPv6. > >>> > >>> In order to route an /48 IPv6 subnet to the VR you have a few > >>> possibilities: > >>> > >>> - Static route from the upperlying routers which are outside > of > >>> CloudStack > >>> - BGP > >>> - OSPFv3 (broken in most cases!) > >>> - DHCPv6 Prefix Delegation > >>> > >>> BGP and/or Static routes are still the best bet here. > >>> > >>> So what you do is that you tell CloudStack that you will route > >>> 2001:db8::/48 to the VR, the VR can then use that to split it > up > >>> into > >>> multiple /64 subnets going towards the instances: > >>> > >>> - 2001:db8::/64 > >>> - 2001:db8:1::/64 > >>> - 2001:db8:2::/64 > >>> ... > >>> - 2001:db8:f::/64 > >>> > >>> And go on. > >>> > >>> In case of BGP you indeed have to tell the VR a few things: > >>> > >>> - It's own AS number > >>> - The peer's address(es) > >>> > >>> With FRR you can simply say: > >>> > >>> neighbor 2001:db8:4fa::179 remote-as external > >>> > >>> The /48 you need to have at the VR anyway in case of either a > >>> static > >>> route or BGP. > >>> > >>> We just need to add a NullRoute on the VR for that /48 so that > >>> traffic > >>> will not be routed to the upper gateway in case of the VR > can't > >>> find a > >>> route. > >>> > >>> Wido > >>> > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli > >>> > <alex.matti...@shapeblue.com > >>> <mailto:alex.matti...@shapeblue.com> > >>> <mailto:alex.matti...@shapeblue.com > >>> <mailto:alex.matti...@shapeblue.com>>> wrote: > >>> > > >>> > Hi Wido, > >>> > That's pretty much in line with our thoughts, thanks > for > >>> the input. > >>> > I believe we agree on the following points then: > >>> > > >>> > - FRR with BGP (no OSPF) > >>> > - Route /48 (or/56) down to the VR > >>> > - /64 per network > >>> > - SLACC for IP addressing > >>> > > >>> > I believe the next big question is then "on which level > >>> of ACS do we > >>> > manage AS numbers?". I see two options: > >>> > 1) Private AS number on a per-zone basis > >>> > 2) Root Admin assigned AS number on a domain/account > basis > >>> > 3) End-user driven AS number on a per network basis > (for > >>> bring your > >>> > own AS and IP scenario) > >>> > > >>> > Thoughts? > >>> > > >>> > Cheers > >>> > Alex > >>> > > >>> > > >>> > > >>> > > >>> > -----Original Message----- > >>> > From: Wido den Hollander <w...@widodh.nl > >>> <mailto:w...@widodh.nl> <mailto:w...@widodh.nl > >>> <mailto:w...@widodh.nl>>> > >>> > Sent: 13 July 2021 15:08 > >>> > To: dev@cloudstack.apache.org > >>> <mailto:dev@cloudstack.apache.org> > >>> <mailto:dev@cloudstack.apache.org > >>> <mailto:dev@cloudstack.apache.org>>; > >>> > Alex Mattioli <alex.matti...@shapeblue.com > >>> <mailto:alex.matti...@shapeblue.com> > >>> > <mailto:alex.matti...@shapeblue.com > >>> <mailto:alex.matti...@shapeblue.com>>> > >>> > Cc: Wei Zhou <wei.z...@shapeblue.com > >>> <mailto:wei.z...@shapeblue.com> > >>> > <mailto:wei.z...@shapeblue.com > >>> <mailto:wei.z...@shapeblue.com>>>; Rohit Yadav > >>> > <rohit.ya...@shapeblue.com > >>> <mailto:rohit.ya...@shapeblue.com> > >>> <mailto:rohit.ya...@shapeblue.com > >>> <mailto:rohit.ya...@shapeblue.com>>>; > >>> > Gabriel Beims Bräscher <gabr...@pcextreme.nl > >>> <mailto:gabr...@pcextreme.nl> > >>> > <mailto:gabr...@pcextreme.nl <mailto: > gabr...@pcextreme.nl > >>>>> > >>> > Subject: Re: IPV6 in Isolated/VPC networks > >>> > > >>> > > >>> > > >>> > On 7/7/21 1:16 PM, Alex Mattioli wrote: > >>> > > Hi all, > >>> > > @Wei Zhou<mailto:wei.z...@shapeblue.com > >>> <mailto:wei.z...@shapeblue.com> > >>> > <mailto:wei.z...@shapeblue.com > >>> <mailto:wei.z...@shapeblue.com>>> @Rohit > >>> > Yadav<mailto:rohit.ya...@shapeblue.com > >>> <mailto:rohit.ya...@shapeblue.com> > >>> > <mailto:rohit.ya...@shapeblue.com > >>> <mailto:rohit.ya...@shapeblue.com>>> and myself are > >>> investigating how > >>> > to enable IPV6 support on Isolated and VPC networks and > >>> would like > >>> > your input on it. > >>> > > At the moment we are looking at implementing FRR > with > >>> BGP (and > >>> > possibly OSPF) on the ACS VR. > >>> > > > >>> > > We are looking for requirements, recommendations, > >>> ideas, rants, > >>> > etc...etc... > >>> > > > >>> > > >>> > Ok! Here we go. > >>> > > >>> > I think that you mean that the VR will actually route > the > >>> IPv6 > >>> > traffic and for that you need to have a way of getting > a > >>> subnet > >>> > routed to the VR. > >>> > > >>> > BGP is probably you best bet here. Although OSPFv3 > >>> technically > >>> > supports this it is very badly implemented in Frr for > >>> example. > >>> > > >>> > Now FRR is a very good router and one of the fancy > >>> features it > >>> > supports is BGP Unnumered. This allows for auto > >>> configuration of BGP > >>> > over a L2 network when both sides are sending Router > >>> Advertisements. > >>> > This is very easy for flexible BGP configurations where > >>> both sides > >>> > have dynamic IPs. > >>> > > >>> > What you want to do is that you get a /56, /48 or > >>> something which is > >>> > >/64 bits routed to the VR. > >>> > > >>> > Now you can sub-segment this into separate /64 subnets. > >>> You don't > >>> > want to go smaller then a /64 is that prevents you from > >>> using SLAAC > >>> > for IPv6 address configuration. This is how it works > for > >>> Shared > >>> > Networks now in Basic and Advanced Zones. > >>> > > >>> > FRR can now also send out the Router Advertisements on > >>> the downlinks > >>> > sending out: > >>> > > >>> > - DNS servers > >>> > - DNS domain > >>> > - Prefix (/64) to be used > >>> > > >>> > There is no need for DHCPv6. You can calculate the IPv6 > >>> address the > >>> > VM will obtain by using the MAC and the prefix. > >>> > > >>> > So in short: > >>> > > >>> > - Using BGP you routed a /48 to the VR > >>> > - Now you split this into /64 subnets towards the > >>> isolated networks > >>> > > >>> > Wido > >>> > > >>> > > Alex Mattioli > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> > -- > >>> > Regards, > >>> > Hean Seng > >>> > >>> > >>> > >>> -- > >>> Regards, > >>> Hean Seng > >>> > >>> > >>> > >>> -- > >>> Regards, > >>> Hean Seng > > > > > > > -- Regards, Hean Seng
