Hi Wido, In context of size I would recommend sticking with RIPE guidelines. https://www.ripe.net/publications/docs/ripe-690
Reminder: /56 256 LAN segments /48 65,536 LAN segments Maybe it will sound counter intuitive but I’m also not excited about OSPFv3 or BGP on VR. As in our case it will be around 300 to 500 potential problems where networking department must be involved if it fails. One of the things I like about VR “isolated/vpc” now is that its stateless, all things that it must know coms from ACS. There is little to non decision making on it. By introducing dynamic routing it will start making decisions- important decisions that will bypass ACS. Now when you reboot VR you do it without hesitation and doubt that it will not work. With dynamics routing you probably will need to confirm that routing is up and its working as it was working before reboot. Also have some one available that can actually fix dynamic routing problems if they appear. You will need to handle dynamic routing problems as daily operations. I’m not saying no to OSPFv3 and BGP in my opinion manual mode also should be treated as first class citizen. On 2021/07/19 09:08:10, Wido den Hollander <w...@widodh.nl> wrote: > > > Op 17-07-2021 om 06:28 schreef Hean Seng: > > I think if doing this way , since you were to implement on peering ip > > between vr and phsical router , then would need keep /56 or 48 at > > Clodustack ? We can only add /64 subnet to Cloudstack only (instead of > > keep the /56 or 48 there). > > > > We can have a /64 at CloudStack in which all VRs talk with the router of > the ISP. > > That is large enough as a interconnect subnet between the VRs and routers. > > From there you can route /56, /48 or which size you want towards the VR. > > From the interconnect /64 you can also grab IPs which you use for > loadbalancing purposes over different VMs. > > > > I saw other software provider do is adding /64 subnet to their system, > > and after that allocate subnet to the VM (from the previous added list). > > > > May be considering the OSPF if really on this. It really a nightmare for > > maintaining 1000 or few thousand of BGP session. You can imagine your > > Cisco Router list of few thousand BGP session there. > > > > Yes, but I would suggest that both OSPFv3 and BGP should work. Not > everybody will have 1000 accounts on their environment. > > Even static routes should be supported. > > Wido > > > > > > > > > > > On Fri, Jul 16, 2021 at 11:17 PM Wido den Hollander <w...@widodh.nl> wrote: > > > >> > >> > >> Op 16-07-2021 om 16:42 schreef Hean Seng: > >>> Hi Wido, > >>> > >>> In current setup, each Cloudstack have own VR, so in this new IPv6 > >> subnet > >>> allocation , each VR (which have Frr) will need to have peering with ISP > >>> router (and either BGP or Static Route) , and there is 1000 Acocunts, it > >>> will 1000 BGP session with ISP router , Am I right for this ? or I > >>> understand wrong . > >>> > >> > >> Yes, that is correct. A /56 would also be sufficient or a /60 which is > >> enough to allocate a few /64 subnets. > >> > >> 1000 BGP connections isn't really a problem for a proper router at the > >> ISP. OSPF(v3) would be better, but as I said that's poorly supported. > >> > >> The ISP could also install 1000 static routes, but that means that the > >> ISP's router needs to have those configured. > >> > >> http://docs.frrouting.org/en/latest/ospf6d.html > >> (While looking up this URL I see that Frr recently put in a lot of work > >> in OSPFv3, seems better now) > >> > >>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each > >>> devices have own IP. It just how to realize in easy way. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <w...@widodh.nl> > >> wrote: > >>> > >>>> > >>>> > >>>> Op 16-07-2021 om 05:54 schreef Hean Seng: > >>>>> Hi Wido, > >>>>> > >>>>> My initial thought is not like this, it is the /48 at ISP router, and > >>>> /64 > >>>>> subnet assign to AdvanceZoneVR, AdvanceZoneVR responsible is > >>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM, and not > >> routing > >>>>> the traffic, in the VM that get the IPv6 IP will default route to ISP > >>>>> router as gw. It can may be a bridge over via Advancezone-VR. > >>>>> > >>>> > >>>> How would you bridge this? That sounds like NAT? > >>>> > >>>> IPv6 is meant to be routed. Not to be translated or bridged in any way. > >>>> > >>>> The way a made the drawing is exactly how IPv6 should work in a VPC > >>>> environment. > >>>> > >>>> Traffic flows through the VR where it can do firewalling of the traffic. > >>>> > >>>>> However, If do as the way described in the drawing, then i suppose will > >>>> be > >>>>> another kind of virtual router going to introduce , to get hold the /48 > >>>> in > >>>>> this virtual router right ? > >>>>> > >>>> > >>>> It can be the same VR. But keep in mind that IPv6 != IPv4. > >>>> > >>>> The VR will get Frr as a new daemon which can talk BGP with the upper > >>>> network to route traffic. > >>>> > >>>>> After this, The Advance Zone, NAT's VR will peer with this new IPv6 > >> VR > >>>>> for getting the IPv6 /64 prefix ? > >>>>> > >>>> > >>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT. > >>>> > >>>>> If do in this way, then I guess you just only need Static route, with > >>>>> peering ip both end as one /48 can have a lot of /64 on it. And > >>>> hardware > >>>>> budgeting for new IPv6-VR will become very important, as all traffic > >> will > >>>>> need to pass over it . > >>>>> > >>>> > >>>> Routing or NAT is the same for the VR. You don't need a very beefy VR > >>>> for this. > >>>> > >>>>> It will be like > >>>>> > >>>>> ISP Router ------ > (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM > >>>>> > >>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering > >> on > >>>>> OSPF instead of BGP , otherwise few thousand of AdvanceZone-VR wil > >> have > >>>>> few thousand of BGP session. on new-IPv6-VR > >>>>> > >>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct, > >> , > >>>>> otherwise ISP router will be full of /64 prefix route either on BGP( > >> Many > >>>>> BGP Session) , or Many Static route . If few thousand account, ti > >> will > >>>>> be few thousand of BGP session with ISP router or few thousand static > >>>> route > >>>>> which is not possible . > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <w...@widodh.nl> > >>>> wrote: > >>>>> > >>>>>> But you still need routing. See the attached PNG (and draw.io XML). > >>>>>> > >>>>>> You need to route the /48 subnet TO the VR which can then route it to > >>>>>> the Virtual Networks behind the VR. > >>>>>> > >>>>>> There is no other way then routing with either BGP or a Static route. > >>>>>> > >>>>>> Wido > >>>>>> > >>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng: > >>>>>>> Or explain like this : > >>>>>>> > >>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin > >>>>>>> assigned to Cloudstack > >>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to > >>>> Virtual > >>>>>>> Router, one Virtual Router have one subniet /64 > >>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64 > >>>>>>> allocated to VR) to VM > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heans...@gmail.com > >>>>>>> <mailto:heans...@gmail.com>> wrote: > >>>>>>> > >>>>>>> Hi Wido, > >>>>>>> > >>>>>>> I think the /48 is at physical router as gateway , and subnet > >> of > >>>> /64 > >>>>>>> at VR of Cloudstack. Cloudstack only keep which /48 prefix > >> and > >>>>>>> vlan information of this /48 to be later split the /64. to VR. > >>>>>>> > >>>>>>> And the instances is getting singe IPv6 of /64 IP. The VR is > >>>>>>> getting /64. The default gateway shall goes to /48 of physical > >>>>>>> router ip . In this case ,does not need any BGP router . > >>>>>>> > >>>>>>> > >>>>>>> Similar concept as IPv4 : > >>>>>>> > >>>>>>> /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 > >>>> that > >>>>>>> created in Network. > >>>>>>> and /64 of IPv6 is equivalent to single IP of IPv4 assign to > >> VM. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander < > >>>> w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Op 14-07-2021 om 16:44 schreef Hean Seng: > >>>>>>> > Hi > >>>>>>> > > >>>>>>> > I replied in another thread, i think do not need > >> implement > >>>>>>> BGP or OSPF, > >>>>>>> > that would be complicated . > >>>>>>> > > >>>>>>> > We only need assign IPv6 's /64 prefix to Virtual > >> Router > >>>>>>> (VR) in NAT > >>>>>>> > zone, and the VR responsible to deliver single IPv6 to > >> VM > >>>> via > >>>>>>> DHCP6. > >>>>>>> > > >>>>>>> > In VR, you need to have Default IPv6 route to Physical > >>>>>>> Router's /48. IP > >>>>>>> > as IPv6 Gateway. Thens should be done . > >>>>>>> > > >>>>>>> > Example : > >>>>>>> > Physical Router Interface > >>>>>>> > IPv6 IP : 2000:aaaa::1/48 > >>>>>>> > > >>>>>>> > Cloudstack virtual router : 2000:aaaa:200:201::1/64 > >> with > >>>>>>> default ipv6 > >>>>>>> > route to router ip 2000:aaaa::1 > >>>>>>> > and Clodustack Virtual router dhcp allocate IP to VM , > >> and > >>>>>>> VM will have > >>>>>>> > default route to VR. IPv6 2000:aaaa:200:201::1 > >>>>>>> > > >>>>>>> > So in cloudstack need to allow user to enter , IPv6 > >>>>>>> gwateway , and > >>>>>>> > the /48 Ipv6 prefix , then it will self allocate the > >> /64 > >>>> ip > >>>>>>> to the VR , > >>>>>>> > and maintain make sure not ovelap allocation > >>>>>>> > > >>>>>>> > > >>>>>>> > >>>>>>> But NAT is truly not the solution with IPv6. IPv6 is > >> supposed > >>>> to > >>>>>> be > >>>>>>> routable. In addition you should avoid DHCPv6 as much as > >>>>>>> possible as > >>>>>>> that's not really the intended use-case for address > >> allocation > >>>>>>> with IPv6. > >>>>>>> > >>>>>>> In order to route an /48 IPv6 subnet to the VR you have a > >> few > >>>>>>> possibilities: > >>>>>>> > >>>>>>> - Static route from the upperlying routers which are > >> outside > >>>> of > >>>>>>> CloudStack > >>>>>>> - BGP > >>>>>>> - OSPFv3 (broken in most cases!) > >>>>>>> - DHCPv6 Prefix Delegation > >>>>>>> > >>>>>>> BGP and/or Static routes are still the best bet here. > >>>>>>> > >>>>>>> So what you do is that you tell CloudStack that you will > >> route > >>>>>>> 2001:db8::/48 to the VR, the VR can then use that to split > >> it > >>>> up > >>>>>>> into > >>>>>>> multiple /64 subnets going towards the instances: > >>>>>>> > >>>>>>> - 2001:db8::/64 > >>>>>>> - 2001:db8:1::/64 > >>>>>>> - 2001:db8:2::/64 > >>>>>>> ... > >>>>>>> - 2001:db8:f::/64 > >>>>>>> > >>>>>>> And go on. > >>>>>>> > >>>>>>> In case of BGP you indeed have to tell the VR a few things: > >>>>>>> > >>>>>>> - It's own AS number > >>>>>>> - The peer's address(es) > >>>>>>> > >>>>>>> With FRR you can simply say: > >>>>>>> > >>>>>>> neighbor 2001:db8:4fa::179 remote-as external > >>>>>>> > >>>>>>> The /48 you need to have at the VR anyway in case of > >> either a > >>>>>>> static > >>>>>>> route or BGP. > >>>>>>> > >>>>>>> We just need to add a NullRoute on the VR for that /48 so > >> that > >>>>>>> traffic > >>>>>>> will not be routed to the upper gateway in case of the VR > >>>> can't > >>>>>>> find a > >>>>>>> route. > >>>>>>> > >>>>>>> Wido > >>>>>>> > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli > >>>>>>> > <alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com> > >>>>>>> <mailto:alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com>>> wrote: > >>>>>>> > > >>>>>>> > Hi Wido, > >>>>>>> > That's pretty much in line with our thoughts, thanks > >>>> for > >>>>>>> the input. > >>>>>>> > I believe we agree on the following points then: > >>>>>>> > > >>>>>>> > - FRR with BGP (no OSPF) > >>>>>>> > - Route /48 (or/56) down to the VR > >>>>>>> > - /64 per network > >>>>>>> > - SLACC for IP addressing > >>>>>>> > > >>>>>>> > I believe the next big question is then "on which > >> level > >>>>>>> of ACS do we > >>>>>>> > manage AS numbers?". I see two options: > >>>>>>> > 1) Private AS number on a per-zone basis > >>>>>>> > 2) Root Admin assigned AS number on a domain/account > >>>> basis > >>>>>>> > 3) End-user driven AS number on a per network basis > >>>> (for > >>>>>>> bring your > >>>>>>> > own AS and IP scenario) > >>>>>>> > > >>>>>>> > Thoughts? > >>>>>>> > > >>>>>>> > Cheers > >>>>>>> > Alex > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -----Original Message----- > >>>>>>> > From: Wido den Hollander <w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl> <mailto:w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl>>> > >>>>>>> > Sent: 13 July 2021 15:08 > >>>>>>> > To: dev@cloudstack.apache.org > >>>>>>> <mailto:dev@cloudstack.apache.org> > >>>>>>> <mailto:dev@cloudstack.apache.org > >>>>>>> <mailto:dev@cloudstack.apache.org>>; > >>>>>>> > Alex Mattioli <alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com> > >>>>>>> > <mailto:alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com>>> > >>>>>>> > Cc: Wei Zhou <wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com> > >>>>>>> > <mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com>>>; Rohit Yadav > >>>>>>> > <rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>>>> <mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com>>>; > >>>>>>> > Gabriel Beims Bräscher <gabr...@pcextreme.nl > >>>>>>> <mailto:gabr...@pcextreme.nl> > >>>>>>> > <mailto:gabr...@pcextreme.nl <mailto: > >>>> gabr...@pcextreme.nl > >>>>>>>>> > >>>>>>> > Subject: Re: IPV6 in Isolated/VPC networks > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > On 7/7/21 1:16 PM, Alex Mattioli wrote: > >>>>>>> > > Hi all, > >>>>>>> > > @Wei Zhou<mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com> > >>>>>>> > <mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com>>> @Rohit > >>>>>>> > Yadav<mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>>>> > <mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com>>> and myself are > >>>>>>> investigating how > >>>>>>> > to enable IPV6 support on Isolated and VPC networks > >> and > >>>>>>> would like > >>>>>>> > your input on it. > >>>>>>> > > At the moment we are looking at implementing FRR > >>>> with > >>>>>>> BGP (and > >>>>>>> > possibly OSPF) on the ACS VR. > >>>>>>> > > > >>>>>>> > > We are looking for requirements, recommendations, > >>>>>>> ideas, rants, > >>>>>>> > etc...etc... > >>>>>>> > > > >>>>>>> > > >>>>>>> > Ok! Here we go. > >>>>>>> > > >>>>>>> > I think that you mean that the VR will actually > >> route > >>>> the > >>>>>>> IPv6 > >>>>>>> > traffic and for that you need to have a way of > >> getting > >>>> a > >>>>>>> subnet > >>>>>>> > routed to the VR. > >>>>>>> > > >>>>>>> > BGP is probably you best bet here. Although OSPFv3 > >>>>>>> technically > >>>>>>> > supports this it is very badly implemented in Frr > >> for > >>>>>>> example. > >>>>>>> > > >>>>>>> > Now FRR is a very good router and one of the fancy > >>>>>>> features it > >>>>>>> > supports is BGP Unnumered. This allows for auto > >>>>>>> configuration of BGP > >>>>>>> > over a L2 network when both sides are sending Router > >>>>>>> Advertisements. > >>>>>>> > This is very easy for flexible BGP configurations > >> where > >>>>>>> both sides > >>>>>>> > have dynamic IPs. > >>>>>>> > > >>>>>>> > What you want to do is that you get a /56, /48 or > >>>>>>> something which is > >>>>>>> > >/64 bits routed to the VR. > >>>>>>> > > >>>>>>> > Now you can sub-segment this into separate /64 > >> subnets. > >>>>>>> You don't > >>>>>>> > want to go smaller then a /64 is that prevents you > >> from > >>>>>>> using SLAAC > >>>>>>> > for IPv6 address configuration. This is how it works > >>>> for > >>>>>>> Shared > >>>>>>> > Networks now in Basic and Advanced Zones. > >>>>>>> > > >>>>>>> > FRR can now also send out the Router Advertisements > >> on > >>>>>>> the downlinks > >>>>>>> > sending out: > >>>>>>> > > >>>>>>> > - DNS servers > >>>>>>> > - DNS domain > >>>>>>> > - Prefix (/64) to be used > >>>>>>> > > >>>>>>> > There is no need for DHCPv6. You can calculate the > >> IPv6 > >>>>>>> address the > >>>>>>> > VM will obtain by using the MAC and the prefix. > >>>>>>> > > >>>>>>> > So in short: > >>>>>>> > > >>>>>>> > - Using BGP you routed a /48 to the VR > >>>>>>> > - Now you split this into /64 subnets towards the > >>>>>>> isolated networks > >>>>>>> > > >>>>>>> > Wido > >>>>>>> > > >>>>>>> > > Alex Mattioli > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Regards, > >>>>>>> > Hean Seng > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Regards, > >>>>>>> Hean Seng > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Regards, > >>>>>>> Hean Seng > >>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > >
