Re: RE: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi,

One example why loosely implemented BGP on VR could lead to tsunami of
problems.
Why I’m proposing to do IPv6 without using BGP.

Zebra/Quagga/FRR over BGP will advertise all local routing table to peer.
All content of ip route.

Usually there are some BGP filters in place to protect from common human
errors.
No one will prohibit me creating interfaces with
1.1.1.1/1.0.0.1/.8.8.8.8/8.4.4.4/and ipv6 counterparts.

They will go to local routing table and then will be advertised over bgp to
peer.
If there will be no filters/protection in place metric at least for DC
where VR is located
will be better and all DNS/NTP/etc traffic will start flowing into my
VR direction.


Networking departments will come up with even more proficient ways of
exploiting loose BGP implementations.

On Thu, 15 Jul 2021 at 09:39, Rohit Yadav  wrote:

> Hi Kristaps,
>
> I think the public list may not allow you larger email and
> doc/attachments. If there are any documents you want to share, can you say
> put them on Google docs and share the link with dev list? Thanks.
>
>
> Regards.
>
> 
> From: Kristaps Cudars 
> Sent: Wednesday, July 14, 2021 21:09
> To: dev@cloudstack.apache.org 
> Subject: Re: RE: IPV6 in Isolated/VPC networks
>
> Pony Mail is blocking me :(
>
> Hi,
>
> Elaborating on my previous email.
>
> In my opinion SLAAC (StateLess Address Auto Configuration) is not good
> candidate for VR as it was created for situations of connecting million
> devices with less amount of effort and no micromanagement needed.
>
> One example that coms in mind is- I want to reassign IPv6 from one instance
> to another with SLAAC I will need to change mac address on instance. This
> will introduce new workflows in ACS.
>
> Suggestion would be to use same IPv4 services for IPv6 and have feature
> parity between both protocols in dual-stack configuration as much as
> possible. IPv4 will stay with NAT and IPv6 will be routable. Form ACS
> interface and workflow perspective everything will stay same for both
> protocols with excursion that IPv6 will not have source and destination
> NAT.
>
> In Primate it can be handled with protocols selection dropdown, checkboxes,
> additional fields in same isolated/vpc workflow.
>
> Vision from user/admin perspective could look as following:
>
> For Zones > Traffic Types > Public for existing or new add/edit IPv6
>
> When isolated/vpc network is created you are presented with option to
> enable or disable both protocols. (One of them should be enabled)
>
> For IPv6 there must be predefined list of /64 networks that can be
> automatically assigned.
> In VR creation menu IPv6 is prefilled and not changeable by user.
> It could be in Zones > Traffic Types > Internal
>
> In case of isolated network public IP must be assigned in case of vpc its
> already assigned.
>
> At this point ACS should do api call/or something to insert route in L3
> element.
> You have two variables internal IPv6 network and public IPv6 address that
> will form route entry.
>
> In our case we are using Cisco NX-OS and it has NX-API. I stand for choice
> and there are many more grate L3 devices. Inserting those variables in to
> ansible that can play route add/remove against many vendors is better. List
> can be git repository.
> It also could be ACS api where you can get list public and internal ip
> relations and how you form/create routing entries is up to you.
> This can be something else proposed by others.
>
> Or do it manually is also option for beginning/ small deployments/ lab.
> Nothing bad happens if you do not create route instantly. It simply will
> not work until route is created.
>
> Probably it would be smart to have global option that enables/disable IPv6.
>
> I’m accepting questions, something must be explained in more detail, rant…
> that this is not good.
>
>
>

[GitHub] [cloudstack-documentation] shwstppr opened a new pull request #231: network: note on shared networks with IPv6

[GitBox]

shwstppr opened a new pull request #231:
URL: https://github.com/apache/cloudstack-documentation/pull/231


   Shared networks with IPv6 only cannot work. NPE is observed with MAC address 
of VM's NIC when a VM is deployed or added to such networks.
   
   ```
   
TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-DirectNetworkGuru.allocateDirectIp:280-DirectNetworkGuru.allocate:253-NetworkOrchestrator.allocateNic:951
   2021-07-14 05:21:19,724 ERROR [c.c.a.ApiServer] 
(qtp109228794-1826:ctx-0f190552 ctx-dde78569) (logid:60f9ec82) unhandled 
exception executing api command: [Ljava.lang.String;@3f6b9f6b
   java.lang.NullPointerException
at com.cloud.utils.net.NetUtils.EUI64Address(NetUtils.java:1626)
   ```
   
   Also discussed in 
https://github.com/apache/cloudstack/issues/4563#issuecomment-794024295


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-documentation] shwstppr commented on pull request #230: trusty is in official docs and it is deprecated

[GitBox]

shwstppr commented on pull request #230:
URL: 
https://github.com/apache/cloudstack-documentation/pull/230#issuecomment-880531384


   @abdelouahabb conflicts here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-documentation] shwstppr commented on pull request #231: network: note on shared networks with IPv6

[GitBox]

shwstppr commented on pull request #231:
URL: 
https://github.com/apache/cloudstack-documentation/pull/231#issuecomment-880531503


   @blueorangutan docbuild


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-documentation] blueorangutan commented on pull request #231: network: note on shared networks with IPv6

[GitBox]

blueorangutan commented on pull request #231:
URL: 
https://github.com/apache/cloudstack-documentation/pull/231#issuecomment-880531834


   @shwstppr a Jenkins job has been kicked to build the document. I'll keep you 
posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [cloudstack-documentation] blueorangutan commented on pull request #231: network: note on shared networks with IPv6

[GitBox]

blueorangutan commented on pull request #231:
URL: 
https://github.com/apache/cloudstack-documentation/pull/231#issuecomment-880532844


   Doc build preview: http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/231. 
(SL-JID 112)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: IPV6 in Isolated/VPC networks

[Wido den Hollander]

Op 14-07-2021 om 16:44 schreef Hean Seng:

Hi

I replied in another thread, i think do not need implement BGP or OSPF, 
that would be complicated .


We only need assign  IPv6 's /64 prefix to Virtual Router (VR) in NAT 
zone, and the VR responsible to deliver single IPv6 to VM via DHCP6.


In VR, you need to have Default IPv6 route to  Physical Router's /48. IP 
as IPv6 Gateway.  Thens should be done .


Example :
Physical Router Interface
  IPv6 IP : 2000:::1/48

Cloudstack  virtual router : 2000::200:201::1/64 with default ipv6 
route to router ip 2000:::1
and Clodustack Virtual router dhcp allocate IP to VM , and  VM will have 
default route to VR. IPv6 2000::200:201::1


So in cloudstack need to allow  user to enter ,  IPv6 gwateway , and 
the  /48 Ipv6 prefix , then it will self allocate the /64 ip to the VR , 
and maintain make sure not ovelap allocation





But NAT is truly not the solution with IPv6. IPv6 is supposed to be 
routable. In addition you should avoid DHCPv6 as much as possible as 
that's not really the intended use-case for address allocation with IPv6.


In order to route an /48 IPv6 subnet to the VR you have a few possibilities:

- Static route from the upperlying routers which are outside of CloudStack
- BGP
- OSPFv3 (broken in most cases!)
- DHCPv6 Prefix Delegation

BGP and/or Static routes are still the best bet here.

So what you do is that you tell CloudStack that you will route 
2001:db8::/48 to the VR, the VR can then use that to split it up into 
multiple /64 subnets going towards the instances:


- 2001:db8::/64
- 2001:db8:1::/64
- 2001:db8:2::/64
...
- 2001:db8:f::/64

And go on.

In case of BGP you indeed have to tell the VR a few things:

- It's own AS number
- The peer's address(es)

With FRR you can simply say:

neighbor 2001:db8:4fa::179 remote-as external

The /48 you need to have at the VR anyway in case of either a static 
route or BGP.


We just need to add a NullRoute on the VR for that /48 so that traffic 
will not be routed to the upper gateway in case of the VR can't find a 
route.


Wido







On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli 
mailto:alex.matti...@shapeblue.com>> wrote:


Hi Wido,
That's pretty much in line with our thoughts, thanks for the input. 
I believe we agree on the following points then:


- FRR with BGP (no OSPF)
- Route /48 (or/56) down to the VR
- /64 per network
- SLACC for IP addressing

I believe the next big question is then "on which level of ACS do we
manage AS numbers?".  I see two options:
1) Private AS number on a per-zone basis
2) Root Admin assigned AS number on a domain/account basis
3) End-user driven AS number on a per network basis (for bring your
own AS and IP scenario)

Thoughts?

Cheers
Alex




-Original Message-
From: Wido den Hollander mailto:w...@widodh.nl>>
Sent: 13 July 2021 15:08
To: dev@cloudstack.apache.org ;
Alex Mattioli mailto:alex.matti...@shapeblue.com>>
Cc: Wei Zhou mailto:wei.z...@shapeblue.com>>; Rohit Yadav
mailto:rohit.ya...@shapeblue.com>>;
Gabriel Beims Bräscher mailto:gabr...@pcextreme.nl>>
Subject: Re: IPV6 in Isolated/VPC networks



On 7/7/21 1:16 PM, Alex Mattioli wrote:
 > Hi all,
 > @Wei Zhou> @Rohit
Yadav> and myself are investigating how
to enable IPV6 support on Isolated and VPC networks and would like
your input on it.
 > At the moment we are looking at implementing FRR with BGP (and
possibly OSPF) on the ACS VR.
 >
 > We are looking for requirements, recommendations, ideas, rants,
etc...etc...
 >

Ok! Here we go.

I think that you mean that the VR will actually route the IPv6
traffic and for that you need to have a way of getting a subnet
routed to the VR.

BGP is probably you best bet here. Although OSPFv3 technically
supports this it is very badly implemented in Frr for example.

Now FRR is a very good router and one of the fancy features it
supports is BGP Unnumered. This allows for auto configuration of BGP
over a L2 network when both sides are sending Router Advertisements.
This is very easy for flexible BGP configurations where both sides
have dynamic IPs.

What you want to do is that you get a /56, /48 or something which is
 >/64 bits routed to the VR.

Now you can sub-segment this into separate /64 subnets. You don't
want to go smaller then a /64 is that prevents you from using SLAAC
for IPv6 address configuration. This is how it works for Shared
Networks now in Basic and Advanced Zones.

FRR can now also send out the Router Advertisements on the downlinks
sending out:

- DNS servers
- DNS domain
- Prefix (/64) to be used

There is no need for

Re: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi Wido,

Can you explain why “DHCPv6 as much as possible as that's not really the 
intended use-case” it’s not intended use-case?


On 2021/07/15 09:31:26, Wido den Hollander  wrote: 
> 
> 
> Op 14-07-2021 om 16:44 schreef Hean Seng:
> > Hi
> > 
> > I replied in another thread, i think do not need implement BGP or OSPF, 
> > that would be complicated .
> > 
> > We only need assign  IPv6 's /64 prefix to Virtual Router (VR) in NAT 
> > zone, and the VR responsible to deliver single IPv6 to VM via DHCP6.
> > 
> > In VR, you need to have Default IPv6 route to  Physical Router's /48. IP 
> > as IPv6 Gateway.  Thens should be done .
> > 
> > Example :
> > Physical Router Interface
> >   IPv6 IP : 2000:::1/48
> > 
> > Cloudstack  virtual router : 2000::200:201::1/64 with default ipv6 
> > route to router ip 2000:::1
> > and Clodustack Virtual router dhcp allocate IP to VM , and  VM will have 
> > default route to VR. IPv6 2000::200:201::1
> > 
> > So in cloudstack need to allow  user to enter ,  IPv6 gwateway , and 
> > the  /48 Ipv6 prefix , then it will self allocate the /64 ip to the VR , 
> > and maintain make sure not ovelap allocation
> > 
> > 
> 
> But NAT is truly not the solution with IPv6. IPv6 is supposed to be 
> routable. In addition you should avoid DHCPv6 as much as possible as 
> that's not really the intended use-case for address allocation with IPv6.
> 
> In order to route an /48 IPv6 subnet to the VR you have a few possibilities:
> 
> - Static route from the upperlying routers which are outside of CloudStack
> - BGP
> - OSPFv3 (broken in most cases!)
> - DHCPv6 Prefix Delegation
> 
> BGP and/or Static routes are still the best bet here.
> 
> So what you do is that you tell CloudStack that you will route 
> 2001:db8::/48 to the VR, the VR can then use that to split it up into 
> multiple /64 subnets going towards the instances:
> 
> - 2001:db8::/64
> - 2001:db8:1::/64
> - 2001:db8:2::/64
> ...
> - 2001:db8:f::/64
> 
> And go on.
> 
> In case of BGP you indeed have to tell the VR a few things:
> 
> - It's own AS number
> - The peer's address(es)
> 
> With FRR you can simply say:
> 
> neighbor 2001:db8:4fa::179 remote-as external
> 
> The /48 you need to have at the VR anyway in case of either a static 
> route or BGP.
> 
> We just need to add a NullRoute on the VR for that /48 so that traffic 
> will not be routed to the upper gateway in case of the VR can't find a 
> route.
> 
> Wido
> 
> > 
> > 
> > 
> > 
> > 
> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli 
> > mailto:alex.matti...@shapeblue.com>> wrote:
> > 
> > Hi Wido,
> > That's pretty much in line with our thoughts, thanks for the input. 
> > I believe we agree on the following points then:
> > 
> > - FRR with BGP (no OSPF)
> > - Route /48 (or/56) down to the VR
> > - /64 per network
> > - SLACC for IP addressing
> > 
> > I believe the next big question is then "on which level of ACS do we
> > manage AS numbers?".  I see two options:
> > 1) Private AS number on a per-zone basis
> > 2) Root Admin assigned AS number on a domain/account basis
> > 3) End-user driven AS number on a per network basis (for bring your
> > own AS and IP scenario)
> > 
> > Thoughts?
> > 
> > Cheers
> > Alex
> > 
> > 
> > 
> > 
> > -Original Message-
> > From: Wido den Hollander mailto:w...@widodh.nl>>
> > Sent: 13 July 2021 15:08
> > To: dev@cloudstack.apache.org ;
> > Alex Mattioli  > >
> > Cc: Wei Zhou  > >; Rohit Yadav
> > mailto:rohit.ya...@shapeblue.com>>;
> > Gabriel Beims Bräscher  > >
> > Subject: Re: IPV6 in Isolated/VPC networks
> > 
> > 
> > 
> > On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >  > Hi all,
> >  > @Wei Zhou > > @Rohit
> > Yadav > > and myself are investigating how
> > to enable IPV6 support on Isolated and VPC networks and would like
> > your input on it.
> >  > At the moment we are looking at implementing FRR with BGP (and
> > possibly OSPF) on the ACS VR.
> >  >
> >  > We are looking for requirements, recommendations, ideas, rants,
> > etc...etc...
> >  >
> > 
> > Ok! Here we go.
> > 
> > I think that you mean that the VR will actually route the IPv6
> > traffic and for that you need to have a way of getting a subnet
> > routed to the VR.
> > 
> > BGP is probably you best bet here. Although OSPFv3 technically
> > supports this it is very badly implemented in Frr for example.
> > 
> > Now FRR is a very good router and one of the fancy features it
> > supports is BGP Unnumered. This allows for auto configuration of BGP
> > over a L2 network when both sides are

Re: IPV6 in Isolated/VPC networks

[Hean Seng]

Or explain like this :

1) Cloudstack generate list of /64 subnet from /48 that Network admin
assigned to Cloudstack
2) Cloudsack allocated the subnet (that generated from step1) to Virtual
Router, one Virtual Router have one subniet /64
3) Virtual Router allocate single IPv6 (within the range of /64 allocated
to VR)  to VM






On Thu, Jul 15, 2021 at 6:25 PM Hean Seng  wrote:

> Hi Wido,
>
> I think the /48 is at physical router as gateway , and subnet of /64 at VR
> of Cloudstack.   Cloudstack only keep which /48 prefix and vlan information
> of this /48 to be later split the  /64. to VR.
>
> And the instances is getting singe IPv6 of /64  IP.   The VR is getting
> /64.  The default gateway shall goes to /48 of physical router ip .   In
> this case ,does not need any BGP router .
>
>
> Similar concept as IPv4 :
>
> /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
> created in Network.
> and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>
>
>
>
> On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  wrote:
>
>>
>>
>> Op 14-07-2021 om 16:44 schreef Hean Seng:
>> > Hi
>> >
>> > I replied in another thread, i think do not need implement BGP or OSPF,
>> > that would be complicated .
>> >
>> > We only need assign  IPv6 's /64 prefix to Virtual Router (VR) in NAT
>> > zone, and the VR responsible to deliver single IPv6 to VM via DHCP6.
>> >
>> > In VR, you need to have Default IPv6 route to  Physical Router's /48.
>> IP
>> > as IPv6 Gateway.  Thens should be done .
>> >
>> > Example :
>> > Physical Router Interface
>> >   IPv6 IP : 2000:::1/48
>> >
>> > Cloudstack  virtual router : 2000::200:201::1/64 with default ipv6
>> > route to router ip 2000:::1
>> > and Clodustack Virtual router dhcp allocate IP to VM , and  VM will
>> have
>> > default route to VR. IPv6 2000::200:201::1
>> >
>> > So in cloudstack need to allow  user to enter ,  IPv6 gwateway , and
>> > the  /48 Ipv6 prefix , then it will self allocate the /64 ip to the VR
>> ,
>> > and maintain make sure not ovelap allocation
>> >
>> >
>>
>> But NAT is truly not the solution with IPv6. IPv6 is supposed to be
>> routable. In addition you should avoid DHCPv6 as much as possible as
>> that's not really the intended use-case for address allocation with IPv6.
>>
>> In order to route an /48 IPv6 subnet to the VR you have a few
>> possibilities:
>>
>> - Static route from the upperlying routers which are outside of CloudStack
>> - BGP
>> - OSPFv3 (broken in most cases!)
>> - DHCPv6 Prefix Delegation
>>
>> BGP and/or Static routes are still the best bet here.
>>
>> So what you do is that you tell CloudStack that you will route
>> 2001:db8::/48 to the VR, the VR can then use that to split it up into
>> multiple /64 subnets going towards the instances:
>>
>> - 2001:db8::/64
>> - 2001:db8:1::/64
>> - 2001:db8:2::/64
>> ...
>> - 2001:db8:f::/64
>>
>> And go on.
>>
>> In case of BGP you indeed have to tell the VR a few things:
>>
>> - It's own AS number
>> - The peer's address(es)
>>
>> With FRR you can simply say:
>>
>> neighbor 2001:db8:4fa::179 remote-as external
>>
>> The /48 you need to have at the VR anyway in case of either a static
>> route or BGP.
>>
>> We just need to add a NullRoute on the VR for that /48 so that traffic
>> will not be routed to the upper gateway in case of the VR can't find a
>> route.
>>
>> Wido
>>
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>> > mailto:alex.matti...@shapeblue.com>>
>> wrote:
>> >
>> > Hi Wido,
>> > That's pretty much in line with our thoughts, thanks for the input.
>> > I believe we agree on the following points then:
>> >
>> > - FRR with BGP (no OSPF)
>> > - Route /48 (or/56) down to the VR
>> > - /64 per network
>> > - SLACC for IP addressing
>> >
>> > I believe the next big question is then "on which level of ACS do we
>> > manage AS numbers?".  I see two options:
>> > 1) Private AS number on a per-zone basis
>> > 2) Root Admin assigned AS number on a domain/account basis
>> > 3) End-user driven AS number on a per network basis (for bring your
>> > own AS and IP scenario)
>> >
>> > Thoughts?
>> >
>> > Cheers
>> > Alex
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Wido den Hollander mailto:w...@widodh.nl>>
>> > Sent: 13 July 2021 15:08
>> > To: dev@cloudstack.apache.org ;
>> > Alex Mattioli > > >
>> > Cc: Wei Zhou > > >; Rohit Yadav
>> > mailto:rohit.ya...@shapeblue.com>>;
>> > Gabriel Beims Bräscher > > >
>> > Subject: Re: IPV6 in Isolated/VPC networks
>> >
>> >
>> >
>> > On 7/7/21 1:16 PM, Alex Mattioli wrote:
>> >  > Hi all,
>> >  > @Wei Zhou> > > @Rohit
>> > Yadav

Re: IPV6 in Isolated/VPC networks

[Hean Seng]

Hi Wido,

I think the /48 is at physical router as gateway , and subnet of /64 at VR
of Cloudstack.   Cloudstack only keep which /48 prefix and vlan information
of this /48 to be later split the  /64. to VR.

And the instances is getting singe IPv6 of /64  IP.   The VR is getting
/64.  The default gateway shall goes to /48 of physical router ip .   In
this case ,does not need any BGP router .


Similar concept as IPv4 :

/48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that created
in Network.
and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.




On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  wrote:

>
>
> Op 14-07-2021 om 16:44 schreef Hean Seng:
> > Hi
> >
> > I replied in another thread, i think do not need implement BGP or OSPF,
> > that would be complicated .
> >
> > We only need assign  IPv6 's /64 prefix to Virtual Router (VR) in NAT
> > zone, and the VR responsible to deliver single IPv6 to VM via DHCP6.
> >
> > In VR, you need to have Default IPv6 route to  Physical Router's /48. IP
> > as IPv6 Gateway.  Thens should be done .
> >
> > Example :
> > Physical Router Interface
> >   IPv6 IP : 2000:::1/48
> >
> > Cloudstack  virtual router : 2000::200:201::1/64 with default ipv6
> > route to router ip 2000:::1
> > and Clodustack Virtual router dhcp allocate IP to VM , and  VM will have
> > default route to VR. IPv6 2000::200:201::1
> >
> > So in cloudstack need to allow  user to enter ,  IPv6 gwateway , and
> > the  /48 Ipv6 prefix , then it will self allocate the /64 ip to the VR ,
> > and maintain make sure not ovelap allocation
> >
> >
>
> But NAT is truly not the solution with IPv6. IPv6 is supposed to be
> routable. In addition you should avoid DHCPv6 as much as possible as
> that's not really the intended use-case for address allocation with IPv6.
>
> In order to route an /48 IPv6 subnet to the VR you have a few
> possibilities:
>
> - Static route from the upperlying routers which are outside of CloudStack
> - BGP
> - OSPFv3 (broken in most cases!)
> - DHCPv6 Prefix Delegation
>
> BGP and/or Static routes are still the best bet here.
>
> So what you do is that you tell CloudStack that you will route
> 2001:db8::/48 to the VR, the VR can then use that to split it up into
> multiple /64 subnets going towards the instances:
>
> - 2001:db8::/64
> - 2001:db8:1::/64
> - 2001:db8:2::/64
> ...
> - 2001:db8:f::/64
>
> And go on.
>
> In case of BGP you indeed have to tell the VR a few things:
>
> - It's own AS number
> - The peer's address(es)
>
> With FRR you can simply say:
>
> neighbor 2001:db8:4fa::179 remote-as external
>
> The /48 you need to have at the VR anyway in case of either a static
> route or BGP.
>
> We just need to add a NullRoute on the VR for that /48 so that traffic
> will not be routed to the upper gateway in case of the VR can't find a
> route.
>
> Wido
>
> >
> >
> >
> >
> >
> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> > mailto:alex.matti...@shapeblue.com>>
> wrote:
> >
> > Hi Wido,
> > That's pretty much in line with our thoughts, thanks for the input.
> > I believe we agree on the following points then:
> >
> > - FRR with BGP (no OSPF)
> > - Route /48 (or/56) down to the VR
> > - /64 per network
> > - SLACC for IP addressing
> >
> > I believe the next big question is then "on which level of ACS do we
> > manage AS numbers?".  I see two options:
> > 1) Private AS number on a per-zone basis
> > 2) Root Admin assigned AS number on a domain/account basis
> > 3) End-user driven AS number on a per network basis (for bring your
> > own AS and IP scenario)
> >
> > Thoughts?
> >
> > Cheers
> > Alex
> >
> >
> >
> >
> > -Original Message-
> > From: Wido den Hollander mailto:w...@widodh.nl>>
> > Sent: 13 July 2021 15:08
> > To: dev@cloudstack.apache.org ;
> > Alex Mattioli  > >
> > Cc: Wei Zhou  > >; Rohit Yadav
> > mailto:rohit.ya...@shapeblue.com>>;
> > Gabriel Beims Bräscher  > >
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> >
> >
> > On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >  > Hi all,
> >  > @Wei Zhou > > @Rohit
> > Yadav > > and myself are investigating how
> > to enable IPV6 support on Isolated and VPC networks and would like
> > your input on it.
> >  > At the moment we are looking at implementing FRR with BGP (and
> > possibly OSPF) on the ACS VR.
> >  >
> >  > We are looking for requirements, recommendations, ideas, rants,
> > etc...etc...
> >  >
> >
> > Ok! Here we go.
> >
> > I think that you mean that the VR will actually route the IPv6
> > traffic and for that you need to 

Re: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi Hean,

You still need to create route on L3 SW that will point /64 VM


On 2021/07/15 10:39:13, Hean Seng  wrote: 
> Or explain like this :
> 
> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
> assigned to Cloudstack
> 2) Cloudsack allocated the subnet (that generated from step1) to Virtual
> Router, one Virtual Router have one subniet /64
> 3) Virtual Router allocate single IPv6 (within the range of /64 allocated
> to VR)  to VM
> 
> 
> 
> 
> 
> 
> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng  wrote:
> 
> > Hi Wido,
> >
> > I think the /48 is at physical router as gateway , and subnet of /64 at VR
> > of Cloudstack.   Cloudstack only keep which /48 prefix and vlan information
> > of this /48 to be later split the  /64. to VR.
> >
> > And the instances is getting singe IPv6 of /64  IP.   The VR is getting
> > /64.  The default gateway shall goes to /48 of physical router ip .   In
> > this case ,does not need any BGP router .
> >
> >
> > Similar concept as IPv4 :
> >
> > /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
> > created in Network.
> > and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
> >
> >
> >
> >
> > On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  wrote:
> >
> >>
> >>
> >> Op 14-07-2021 om 16:44 schreef Hean Seng:
> >> > Hi
> >> >
> >> > I replied in another thread, i think do not need implement BGP or OSPF,
> >> > that would be complicated .
> >> >
> >> > We only need assign  IPv6 's /64 prefix to Virtual Router (VR) in NAT
> >> > zone, and the VR responsible to deliver single IPv6 to VM via DHCP6.
> >> >
> >> > In VR, you need to have Default IPv6 route to  Physical Router's /48.
> >> IP
> >> > as IPv6 Gateway.  Thens should be done .
> >> >
> >> > Example :
> >> > Physical Router Interface
> >> >   IPv6 IP : 2000:::1/48
> >> >
> >> > Cloudstack  virtual router : 2000::200:201::1/64 with default ipv6
> >> > route to router ip 2000:::1
> >> > and Clodustack Virtual router dhcp allocate IP to VM , and  VM will
> >> have
> >> > default route to VR. IPv6 2000::200:201::1
> >> >
> >> > So in cloudstack need to allow  user to enter ,  IPv6 gwateway , and
> >> > the  /48 Ipv6 prefix , then it will self allocate the /64 ip to the VR
> >> ,
> >> > and maintain make sure not ovelap allocation
> >> >
> >> >
> >>
> >> But NAT is truly not the solution with IPv6. IPv6 is supposed to be
> >> routable. In addition you should avoid DHCPv6 as much as possible as
> >> that's not really the intended use-case for address allocation with IPv6.
> >>
> >> In order to route an /48 IPv6 subnet to the VR you have a few
> >> possibilities:
> >>
> >> - Static route from the upperlying routers which are outside of CloudStack
> >> - BGP
> >> - OSPFv3 (broken in most cases!)
> >> - DHCPv6 Prefix Delegation
> >>
> >> BGP and/or Static routes are still the best bet here.
> >>
> >> So what you do is that you tell CloudStack that you will route
> >> 2001:db8::/48 to the VR, the VR can then use that to split it up into
> >> multiple /64 subnets going towards the instances:
> >>
> >> - 2001:db8::/64
> >> - 2001:db8:1::/64
> >> - 2001:db8:2::/64
> >> ...
> >> - 2001:db8:f::/64
> >>
> >> And go on.
> >>
> >> In case of BGP you indeed have to tell the VR a few things:
> >>
> >> - It's own AS number
> >> - The peer's address(es)
> >>
> >> With FRR you can simply say:
> >>
> >> neighbor 2001:db8:4fa::179 remote-as external
> >>
> >> The /48 you need to have at the VR anyway in case of either a static
> >> route or BGP.
> >>
> >> We just need to add a NullRoute on the VR for that /48 so that traffic
> >> will not be routed to the upper gateway in case of the VR can't find a
> >> route.
> >>
> >> Wido
> >>
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >> > mailto:alex.matti...@shapeblue.com>>
> >> wrote:
> >> >
> >> > Hi Wido,
> >> > That's pretty much in line with our thoughts, thanks for the input.
> >> > I believe we agree on the following points then:
> >> >
> >> > - FRR with BGP (no OSPF)
> >> > - Route /48 (or/56) down to the VR
> >> > - /64 per network
> >> > - SLACC for IP addressing
> >> >
> >> > I believe the next big question is then "on which level of ACS do we
> >> > manage AS numbers?".  I see two options:
> >> > 1) Private AS number on a per-zone basis
> >> > 2) Root Admin assigned AS number on a domain/account basis
> >> > 3) End-user driven AS number on a per network basis (for bring your
> >> > own AS and IP scenario)
> >> >
> >> > Thoughts?
> >> >
> >> > Cheers
> >> > Alex
> >> >
> >> >
> >> >
> >> >
> >> > -Original Message-
> >> > From: Wido den Hollander mailto:w...@widodh.nl>>
> >> > Sent: 13 July 2021 15:08
> >> > To: dev@cloudstack.apache.org ;
> >> > Alex Mattioli  >> > >
> >> > Cc: Wei Zhou  >> > 

Security groups support in advanced zone

[Wei ZHOU]

Hi all,

We are investigating security groups support in Advanced zones. it would be
nice to get your feedback.

Some background knowledge:
(1) there are 3 types of zones in cloudstack: Basic, Advanced, Advanced
with Security groups (support KVM and Xenserver. vmware is not supported)
(2) Admins can create shared networks in all 3 zones, but they can only
create isolated networks in the Advanced zones.
(3) For vms on shared networks in Basic zone or Advanced zone with Security
groups. users can create security groups and associate it to the VMs as
firewalls. The security group rules are applied on hypervisors
(kvm/xenserver)
(4) For vms on shared networks in an Advanced zone , there are no security
group rules. all ports are open, users have to configure firewall inside
vms.

I am wondering if it is necessary to add security groups for shared
networks in Advanced zones. Here are some advantages
(1) vms on shared networks will have firewall rules (applied on hypervisor).
(2) we do not need to manage multiple zone types.
Basic zones = advanced zone with only 1 shared network
Advanced zone with security groups = advanced zone with multiple shared
networks.

What's your opinion ? Will it be helpful for you ?

Kind regards,
Wei

Re: [PROPOSE] RM for CloudStack Kubernetes Provider v1.0

[Simon Weller]

+1



From: David Jumani 
Sent: Thursday, July 15, 2021 1:31 AM
To: users ; dev@cloudstack.apache.org 

Subject: [PROPOSE] RM for CloudStack Kubernetes Provider v1.0

Hi,

I'd like to put myself forward as the release manager for CloudStack Kubernetes 
Provider v1.0.

This will be the first release of CloudStack Kubernetes Provider which 
facilitates Kubernetes deployments on Cloudstack.
It allows Kubernetes to dynamically allocate IP addresses and the respective 
networking rules on CloudStack to ensure seamless TCP, UDP and TCP-Proxy 
LoadBalancer deployments on Kubernetes.

It was initially the Cloudstack provider in Kubernetes which was later 
extracted to allow for pluggable providers.
A lot of work and effort has gone into developing it, and we are looking 
forward to its grand debut.

The list of open issues triaged for the v1.0 milestone can be found at 
https://github.com/apache/cloudstack-kubernetes-provider/milestone/1
If you encounter any issues, please do report them at 
https://github.com/apache/cloudstack-kubernetes-provider/issues

Looking forward to your support

Thanks,
David

Re: IPV6 in Isolated/VPC networks

[Wido den Hollander]

But you still need routing. See the attached PNG (and draw.io XML).

You need to route the /48 subnet TO the VR which can then route it to 
the Virtual Networks behind the VR.


There is no other way then routing with either BGP or a Static route.

Wido

Op 15-07-2021 om 12:39 schreef Hean Seng:

Or explain like this :

1) Cloudstack generate list of /64 subnet from /48 that Network admin 
assigned to Cloudstack
2) Cloudsack allocated the subnet (that generated from step1) to Virtual 
Router, one Virtual Router have one subniet /64
3) Virtual Router allocate single IPv6 (within the range of /64 
allocated to VR)  to VM







On Thu, Jul 15, 2021 at 6:25 PM Hean Seng > wrote:


Hi Wido,

I think the /48 is at physical router as gateway , and subnet of /64
at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
vlan information of this /48 to be later split the  /64. to VR.

And the instances is getting singe IPv6 of /64  IP.   The VR is
getting /64.  The default gateway shall goes to /48 of physical
router ip .   In this case ,does not need any BGP router .


Similar concept as IPv4 :

/48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
created in Network.
and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.




On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander mailto:w...@widodh.nl>> wrote:



Op 14-07-2021 om 16:44 schreef Hean Seng:
 > Hi
 >
 > I replied in another thread, i think do not need implement
BGP or OSPF,
 > that would be complicated .
 >
 > We only need assign  IPv6 's /64 prefix to Virtual Router
(VR) in NAT
 > zone, and the VR responsible to deliver single IPv6 to VM via
DHCP6.
 >
 > In VR, you need to have Default IPv6 route to  Physical
Router's /48. IP
 > as IPv6 Gateway.  Thens should be done .
 >
 > Example :
 > Physical Router Interface
 >   IPv6 IP : 2000:::1/48
 >
 > Cloudstack  virtual router : 2000::200:201::1/64 with
default ipv6
 > route to router ip 2000:::1
 > and Clodustack Virtual router dhcp allocate IP to VM , and 
VM will have

 > default route to VR. IPv6 2000::200:201::1
 >
 > So in cloudstack need to allow  user to enter ,  IPv6
gwateway , and
 > the  /48 Ipv6 prefix , then it will self allocate the /64 ip
to the VR ,
 > and maintain make sure not ovelap allocation
 >
 >

But NAT is truly not the solution with IPv6. IPv6 is supposed to be
routable. In addition you should avoid DHCPv6 as much as
possible as
that's not really the intended use-case for address allocation
with IPv6.

In order to route an /48 IPv6 subnet to the VR you have a few
possibilities:

- Static route from the upperlying routers which are outside of
CloudStack
- BGP
- OSPFv3 (broken in most cases!)
- DHCPv6 Prefix Delegation

BGP and/or Static routes are still the best bet here.

So what you do is that you tell CloudStack that you will route
2001:db8::/48 to the VR, the VR can then use that to split it up
into
multiple /64 subnets going towards the instances:

- 2001:db8::/64
- 2001:db8:1::/64
- 2001:db8:2::/64
...
- 2001:db8:f::/64

And go on.

In case of BGP you indeed have to tell the VR a few things:

- It's own AS number
- The peer's address(es)

With FRR you can simply say:

neighbor 2001:db8:4fa::179 remote-as external

The /48 you need to have at the VR anyway in case of either a
static
route or BGP.

We just need to add a NullRoute on the VR for that /48 so that
traffic
will not be routed to the upper gateway in case of the VR can't
find a
route.

Wido

 >
 >
 >
 >
 >
 > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
 > mailto:alex.matti...@shapeblue.com>
>> wrote:
 >
 >     Hi Wido,
 >     That's pretty much in line with our thoughts, thanks for
the input.
 >     I believe we agree on the following points then:
 >
 >     - FRR with BGP (no OSPF)
 >     - Route /48 (or/56) down to the VR
 >     - /64 per network
 >     - SLACC for IP addressing
 >
 >     I believe the next big question is then "on which level
of ACS do we
 >     manage AS numbers?".  I see two options:
 >     1) Private AS number on a per-zone basis
 >     2) Root Admin as

Re: IPV6 in Isolated/VPC networks

[Wido den Hollander]

Op 14-07-2021 om 14:59 schreef Alex Mattioli:

Hi Kristaps,
Thanks for the nice schematic, pretty much where we were going.

I just didn't understand your first statement " I would like to argue that 
implementer dynamic routing protocol and associated security problems/challenges with it 
to have IPv6 route inserted in L3 router/s is not a good goal."

Would you mind clarifying/expanding on it please?


I would like to know that as well. Because protocols like BGP and OSPF 
are intended for that use-case.


I don't see ACS logging into our Juniper MX routers to program a static 
route.


BGP doesn't have to be used for something like Anycast or multiple 
datacenter availability.


The reason I said DHCPv6 should be avoided is because of the limited 
support. You also need to keep a database of IP addresses while SLAAC 
exactly does what you want.


Router Advertisements with SLAAC is much better supported in Operating 
Systems then DHCPv6 is.


Wido



Thanks
Alex

  



-Original Message-
From: Kristaps Cudars 
Sent: 13 July 2021 20:44
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi,

I would like to argue that implementer dynamic routing protocol and associated 
security problems/challenges with it to have IPv6 route inserted in L3 router/s 
is not a good goal.

In my opinion dynamic routing on VR would be interesting to scale availability 
of service across several datacenter if they participate in same AS. With BGP 
you could advertise same IP form different VR located in different DC IPv6/128 
or/and IPv4/32.

I would delegate task of router creation to ACS somewhere at moment of VR 
creation.
It could happen over ssh/snmp/api rest or ansible- something that supports wide 
variety of vendors/devices.

Have created rough schematic on how it could look on VR side: 
https://dice.lv/acs/ACS_router_v2.pdf


On 2021/07/13 13:08:20, Wido den Hollander  wrote:



On 7/7/21 1:16 PM, Alex Mattioli wrote:

Hi all,
@Wei Zhou @Rohit 
Yadav and myself are investigating how to enable 
IPV6 support on Isolated and VPC networks and would like your input on it.
At the moment we are looking at implementing FRR with BGP (and possibly OSPF) 
on the ACS VR.

We are looking for requirements, recommendations, ideas, rants, etc...etc...



Ok! Here we go.

I think that you mean that the VR will actually route the IPv6 traffic
and for that you need to have a way of getting a subnet routed to the VR.

BGP is probably you best bet here. Although OSPFv3 technically
supports this it is very badly implemented in Frr for example.

Now FRR is a very good router and one of the fancy features it
supports is BGP Unnumered. This allows for auto configuration of BGP
over a L2 network when both sides are sending Router Advertisements.
This is very easy for flexible BGP configurations where both sides have dynamic 
IPs.

What you want to do is that you get a /56, /48 or something which is

/64 bits routed to the VR.


Now you can sub-segment this into separate /64 subnets. You don't want
to go smaller then a /64 is that prevents you from using SLAAC for
IPv6 address configuration. This is how it works for Shared Networks
now in Basic and Advanced Zones.

FRR can now also send out the Router Advertisements on the downlinks
sending out:

- DNS servers
- DNS domain
- Prefix (/64) to be used

There is no need for DHCPv6. You can calculate the IPv6 address the VM
will obtain by using the MAC and the prefix.

So in short:

- Using BGP you routed a /48 to the VR
- Now you split this into /64 subnets towards the isolated networks

Wido


Alex Mattioli

  

Re: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi Wido,

What is benefit of using Route Advertisement on internal VR networks? 

In drawing VR is in VPC mode how it will work for isolated network where 
external link/ip is not assigned initially?


On 2021/07/15 14:47:24, Wido den Hollander  wrote: 
> But you still need routing. See the attached PNG (and draw.io XML).
> 
> You need to route the /48 subnet TO the VR which can then route it to 
> the Virtual Networks behind the VR.
> 
> There is no other way then routing with either BGP or a Static route.
> 
> Wido
> 
> Op 15-07-2021 om 12:39 schreef Hean Seng:
> > Or explain like this :
> > 
> > 1) Cloudstack generate list of /64 subnet from /48 that Network admin 
> > assigned to Cloudstack
> > 2) Cloudsack allocated the subnet (that generated from step1) to Virtual 
> > Router, one Virtual Router have one subniet /64
> > 3) Virtual Router allocate single IPv6 (within the range of /64 
> > allocated to VR)  to VM
> > 
> > 
> > 
> > 
> > 
> > 
> > On Thu, Jul 15, 2021 at 6:25 PM Hean Seng  > > wrote:
> > 
> > Hi Wido,
> > 
> > I think the /48 is at physical router as gateway , and subnet of /64
> > at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
> > vlan information of this /48 to be later split the  /64. to VR.
> > 
> > And the instances is getting singe IPv6 of /64  IP.   The VR is
> > getting /64.  The default gateway shall goes to /48 of physical
> > router ip .   In this case ,does not need any BGP router .
> > 
> > 
> > Similar concept as IPv4 :
> > 
> > /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
> > created in Network.
> > and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
> > 
> > 
> > 
> > 
> > On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  > > wrote:
> > 
> > 
> > 
> > Op 14-07-2021 om 16:44 schreef Hean Seng:
> >  > Hi
> >  >
> >  > I replied in another thread, i think do not need implement
> > BGP or OSPF,
> >  > that would be complicated .
> >  >
> >  > We only need assign  IPv6 's /64 prefix to Virtual Router
> > (VR) in NAT
> >  > zone, and the VR responsible to deliver single IPv6 to VM via
> > DHCP6.
> >  >
> >  > In VR, you need to have Default IPv6 route to  Physical
> > Router's /48. IP
> >  > as IPv6 Gateway.  Thens should be done .
> >  >
> >  > Example :
> >  > Physical Router Interface
> >  >   IPv6 IP : 2000:::1/48
> >  >
> >  > Cloudstack  virtual router : 2000::200:201::1/64 with
> > default ipv6
> >  > route to router ip 2000:::1
> >  > and Clodustack Virtual router dhcp allocate IP to VM , and 
> > VM will have
> >  > default route to VR. IPv6 2000::200:201::1
> >  >
> >  > So in cloudstack need to allow  user to enter ,  IPv6
> > gwateway , and
> >  > the  /48 Ipv6 prefix , then it will self allocate the /64 ip
> > to the VR ,
> >  > and maintain make sure not ovelap allocation
> >  >
> >  >
> > 
> > But NAT is truly not the solution with IPv6. IPv6 is supposed to be
> > routable. In addition you should avoid DHCPv6 as much as
> > possible as
> > that's not really the intended use-case for address allocation
> > with IPv6.
> > 
> > In order to route an /48 IPv6 subnet to the VR you have a few
> > possibilities:
> > 
> > - Static route from the upperlying routers which are outside of
> > CloudStack
> > - BGP
> > - OSPFv3 (broken in most cases!)
> > - DHCPv6 Prefix Delegation
> > 
> > BGP and/or Static routes are still the best bet here.
> > 
> > So what you do is that you tell CloudStack that you will route
> > 2001:db8::/48 to the VR, the VR can then use that to split it up
> > into
> > multiple /64 subnets going towards the instances:
> > 
> > - 2001:db8::/64
> > - 2001:db8:1::/64
> > - 2001:db8:2::/64
> > ...
> > - 2001:db8:f::/64
> > 
> > And go on.
> > 
> > In case of BGP you indeed have to tell the VR a few things:
> > 
> > - It's own AS number
> > - The peer's address(es)
> > 
> > With FRR you can simply say:
> > 
> > neighbor 2001:db8:4fa::179 remote-as external
> > 
> > The /48 you need to have at the VR anyway in case of either a
> > static
> > route or BGP.
> > 
> > We just need to add a NullRoute on the VR for that /48 so that
> > traffic
> > will not be routed to the upper gateway in case of the VR can't
> > find a
> > route.
> > 
> > Wido
> > 
> >  >
> >  >
> >  >
> >  >
> >  >
> >   

Re: IPV6 in Isolated/VPC networks

[Wido den Hollander]

Op 15-07-2021 om 17:05 schreef Kristaps Cudars:

Hi Wido,

What is benefit of using Route Advertisement on internal VR networks?



The VMs need the Router Advertisement to learn their default gateway. 
That's the only way with IPv6.


The RA also contains the prefix (/64) which the VMs can use to calculate 
their IPv6 address (SLAAC).



In drawing VR is in VPC mode how it will work for isolated network where 
external link/ip is not assigned initially?


On 2021/07/15 14:47:24, Wido den Hollander  wrote:

But you still need routing. See the attached PNG (and draw.io XML).

You need to route the /48 subnet TO the VR which can then route it to
the Virtual Networks behind the VR.

There is no other way then routing with either BGP or a Static route.

Wido

Op 15-07-2021 om 12:39 schreef Hean Seng:

Or explain like this :

1) Cloudstack generate list of /64 subnet from /48 that Network admin
assigned to Cloudstack
2) Cloudsack allocated the subnet (that generated from step1) to Virtual
Router, one Virtual Router have one subniet /64
3) Virtual Router allocate single IPv6 (within the range of /64
allocated to VR)  to VM






On Thu, Jul 15, 2021 at 6:25 PM Hean Seng mailto:heans...@gmail.com>> wrote:

 Hi Wido,

 I think the /48 is at physical router as gateway , and subnet of /64
 at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
 vlan information of this /48 to be later split the  /64. to VR.

 And the instances is getting singe IPv6 of /64  IP.   The VR is
 getting /64.  The default gateway shall goes to /48 of physical
 router ip .   In this case ,does not need any BGP router .


 Similar concept as IPv4 :

 /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
 created in Network.
 and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.




 On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander mailto:w...@widodh.nl>> wrote:



 Op 14-07-2021 om 16:44 schreef Hean Seng:
  > Hi
  >
  > I replied in another thread, i think do not need implement
 BGP or OSPF,
  > that would be complicated .
  >
  > We only need assign  IPv6 's /64 prefix to Virtual Router
 (VR) in NAT
  > zone, and the VR responsible to deliver single IPv6 to VM via
 DHCP6.
  >
  > In VR, you need to have Default IPv6 route to  Physical
 Router's /48. IP
  > as IPv6 Gateway.  Thens should be done .
  >
  > Example :
  > Physical Router Interface
  >   IPv6 IP : 2000:::1/48
  >
  > Cloudstack  virtual router : 2000::200:201::1/64 with
 default ipv6
  > route to router ip 2000:::1
  > and Clodustack Virtual router dhcp allocate IP to VM , and
 VM will have
  > default route to VR. IPv6 2000::200:201::1
  >
  > So in cloudstack need to allow  user to enter ,  IPv6
 gwateway , and
  > the  /48 Ipv6 prefix , then it will self allocate the /64 ip
 to the VR ,
  > and maintain make sure not ovelap allocation
  >
  >

 But NAT is truly not the solution with IPv6. IPv6 is supposed to be
 routable. In addition you should avoid DHCPv6 as much as
 possible as
 that's not really the intended use-case for address allocation
 with IPv6.

 In order to route an /48 IPv6 subnet to the VR you have a few
 possibilities:

 - Static route from the upperlying routers which are outside of
 CloudStack
 - BGP
 - OSPFv3 (broken in most cases!)
 - DHCPv6 Prefix Delegation

 BGP and/or Static routes are still the best bet here.

 So what you do is that you tell CloudStack that you will route
 2001:db8::/48 to the VR, the VR can then use that to split it up
 into
 multiple /64 subnets going towards the instances:

 - 2001:db8::/64
 - 2001:db8:1::/64
 - 2001:db8:2::/64
 ...
 - 2001:db8:f::/64

 And go on.

 In case of BGP you indeed have to tell the VR a few things:

 - It's own AS number
 - The peer's address(es)

 With FRR you can simply say:

 neighbor 2001:db8:4fa::179 remote-as external

 The /48 you need to have at the VR anyway in case of either a
 static
 route or BGP.

 We just need to add a NullRoute on the VR for that /48 so that
 traffic
 will not be routed to the upper gateway in case of the VR can't
 find a
 route.

 Wido

  >
  >
  >
  >
  >
  > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
  > mailto:alex.matti...@shapeblue.com>
 >> wrote:
  >
 

Re: [PROPOSE] RM for CloudStack Kubernetes Provider v1.0

[Suresh Anaparti]

+1

Good luck David!

Regards,
Suresh

On 15/07/21, 12:02 PM, "David Jumani"  wrote:

Hi,

I'd like to put myself forward as the release manager for CloudStack 
Kubernetes Provider 
v1.0.

This will be the first release of CloudStack Kubernetes Provider which 
facilitates Kubernetes deployments on Cloudstack.
It allows Kubernetes to dynamically allocate IP addresses and the 
respective networking rules on CloudStack to ensure seamless TCP, UDP and 
TCP-Proxy LoadBalancer deployments on Kubernetes.

It was initially the Cloudstack provider in Kubernetes which was later 
extracted to allow for pluggable providers.
A lot of work and effort has gone into developing it, and we are looking 
forward to its grand debut.

The list of open issues triaged for the v1.0 milestone can be found at 
https://github.com/apache/cloudstack-kubernetes-provider/milestone/1
If you encounter any issues, please do report them at 
https://github.com/apache/cloudstack-kubernetes-provider/issues

Looking forward to your support

Thanks,
David





 

Re: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi Wido,

ACS must know what ip`s it should assignee for external and internal networks 
on VR isolated and vpc.

ACS will know external IP of VR and it will also know subnet or subnets that 
has been assigned. From this you can form route.

This information can be exposed by ACS api. IT can be used by ansible/shell 
script/etc.. to form routing table and submit it to any L3 device and also 
Juniper MX. Junos has exceptional API.

Anycast and multicast is grate when you control all network and your app is 
aware of it to some extent or you have other means to control it.

The only place I know DHCPv6 are not working is Android as there is gentlemen 
Lorenzo Colitti who has decided that back work compatibility is not a thing. 
https://issuetracker.google.com/issues/36949085?pli=1

What are other examples?

Every OS that wants to run in enterprise environment must support DHCPv6 as 
enterprise still needs:

Ability to assign suffix such as git.com
Register hosts in DNS
Keep track of what host had what IP at a certain time
Image deployment via PXE (think DHCP options)
Other DHCP options used.
Ability to easily swap DNS server in entire network.
Dot1X deployment where you want RADIUS server to see DHCP request

There are workarounds to some of them.


On 2021/07/15 14:51:08, Wido den Hollander  wrote: 
> 
> 
> Op 14-07-2021 om 14:59 schreef Alex Mattioli:
> > Hi Kristaps,
> > Thanks for the nice schematic, pretty much where we were going.
> > 
> > I just didn't understand your first statement " I would like to argue that 
> > implementer dynamic routing protocol and associated security 
> > problems/challenges with it to have IPv6 route inserted in L3 router/s is 
> > not a good goal."
> > 
> > Would you mind clarifying/expanding on it please?
> 
> I would like to know that as well. Because protocols like BGP and OSPF 
> are intended for that use-case.
> 
> I don't see ACS logging into our Juniper MX routers to program a static 
> route.
> 
> BGP doesn't have to be used for something like Anycast or multiple 
> datacenter availability.
> 
> The reason I said DHCPv6 should be avoided is because of the limited 
> support. You also need to keep a database of IP addresses while SLAAC 
> exactly does what you want.
> 
> Router Advertisements with SLAAC is much better supported in Operating 
> Systems then DHCPv6 is.
> 
> Wido
> 
> > 
> > Thanks
> > Alex
> > 
> >   
> > 
> > 
> > -Original Message-
> > From: Kristaps Cudars 
> > Sent: 13 July 2021 20:44
> > To: dev@cloudstack.apache.org
> > Subject: Re: IPV6 in Isolated/VPC networks
> > 
> > Hi,
> > 
> > I would like to argue that implementer dynamic routing protocol and 
> > associated security problems/challenges with it to have IPv6 route inserted 
> > in L3 router/s is not a good goal.
> > 
> > In my opinion dynamic routing on VR would be interesting to scale 
> > availability of service across several datacenter if they participate in 
> > same AS. With BGP you could advertise same IP form different VR located in 
> > different DC IPv6/128 or/and IPv4/32.
> > 
> > I would delegate task of router creation to ACS somewhere at moment of VR 
> > creation.
> > It could happen over ssh/snmp/api rest or ansible- something that supports 
> > wide variety of vendors/devices.
> > 
> > Have created rough schematic on how it could look on VR side: 
> > https://dice.lv/acs/ACS_router_v2.pdf
> > 
> > 
> > On 2021/07/13 13:08:20, Wido den Hollander  wrote:
> >>
> >>
> >> On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>> Hi all,
> >>> @Wei Zhou @Rohit 
> >>> Yadav and myself are investigating how 
> >>> to enable IPV6 support on Isolated and VPC networks and would like your 
> >>> input on it.
> >>> At the moment we are looking at implementing FRR with BGP (and possibly 
> >>> OSPF) on the ACS VR.
> >>>
> >>> We are looking for requirements, recommendations, ideas, rants, 
> >>> etc...etc...
> >>>
> >>
> >> Ok! Here we go.
> >>
> >> I think that you mean that the VR will actually route the IPv6 traffic
> >> and for that you need to have a way of getting a subnet routed to the VR.
> >>
> >> BGP is probably you best bet here. Although OSPFv3 technically
> >> supports this it is very badly implemented in Frr for example.
> >>
> >> Now FRR is a very good router and one of the fancy features it
> >> supports is BGP Unnumered. This allows for auto configuration of BGP
> >> over a L2 network when both sides are sending Router Advertisements.
> >> This is very easy for flexible BGP configurations where both sides have 
> >> dynamic IPs.
> >>
> >> What you want to do is that you get a /56, /48 or something which is
> >>> /64 bits routed to the VR.
> >>
> >> Now you can sub-segment this into separate /64 subnets. You don't want
> >> to go smaller then a /64 is that prevents you from using SLAAC for
> >> IPv6 address configuration. This is how it works for Shared Networks
> >> now in Basic and Advanced Zones.
> >>
> >> 

Question about SolidFire plugin with KVM hypervisor

[Slavka Peleva]

Hi all,

Is there someone of you who uses the CS 4.15.1.0 or latest with SolidFire
primary storage over a KVM hypervisor? If there is someone, is it possible
to share the volumes' format in the DB?

Probably more questions will appear after this :)

Best regards,
Slavka

Re: IPV6 in Isolated/VPC networks

[Kristaps Cudars]

Hi Wido,

DHCPv6 is not an option?
It enables feature parity between IPv4 and IPv6 in context of VR.

Or there are some advantages in RA and SLAAC?



On 2021/07/15 15:10:38, Wido den Hollander  wrote: 
> 
> 
> Op 15-07-2021 om 17:05 schreef Kristaps Cudars:
> > Hi Wido,
> > 
> > What is benefit of using Route Advertisement on internal VR networks?
> > 
> 
> The VMs need the Router Advertisement to learn their default gateway. 
> That's the only way with IPv6.
> 
> The RA also contains the prefix (/64) which the VMs can use to calculate 
> their IPv6 address (SLAAC).
> 
> > In drawing VR is in VPC mode how it will work for isolated network where 
> > external link/ip is not assigned initially?
> > 
> > 
> > On 2021/07/15 14:47:24, Wido den Hollander  wrote:
> >> But you still need routing. See the attached PNG (and draw.io XML).
> >>
> >> You need to route the /48 subnet TO the VR which can then route it to
> >> the Virtual Networks behind the VR.
> >>
> >> There is no other way then routing with either BGP or a Static route.
> >>
> >> Wido
> >>
> >> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>> Or explain like this :
> >>>
> >>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
> >>> assigned to Cloudstack
> >>> 2) Cloudsack allocated the subnet (that generated from step1) to Virtual
> >>> Router, one Virtual Router have one subniet /64
> >>> 3) Virtual Router allocate single IPv6 (within the range of /64
> >>> allocated to VR)  to VM
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng  >>> > wrote:
> >>>
> >>>  Hi Wido,
> >>>
> >>>  I think the /48 is at physical router as gateway , and subnet of /64
> >>>  at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
> >>>  vlan information of this /48 to be later split the  /64. to VR.
> >>>
> >>>  And the instances is getting singe IPv6 of /64  IP.   The VR is
> >>>  getting /64.  The default gateway shall goes to /48 of physical
> >>>  router ip .   In this case ,does not need any BGP router .
> >>>
> >>>
> >>>  Similar concept as IPv4 :
> >>>
> >>>  /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
> >>>  created in Network.
> >>>  and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
> >>>
> >>>
> >>>
> >>>
> >>>  On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  >>>  > wrote:
> >>>
> >>>
> >>>
> >>>  Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>   > Hi
> >>>   >
> >>>   > I replied in another thread, i think do not need implement
> >>>  BGP or OSPF,
> >>>   > that would be complicated .
> >>>   >
> >>>   > We only need assign  IPv6 's /64 prefix to Virtual Router
> >>>  (VR) in NAT
> >>>   > zone, and the VR responsible to deliver single IPv6 to VM via
> >>>  DHCP6.
> >>>   >
> >>>   > In VR, you need to have Default IPv6 route to  Physical
> >>>  Router's /48. IP
> >>>   > as IPv6 Gateway.  Thens should be done .
> >>>   >
> >>>   > Example :
> >>>   > Physical Router Interface
> >>>   >   IPv6 IP : 2000:::1/48
> >>>   >
> >>>   > Cloudstack  virtual router : 2000::200:201::1/64 with
> >>>  default ipv6
> >>>   > route to router ip 2000:::1
> >>>   > and Clodustack Virtual router dhcp allocate IP to VM , and
> >>>  VM will have
> >>>   > default route to VR. IPv6 2000::200:201::1
> >>>   >
> >>>   > So in cloudstack need to allow  user to enter ,  IPv6
> >>>  gwateway , and
> >>>   > the  /48 Ipv6 prefix , then it will self allocate the /64 ip
> >>>  to the VR ,
> >>>   > and maintain make sure not ovelap allocation
> >>>   >
> >>>   >
> >>>
> >>>  But NAT is truly not the solution with IPv6. IPv6 is supposed to 
> >>> be
> >>>  routable. In addition you should avoid DHCPv6 as much as
> >>>  possible as
> >>>  that's not really the intended use-case for address allocation
> >>>  with IPv6.
> >>>
> >>>  In order to route an /48 IPv6 subnet to the VR you have a few
> >>>  possibilities:
> >>>
> >>>  - Static route from the upperlying routers which are outside of
> >>>  CloudStack
> >>>  - BGP
> >>>  - OSPFv3 (broken in most cases!)
> >>>  - DHCPv6 Prefix Delegation
> >>>
> >>>  BGP and/or Static routes are still the best bet here.
> >>>
> >>>  So what you do is that you tell CloudStack that you will route
> >>>  2001:db8::/48 to the VR, the VR can then use that to split it up
> >>>  into
> >>>  multiple /64 subnets going towards the instances:
> >>>
> >>>  - 2001:db8::/64
> >>>  - 2001:db8:1::/64
> >>>  - 2001:db8:2::/6

Re: IPV6 in Isolated/VPC networks

[Hean Seng]

Hi Wido,

My initial thought is not like this,  it is the /48 at ISP router, and /64
subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
the traffic,   in the VM that get the IPv6 IP will default route to ISP
router as gw.   It can may be a bridge over via Advancezone-VR.

However, If do as the way described in the drawing, then i suppose will be
another kind of virtual router going to introduce , to get hold the /48 in
this virtual router right ?

After this,  The Advance Zone, NAT's  VR will peer with this new IPv6 VR
for getting the IPv6 /64 prefix ?

If do in this way, then I guess  you just only need Static route, with
peering ip both end  as one /48 can have a lot of /64 on it.  And hardware
budgeting for new IPv6-VR will become very important, as all traffic will
need to pass over it .

It will be like

ISP Router  -- >  (new IPV6-VR )  > AdvanceZone-VR > VM

Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on
OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil have
few thousand of BGP session. on new-IPv6-VR

Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
otherwise ISP router will be full of /64 prefix route either on BGP( Many
BGP Session) , or  Many Static route .   If few thousand account, ti will
be few thousand of BGP session with ISP router or few thousand static route
which  is not possible .






On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander  wrote:

> But you still need routing. See the attached PNG (and draw.io XML).
>
> You need to route the /48 subnet TO the VR which can then route it to
> the Virtual Networks behind the VR.
>
> There is no other way then routing with either BGP or a Static route.
>
> Wido
>
> Op 15-07-2021 om 12:39 schreef Hean Seng:
> > Or explain like this :
> >
> > 1) Cloudstack generate list of /64 subnet from /48 that Network admin
> > assigned to Cloudstack
> > 2) Cloudsack allocated the subnet (that generated from step1) to Virtual
> > Router, one Virtual Router have one subniet /64
> > 3) Virtual Router allocate single IPv6 (within the range of /64
> > allocated to VR)  to VM
> >
> >
> >
> >
> >
> >
> > On Thu, Jul 15, 2021 at 6:25 PM Hean Seng  > > wrote:
> >
> > Hi Wido,
> >
> > I think the /48 is at physical router as gateway , and subnet of /64
> > at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
> > vlan information of this /48 to be later split the  /64. to VR.
> >
> > And the instances is getting singe IPv6 of /64  IP.   The VR is
> > getting /64.  The default gateway shall goes to /48 of physical
> > router ip .   In this case ,does not need any BGP router .
> >
> >
> > Similar concept as IPv4 :
> >
> > /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 that
> > created in Network.
> > and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
> >
> >
> >
> >
> > On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander  > > wrote:
> >
> >
> >
> > Op 14-07-2021 om 16:44 schreef Hean Seng:
> >  > Hi
> >  >
> >  > I replied in another thread, i think do not need implement
> > BGP or OSPF,
> >  > that would be complicated .
> >  >
> >  > We only need assign  IPv6 's /64 prefix to Virtual Router
> > (VR) in NAT
> >  > zone, and the VR responsible to deliver single IPv6 to VM via
> > DHCP6.
> >  >
> >  > In VR, you need to have Default IPv6 route to  Physical
> > Router's /48. IP
> >  > as IPv6 Gateway.  Thens should be done .
> >  >
> >  > Example :
> >  > Physical Router Interface
> >  >   IPv6 IP : 2000:::1/48
> >  >
> >  > Cloudstack  virtual router : 2000::200:201::1/64 with
> > default ipv6
> >  > route to router ip 2000:::1
> >  > and Clodustack Virtual router dhcp allocate IP to VM , and
> > VM will have
> >  > default route to VR. IPv6 2000::200:201::1
> >  >
> >  > So in cloudstack need to allow  user to enter ,  IPv6
> > gwateway , and
> >  > the  /48 Ipv6 prefix , then it will self allocate the /64 ip
> > to the VR ,
> >  > and maintain make sure not ovelap allocation
> >  >
> >  >
> >
> > But NAT is truly not the solution with IPv6. IPv6 is supposed to
> be
> > routable. In addition you should avoid DHCPv6 as much as
> > possible as
> > that's not really the intended use-case for address allocation
> > with IPv6.
> >
> > In order to route an /48 IPv6 subnet to the VR you have a few
> > possibilities:
> >
> > - Static route from the upperlying routers which are outside of
> > CloudStack
> > - BG