Good info. I think seeing another use case helps us narrow the requirements a bit more.
On Thu, Jan 9, 2014 at 8:04 AM, Daan Hoogland <dhoogl...@schubergphilis.com> wrote: > H Marcus, We had a small session on how we plan to go about ipv6 > configuration with your bullet list present. Comments are still brainstorming > phase quality but hopefully they will lead to something. > >> * VPC has no global IPv6 prefix (super CIDR as current private space), it's >> simply IPv6 enabled or not. Admins can choose to route a /60 or a /48 to a >> vpc and carve it up among the networks there, but it's not required or >> enforced by cloudstack. > The idea at Schuberg Philis is that VPCs should have a standard size network > assigned to be taken from a pool. We should be able to configure the width of > the network. The idea is that per level (Physical network/vpc/tier) only one > width will be used but the size of it is configurable. Default block for a VR > / VPC would be a /56 and each individual network would use a /64. Note that > in a typical isolated network we only use the first. Ok, good. Just to clarify, this is your use case, you're not saying cloudstack should enforce a static size, just that you would deploy it that way, correct? Do you have any specific reasoning or need for the VPC itself to have a configured contiguous block, rather than just assign the /64s the networks? > >> * VPC networks get assigned one or multiple IPv6 prefixes, each prefix can >> be marked 'SLAAC', 'DHCP', or 'Manual'. > For networks we are in favor of using SLAAC for configuring routing and > addressing, but would advise DHCPv6 to configure DNS and potentially other > things (NTP?). No need to register the addresses in CloudStack. We already > store the MAC of the NIC so we know which address will be configured on the > interface with the configured prefix. Only the router will receive a specific > address (prefix::1/64) I agree that we would want to publish DNS via DHCPv6, but is it necessary, do you think? I assume we're not currently planning IPv6 *only* networks, thus AAAA can be looked up via the VR's DHCPv4 resolver. To me it seems like we'd do it for the sake of completeness, I suppose. I'm just trying to sort things into stages, what do we need to have working IPv6, what is nice to have but not necessary, and which dependent features will we want to add. If we start off and get everything but that done, everything would still work, I think. > > As mentioned earlier first implementation will do SLAAC. How will SLAAC be > configured with ACL's? I assume ACLs would work as they do with ipv4, where you can select 'incoming from X location and A-B ports with this protocol to this network' and outgoing. I don't think it has any bearing on SLAAC, DHCP, or manual configurations as currently ACLs apply to a whole prefix. > >> * Mgmt server allows calling external plugins for pushing routes to routers >> upstream of VPCs (SDN or router API), but could be manually done by admins >> as well > We discussed the routing issues in front of the VPC / VR. The goal is to > provide cloudstack with a way to deal with routable addresses inside an > isolated network, as actually this problem is relevant for both IPv4 and > IPv6. However for IPv4 we have a perfectly workable solution with NAT and in > the IPv6 world we don't/. So we need to address this issue on how to route a > network from the provider edge into an isolated network. > > For IPv6 we have several options: > * Dynamic routing protocols, here we can think of iBGP, OSPF or any > other protocol. Basically cloudstack assigns a block to a VR / VPC and the > VPC / VR tells the outside world to route that block to the outside IP of the > router. > * Block allocator, cloudstack knows about the supernet (/48 for > example) and assigns a block (/56) to a VR or VPC. Cloudstack tells the block > allocator to allocate (route) this block to the VR/VPC. Block allocater needs > to know about the device being managed (upstream router like cisco/juniper) > and configured the route using telnet or an API > * Broker, We introduce a new systemvm, the block broker. The admin > routes the /48 to the block broker and CloudStack tells the blok broker to > reroute a specific subblock to this VR / VPC > * Static, Admin configures a list of block/ip pairs in the upstream > router. CloudStack knows about these pairs and assigns the ip to the VR / VPC > and the block will be routed. > We've got some experience in this dept, as we currently route IPv4 public addresses into the VR as well via a little plugin. The simplest implementation would simply be 1) assign a /64 to a network via mgmt server, this goes in a table 2) when assignment occurs, and when router reboots, we send a command to the VR to program the route for that network. All it takes is for the upstream routers to know to send the prefix to that VR and you're done. I may be wrong, and please someone step in and help if needed, but I think BGP and the like would be a lot of work to automate, and wouldn't scale. Assuming you'd want everything under the same ASN or split between a few, the advertisements would scale quadratically (every VR needs to publish to every other VR), that doesn't work for a cloud provider who expects thousands or tens of thousands of VPCs. If you wanted to assign ASNs to VPCs instead, that would be hard to automate, I think. If that's not the case, then yeah, let's just set up quagga and go. The block broker sounds interesting, but ultimately it's going to be a big bottleneck as a VPC router to rule all VPCs (so to speak). Actually, this is not dissimilar from what I was suggesting, which was to allow for plugins so that external routers could be programmed with the relevant route info during assignment and vpc router start/stop, just that we take the burden of doing the upstream routing away from a cloudstack VR and allow whatever hardware or setup the local admins choose. We could even make it easier to integrate; in addition to the plugin capability we could publish route info via the message bus, so any sysadmin is just a python script away from grabbing that data and programming their upstream routers through whatever tools they use. We could also do both, for example right now we allow people to bring up isolated networks without a VR, assuming they'll have their own. We could both publish the data via plugin/message bus and allow for a broker VM, and those could be done separately. Honestly, I have no interest in the block broker, but I think it would be a nice addition for small setups, and I do think not having that or automated route advertisement (BGP) is a big hurdle for small and/or less experienced teams to deploy IPv6. > >> * Work could be done in stages, e.g. SLAAC/manual network ranges would be >> fairly straightforward, whereas DHCP ranges would require programming >> scripts and ip allocation code. > >> * An issue was raised about privacy concerns with SLAAC using MAC, but we >> think this revolves more around clients than servers; that is, a client >> moving around the country would be traceable because of the MAC, but a >> server always has the same address anyway. > >> * Still need to figure out what to do about ACLs, that's a whole separate >> issue, but a plan needs to be in place. > Are we going to put ACL's on networks or hosts or both? What is easiest to > implement and enforce? > > Do we care about portforwarding, static NAT, Load Balancing etc as well? Or > at least think > about what impact these decisions have. > This is not a version 1 consideration?!? Of course it is. None of these are things that need to be done in version 1, I don't think, but we need to spec it out such that we don't have to rebuild it to do these things. > >> * We assume there will be an ipv6 public range assignable, allocated for VPC >> routers/static nat/loadbalancers to pull from. How will this range be made >> available? > Is a cidr configured? > Will we preconfigure all ranges or have a dynamic availability check? > > Radvd package is added to the systemvm template. (Done by Hugo this morning) > > Some standard UI components are needed. > > (test-)attention needs to go to: > > HaProxy version/feature matrix has to be reviewed to check for ipv6 > compatability. > > Does dnsmasq work with ipv6? > > Does keepalived work seemslessly? > > Ipv6 tables configuration needs to be pushed to the VRs > > -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: maandag 6 januari 2014 9:11 > To: cloudstack-...@incubator.apache.org > Subject: IPv6 in VPC (was Re: IPv6 plan - questions) > > I've discussed this a bit with various subject matter experts at our > datacenters/business, and so far we're leaning toward a rollout like > this: > > * VPC has no global IPv6 prefix (super CIDR as current private space), it's > simply IPv6 enabled or not. Admins can choose to route a /60 or a > /48 to a vpc and carve it up among the networks there, but it's not required > or enforced by cloudstack. > > * VPC networks get assigned one or multiple IPv6 prefixes, each prefix can be > marked 'SLAAC', 'DHCP', or 'Manual'. > > * Mgmt server allows calling external plugins for pushing routes to routers > upstream of VPCs (SDN or router API), but could be manually done by admins as > well > > * Work could be done in stages, e.g. SLAAC/manual network ranges would be > fairly straightforward, whereas DHCP ranges would require programming scripts > and ip allocation code. > > * An issue was raised about privacy concerns with SLAAC using MAC, but we > think this revolves more around clients than servers; that is, a client > moving around the country would be traceable because of the MAC, but a server > always has the same address anyway. > > * Still need to figure out what to do about ACLs, that's a whole separate > issue, but a plan needs to be in place. Do we care about port forwarding, > static NAT, Load Balancing etc as well? Or at least think about what impact > these decisions have. > > * We assume there will be an ipv6 public range assignable, allocated for VPC > routers/static nat/loadbalancers to pull from. > > On Sat, Jan 4, 2014 at 6:11 AM, Marcus Sorensen <shadow...@gmail.com> wrote: >> I've put together a rough draft spec: >> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+in+VPC+Rou >> ter >> >> I basically just laid out some rough ideas. I know there has been a >> lot of discussion in the past about DHCPv6, etc. My hope is that we >> can at least decide on a spec, for future reference. >> >> >> On Fri, Jan 3, 2014 at 9:53 PM, Marcus Sorensen <shadow...@gmail.com> wrote: >>> It's been a long time since I've heard anything in regards to IPv6, >>> let alone VPC support. Does anyone have plans for this at all? We'd >>> like to support IPv6, and we have enough CS knowledge and external >>> tools to hack something together, but I'd much prefer to build with >>> the community and/or be forward compatible with what it deploys. >>> >>> I'd like to start with something simple, like perhaps optionally >>> providing a /64 or larger as a parameter when creating a VPC (or a >>> separate call to add an IPV6 block), and network on the vpc. Then it >>> sounds like there's already a mechanism in place for tracking ipv6 >>> assignments to nics, that could be leveraged to pass dhcp assignments >>> to routers. >>> >>> Then there's the whole acl thing, that seems like at least as big of >>> a project as mentioned previously. >>> >>> On Mon, Aug 12, 2013 at 3:47 PM, Marcus Sorensen <shadow...@gmail.com> >>> wrote: >>>> has there been any further discussion that I might have missed >>>> around >>>> ipv6 in VPC? >>>> >>>> On Thu, Mar 7, 2013 at 12:09 PM, Sheng Yang <sh...@yasker.org> wrote: >>>>> Hi Dave, >>>>> >>>>> I am glad it fits your need. That's our target. :) >>>>> >>>>> --Sheng >>>>> >>>>> On Thu, Mar 7, 2013 at 2:14 AM, Dave Cahill <dcah...@midokura.com> wrote: >>>>>> Hi Sheng, >>>>>> >>>>>> Thanks for the quick reply, that helps a lot. >>>>>> >>>>>> My main purpose was to figure out how these changes affect virtual >>>>>> networking and pluggability. Having read through the IPv6 code >>>>>> today, it looks like it will work very nicely with virtual networks. >>>>>> >>>>>> For example, when VMs are assigned an IPv6 address, the IPv6 >>>>>> address is stored in the NicProfile object. So, taking DHCP as an >>>>>> example, if the MidoNet plugin implements the DHCPServiceProvider >>>>>> interface, it will receive the NicProfile as one of the parameters of >>>>>> addDhcpEntry. >>>>>> If we want to implement IPv6, we can then take the IPv6 address >>>>>> from the NicProfile, and just use it as needed. >>>>>> >>>>>> Thanks again for taking the time to respond, and for the detailed FS. >>>>>> >>>>>> Dave. >>>>>> >>>>>> On Thu, Mar 7, 2013 at 4:57 AM, Sheng Yang <sh...@yasker.org> wrote: >>>>>> >>>>>>> On Wed, Mar 6, 2013 at 1:36 AM, Dave Cahill <dcah...@midokura.com> >>>>>>> wrote: >>>>>>> > Hi, >>>>>>> >>>>>>> Hi Dave, >>>>>>> > >>>>>>> > I've been catching up on IPv6 plans by reading the functional >>>>>>> > specs and Jira tickets - it's great to have so much material to refer >>>>>>> > to. >>>>>>> > >>>>>>> > I still have a few questions though, and I'm hoping someone >>>>>>> > involved with the feature can enlighten me. >>>>>>> > >>>>>>> > *[Support for Providers other than Virtual Router]* In [3], the >>>>>>> > spec says "No external device support in plan." >>>>>>> > What does this mean exactly? >>>>>>> >>>>>>> Because CloudStack also supports using external devices as >>>>>>> network controller e.g. Juniper SRX as firewall and NetScaler as >>>>>>> load balancer. The words here said is just we don't support these >>>>>>> devices when using IPv6. >>>>>>> > >>>>>>> > For example, if using Providers other than the Virtual Router, >>>>>>> > does the UI still allow setting IPv6 addresses? >>>>>>> > >>>>>>> > If so, do we attempt to pass IPv6 addresses to the Providers no >>>>>>> > matter what, or do we check whether the Provider has IPv6 support? >>>>>>> >>>>>>> Yes, we checked it when you try to create a IPv6 >>>>>>> network(currently only support advance shared network). >>>>>>> >>>>>>> > >>>>>>> > *[Networking Modes]* >>>>>>> > Advanced Shared mode and Basic mode are mentioned in the Jira >>>>>>> > ticket [1] - "Isolated Network" is mentioned briefly in [2], >>>>>>> > but I wanted to check if the Advanced Isolated and VPC modes >>>>>>> > are on the roadmap? >>>>>>> >>>>>>> There is no "basic isolated" network, so "Isolated" network is >>>>>>> what we're talking about. We haven't got plan for VPC yet. >>>>>>> >>>>>>> And one correction: we didn't support "basic" mode for phase 1. >>>>>>> We support only "advance shared network" in phase 1. The >>>>>>> supported cases are described in FS. Jira ticket only provided a >>>>>>> rough idea at the time. >>>>>>> > >>>>>>> > *[IP Address Management / IPAM]* From [1], re: handing out IPv6 >>>>>>> > addresses: "One way could be that the network admin creates a >>>>>>> > static route for a /48 towards a Virtual Router and then the VR >>>>>>> > can hand out /64s to Instances." >>>>>>> > >>>>>>> > With IPv4, IPAM is handled by the CloudStack management server, >>>>>>> > and the VR is told which IP address to give to the VM over >>>>>>> > DHCP. Would this change with IPv6? "The VR can hand out /64s to >>>>>>> > instances" sounds like the VR is handling IPAM to some extent. >>>>>>> >>>>>>> Well, it's not how it works now. Please refer to the FS. The >>>>>>> current implementation works like before. VR get a /64 then >>>>>>> handle out IPv6 addresses to VM. >>>>>>> > >>>>>>> > From [3], "Router advertisement should be sent by public >>>>>>> > gateway in the network." - to double-check, does this mean the >>>>>>> > router outside the CloudStack network should send RAs, but the VR >>>>>>> > won't send RAs? >>>>>>> >>>>>>> Yes. Because in phase 1, we support only "advance shared >>>>>>> network", in which case, VR is NOT the gateway. So we assume the >>>>>>> gateway router outside CloudStack should send out RA to the VMs. >>>>>>> >>>>>>> But in the phase 2, VR would acting as gateway, then it would send out >>>>>>> RAs. >>>>>>> >>>>>>> --Sheng >>>>>>> > >>>>>>> > Thanks, >>>>>>> > Dave. >>>>>>> > >>>>>>> > [1] IPv6 Support main Jira ticket >>>>>>> > https://issues.apache.org/jira/browse/CLOUDSTACK-452 >>>>>>> > >>>>>>> > [2] IPv6 Support in CloudStack FS >>>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support-in-cloudstack. >>>>>>> > html >>>>>>> > >>>>>>> > [3] IPv6 Support FS >>>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support.html >>>>>>>
