On 8/29/12 1:34 PM, "Edison Su" <edison...@citrix.com> wrote:
>Hi Anthony & Alena, > Could you help to provide information about VPC, how it works, Here is the functional spec on the feature: wiki.cloudstack.org/display/RelOps/Inter-VLAN+Routing+functional+spec VpcVirtualNetworkApplianceManagerImpl is the manager responsible for VPC Virtual router operations (plug/unplug nics, etc) > which commands needed to implemented on the hypervisor side? 1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug (currently works for VR vm only). In VPC called when add nic for Public/Guest networks. 2) SetupGuestNetworkCommand - sets up dhcp range, dns information, networkDomain information on the Nic to make it 3) SetNetworkACLCommand - creates network ACL on the virtual router 4) SetSourceNatCommand - used for setting source nat on the Public IP on the VPC VR. 5) Site2SiteVpnCfgCommand - for setting up S2S VPN Anthony/Kelven/Vijay did implementation for Xen/vmWare resources, they can help you answering all hypervisor related questions. If you need more details on business logic + Vpc VR management, I can help with that. -Alena. > >> -----Original Message----- >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> Sent: Wednesday, August 29, 2012 10:16 AM >> To: cloudstack-dev@incubator.apache.org >> Subject: Re: VM router spawning multiple public nics >> >> I'd be willing to give it a shot if someone could point me in the >> right direction and be available to answer questions. >> >> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <edison...@citrix.com> >> wrote: >> > Yah, KVM doesn't support VPC yet. Will you help to add VPC support on >> KVM?:) Just implement a few VPC related commands... >> > >> >> -----Original Message----- >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> Sent: Wednesday, August 29, 2012 6:49 AM >> >> To: cloudstack-dev@incubator.apache.org >> >> Subject: Re: VM router spawning multiple public nics >> >> >> >> I can confirm that the patch has fixed my particular issue. >> >> >> >> This is likely unrelated and I think it doesn't even use the same >> >> code, but I began to play with the VPC stuff a bit and noticed that >> I >> >> don't get any interfaces except for link local. I'd probably chalk >> >> that up to it not being ready for KVM, but I thought it was worth a >> >> mention. I'd be happy to try to help get it ready if someone has >> time >> >> to nudge me in the right direction. >> >> >> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su <edison...@citrix.com> >> wrote: >> >> > >> >> > >> >> >> -----Original Message----- >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >> Sent: Tuesday, August 28, 2012 2:00 PM >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >> Subject: Re: VM router spawning multiple public nics >> >> >> >> >> >> I thought about this solution myself, but below this portion of >> >> code >> >> >> it looks like it uses the hash map to determine which nic number >> to >> >> >> add the IP to, so with multiple 'untagged' networks it would have >> no >> >> >> way of knowing which nicnum in the router corresponds with the >> >> correct >> >> >> untagged vlan. >> >> >> >> >> >> nicNum = vlanAllocatedToVM.get(ip.getVlanId()); >> >> >> networkUsage(routerIp, "addVif", "eth" + nicNum); >> >> >> result = >> >> >> _virtRouterResource.assignPublicIpAddress(routerName, >> >> >> routerIp, ip.getPublicIp(), ip.isAdd(), >> >> >> ip.isFirstIP(), >> >> >> ip.isSourceNat(), ip.getVlanId(), >> >> >> ip.getVlanGateway(), >> >> >> ip.getVlanNetmask(), >> ip.getVifMacAddress(), >> >> >> ip.getGuestIp(), nicNum); >> >> >> >> >> >> if ip.getVlanId() returns untagged (as it does on networks with >> no >> >> >> vlan id), and we tried to put multiple untagged keys in >> >> >> vlanAllocatedToVM (as with multiple untagged networks), we get >> the >> >> >> wrong nicNum, no? >> >> > >> >> > In the ipassoc case, if there are multiple untagged networks, all >> of >> >> them are use the same >> >> > Public bridge. Then multiple ip address will be added on eth2 >> inside >> >> router VM. >> >> > If it works physically, then it works. >> >> > >> >> >> >> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <edison...@citrix.com> >> >> wrote: >> >> >> > >> >> >> > >> >> >> >> -----Original Message----- >> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM >> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >> >> Subject: Re: VM router spawning multiple public nics >> >> >> >> >> >> >> >> Yes, that looks like it would work for me, however that's not >> >> >> >> something that would ever make it into master, right? >> Essentially >> >> >> >> killing tagging for the public, private, and guest traffic >> labels? >> >> >> >> There's also still the issue of not being able to >> differentiate >> >> >> >> between multiple untagged networks, if we wanted to add an IP >> to >> >> a >> >> >> >> router it might not know which untagged interface to apply it >> to. >> >> >> > >> >> >> > Physically, all the "untagged" network will be created on >> >> >> public/guest/private bridge(the name we put in >> >> >> private/public/guest.bridge.name in agent.properties"). >> >> >> > Because, there is no way to create a new untagged bridge by >> agent >> >> >> itself. Agent code only knows how to create a new tagged(vlan) >> >> bridge. >> >> >> > So the fix should be pushed into master. >> >> >> > >> >> >> >> >> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su >> <edison...@citrix.com> >> >> >> wrote: >> >> >> >> > >> >> >> >> > >> >> >> >> >> -----Original Message----- >> >> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM >> >> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >> >> >> Subject: Re: VM router spawning multiple public nics >> >> >> >> >> >> >> >> >> >> Thanks for pointing me in the right direction. I've >> reviewed >> >> this >> >> >> >> code >> >> >> >> >> in a bit more detail, and it seems like it's accomplishing >> the >> >> >> >> >> following: >> >> >> >> >> >> >> >> >> >> 1. get all network interfaces currently connected to the >> >> running >> >> >> VM >> >> >> >> >> (a.k.a vnet devices via libvirt) >> >> >> >> >> 2. find out which vlans these network interfaces are >> bridged >> >> to, >> >> >> >> store >> >> >> >> >> this in a hash map of vlan ids and nics >> >> >> >> >> 3. get all ip addresses to be added to the VM >> >> >> >> >> 4. for each ip, get the configured vlan id for the ip, >> compare >> >> it >> >> >> to >> >> >> >> >> the hash map of existing vlan ids and nics >> >> >> >> >> 5. if the required vlan id is not found in the hash map, >> >> create a >> >> >> >> new >> >> >> >> >> nic >> >> >> >> >> 6. assign the ip to the nic identified by the vlan id key >> in >> >> the >> >> >> >> hash >> >> >> >> >> map >> >> >> >> >> >> >> >> >> >> In this case, we're getting a vlan id returned in step 2 >> for a >> >> >> >> bridged >> >> >> >> >> nic whose network is defined as untagged in the cloudstack >> db, >> >> >> >> >> therefore in step 5 we never match as already having a nic >> for >> >> >> >> >> 'untagged'. I wrote a big long response discussing this >> issue, >> >> >> but >> >> >> >> as >> >> >> >> >> I began to dig further I realized that aside from my >> >> particular >> >> >> case, >> >> >> >> >> untagged vlans in general are just broken (for example they >> >> can't >> >> >> be >> >> >> >> >> dealt with uniquely in the current IpAssocCommand code, >> given >> >> the >> >> >> >> hash >> >> >> >> >> map) and it would require more effort than I have time for >> now >> >> to >> >> >> >> make >> >> >> >> >> things work. If the code were already in place to >> >> differentiate >> >> >> >> >> between multiple untagged nics I think that fixing my >> problem >> >> >> would >> >> >> >> be >> >> >> >> >> trivial, but since its not, I'll just find an alternative >> >> >> solution. >> >> >> >> >> >> >> >> >> > >> >> >> >> > The untagged network usually means "untagged", no vlan on >> the >> >> >> >> bridge... >> >> >> >> > In your case, the untagged network actually has vlan(tagged) >> on >> >> >> the >> >> >> >> bridge, thus getting things confused. >> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp) work for you? >> >> >> >> > >> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen >> >> >> >> <shadow...@gmail.com> >> >> >> >> >> wrote: >> >> >> >> >> > ... >> >> >> >> >> > Integer nicPos = 0; >> >> >> >> >> > for (InterfaceDef nic : nics) { >> >> >> >> >> > if >> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) { >> >> >> >> >> > vlanAllocatedToVM.put("LinkLocal", >> >> nicPos); >> >> >> >> >> > } else { >> >> >> >> >> > String vlanId = >> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >> >> >> > if (vlanId != null) { >> >> >> >> >> > vlanAllocatedToVM.put(vlanId, >> >> nicPos); >> >> >> >> >> > } else { >> >> >> >> >> > >> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >> >> nicPos); >> >> >> >> >> > } >> >> >> >> >> > } >> >> >> >> >> > nicPos++; >> >> >> >> >> > } >> >> >> >> >> > IpAddressTO[] ips = cmd.getIpAddresses(); >> >> >> >> >> > int i = 0; >> >> >> >> >> > String result = null; >> >> >> >> >> > int nicNum = 0; >> >> >> >> >> > for (IpAddressTO ip : ips) { >> >> >> >> >> > if >> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) >> >> >> >> { >> >> >> >> >> > /* plug a vif into router */ >> >> >> >> >> > VifHotPlug(conn, routerName, >> >> ip.getVlanId(), >> >> >> >> >> > ip.getVifMacAddress()); >> >> >> >> >> > vlanAllocatedToVM.put(ip.getVlanId(), >> >> >> >> nicPos++); >> >> >> >> >> > } >> >> >> >> >> > ... >> >> >> >> >> > >> >> >> >> >> > Looks like the getVlanIdFromBridge might be a bit >> misleading. >> >> I >> >> >> am >> >> >> >> >> > running my guest public traffic on a 'cloudbr470', which >> is >> >> a >> >> >> >> bridge >> >> >> >> >> > to eth2.470, yet I configured this network as 'untagged' >> >> >> because I >> >> >> >> >> > have a vlan 470 available on eth3 for cloudstack to >> >> autoassign >> >> >> >> (eth3 >> >> >> >> >> > is where all of my stuff will be autoassigned). So I'm >> not >> >> 100% >> >> >> >> sure >> >> >> >> >> > yet what's going on here but it seems as though the above >> is >> >> >> not >> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan number >> >> for >> >> >> >> >> > eth2.470, but when it enumerates the IPs for the router, >> it >> >> >> then >> >> >> >> runs >> >> >> >> >> > ip.getVlanId() and doesn't find a nic for the untagged IP >> >> and >> >> >> >> creates >> >> >> >> >> > one. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > I realize this is perhaps an uncommon case, but a bug >> >> >> nonetheless. >> >> >> >> >> > I'll play with the code a bit and see if I can come up >> with >> >> a >> >> >> >> >> > solution. I'm thinking I can look at the nic's broadcast >> URI >> >> >> and >> >> >> >> see >> >> >> >> >> > if it's supposed to be untagged, then add to >> >> vlanAllocatedToVM >> >> >> >> >> > appropriately, off the top of my head something like: >> >> >> >> >> > >> >> >> >> >> > String vlanId = >> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >> >> >> > if (vlanId != null && >> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged") { >> >> >> >> >> > vlanAllocatedToVM.put(vlanId, >> >> nicPos); >> >> >> >> >> > } else { >> >> >> >> >> > >> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >> >> nicPos); >> >> >> >> >> > } >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su >> >> >> <edison...@citrix.com> >> >> >> >> >> wrote: >> >> >> >> >> >> Possible bug in in kvm code: LibvirtComputingResource- >> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is only >> place >> >> >> >> adding >> >> >> >> >> nic into router vm. >> >> >> >> >> >> Turn on agent log, then take a look what happened. >> >> >> >> >> >> >> >> >> >> >> >>> -----Original Message----- >> >> >> >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM >> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org >> >> >> >> >> >>> Subject: VM router spawning multiple public nics >> >> >> >> >> >>> >> >> >> >> >> >>> I've got two zones running the same build of cloudstack >> (a >> >> >> >> recent >> >> >> >> >> copy >> >> >> >> >> >>> of master). One of them creates routers that turn into >> >> ugly >> >> >> >> >> >>> multi-headed beasts, and by that I mean that any time I >> >> >> create a >> >> >> >> >> port >> >> >> >> >> >>> forwarding or iptables rule for that router I get a new >> >> >> public >> >> >> >> NIC >> >> >> >> >> >>> with an identical IP address, I have an instance with a >> >> few >> >> >> tens >> >> >> >> of >> >> >> >> >> >>> NICs. My guess is that some script isn't detecting >> that >> >> >> there's >> >> >> >> >> >>> already a NIC with the public IP on it. It looks fine >> in >> >> the >> >> >> >> >> >>> database, there is only one public NIC defined in the >> nics >> >> >> table. >> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if anyone knows >> where I >> >> >> >> should >> >> >> >> >> >>> begin the headstart would be appreciated. >> >> >> >> >> >>> >> >> >> >> >> >>> Thanks >
