Just want to clarify: "We can add implementation of SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2SiteVpnCfgCommand in VirtualRoutingResource."
So the Xen implementation of SetupGuestNetworkCommand is in CitrixResourceBase.java, and the VMware one is in VmwareResource.java, but you're saying I should put an implementation of these in VirtualRoutingResource.java for KVM? Sorry, still trying to piece together how everything relates. On Thu, Aug 30, 2012 at 10:42 AM, Edison Su <edison...@citrix.com> wrote: > Yes, it is. > >> -----Original Message----- >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> Sent: Thursday, August 30, 2012 8:45 AM >> To: Edison Su >> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; >> Vijayendra Bhamidipati >> Subject: VPC for KVM (was VM router spawning multiple public nics) >> >> I notice there's also an 'IpAssocVpcCommand', I'm assuming that should >> be added to the list? >> >> On Wed, Aug 29, 2012 at 6:37 PM, Edison Su <edison...@citrix.com> wrote: >> > You can find the VPC reference implementation from >> CitrixResourceBase.java, which is the implementation for Xenserver. >> Just take a look at how the VPC related commands are implemented. >> > Take SetNetworkACLCommand as an example: >> > The function execute(SetNetworkACLCommand cmd) in citrixResourceBase: >> > 1. parse SetNetworkACLCommand >> > 2. call scripts/vm/hypervisor/xenserver/vmops, function routerProxy >> > 3. routerproxy will call scripts/network/domr/router_proxy.sh >> > 4. router_proxy.sh will login into router vm, execute a shell script >> inside router VM to program rules. >> > >> > In KVM, we can directly call router_proxy.sh or directly login into >> router vm, we just need to prepare the parameters for this script. >> > The reference code is in VirtualRoutingResource.java, all the network >> related command(extended from NetworkElementCommand) will be redirected >> to VirtualRoutingResource. >> > We can add implementation of >> SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2 >> SiteVpnCfgCommand in VirtualRoutingResource. >> > >> > >> >> -----Original Message----- >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> Sent: Wednesday, August 29, 2012 4:34 PM >> >> To: cloudstack-dev@incubator.apache.org >> >> Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra Bhamidipati >> >> Subject: Re: VM router spawning multiple public nics >> >> >> >> Great, thanks. I've already got a working implementation of >> >> adding/removing nics from regular instances that I've been playing >> >> with, so I'm getting vaguely familiar with the various data types >> and >> >> things surrounding the networking. I don't know if this is quite >> >> within my reach just yet but I'll see how far I get. >> >> >> >> On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk >> >> <alena.prokharc...@citrix.com> wrote: >> >> > On 8/29/12 1:34 PM, "Edison Su" <edison...@citrix.com> wrote: >> >> > >> >> >>Hi Anthony & Alena, >> >> >> Could you help to provide information about VPC, how it works, >> >> > >> >> > >> >> > Here is the functional spec on the feature: >> >> > >> >> > wiki.cloudstack.org/display/RelOps/Inter- >> VLAN+Routing+functional+spec >> >> > >> >> > >> >> > VpcVirtualNetworkApplianceManagerImpl is the manager responsible >> for >> >> VPC >> >> > Virtual router operations (plug/unplug nics, etc) >> >> > >> >> > >> >> > >> >> >> which commands needed to implemented on the hypervisor side? >> >> > >> >> > >> >> > >> >> > 1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug >> >> (currently >> >> > works for VR vm only). In VPC called when add nic for Public/Guest >> >> > networks. >> >> > 2) SetupGuestNetworkCommand - sets up dhcp range, dns information, >> >> > networkDomain information on the Nic to make it >> >> > 3) SetNetworkACLCommand - creates network ACL on the virtual >> router >> >> > 4) SetSourceNatCommand - used for setting source nat on the Public >> IP >> >> on >> >> > the VPC VR. >> >> > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN >> >> > >> >> > >> >> > Anthony/Kelven/Vijay did implementation for Xen/vmWare resources, >> >> they can >> >> > help you answering all hypervisor related questions. If you need >> more >> >> > details on business logic + Vpc VR management, I can help with >> that. >> >> > >> >> > >> >> > -Alena. >> >> > >> >> >> >> >> >>> -----Original Message----- >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> Sent: Wednesday, August 29, 2012 10:16 AM >> >> >>> To: cloudstack-dev@incubator.apache.org >> >> >>> Subject: Re: VM router spawning multiple public nics >> >> >>> >> >> >>> I'd be willing to give it a shot if someone could point me in >> the >> >> >>> right direction and be available to answer questions. >> >> >>> >> >> >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su >> <edison...@citrix.com> >> >> >>> wrote: >> >> >>> > Yah, KVM doesn't support VPC yet. Will you help to add VPC >> >> support on >> >> >>> KVM?:) Just implement a few VPC related commands... >> >> >>> > >> >> >>> >> -----Original Message----- >> >> >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >> Sent: Wednesday, August 29, 2012 6:49 AM >> >> >>> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >> Subject: Re: VM router spawning multiple public nics >> >> >>> >> >> >> >>> >> I can confirm that the patch has fixed my particular issue. >> >> >>> >> >> >> >>> >> This is likely unrelated and I think it doesn't even use the >> >> same >> >> >>> >> code, but I began to play with the VPC stuff a bit and >> noticed >> >> that >> >> >>> I >> >> >>> >> don't get any interfaces except for link local. I'd probably >> >> chalk >> >> >>> >> that up to it not being ready for KVM, but I thought it was >> >> worth a >> >> >>> >> mention. I'd be happy to try to help get it ready if someone >> >> has >> >> >>> time >> >> >>> >> to nudge me in the right direction. >> >> >>> >> >> >> >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su >> >> <edison...@citrix.com> >> >> >>> wrote: >> >> >>> >> > >> >> >>> >> > >> >> >>> >> >> -----Original Message----- >> >> >>> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM >> >> >>> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >> >> Subject: Re: VM router spawning multiple public nics >> >> >>> >> >> >> >> >>> >> >> I thought about this solution myself, but below this >> portion >> >> of >> >> >>> >> code >> >> >>> >> >> it looks like it uses the hash map to determine which nic >> >> number >> >> >>> to >> >> >>> >> >> add the IP to, so with multiple 'untagged' networks it >> would >> >> have >> >> >>> no >> >> >>> >> >> way of knowing which nicnum in the router corresponds with >> >> the >> >> >>> >> correct >> >> >>> >> >> untagged vlan. >> >> >>> >> >> >> >> >>> >> >> nicNum = >> >> vlanAllocatedToVM.get(ip.getVlanId()); >> >> >>> >> >> networkUsage(routerIp, "addVif", "eth" + >> >> nicNum); >> >> >>> >> >> result = >> >> >>> >> >> _virtRouterResource.assignPublicIpAddress(routerName, >> >> >>> >> >> routerIp, ip.getPublicIp(), >> >> ip.isAdd(), >> >> >>> >> >> ip.isFirstIP(), >> >> >>> >> >> ip.isSourceNat(), ip.getVlanId(), >> >> >>> >> >> ip.getVlanGateway(), >> >> >>> >> >> ip.getVlanNetmask(), >> >> >>> ip.getVifMacAddress(), >> >> >>> >> >> ip.getGuestIp(), nicNum); >> >> >>> >> >> >> >> >>> >> >> if ip.getVlanId() returns untagged (as it does on networks >> >> with >> >> >>> no >> >> >>> >> >> vlan id), and we tried to put multiple untagged keys in >> >> >>> >> >> vlanAllocatedToVM (as with multiple untagged networks), we >> >> get >> >> >>> the >> >> >>> >> >> wrong nicNum, no? >> >> >>> >> > >> >> >>> >> > In the ipassoc case, if there are multiple untagged >> networks, >> >> all >> >> >>> of >> >> >>> >> them are use the same >> >> >>> >> > Public bridge. Then multiple ip address will be added on >> eth2 >> >> >>> inside >> >> >>> >> router VM. >> >> >>> >> > If it works physically, then it works. >> >> >>> >> > >> >> >>> >> >> >> >> >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su >> >> <edison...@citrix.com> >> >> >>> >> wrote: >> >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> >> -----Original Message----- >> >> >>> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM >> >> >>> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >> >> >> Subject: Re: VM router spawning multiple public nics >> >> >>> >> >> >> >> >> >>> >> >> >> Yes, that looks like it would work for me, however >> that's >> >> not >> >> >>> >> >> >> something that would ever make it into master, right? >> >> >>> Essentially >> >> >>> >> >> >> killing tagging for the public, private, and guest >> traffic >> >> >>> labels? >> >> >>> >> >> >> There's also still the issue of not being able to >> >> >>> differentiate >> >> >>> >> >> >> between multiple untagged networks, if we wanted to add >> an >> >> IP >> >> >>> to >> >> >>> >> a >> >> >>> >> >> >> router it might not know which untagged interface to >> apply >> >> it >> >> >>> to. >> >> >>> >> >> > >> >> >>> >> >> > Physically, all the "untagged" network will be created >> on >> >> >>> >> >> public/guest/private bridge(the name we put in >> >> >>> >> >> private/public/guest.bridge.name in agent.properties"). >> >> >>> >> >> > Because, there is no way to create a new untagged bridge >> by >> >> >>> agent >> >> >>> >> >> itself. Agent code only knows how to create a new >> tagged(vlan) >> >> >>> >> bridge. >> >> >>> >> >> > So the fix should be pushed into master. >> >> >>> >> >> > >> >> >>> >> >> >> >> >> >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su >> >> >>> <edison...@citrix.com> >> >> >>> >> >> wrote: >> >> >>> >> >> >> > >> >> >>> >> >> >> > >> >> >>> >> >> >> >> -----Original Message----- >> >> >>> >> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM >> >> >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >> >> >> >> Subject: Re: VM router spawning multiple public nics >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> Thanks for pointing me in the right direction. I've >> >> >>> reviewed >> >> >>> >> this >> >> >>> >> >> >> code >> >> >>> >> >> >> >> in a bit more detail, and it seems like it's >> >> accomplishing >> >> >>> the >> >> >>> >> >> >> >> following: >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> 1. get all network interfaces currently connected to >> >> the >> >> >>> >> running >> >> >>> >> >> VM >> >> >>> >> >> >> >> (a.k.a vnet devices via libvirt) >> >> >>> >> >> >> >> 2. find out which vlans these network interfaces are >> >> >>> bridged >> >> >>> >> to, >> >> >>> >> >> >> store >> >> >>> >> >> >> >> this in a hash map of vlan ids and nics >> >> >>> >> >> >> >> 3. get all ip addresses to be added to the VM >> >> >>> >> >> >> >> 4. for each ip, get the configured vlan id for the >> ip, >> >> >>> compare >> >> >>> >> it >> >> >>> >> >> to >> >> >>> >> >> >> >> the hash map of existing vlan ids and nics >> >> >>> >> >> >> >> 5. if the required vlan id is not found in the hash >> map, >> >> >>> >> create a >> >> >>> >> >> >> new >> >> >>> >> >> >> >> nic >> >> >>> >> >> >> >> 6. assign the ip to the nic identified by the vlan >> id >> >> key >> >> >>> in >> >> >>> >> the >> >> >>> >> >> >> hash >> >> >>> >> >> >> >> map >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> In this case, we're getting a vlan id returned in >> step >> >> 2 >> >> >>> for a >> >> >>> >> >> >> bridged >> >> >>> >> >> >> >> nic whose network is defined as untagged in the >> >> cloudstack >> >> >>> db, >> >> >>> >> >> >> >> therefore in step 5 we never match as already having >> a >> >> nic >> >> >>> for >> >> >>> >> >> >> >> 'untagged'. I wrote a big long response discussing >> this >> >> >>> issue, >> >> >>> >> >> but >> >> >>> >> >> >> as >> >> >>> >> >> >> >> I began to dig further I realized that aside from my >> >> >>> >> particular >> >> >>> >> >> case, >> >> >>> >> >> >> >> untagged vlans in general are just broken (for >> example >> >> they >> >> >>> >> can't >> >> >>> >> >> be >> >> >>> >> >> >> >> dealt with uniquely in the current IpAssocCommand >> code, >> >> >>> given >> >> >>> >> the >> >> >>> >> >> >> hash >> >> >>> >> >> >> >> map) and it would require more effort than I have >> time >> >> for >> >> >>> now >> >> >>> >> to >> >> >>> >> >> >> make >> >> >>> >> >> >> >> things work. If the code were already in place to >> >> >>> >> differentiate >> >> >>> >> >> >> >> between multiple untagged nics I think that fixing >> my >> >> >>> problem >> >> >>> >> >> would >> >> >>> >> >> >> be >> >> >>> >> >> >> >> trivial, but since its not, I'll just find an >> >> alternative >> >> >>> >> >> solution. >> >> >>> >> >> >> >> >> >> >>> >> >> >> > >> >> >>> >> >> >> > The untagged network usually means "untagged", no >> vlan >> >> on >> >> >>> the >> >> >>> >> >> >> bridge... >> >> >>> >> >> >> > In your case, the untagged network actually has >> >> vlan(tagged) >> >> >>> on >> >> >>> >> >> the >> >> >>> >> >> >> bridge, thus getting things confused. >> >> >>> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp) work >> for >> >> you? >> >> >>> >> >> >> > >> >> >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen >> >> >>> >> >> >> <shadow...@gmail.com> >> >> >>> >> >> >> >> wrote: >> >> >>> >> >> >> >> > ... >> >> >>> >> >> >> >> > Integer nicPos = 0; >> >> >>> >> >> >> >> > for (InterfaceDef nic : nics) { >> >> >>> >> >> >> >> > if >> >> >>> >> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) >> >> { >> >> >>> >> >> >> >> > >> vlanAllocatedToVM.put("LinkLocal", >> >> >>> >> nicPos); >> >> >>> >> >> >> >> > } else { >> >> >>> >> >> >> >> > String vlanId = >> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >>> >> >> >> >> > if (vlanId != null) { >> >> >>> >> >> >> >> > >> vlanAllocatedToVM.put(vlanId, >> >> >>> >> nicPos); >> >> >>> >> >> >> >> > } else { >> >> >>> >> >> >> >> > >> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >>> >> >> >> nicPos); >> >> >>> >> >> >> >> > } >> >> >>> >> >> >> >> > } >> >> >>> >> >> >> >> > nicPos++; >> >> >>> >> >> >> >> > } >> >> >>> >> >> >> >> > IpAddressTO[] ips = >> cmd.getIpAddresses(); >> >> >>> >> >> >> >> > int i = 0; >> >> >>> >> >> >> >> > String result = null; >> >> >>> >> >> >> >> > int nicNum = 0; >> >> >>> >> >> >> >> > for (IpAddressTO ip : ips) { >> >> >>> >> >> >> >> > if >> >> >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) >> >> >>> >> >> >> { >> >> >>> >> >> >> >> > /* plug a vif into router */ >> >> >>> >> >> >> >> > VifHotPlug(conn, routerName, >> >> >>> >> ip.getVlanId(), >> >> >>> >> >> >> >> > ip.getVifMacAddress()); >> >> >>> >> >> >> >> > >> >> vlanAllocatedToVM.put(ip.getVlanId(), >> >> >>> >> >> >> nicPos++); >> >> >>> >> >> >> >> > } >> >> >>> >> >> >> >> > ... >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > Looks like the getVlanIdFromBridge might be a bit >> >> >>> misleading. >> >> >>> >> I >> >> >>> >> >> am >> >> >>> >> >> >> >> > running my guest public traffic on a 'cloudbr470', >> >> which >> >> >>> is >> >> >>> >> a >> >> >>> >> >> >> bridge >> >> >>> >> >> >> >> > to eth2.470, yet I configured this network as >> >> 'untagged' >> >> >>> >> >> because I >> >> >>> >> >> >> >> > have a vlan 470 available on eth3 for cloudstack >> to >> >> >>> >> autoassign >> >> >>> >> >> >> (eth3 >> >> >>> >> >> >> >> > is where all of my stuff will be autoassigned). So >> >> I'm >> >> >>> not >> >> >>> >> 100% >> >> >>> >> >> >> sure >> >> >>> >> >> >> >> > yet what's going on here but it seems as though >> the >> >> above >> >> >>> is >> >> >>> >> >> not >> >> >>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan >> >> number >> >> >>> >> for >> >> >>> >> >> >> >> > eth2.470, but when it enumerates the IPs for the >> >> router, >> >> >>> it >> >> >>> >> >> then >> >> >>> >> >> >> runs >> >> >>> >> >> >> >> > ip.getVlanId() and doesn't find a nic for the >> >> untagged IP >> >> >>> >> and >> >> >>> >> >> >> creates >> >> >>> >> >> >> >> > one. >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > I realize this is perhaps an uncommon case, but a >> bug >> >> >>> >> >> nonetheless. >> >> >>> >> >> >> >> > I'll play with the code a bit and see if I can >> come >> >> up >> >> >>> with >> >> >>> >> a >> >> >>> >> >> >> >> > solution. I'm thinking I can look at the nic's >> >> broadcast >> >> >>> URI >> >> >>> >> >> and >> >> >>> >> >> >> see >> >> >>> >> >> >> >> > if it's supposed to be untagged, then add to >> >> >>> >> vlanAllocatedToVM >> >> >>> >> >> >> >> > appropriately, off the top of my head something >> like: >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > String vlanId = >> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >>> >> >> >> >> > if (vlanId != null && >> >> >>> >> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged") >> >> { >> >> >>> >> >> >> >> > >> vlanAllocatedToVM.put(vlanId, >> >> >>> >> nicPos); >> >> >>> >> >> >> >> > } else { >> >> >>> >> >> >> >> > >> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >>> >> >> >> nicPos); >> >> >>> >> >> >> >> > } >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > >> >> >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su >> >> >>> >> >> <edison...@citrix.com> >> >> >>> >> >> >> >> wrote: >> >> >>> >> >> >> >> >> Possible bug in in kvm code: >> >> LibvirtComputingResource- >> >> >>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is >> >> only >> >> >>> place >> >> >>> >> >> >> adding >> >> >>> >> >> >> >> nic into router vm. >> >> >>> >> >> >> >> >> Turn on agent log, then take a look what happened. >> >> >>> >> >> >> >> >> >> >> >>> >> >> >> >> >>> -----Original Message----- >> >> >>> >> >> >> >> >>> From: Marcus Sorensen >> [mailto:shadow...@gmail.com] >> >> >>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM >> >> >>> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org >> >> >>> >> >> >> >> >>> Subject: VM router spawning multiple public nics >> >> >>> >> >> >> >> >>> >> >> >>> >> >> >> >> >>> I've got two zones running the same build of >> >> cloudstack >> >> >>> (a >> >> >>> >> >> >> recent >> >> >>> >> >> >> >> copy >> >> >>> >> >> >> >> >>> of master). One of them creates routers that >> turn >> >> into >> >> >>> >> ugly >> >> >>> >> >> >> >> >>> multi-headed beasts, and by that I mean that any >> >> time I >> >> >>> >> >> create a >> >> >>> >> >> >> >> port >> >> >>> >> >> >> >> >>> forwarding or iptables rule for that router I >> get a >> >> new >> >> >>> >> >> public >> >> >>> >> >> >> NIC >> >> >>> >> >> >> >> >>> with an identical IP address, I have an instance >> >> with a >> >> >>> >> few >> >> >>> >> >> tens >> >> >>> >> >> >> of >> >> >>> >> >> >> >> >>> NICs. My guess is that some script isn't >> detecting >> >> >>> that >> >> >>> >> >> there's >> >> >>> >> >> >> >> >>> already a NIC with the public IP on it. It >> looks >> >> fine >> >> >>> in >> >> >>> >> the >> >> >>> >> >> >> >> >>> database, there is only one public NIC defined >> in >> >> the >> >> >>> nics >> >> >>> >> >> table. >> >> >>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if anyone >> knows >> >> >>> where I >> >> >>> >> >> >> should >> >> >>> >> >> >> >> >>> begin the headstart would be appreciated. >> >> >>> >> >> >> >> >>> >> >> >>> >> >> >> >> >>> Thanks >> >> >> >> >> > >> >> >
