You can find the VPC reference implementation from CitrixResourceBase.java, which is the implementation for Xenserver. Just take a look at how the VPC related commands are implemented. Take SetNetworkACLCommand as an example: The function execute(SetNetworkACLCommand cmd) in citrixResourceBase: 1. parse SetNetworkACLCommand 2. call scripts/vm/hypervisor/xenserver/vmops, function routerProxy 3. routerproxy will call scripts/network/domr/router_proxy.sh 4. router_proxy.sh will login into router vm, execute a shell script inside router VM to program rules.
In KVM, we can directly call router_proxy.sh or directly login into router vm, we just need to prepare the parameters for this script. The reference code is in VirtualRoutingResource.java, all the network related command(extended from NetworkElementCommand) will be redirected to VirtualRoutingResource. We can add implementation of SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2SiteVpnCfgCommand in VirtualRoutingResource. > -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Wednesday, August 29, 2012 4:34 PM > To: cloudstack-dev@incubator.apache.org > Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra Bhamidipati > Subject: Re: VM router spawning multiple public nics > > Great, thanks. I've already got a working implementation of > adding/removing nics from regular instances that I've been playing > with, so I'm getting vaguely familiar with the various data types and > things surrounding the networking. I don't know if this is quite > within my reach just yet but I'll see how far I get. > > On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk > <alena.prokharc...@citrix.com> wrote: > > On 8/29/12 1:34 PM, "Edison Su" <edison...@citrix.com> wrote: > > > >>Hi Anthony & Alena, > >> Could you help to provide information about VPC, how it works, > > > > > > Here is the functional spec on the feature: > > > > wiki.cloudstack.org/display/RelOps/Inter-VLAN+Routing+functional+spec > > > > > > VpcVirtualNetworkApplianceManagerImpl is the manager responsible for > VPC > > Virtual router operations (plug/unplug nics, etc) > > > > > > > >> which commands needed to implemented on the hypervisor side? > > > > > > > > 1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug > (currently > > works for VR vm only). In VPC called when add nic for Public/Guest > > networks. > > 2) SetupGuestNetworkCommand - sets up dhcp range, dns information, > > networkDomain information on the Nic to make it > > 3) SetNetworkACLCommand - creates network ACL on the virtual router > > 4) SetSourceNatCommand - used for setting source nat on the Public IP > on > > the VPC VR. > > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN > > > > > > Anthony/Kelven/Vijay did implementation for Xen/vmWare resources, > they can > > help you answering all hypervisor related questions. If you need more > > details on business logic + Vpc VR management, I can help with that. > > > > > > -Alena. > > > >> > >>> -----Original Message----- > >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> Sent: Wednesday, August 29, 2012 10:16 AM > >>> To: cloudstack-dev@incubator.apache.org > >>> Subject: Re: VM router spawning multiple public nics > >>> > >>> I'd be willing to give it a shot if someone could point me in the > >>> right direction and be available to answer questions. > >>> > >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <edison...@citrix.com> > >>> wrote: > >>> > Yah, KVM doesn't support VPC yet. Will you help to add VPC > support on > >>> KVM?:) Just implement a few VPC related commands... > >>> > > >>> >> -----Original Message----- > >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> >> Sent: Wednesday, August 29, 2012 6:49 AM > >>> >> To: cloudstack-dev@incubator.apache.org > >>> >> Subject: Re: VM router spawning multiple public nics > >>> >> > >>> >> I can confirm that the patch has fixed my particular issue. > >>> >> > >>> >> This is likely unrelated and I think it doesn't even use the > same > >>> >> code, but I began to play with the VPC stuff a bit and noticed > that > >>> I > >>> >> don't get any interfaces except for link local. I'd probably > chalk > >>> >> that up to it not being ready for KVM, but I thought it was > worth a > >>> >> mention. I'd be happy to try to help get it ready if someone > has > >>> time > >>> >> to nudge me in the right direction. > >>> >> > >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su > <edison...@citrix.com> > >>> wrote: > >>> >> > > >>> >> > > >>> >> >> -----Original Message----- > >>> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM > >>> >> >> To: cloudstack-dev@incubator.apache.org > >>> >> >> Subject: Re: VM router spawning multiple public nics > >>> >> >> > >>> >> >> I thought about this solution myself, but below this portion > of > >>> >> code > >>> >> >> it looks like it uses the hash map to determine which nic > number > >>> to > >>> >> >> add the IP to, so with multiple 'untagged' networks it would > have > >>> no > >>> >> >> way of knowing which nicnum in the router corresponds with > the > >>> >> correct > >>> >> >> untagged vlan. > >>> >> >> > >>> >> >> nicNum = > vlanAllocatedToVM.get(ip.getVlanId()); > >>> >> >> networkUsage(routerIp, "addVif", "eth" + > nicNum); > >>> >> >> result = > >>> >> >> _virtRouterResource.assignPublicIpAddress(routerName, > >>> >> >> routerIp, ip.getPublicIp(), > ip.isAdd(), > >>> >> >> ip.isFirstIP(), > >>> >> >> ip.isSourceNat(), ip.getVlanId(), > >>> >> >> ip.getVlanGateway(), > >>> >> >> ip.getVlanNetmask(), > >>> ip.getVifMacAddress(), > >>> >> >> ip.getGuestIp(), nicNum); > >>> >> >> > >>> >> >> if ip.getVlanId() returns untagged (as it does on networks > with > >>> no > >>> >> >> vlan id), and we tried to put multiple untagged keys in > >>> >> >> vlanAllocatedToVM (as with multiple untagged networks), we > get > >>> the > >>> >> >> wrong nicNum, no? > >>> >> > > >>> >> > In the ipassoc case, if there are multiple untagged networks, > all > >>> of > >>> >> them are use the same > >>> >> > Public bridge. Then multiple ip address will be added on eth2 > >>> inside > >>> >> router VM. > >>> >> > If it works physically, then it works. > >>> >> > > >>> >> >> > >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su > <edison...@citrix.com> > >>> >> wrote: > >>> >> >> > > >>> >> >> > > >>> >> >> >> -----Original Message----- > >>> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM > >>> >> >> >> To: cloudstack-dev@incubator.apache.org > >>> >> >> >> Subject: Re: VM router spawning multiple public nics > >>> >> >> >> > >>> >> >> >> Yes, that looks like it would work for me, however that's > not > >>> >> >> >> something that would ever make it into master, right? > >>> Essentially > >>> >> >> >> killing tagging for the public, private, and guest traffic > >>> labels? > >>> >> >> >> There's also still the issue of not being able to > >>> differentiate > >>> >> >> >> between multiple untagged networks, if we wanted to add an > IP > >>> to > >>> >> a > >>> >> >> >> router it might not know which untagged interface to apply > it > >>> to. > >>> >> >> > > >>> >> >> > Physically, all the "untagged" network will be created on > >>> >> >> public/guest/private bridge(the name we put in > >>> >> >> private/public/guest.bridge.name in agent.properties"). > >>> >> >> > Because, there is no way to create a new untagged bridge by > >>> agent > >>> >> >> itself. Agent code only knows how to create a new tagged(vlan) > >>> >> bridge. > >>> >> >> > So the fix should be pushed into master. > >>> >> >> > > >>> >> >> >> > >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su > >>> <edison...@citrix.com> > >>> >> >> wrote: > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> >> -----Original Message----- > >>> >> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM > >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org > >>> >> >> >> >> Subject: Re: VM router spawning multiple public nics > >>> >> >> >> >> > >>> >> >> >> >> Thanks for pointing me in the right direction. I've > >>> reviewed > >>> >> this > >>> >> >> >> code > >>> >> >> >> >> in a bit more detail, and it seems like it's > accomplishing > >>> the > >>> >> >> >> >> following: > >>> >> >> >> >> > >>> >> >> >> >> 1. get all network interfaces currently connected to > the > >>> >> running > >>> >> >> VM > >>> >> >> >> >> (a.k.a vnet devices via libvirt) > >>> >> >> >> >> 2. find out which vlans these network interfaces are > >>> bridged > >>> >> to, > >>> >> >> >> store > >>> >> >> >> >> this in a hash map of vlan ids and nics > >>> >> >> >> >> 3. get all ip addresses to be added to the VM > >>> >> >> >> >> 4. for each ip, get the configured vlan id for the ip, > >>> compare > >>> >> it > >>> >> >> to > >>> >> >> >> >> the hash map of existing vlan ids and nics > >>> >> >> >> >> 5. if the required vlan id is not found in the hash map, > >>> >> create a > >>> >> >> >> new > >>> >> >> >> >> nic > >>> >> >> >> >> 6. assign the ip to the nic identified by the vlan id > key > >>> in > >>> >> the > >>> >> >> >> hash > >>> >> >> >> >> map > >>> >> >> >> >> > >>> >> >> >> >> In this case, we're getting a vlan id returned in step > 2 > >>> for a > >>> >> >> >> bridged > >>> >> >> >> >> nic whose network is defined as untagged in the > cloudstack > >>> db, > >>> >> >> >> >> therefore in step 5 we never match as already having a > nic > >>> for > >>> >> >> >> >> 'untagged'. I wrote a big long response discussing this > >>> issue, > >>> >> >> but > >>> >> >> >> as > >>> >> >> >> >> I began to dig further I realized that aside from my > >>> >> particular > >>> >> >> case, > >>> >> >> >> >> untagged vlans in general are just broken (for example > they > >>> >> can't > >>> >> >> be > >>> >> >> >> >> dealt with uniquely in the current IpAssocCommand code, > >>> given > >>> >> the > >>> >> >> >> hash > >>> >> >> >> >> map) and it would require more effort than I have time > for > >>> now > >>> >> to > >>> >> >> >> make > >>> >> >> >> >> things work. If the code were already in place to > >>> >> differentiate > >>> >> >> >> >> between multiple untagged nics I think that fixing my > >>> problem > >>> >> >> would > >>> >> >> >> be > >>> >> >> >> >> trivial, but since its not, I'll just find an > alternative > >>> >> >> solution. > >>> >> >> >> >> > >>> >> >> >> > > >>> >> >> >> > The untagged network usually means "untagged", no vlan > on > >>> the > >>> >> >> >> bridge... > >>> >> >> >> > In your case, the untagged network actually has > vlan(tagged) > >>> on > >>> >> >> the > >>> >> >> >> bridge, thus getting things confused. > >>> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp) work for > you? > >>> >> >> >> > > >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen > >>> >> >> >> <shadow...@gmail.com> > >>> >> >> >> >> wrote: > >>> >> >> >> >> > ... > >>> >> >> >> >> > Integer nicPos = 0; > >>> >> >> >> >> > for (InterfaceDef nic : nics) { > >>> >> >> >> >> > if > >>> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) > { > >>> >> >> >> >> > vlanAllocatedToVM.put("LinkLocal", > >>> >> nicPos); > >>> >> >> >> >> > } else { > >>> >> >> >> >> > String vlanId = > >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >>> >> >> >> >> > if (vlanId != null) { > >>> >> >> >> >> > vlanAllocatedToVM.put(vlanId, > >>> >> nicPos); > >>> >> >> >> >> > } else { > >>> >> >> >> >> > > >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, > >>> >> >> >> nicPos); > >>> >> >> >> >> > } > >>> >> >> >> >> > } > >>> >> >> >> >> > nicPos++; > >>> >> >> >> >> > } > >>> >> >> >> >> > IpAddressTO[] ips = cmd.getIpAddresses(); > >>> >> >> >> >> > int i = 0; > >>> >> >> >> >> > String result = null; > >>> >> >> >> >> > int nicNum = 0; > >>> >> >> >> >> > for (IpAddressTO ip : ips) { > >>> >> >> >> >> > if > >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) > >>> >> >> >> { > >>> >> >> >> >> > /* plug a vif into router */ > >>> >> >> >> >> > VifHotPlug(conn, routerName, > >>> >> ip.getVlanId(), > >>> >> >> >> >> > ip.getVifMacAddress()); > >>> >> >> >> >> > > vlanAllocatedToVM.put(ip.getVlanId(), > >>> >> >> >> nicPos++); > >>> >> >> >> >> > } > >>> >> >> >> >> > ... > >>> >> >> >> >> > > >>> >> >> >> >> > Looks like the getVlanIdFromBridge might be a bit > >>> misleading. > >>> >> I > >>> >> >> am > >>> >> >> >> >> > running my guest public traffic on a 'cloudbr470', > which > >>> is > >>> >> a > >>> >> >> >> bridge > >>> >> >> >> >> > to eth2.470, yet I configured this network as > 'untagged' > >>> >> >> because I > >>> >> >> >> >> > have a vlan 470 available on eth3 for cloudstack to > >>> >> autoassign > >>> >> >> >> (eth3 > >>> >> >> >> >> > is where all of my stuff will be autoassigned). So > I'm > >>> not > >>> >> 100% > >>> >> >> >> sure > >>> >> >> >> >> > yet what's going on here but it seems as though the > above > >>> is > >>> >> >> not > >>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan > number > >>> >> for > >>> >> >> >> >> > eth2.470, but when it enumerates the IPs for the > router, > >>> it > >>> >> >> then > >>> >> >> >> runs > >>> >> >> >> >> > ip.getVlanId() and doesn't find a nic for the > untagged IP > >>> >> and > >>> >> >> >> creates > >>> >> >> >> >> > one. > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > I realize this is perhaps an uncommon case, but a bug > >>> >> >> nonetheless. > >>> >> >> >> >> > I'll play with the code a bit and see if I can come > up > >>> with > >>> >> a > >>> >> >> >> >> > solution. I'm thinking I can look at the nic's > broadcast > >>> URI > >>> >> >> and > >>> >> >> >> see > >>> >> >> >> >> > if it's supposed to be untagged, then add to > >>> >> vlanAllocatedToVM > >>> >> >> >> >> > appropriately, off the top of my head something like: > >>> >> >> >> >> > > >>> >> >> >> >> > String vlanId = > >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >>> >> >> >> >> > if (vlanId != null && > >>> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged") > { > >>> >> >> >> >> > vlanAllocatedToVM.put(vlanId, > >>> >> nicPos); > >>> >> >> >> >> > } else { > >>> >> >> >> >> > > >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, > >>> >> >> >> nicPos); > >>> >> >> >> >> > } > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su > >>> >> >> <edison...@citrix.com> > >>> >> >> >> >> wrote: > >>> >> >> >> >> >> Possible bug in in kvm code: > LibvirtComputingResource- > >>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is > only > >>> place > >>> >> >> >> adding > >>> >> >> >> >> nic into router vm. > >>> >> >> >> >> >> Turn on agent log, then take a look what happened. > >>> >> >> >> >> >> > >>> >> >> >> >> >>> -----Original Message----- > >>> >> >> >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM > >>> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org > >>> >> >> >> >> >>> Subject: VM router spawning multiple public nics > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> I've got two zones running the same build of > cloudstack > >>> (a > >>> >> >> >> recent > >>> >> >> >> >> copy > >>> >> >> >> >> >>> of master). One of them creates routers that turn > into > >>> >> ugly > >>> >> >> >> >> >>> multi-headed beasts, and by that I mean that any > time I > >>> >> >> create a > >>> >> >> >> >> port > >>> >> >> >> >> >>> forwarding or iptables rule for that router I get a > new > >>> >> >> public > >>> >> >> >> NIC > >>> >> >> >> >> >>> with an identical IP address, I have an instance > with a > >>> >> few > >>> >> >> tens > >>> >> >> >> of > >>> >> >> >> >> >>> NICs. My guess is that some script isn't detecting > >>> that > >>> >> >> there's > >>> >> >> >> >> >>> already a NIC with the public IP on it. It looks > fine > >>> in > >>> >> the > >>> >> >> >> >> >>> database, there is only one public NIC defined in > the > >>> nics > >>> >> >> table. > >>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if anyone knows > >>> where I > >>> >> >> >> should > >>> >> >> >> >> >>> begin the headstart would be appreciated. > >>> >> >> >> >> >>> > >>> >> >> >> >> >>> Thanks > >> > > > >
