Hi Anthony & Alena, Could you help to provide information about VPC, how it works, which commands needed to implemented on the hypervisor side?
> -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Wednesday, August 29, 2012 10:16 AM > To: cloudstack-dev@incubator.apache.org > Subject: Re: VM router spawning multiple public nics > > I'd be willing to give it a shot if someone could point me in the > right direction and be available to answer questions. > > On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <edison...@citrix.com> > wrote: > > Yah, KVM doesn't support VPC yet. Will you help to add VPC support on > KVM?:) Just implement a few VPC related commands... > > > >> -----Original Message----- > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> Sent: Wednesday, August 29, 2012 6:49 AM > >> To: cloudstack-dev@incubator.apache.org > >> Subject: Re: VM router spawning multiple public nics > >> > >> I can confirm that the patch has fixed my particular issue. > >> > >> This is likely unrelated and I think it doesn't even use the same > >> code, but I began to play with the VPC stuff a bit and noticed that > I > >> don't get any interfaces except for link local. I'd probably chalk > >> that up to it not being ready for KVM, but I thought it was worth a > >> mention. I'd be happy to try to help get it ready if someone has > time > >> to nudge me in the right direction. > >> > >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su <edison...@citrix.com> > wrote: > >> > > >> > > >> >> -----Original Message----- > >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> Sent: Tuesday, August 28, 2012 2:00 PM > >> >> To: cloudstack-dev@incubator.apache.org > >> >> Subject: Re: VM router spawning multiple public nics > >> >> > >> >> I thought about this solution myself, but below this portion of > >> code > >> >> it looks like it uses the hash map to determine which nic number > to > >> >> add the IP to, so with multiple 'untagged' networks it would have > no > >> >> way of knowing which nicnum in the router corresponds with the > >> correct > >> >> untagged vlan. > >> >> > >> >> nicNum = vlanAllocatedToVM.get(ip.getVlanId()); > >> >> networkUsage(routerIp, "addVif", "eth" + nicNum); > >> >> result = > >> >> _virtRouterResource.assignPublicIpAddress(routerName, > >> >> routerIp, ip.getPublicIp(), ip.isAdd(), > >> >> ip.isFirstIP(), > >> >> ip.isSourceNat(), ip.getVlanId(), > >> >> ip.getVlanGateway(), > >> >> ip.getVlanNetmask(), > ip.getVifMacAddress(), > >> >> ip.getGuestIp(), nicNum); > >> >> > >> >> if ip.getVlanId() returns untagged (as it does on networks with > no > >> >> vlan id), and we tried to put multiple untagged keys in > >> >> vlanAllocatedToVM (as with multiple untagged networks), we get > the > >> >> wrong nicNum, no? > >> > > >> > In the ipassoc case, if there are multiple untagged networks, all > of > >> them are use the same > >> > Public bridge. Then multiple ip address will be added on eth2 > inside > >> router VM. > >> > If it works physically, then it works. > >> > > >> >> > >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <edison...@citrix.com> > >> wrote: > >> >> > > >> >> > > >> >> >> -----Original Message----- > >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM > >> >> >> To: cloudstack-dev@incubator.apache.org > >> >> >> Subject: Re: VM router spawning multiple public nics > >> >> >> > >> >> >> Yes, that looks like it would work for me, however that's not > >> >> >> something that would ever make it into master, right? > Essentially > >> >> >> killing tagging for the public, private, and guest traffic > labels? > >> >> >> There's also still the issue of not being able to > differentiate > >> >> >> between multiple untagged networks, if we wanted to add an IP > to > >> a > >> >> >> router it might not know which untagged interface to apply it > to. > >> >> > > >> >> > Physically, all the "untagged" network will be created on > >> >> public/guest/private bridge(the name we put in > >> >> private/public/guest.bridge.name in agent.properties"). > >> >> > Because, there is no way to create a new untagged bridge by > agent > >> >> itself. Agent code only knows how to create a new tagged(vlan) > >> bridge. > >> >> > So the fix should be pushed into master. > >> >> > > >> >> >> > >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su > <edison...@citrix.com> > >> >> wrote: > >> >> >> > > >> >> >> > > >> >> >> >> -----Original Message----- > >> >> >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM > >> >> >> >> To: cloudstack-dev@incubator.apache.org > >> >> >> >> Subject: Re: VM router spawning multiple public nics > >> >> >> >> > >> >> >> >> Thanks for pointing me in the right direction. I've > reviewed > >> this > >> >> >> code > >> >> >> >> in a bit more detail, and it seems like it's accomplishing > the > >> >> >> >> following: > >> >> >> >> > >> >> >> >> 1. get all network interfaces currently connected to the > >> running > >> >> VM > >> >> >> >> (a.k.a vnet devices via libvirt) > >> >> >> >> 2. find out which vlans these network interfaces are > bridged > >> to, > >> >> >> store > >> >> >> >> this in a hash map of vlan ids and nics > >> >> >> >> 3. get all ip addresses to be added to the VM > >> >> >> >> 4. for each ip, get the configured vlan id for the ip, > compare > >> it > >> >> to > >> >> >> >> the hash map of existing vlan ids and nics > >> >> >> >> 5. if the required vlan id is not found in the hash map, > >> create a > >> >> >> new > >> >> >> >> nic > >> >> >> >> 6. assign the ip to the nic identified by the vlan id key > in > >> the > >> >> >> hash > >> >> >> >> map > >> >> >> >> > >> >> >> >> In this case, we're getting a vlan id returned in step 2 > for a > >> >> >> bridged > >> >> >> >> nic whose network is defined as untagged in the cloudstack > db, > >> >> >> >> therefore in step 5 we never match as already having a nic > for > >> >> >> >> 'untagged'. I wrote a big long response discussing this > issue, > >> >> but > >> >> >> as > >> >> >> >> I began to dig further I realized that aside from my > >> particular > >> >> case, > >> >> >> >> untagged vlans in general are just broken (for example they > >> can't > >> >> be > >> >> >> >> dealt with uniquely in the current IpAssocCommand code, > given > >> the > >> >> >> hash > >> >> >> >> map) and it would require more effort than I have time for > now > >> to > >> >> >> make > >> >> >> >> things work. If the code were already in place to > >> differentiate > >> >> >> >> between multiple untagged nics I think that fixing my > problem > >> >> would > >> >> >> be > >> >> >> >> trivial, but since its not, I'll just find an alternative > >> >> solution. > >> >> >> >> > >> >> >> > > >> >> >> > The untagged network usually means "untagged", no vlan on > the > >> >> >> bridge... > >> >> >> > In your case, the untagged network actually has vlan(tagged) > on > >> >> the > >> >> >> bridge, thus getting things confused. > >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp) work for you? > >> >> >> > > >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen > >> >> >> <shadow...@gmail.com> > >> >> >> >> wrote: > >> >> >> >> > ... > >> >> >> >> > Integer nicPos = 0; > >> >> >> >> > for (InterfaceDef nic : nics) { > >> >> >> >> > if > >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) { > >> >> >> >> > vlanAllocatedToVM.put("LinkLocal", > >> nicPos); > >> >> >> >> > } else { > >> >> >> >> > String vlanId = > >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >> >> >> >> > if (vlanId != null) { > >> >> >> >> > vlanAllocatedToVM.put(vlanId, > >> nicPos); > >> >> >> >> > } else { > >> >> >> >> > > vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> >> >> nicPos); > >> >> >> >> > } > >> >> >> >> > } > >> >> >> >> > nicPos++; > >> >> >> >> > } > >> >> >> >> > IpAddressTO[] ips = cmd.getIpAddresses(); > >> >> >> >> > int i = 0; > >> >> >> >> > String result = null; > >> >> >> >> > int nicNum = 0; > >> >> >> >> > for (IpAddressTO ip : ips) { > >> >> >> >> > if > >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) > >> >> >> { > >> >> >> >> > /* plug a vif into router */ > >> >> >> >> > VifHotPlug(conn, routerName, > >> ip.getVlanId(), > >> >> >> >> > ip.getVifMacAddress()); > >> >> >> >> > vlanAllocatedToVM.put(ip.getVlanId(), > >> >> >> nicPos++); > >> >> >> >> > } > >> >> >> >> > ... > >> >> >> >> > > >> >> >> >> > Looks like the getVlanIdFromBridge might be a bit > misleading. > >> I > >> >> am > >> >> >> >> > running my guest public traffic on a 'cloudbr470', which > is > >> a > >> >> >> bridge > >> >> >> >> > to eth2.470, yet I configured this network as 'untagged' > >> >> because I > >> >> >> >> > have a vlan 470 available on eth3 for cloudstack to > >> autoassign > >> >> >> (eth3 > >> >> >> >> > is where all of my stuff will be autoassigned). So I'm > not > >> 100% > >> >> >> sure > >> >> >> >> > yet what's going on here but it seems as though the above > is > >> >> not > >> >> >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan number > >> for > >> >> >> >> > eth2.470, but when it enumerates the IPs for the router, > it > >> >> then > >> >> >> runs > >> >> >> >> > ip.getVlanId() and doesn't find a nic for the untagged IP > >> and > >> >> >> creates > >> >> >> >> > one. > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > I realize this is perhaps an uncommon case, but a bug > >> >> nonetheless. > >> >> >> >> > I'll play with the code a bit and see if I can come up > with > >> a > >> >> >> >> > solution. I'm thinking I can look at the nic's broadcast > URI > >> >> and > >> >> >> see > >> >> >> >> > if it's supposed to be untagged, then add to > >> vlanAllocatedToVM > >> >> >> >> > appropriately, off the top of my head something like: > >> >> >> >> > > >> >> >> >> > String vlanId = > >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >> >> >> >> > if (vlanId != null && > >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged") { > >> >> >> >> > vlanAllocatedToVM.put(vlanId, > >> nicPos); > >> >> >> >> > } else { > >> >> >> >> > > vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> >> >> nicPos); > >> >> >> >> > } > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su > >> >> <edison...@citrix.com> > >> >> >> >> wrote: > >> >> >> >> >> Possible bug in in kvm code: LibvirtComputingResource- > >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is only > place > >> >> >> adding > >> >> >> >> nic into router vm. > >> >> >> >> >> Turn on agent log, then take a look what happened. > >> >> >> >> >> > >> >> >> >> >>> -----Original Message----- > >> >> >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM > >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org > >> >> >> >> >>> Subject: VM router spawning multiple public nics > >> >> >> >> >>> > >> >> >> >> >>> I've got two zones running the same build of cloudstack > (a > >> >> >> recent > >> >> >> >> copy > >> >> >> >> >>> of master). One of them creates routers that turn into > >> ugly > >> >> >> >> >>> multi-headed beasts, and by that I mean that any time I > >> >> create a > >> >> >> >> port > >> >> >> >> >>> forwarding or iptables rule for that router I get a new > >> >> public > >> >> >> NIC > >> >> >> >> >>> with an identical IP address, I have an instance with a > >> few > >> >> tens > >> >> >> of > >> >> >> >> >>> NICs. My guess is that some script isn't detecting > that > >> >> there's > >> >> >> >> >>> already a NIC with the public IP on it. It looks fine > in > >> the > >> >> >> >> >>> database, there is only one public NIC defined in the > nics > >> >> table. > >> >> >> >> >>> I'll troubleshoot it tomorrow, but if anyone knows > where I > >> >> >> should > >> >> >> >> >>> begin the headstart would be appreciated. > >> >> >> >> >>> > >> >> >> >> >>> Thanks
