> -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Tuesday, August 28, 2012 2:00 PM > To: cloudstack-dev@incubator.apache.org > Subject: Re: VM router spawning multiple public nics > > I thought about this solution myself, but below this portion of code > it looks like it uses the hash map to determine which nic number to > add the IP to, so with multiple 'untagged' networks it would have no > way of knowing which nicnum in the router corresponds with the correct > untagged vlan. > > nicNum = vlanAllocatedToVM.get(ip.getVlanId()); > networkUsage(routerIp, "addVif", "eth" + nicNum); > result = > _virtRouterResource.assignPublicIpAddress(routerName, > routerIp, ip.getPublicIp(), ip.isAdd(), > ip.isFirstIP(), > ip.isSourceNat(), ip.getVlanId(), > ip.getVlanGateway(), > ip.getVlanNetmask(), ip.getVifMacAddress(), > ip.getGuestIp(), nicNum); > > if ip.getVlanId() returns untagged (as it does on networks with no > vlan id), and we tried to put multiple untagged keys in > vlanAllocatedToVM (as with multiple untagged networks), we get the > wrong nicNum, no?
In the ipassoc case, if there are multiple untagged networks, all of them are use the same Public bridge. Then multiple ip address will be added on eth2 inside router VM. If it works physically, then it works. > > On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <edison...@citrix.com> wrote: > > > > > >> -----Original Message----- > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> Sent: Tuesday, August 28, 2012 1:40 PM > >> To: cloudstack-dev@incubator.apache.org > >> Subject: Re: VM router spawning multiple public nics > >> > >> Yes, that looks like it would work for me, however that's not > >> something that would ever make it into master, right? Essentially > >> killing tagging for the public, private, and guest traffic labels? > >> There's also still the issue of not being able to differentiate > >> between multiple untagged networks, if we wanted to add an IP to a > >> router it might not know which untagged interface to apply it to. > > > > Physically, all the "untagged" network will be created on > public/guest/private bridge(the name we put in > private/public/guest.bridge.name in agent.properties"). > > Because, there is no way to create a new untagged bridge by agent > itself. Agent code only knows how to create a new tagged(vlan) bridge. > > So the fix should be pushed into master. > > > >> > >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su <edison...@citrix.com> > wrote: > >> > > >> > > >> >> -----Original Message----- > >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> Sent: Tuesday, August 28, 2012 12:23 PM > >> >> To: cloudstack-dev@incubator.apache.org > >> >> Subject: Re: VM router spawning multiple public nics > >> >> > >> >> Thanks for pointing me in the right direction. I've reviewed this > >> code > >> >> in a bit more detail, and it seems like it's accomplishing the > >> >> following: > >> >> > >> >> 1. get all network interfaces currently connected to the running > VM > >> >> (a.k.a vnet devices via libvirt) > >> >> 2. find out which vlans these network interfaces are bridged to, > >> store > >> >> this in a hash map of vlan ids and nics > >> >> 3. get all ip addresses to be added to the VM > >> >> 4. for each ip, get the configured vlan id for the ip, compare it > to > >> >> the hash map of existing vlan ids and nics > >> >> 5. if the required vlan id is not found in the hash map, create a > >> new > >> >> nic > >> >> 6. assign the ip to the nic identified by the vlan id key in the > >> hash > >> >> map > >> >> > >> >> In this case, we're getting a vlan id returned in step 2 for a > >> bridged > >> >> nic whose network is defined as untagged in the cloudstack db, > >> >> therefore in step 5 we never match as already having a nic for > >> >> 'untagged'. I wrote a big long response discussing this issue, > but > >> as > >> >> I began to dig further I realized that aside from my particular > case, > >> >> untagged vlans in general are just broken (for example they can't > be > >> >> dealt with uniquely in the current IpAssocCommand code, given the > >> hash > >> >> map) and it would require more effort than I have time for now to > >> make > >> >> things work. If the code were already in place to differentiate > >> >> between multiple untagged nics I think that fixing my problem > would > >> be > >> >> trivial, but since its not, I'll just find an alternative > solution. > >> >> > >> > > >> > The untagged network usually means "untagged", no vlan on the > >> bridge... > >> > In your case, the untagged network actually has vlan(tagged) on > the > >> bridge, thus getting things confused. > >> > Will this patch(http://pastebin.com/HJXzZwKp) work for you? > >> > > >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen > >> <shadow...@gmail.com> > >> >> wrote: > >> >> > ... > >> >> > Integer nicPos = 0; > >> >> > for (InterfaceDef nic : nics) { > >> >> > if > >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) { > >> >> > vlanAllocatedToVM.put("LinkLocal", nicPos); > >> >> > } else { > >> >> > String vlanId = > >> >> getVlanIdFromBridge(nic.getBrName()); > >> >> > if (vlanId != null) { > >> >> > vlanAllocatedToVM.put(vlanId, nicPos); > >> >> > } else { > >> >> > vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> nicPos); > >> >> > } > >> >> > } > >> >> > nicPos++; > >> >> > } > >> >> > IpAddressTO[] ips = cmd.getIpAddresses(); > >> >> > int i = 0; > >> >> > String result = null; > >> >> > int nicNum = 0; > >> >> > for (IpAddressTO ip : ips) { > >> >> > if > (!vlanAllocatedToVM.containsKey(ip.getVlanId())) > >> { > >> >> > /* plug a vif into router */ > >> >> > VifHotPlug(conn, routerName, ip.getVlanId(), > >> >> > ip.getVifMacAddress()); > >> >> > vlanAllocatedToVM.put(ip.getVlanId(), > >> nicPos++); > >> >> > } > >> >> > ... > >> >> > > >> >> > Looks like the getVlanIdFromBridge might be a bit misleading. I > am > >> >> > running my guest public traffic on a 'cloudbr470', which is a > >> bridge > >> >> > to eth2.470, yet I configured this network as 'untagged' > because I > >> >> > have a vlan 470 available on eth3 for cloudstack to autoassign > >> (eth3 > >> >> > is where all of my stuff will be autoassigned). So I'm not 100% > >> sure > >> >> > yet what's going on here but it seems as though the above is > not > >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan number for > >> >> > eth2.470, but when it enumerates the IPs for the router, it > then > >> runs > >> >> > ip.getVlanId() and doesn't find a nic for the untagged IP and > >> creates > >> >> > one. > >> >> > > >> >> > > >> >> > I realize this is perhaps an uncommon case, but a bug > nonetheless. > >> >> > I'll play with the code a bit and see if I can come up with a > >> >> > solution. I'm thinking I can look at the nic's broadcast URI > and > >> see > >> >> > if it's supposed to be untagged, then add to vlanAllocatedToVM > >> >> > appropriately, off the top of my head something like: > >> >> > > >> >> > String vlanId = > >> >> getVlanIdFromBridge(nic.getBrName()); > >> >> > if (vlanId != null && > >> >> > !nic.getBroadcastUri().toString().contains("untagged") { > >> >> > vlanAllocatedToVM.put(vlanId, nicPos); > >> >> > } else { > >> >> > vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> nicPos); > >> >> > } > >> >> > > >> >> > > >> >> > > >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su > <edison...@citrix.com> > >> >> wrote: > >> >> >> Possible bug in in kvm code: LibvirtComputingResource- > >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is only place > >> adding > >> >> nic into router vm. > >> >> >> Turn on agent log, then take a look what happened. > >> >> >> > >> >> >>> -----Original Message----- > >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >> >>> Sent: Monday, August 27, 2012 5:10 PM > >> >> >>> To: cloudstack-dev@incubator.apache.org > >> >> >>> Subject: VM router spawning multiple public nics > >> >> >>> > >> >> >>> I've got two zones running the same build of cloudstack (a > >> recent > >> >> copy > >> >> >>> of master). One of them creates routers that turn into ugly > >> >> >>> multi-headed beasts, and by that I mean that any time I > create a > >> >> port > >> >> >>> forwarding or iptables rule for that router I get a new > public > >> NIC > >> >> >>> with an identical IP address, I have an instance with a few > tens > >> of > >> >> >>> NICs. My guess is that some script isn't detecting that > there's > >> >> >>> already a NIC with the public IP on it. It looks fine in the > >> >> >>> database, there is only one public NIC defined in the nics > table. > >> >> >>> I'll troubleshoot it tomorrow, but if anyone knows where I > >> should > >> >> >>> begin the headstart would be appreciated. > >> >> >>> > >> >> >>> Thanks
