> -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Friday, August 31, 2012 1:58 PM > To: Edison Su > Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; > Vijayendra Bhamidipati > Subject: Re: VPC for KVM (was VM router spawning multiple public nics) > > Ok, I've gotten to most of the list and things seem to be working. > Nics plug, Gateway IPs add, networks create, SNATs working, ACLs > working. The ones I haven't gotten to are: > > *Site2SiteVpnCfgCommand > > and two which weren't on the original list that I just came across > (will check for others): > *SetPortForwardingRulesVpcCommand > *CheckS2SVpnConnectionsCommand > *SetStaticRouteCommand
The commands related to site2site VPN can be finished later, as it needs more complicated setup to test. But I'd like to get SetPortForwardingRulesVpcCommand and SetStaticRouteCommand working. > > In addition, the following observations have been made and not yet > fixed. If anyone has feedback on these (even a 'yeah, that doesn't > work', or 'hmm, that should be working') so I can gauge whether I > should dig into them or not I'd appreciate it. I don't want to break > something that should be working or is working for someone else. > > * listNetworkACLs always returns an empty listnetworkaclsresponse > * not getting default ACLs, not getting any with new tiers/vms, > however setting ACLs via API works > * password server is broken (saw the other thread where it's a dev > binding issue... is someone actively looking at this?) > * static route button in UI doesn't do anything when clicked (I would > expect at least a dialogue or a fail message because the > SetStaticRouteCommand is missing) That's ok, seems all of these "issues" are on the management server side. Alena, could you help to take a look at, are they real issue? > > > On Fri, Aug 31, 2012 at 1:40 PM, Edison Su <edison...@citrix.com> wrote: > > > > > >> -----Original Message----- > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> Sent: Thursday, August 30, 2012 10:21 PM > >> To: Edison Su > >> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; > >> Vijayendra Bhamidipati > >> Subject: Re: VPC for KVM (was VM router spawning multiple public > nics) > >> > >> I've made some good progress today, I'll probably have it all > working > >> tomorrow, but I have a question. Looking at the API command > >> associateIpAddress and the functional spec, if I pass a vpcId it > then > >> calls associateIPToVpc. Unlike its brother, associateIPToNetwork, I > >> don't clearly see where the ip is actually put on the router, I only > >> see it marking the ip allocated in the database. The functional spec > >> reads as though this IP should be on the VPC. Maybe I'm just not > >> getting that bolded statement 'All ip addresses get allocated to the > >> VPC', my first thought is that of course they would, any IP I > allocate > >> gets put on the router... I guess I just need to play with it a bit > >> more. > > > > After reading the functional spec, my guess is that: > > One VPC has multiple guest networks(one guest network is a tenant, > e.g VLAN). On the guest network, there are a lot of VMs running on it. > > All of these VMs will use ip address range allocated from guest > network(createnetwork api), usually they are using internal ip address. > > Regarding to these commands: > > 1. AssociateIPAddrCmd > > "in VPC setup as a result of associateIpAddress all ip addresses get > allocated to VPC " means public ip address(when creating a zone, you'd > specify public ip address range) are belong to a VPC, you can associate > a public address to a guest network which attached to a VPC. > > In the router, it will add a public ip address on an ethX(which is > created on public.network.device on kvm host), add a routing > table(routing guest network gateway to that public ip). So after that, > user VMs created on that guest network should be able to access public > network through that public ip. > > 2. There are other commands(netwrok), which do some magic inside > router vm, basically implement the features like, who can access this > guest network, how to set up a portforwording rule for guest network > etc. > > > >> > >> Not in front of the code at the moment, I just thought of a few more > >> quick questions that would help... What is the expected outcome of > the > >> default rules? no access between private networks, but public access > >> for every network? > > > > By default, all incoming traffic to guest networks is blocked, but > outgoing traffic should work. > > > >> > >> The functional spec mentions public and private gateways, does > public > >> refer to giving someone say a /24 of public address space and then > >> creating a route on our router to their VPC router, i.e. public > >> networks instead of NATed ones? > > > > All of these gateways are referring to gateways inside router VM, not > any physical router/gateway. > > The public refer to the network created on public.network.device on > your kvm host. > > The private gateways referring to network created on > guest.network.device on your kvm host. > > In a VPC setup, the router can have multiple guest network(a.k. > multiple private gateways) and multiple public networks. > > You can call AssociateIPAddrCmd to bind a public ip to a guest > network, then can implement portforwording etc. > > > > I am not VPC expert, my answer is not authoritative as it looks like:) > Correct me if I am wrong. > > > >> > >> On Thu, Aug 30, 2012 at 11:54 AM, Marcus Sorensen > <shadow...@gmail.com> > >> wrote: > >> > Yes, thanks for clarifying. > >> > > >> > On Thu, Aug 30, 2012 at 11:53 AM, Edison Su <edison...@citrix.com> > >> wrote: > >> >> Oh, I see, there are two kinds of commands: > >> >> One doesn't need to get information from libvirt, which just > login > >> into router vm and program rules. For these commands, we can put > them > >> into virtualRoutingResource. > >> >> Another does need to get infor from libvirt, such ipassoc, which > >> need to get nic infor from libvirt, and plug in vif if needed. For > >> these commands, need to be implemented in libvirtcomputingresource. > >> While if it needs to access/program router, then add put that code > in > >> virtualroutingresource. > >> >> > >> >> Does it make sense? > >> >> > >> >>> -----Original Message----- > >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >>> Sent: Thursday, August 30, 2012 10:47 AM > >> >>> To: Edison Su > >> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; > >> >>> Vijayendra Bhamidipati > >> >>> Subject: Re: VPC for KVM (was VM router spawning multiple public > >> nics) > >> >>> > >> >>> Ok, I think I get it. For example when I'm looking at > >> IpAssocCommand > >> >>> it calls _virtRouterResource.assignPublicIpAddress to actually > run > >> the > >> >>> sh. So with SetupGuestNetworkCommand, I'll create that in > >> >>> LibvirtComputingResource, and within that I'll make a call to > >> >>> VirtualRoutingResource to do the work... is that right? > >> >>> > >> >>> On Thu, Aug 30, 2012 at 11:31 AM, Marcus Sorensen > >> <shadow...@gmail.com> > >> >>> wrote: > >> >>> > Just want to clarify: > >> >>> > > >> >>> > "We can add implementation of > >> >>> > > >> >>> > >> > SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2 > >> >>> SiteVpnCfgCommand > >> >>> > in VirtualRoutingResource." > >> >>> > > >> >>> > So the Xen implementation of SetupGuestNetworkCommand is in > >> >>> > CitrixResourceBase.java, and the VMware one is in > >> VmwareResource.java, > >> >>> > but you're saying I should put an implementation of these in > >> >>> > VirtualRoutingResource.java for KVM? Sorry, still trying to > piece > >> >>> > together how everything relates. > >> >>> > > >> >>> > On Thu, Aug 30, 2012 at 10:42 AM, Edison Su > >> <edison...@citrix.com> > >> >>> wrote: > >> >>> >> Yes, it is. > >> >>> >> > >> >>> >>> -----Original Message----- > >> >>> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >>> >>> Sent: Thursday, August 30, 2012 8:45 AM > >> >>> >>> To: Edison Su > >> >>> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven > >> Yang; > >> >>> >>> Vijayendra Bhamidipati > >> >>> >>> Subject: VPC for KVM (was VM router spawning multiple public > >> nics) > >> >>> >>> > >> >>> >>> I notice there's also an 'IpAssocVpcCommand', I'm assuming > that > >> >>> should > >> >>> >>> be added to the list? > >> >>> >>> > >> >>> >>> On Wed, Aug 29, 2012 at 6:37 PM, Edison Su > >> <edison...@citrix.com> > >> >>> wrote: > >> >>> >>> > You can find the VPC reference implementation from > >> >>> >>> CitrixResourceBase.java, which is the implementation for > >> Xenserver. > >> >>> >>> Just take a look at how the VPC related commands are > >> implemented. > >> >>> >>> > Take SetNetworkACLCommand as an example: > >> >>> >>> > The function execute(SetNetworkACLCommand cmd) in > >> >>> citrixResourceBase: > >> >>> >>> > 1. parse SetNetworkACLCommand > >> >>> >>> > 2. call scripts/vm/hypervisor/xenserver/vmops, function > >> >>> routerProxy > >> >>> >>> > 3. routerproxy will call > scripts/network/domr/router_proxy.sh > >> >>> >>> > 4. router_proxy.sh will login into router vm, execute a > shell > >> >>> script > >> >>> >>> inside router VM to program rules. > >> >>> >>> > > >> >>> >>> > In KVM, we can directly call router_proxy.sh or directly > >> login > >> >>> into > >> >>> >>> router vm, we just need to prepare the parameters for this > >> script. > >> >>> >>> > The reference code is in VirtualRoutingResource.java, all > the > >> >>> network > >> >>> >>> related command(extended from NetworkElementCommand) will be > >> >>> redirected > >> >>> >>> to VirtualRoutingResource. > >> >>> >>> > We can add implementation of > >> >>> >>> > >> >>> > >> > SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2 > >> >>> >>> SiteVpnCfgCommand in VirtualRoutingResource. > >> >>> >>> > > >> >>> >>> > > >> >>> >>> >> -----Original Message----- > >> >>> >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >>> >>> >> Sent: Wednesday, August 29, 2012 4:34 PM > >> >>> >>> >> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra > >> Bhamidipati > >> >>> >>> >> Subject: Re: VM router spawning multiple public nics > >> >>> >>> >> > >> >>> >>> >> Great, thanks. I've already got a working implementation > of > >> >>> >>> >> adding/removing nics from regular instances that I've > been > >> >>> playing > >> >>> >>> >> with, so I'm getting vaguely familiar with the various > data > >> >>> types > >> >>> >>> and > >> >>> >>> >> things surrounding the networking. I don't know if this > is > >> quite > >> >>> >>> >> within my reach just yet but I'll see how far I get. > >> >>> >>> >> > >> >>> >>> >> On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk > >> >>> >>> >> <alena.prokharc...@citrix.com> wrote: > >> >>> >>> >> > On 8/29/12 1:34 PM, "Edison Su" <edison...@citrix.com> > >> wrote: > >> >>> >>> >> > > >> >>> >>> >> >>Hi Anthony & Alena, > >> >>> >>> >> >> Could you help to provide information about VPC, how > it > >> >>> works, > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > Here is the functional spec on the feature: > >> >>> >>> >> > > >> >>> >>> >> > wiki.cloudstack.org/display/RelOps/Inter- > >> >>> >>> VLAN+Routing+functional+spec > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > VpcVirtualNetworkApplianceManagerImpl is the manager > >> >>> responsible > >> >>> >>> for > >> >>> >>> >> VPC > >> >>> >>> >> > Virtual router operations (plug/unplug nics, etc) > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> >> which commands needed to implemented on the hypervisor > >> side? > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > 1) PlugNicCommand/UnplugNicCommand - does Nic > >> hotplug/unplug > >> >>> >>> >> (currently > >> >>> >>> >> > works for VR vm only). In VPC called when add nic for > >> >>> Public/Guest > >> >>> >>> >> > networks. > >> >>> >>> >> > 2) SetupGuestNetworkCommand - sets up dhcp range, dns > >> >>> information, > >> >>> >>> >> > networkDomain information on the Nic to make it > >> >>> >>> >> > 3) SetNetworkACLCommand - creates network ACL on the > >> virtual > >> >>> >>> router > >> >>> >>> >> > 4) SetSourceNatCommand - used for setting source nat on > >> the > >> >>> Public > >> >>> >>> IP > >> >>> >>> >> on > >> >>> >>> >> > the VPC VR. > >> >>> >>> >> > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > Anthony/Kelven/Vijay did implementation for Xen/vmWare > >> >>> resources, > >> >>> >>> >> they can > >> >>> >>> >> > help you answering all hypervisor related questions. If > >> you > >> >>> need > >> >>> >>> more > >> >>> >>> >> > details on business logic + Vpc VR management, I can > help > >> with > >> >>> >>> that. > >> >>> >>> >> > > >> >>> >>> >> > > >> >>> >>> >> > -Alena. > >> >>> >>> >> > > >> >>> >>> >> >> > >> >>> >>> >> >>> -----Original Message----- > >> >>> >>> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> Sent: Wednesday, August 29, 2012 10:16 AM > >> >>> >>> >> >>> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> >>> Subject: Re: VM router spawning multiple public nics > >> >>> >>> >> >>> > >> >>> >>> >> >>> I'd be willing to give it a shot if someone could > point > >> me > >> >>> in > >> >>> >>> the > >> >>> >>> >> >>> right direction and be available to answer questions. > >> >>> >>> >> >>> > >> >>> >>> >> >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su > >> >>> >>> <edison...@citrix.com> > >> >>> >>> >> >>> wrote: > >> >>> >>> >> >>> > Yah, KVM doesn't support VPC yet. Will you help to > add > >> VPC > >> >>> >>> >> support on > >> >>> >>> >> >>> KVM?:) Just implement a few VPC related commands... > >> >>> >>> >> >>> > > >> >>> >>> >> >>> >> -----Original Message----- > >> >>> >>> >> >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> >> Sent: Wednesday, August 29, 2012 6:49 AM > >> >>> >>> >> >>> >> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> >>> >> Subject: Re: VM router spawning multiple public > nics > >> >>> >>> >> >>> >> > >> >>> >>> >> >>> >> I can confirm that the patch has fixed my > particular > >> >>> issue. > >> >>> >>> >> >>> >> > >> >>> >>> >> >>> >> This is likely unrelated and I think it doesn't > even > >> use > >> >>> the > >> >>> >>> >> same > >> >>> >>> >> >>> >> code, but I began to play with the VPC stuff a bit > >> and > >> >>> >>> noticed > >> >>> >>> >> that > >> >>> >>> >> >>> I > >> >>> >>> >> >>> >> don't get any interfaces except for link local. > I'd > >> >>> probably > >> >>> >>> >> chalk > >> >>> >>> >> >>> >> that up to it not being ready for KVM, but I > thought > >> it > >> >>> was > >> >>> >>> >> worth a > >> >>> >>> >> >>> >> mention. I'd be happy to try to help get it ready > if > >> >>> someone > >> >>> >>> >> has > >> >>> >>> >> >>> time > >> >>> >>> >> >>> >> to nudge me in the right direction. > >> >>> >>> >> >>> >> > >> >>> >>> >> >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su > >> >>> >>> >> <edison...@citrix.com> > >> >>> >>> >> >>> wrote: > >> >>> >>> >> >>> >> > > >> >>> >>> >> >>> >> > > >> >>> >>> >> >>> >> >> -----Original Message----- > >> >>> >>> >> >>> >> >> From: Marcus Sorensen > [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM > >> >>> >>> >> >>> >> >> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> >>> >> >> Subject: Re: VM router spawning multiple public > >> nics > >> >>> >>> >> >>> >> >> > >> >>> >>> >> >>> >> >> I thought about this solution myself, but > below > >> this > >> >>> >>> portion > >> >>> >>> >> of > >> >>> >>> >> >>> >> code > >> >>> >>> >> >>> >> >> it looks like it uses the hash map to determine > >> which > >> >>> nic > >> >>> >>> >> number > >> >>> >>> >> >>> to > >> >>> >>> >> >>> >> >> add the IP to, so with multiple 'untagged' > >> networks it > >> >>> >>> would > >> >>> >>> >> have > >> >>> >>> >> >>> no > >> >>> >>> >> >>> >> >> way of knowing which nicnum in the router > >> corresponds > >> >>> with > >> >>> >>> >> the > >> >>> >>> >> >>> >> correct > >> >>> >>> >> >>> >> >> untagged vlan. > >> >>> >>> >> >>> >> >> > >> >>> >>> >> >>> >> >> nicNum = > >> >>> >>> >> vlanAllocatedToVM.get(ip.getVlanId()); > >> >>> >>> >> >>> >> >> networkUsage(routerIp, "addVif", > >> "eth" > >> >>> + > >> >>> >>> >> nicNum); > >> >>> >>> >> >>> >> >> result = > >> >>> >>> >> >>> >> >> > >> _virtRouterResource.assignPublicIpAddress(routerName, > >> >>> >>> >> >>> >> >> routerIp, > ip.getPublicIp(), > >> >>> >>> >> ip.isAdd(), > >> >>> >>> >> >>> >> >> ip.isFirstIP(), > >> >>> >>> >> >>> >> >> ip.isSourceNat(), > >> >>> ip.getVlanId(), > >> >>> >>> >> >>> >> >> ip.getVlanGateway(), > >> >>> >>> >> >>> >> >> ip.getVlanNetmask(), > >> >>> >>> >> >>> ip.getVifMacAddress(), > >> >>> >>> >> >>> >> >> ip.getGuestIp(), > nicNum); > >> >>> >>> >> >>> >> >> > >> >>> >>> >> >>> >> >> if ip.getVlanId() returns untagged (as it does > on > >> >>> networks > >> >>> >>> >> with > >> >>> >>> >> >>> no > >> >>> >>> >> >>> >> >> vlan id), and we tried to put multiple untagged > >> keys > >> >>> in > >> >>> >>> >> >>> >> >> vlanAllocatedToVM (as with multiple untagged > >> networks), > >> >>> we > >> >>> >>> >> get > >> >>> >>> >> >>> the > >> >>> >>> >> >>> >> >> wrong nicNum, no? > >> >>> >>> >> >>> >> > > >> >>> >>> >> >>> >> > In the ipassoc case, if there are multiple > untagged > >> >>> >>> networks, > >> >>> >>> >> all > >> >>> >>> >> >>> of > >> >>> >>> >> >>> >> them are use the same > >> >>> >>> >> >>> >> > Public bridge. Then multiple ip address will be > >> added > >> >>> on > >> >>> >>> eth2 > >> >>> >>> >> >>> inside > >> >>> >>> >> >>> >> router VM. > >> >>> >>> >> >>> >> > If it works physically, then it works. > >> >>> >>> >> >>> >> > > >> >>> >>> >> >>> >> >> > >> >>> >>> >> >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su > >> >>> >>> >> <edison...@citrix.com> > >> >>> >>> >> >>> >> wrote: > >> >>> >>> >> >>> >> >> > > >> >>> >>> >> >>> >> >> > > >> >>> >>> >> >>> >> >> >> -----Original Message----- > >> >>> >>> >> >>> >> >> >> From: Marcus Sorensen > >> [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM > >> >>> >>> >> >>> >> >> >> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> >>> >> >> >> Subject: Re: VM router spawning multiple > public > >> >>> nics > >> >>> >>> >> >>> >> >> >> > >> >>> >>> >> >>> >> >> >> Yes, that looks like it would work for me, > >> however > >> >>> >>> that's > >> >>> >>> >> not > >> >>> >>> >> >>> >> >> >> something that would ever make it into > master, > >> >>> right? > >> >>> >>> >> >>> Essentially > >> >>> >>> >> >>> >> >> >> killing tagging for the public, private, and > >> guest > >> >>> >>> traffic > >> >>> >>> >> >>> labels? > >> >>> >>> >> >>> >> >> >> There's also still the issue of not being > able > >> to > >> >>> >>> >> >>> differentiate > >> >>> >>> >> >>> >> >> >> between multiple untagged networks, if we > >> wanted to > >> >>> add > >> >>> >>> an > >> >>> >>> >> IP > >> >>> >>> >> >>> to > >> >>> >>> >> >>> >> a > >> >>> >>> >> >>> >> >> >> router it might not know which untagged > >> interface > >> >>> to > >> >>> >>> apply > >> >>> >>> >> it > >> >>> >>> >> >>> to. > >> >>> >>> >> >>> >> >> > > >> >>> >>> >> >>> >> >> > Physically, all the "untagged" network will > be > >> >>> created > >> >>> >>> on > >> >>> >>> >> >>> >> >> public/guest/private bridge(the name we put in > >> >>> >>> >> >>> >> >> private/public/guest.bridge.name in > >> agent.properties"). > >> >>> >>> >> >>> >> >> > Because, there is no way to create a new > >> untagged > >> >>> bridge > >> >>> >>> by > >> >>> >>> >> >>> agent > >> >>> >>> >> >>> >> >> itself. Agent code only knows how to create a > new > >> >>> >>> tagged(vlan) > >> >>> >>> >> >>> >> bridge. > >> >>> >>> >> >>> >> >> > So the fix should be pushed into master. > >> >>> >>> >> >>> >> >> > > >> >>> >>> >> >>> >> >> >> > >> >>> >>> >> >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su > >> >>> >>> >> >>> <edison...@citrix.com> > >> >>> >>> >> >>> >> >> wrote: > >> >>> >>> >> >>> >> >> >> > > >> >>> >>> >> >>> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> -----Original Message----- > >> >>> >>> >> >>> >> >> >> >> From: Marcus Sorensen > >> >>> [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM > >> >>> >>> >> >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org > >> >>> >>> >> >>> >> >> >> >> Subject: Re: VM router spawning multiple > >> public > >> >>> nics > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> >> >>> >> >> >> >> Thanks for pointing me in the right > >> direction. > >> >>> I've > >> >>> >>> >> >>> reviewed > >> >>> >>> >> >>> >> this > >> >>> >>> >> >>> >> >> >> code > >> >>> >>> >> >>> >> >> >> >> in a bit more detail, and it seems like > it's > >> >>> >>> >> accomplishing > >> >>> >>> >> >>> the > >> >>> >>> >> >>> >> >> >> >> following: > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> >> >>> >> >> >> >> 1. get all network interfaces currently > >> >>> connected to > >> >>> >>> >> the > >> >>> >>> >> >>> >> running > >> >>> >>> >> >>> >> >> VM > >> >>> >>> >> >>> >> >> >> >> (a.k.a vnet devices via libvirt) > >> >>> >>> >> >>> >> >> >> >> 2. find out which vlans these network > >> interfaces > >> >>> are > >> >>> >>> >> >>> bridged > >> >>> >>> >> >>> >> to, > >> >>> >>> >> >>> >> >> >> store > >> >>> >>> >> >>> >> >> >> >> this in a hash map of vlan ids and nics > >> >>> >>> >> >>> >> >> >> >> 3. get all ip addresses to be added to > the > >> VM > >> >>> >>> >> >>> >> >> >> >> 4. for each ip, get the configured vlan > id > >> for > >> >>> the > >> >>> >>> ip, > >> >>> >>> >> >>> compare > >> >>> >>> >> >>> >> it > >> >>> >>> >> >>> >> >> to > >> >>> >>> >> >>> >> >> >> >> the hash map of existing vlan ids and > nics > >> >>> >>> >> >>> >> >> >> >> 5. if the required vlan id is not found > in > >> the > >> >>> hash > >> >>> >>> map, > >> >>> >>> >> >>> >> create a > >> >>> >>> >> >>> >> >> >> new > >> >>> >>> >> >>> >> >> >> >> nic > >> >>> >>> >> >>> >> >> >> >> 6. assign the ip to the nic identified by > >> the > >> >>> vlan > >> >>> >>> id > >> >>> >>> >> key > >> >>> >>> >> >>> in > >> >>> >>> >> >>> >> the > >> >>> >>> >> >>> >> >> >> hash > >> >>> >>> >> >>> >> >> >> >> map > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> >> >>> >> >> >> >> In this case, we're getting a vlan id > >> returned > >> >>> in > >> >>> >>> step > >> >>> >>> >> 2 > >> >>> >>> >> >>> for a > >> >>> >>> >> >>> >> >> >> bridged > >> >>> >>> >> >>> >> >> >> >> nic whose network is defined as untagged > in > >> the > >> >>> >>> >> cloudstack > >> >>> >>> >> >>> db, > >> >>> >>> >> >>> >> >> >> >> therefore in step 5 we never match as > >> already > >> >>> having > >> >>> >>> a > >> >>> >>> >> nic > >> >>> >>> >> >>> for > >> >>> >>> >> >>> >> >> >> >> 'untagged'. I wrote a big long response > >> >>> discussing > >> >>> >>> this > >> >>> >>> >> >>> issue, > >> >>> >>> >> >>> >> >> but > >> >>> >>> >> >>> >> >> >> as > >> >>> >>> >> >>> >> >> >> >> I began to dig further I realized that > aside > >> >>> from my > >> >>> >>> >> >>> >> particular > >> >>> >>> >> >>> >> >> case, > >> >>> >>> >> >>> >> >> >> >> untagged vlans in general are just broken > >> (for > >> >>> >>> example > >> >>> >>> >> they > >> >>> >>> >> >>> >> can't > >> >>> >>> >> >>> >> >> be > >> >>> >>> >> >>> >> >> >> >> dealt with uniquely in the current > >> >>> IpAssocCommand > >> >>> >>> code, > >> >>> >>> >> >>> given > >> >>> >>> >> >>> >> the > >> >>> >>> >> >>> >> >> >> hash > >> >>> >>> >> >>> >> >> >> >> map) and it would require more effort > than I > >> >>> have > >> >>> >>> time > >> >>> >>> >> for > >> >>> >>> >> >>> now > >> >>> >>> >> >>> >> to > >> >>> >>> >> >>> >> >> >> make > >> >>> >>> >> >>> >> >> >> >> things work. If the code were already in > >> place > >> >>> to > >> >>> >>> >> >>> >> differentiate > >> >>> >>> >> >>> >> >> >> >> between multiple untagged nics I think > that > >> >>> fixing > >> >>> >>> my > >> >>> >>> >> >>> problem > >> >>> >>> >> >>> >> >> would > >> >>> >>> >> >>> >> >> >> be > >> >>> >>> >> >>> >> >> >> >> trivial, but since its not, I'll just > find > >> an > >> >>> >>> >> alternative > >> >>> >>> >> >>> >> >> solution. > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> >> >>> >> >> >> > > >> >>> >>> >> >>> >> >> >> > The untagged network usually means > "untagged", > >> no > >> >>> >>> vlan > >> >>> >>> >> on > >> >>> >>> >> >>> the > >> >>> >>> >> >>> >> >> >> bridge... > >> >>> >>> >> >>> >> >> >> > In your case, the untagged network > actually > >> has > >> >>> >>> >> vlan(tagged) > >> >>> >>> >> >>> on > >> >>> >>> >> >>> >> >> the > >> >>> >>> >> >>> >> >> >> bridge, thus getting things confused. > >> >>> >>> >> >>> >> >> >> > Will this > patch(http://pastebin.com/HJXzZwKp) > >> >>> work > >> >>> >>> for > >> >>> >>> >> you? > >> >>> >>> >> >>> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus > >> >>> Sorensen > >> >>> >>> >> >>> >> >> >> <shadow...@gmail.com> > >> >>> >>> >> >>> >> >> >> >> wrote: > >> >>> >>> >> >>> >> >> >> >> > ... > >> >>> >>> >> >>> >> >> >> >> > Integer nicPos = 0; > >> >>> >>> >> >>> >> >> >> >> > for (InterfaceDef nic : > nics) > >> { > >> >>> >>> >> >>> >> >> >> >> > if > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) > >> >>> >>> >> { > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> vlanAllocatedToVM.put("LinkLocal", > >> >>> >>> >> >>> >> nicPos); > >> >>> >>> >> >>> >> >> >> >> > } else { > >> >>> >>> >> >>> >> >> >> >> > String vlanId = > >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >> >>> >>> >> >>> >> >> >> >> > if (vlanId != null) > { > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> vlanAllocatedToVM.put(vlanId, > >> >>> >>> >> >>> >> nicPos); > >> >>> >>> >> >>> >> >> >> >> > } else { > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> >>> >>> >> >>> >> >> >> nicPos); > >> >>> >>> >> >>> >> >> >> >> > } > >> >>> >>> >> >>> >> >> >> >> > } > >> >>> >>> >> >>> >> >> >> >> > nicPos++; > >> >>> >>> >> >>> >> >> >> >> > } > >> >>> >>> >> >>> >> >> >> >> > IpAddressTO[] ips = > >> >>> >>> cmd.getIpAddresses(); > >> >>> >>> >> >>> >> >> >> >> > int i = 0; > >> >>> >>> >> >>> >> >> >> >> > String result = null; > >> >>> >>> >> >>> >> >> >> >> > int nicNum = 0; > >> >>> >>> >> >>> >> >> >> >> > for (IpAddressTO ip : ips) > { > >> >>> >>> >> >>> >> >> >> >> > if > >> >>> >>> >> >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) > >> >>> >>> >> >>> >> >> >> { > >> >>> >>> >> >>> >> >> >> >> > /* plug a vif into > >> router > >> >>> */ > >> >>> >>> >> >>> >> >> >> >> > VifHotPlug(conn, > >> >>> routerName, > >> >>> >>> >> >>> >> ip.getVlanId(), > >> >>> >>> >> >>> >> >> >> >> > > >> >>> ip.getVifMacAddress()); > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> vlanAllocatedToVM.put(ip.getVlanId(), > >> >>> >>> >> >>> >> >> >> nicPos++); > >> >>> >>> >> >>> >> >> >> >> > } > >> >>> >>> >> >>> >> >> >> >> > ... > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > Looks like the getVlanIdFromBridge > might > >> be a > >> >>> bit > >> >>> >>> >> >>> misleading. > >> >>> >>> >> >>> >> I > >> >>> >>> >> >>> >> >> am > >> >>> >>> >> >>> >> >> >> >> > running my guest public traffic on a > >> >>> 'cloudbr470', > >> >>> >>> >> which > >> >>> >>> >> >>> is > >> >>> >>> >> >>> >> a > >> >>> >>> >> >>> >> >> >> bridge > >> >>> >>> >> >>> >> >> >> >> > to eth2.470, yet I configured this > network > >> as > >> >>> >>> >> 'untagged' > >> >>> >>> >> >>> >> >> because I > >> >>> >>> >> >>> >> >> >> >> > have a vlan 470 available on eth3 for > >> >>> cloudstack > >> >>> >>> to > >> >>> >>> >> >>> >> autoassign > >> >>> >>> >> >>> >> >> >> (eth3 > >> >>> >>> >> >>> >> >> >> >> > is where all of my stuff will be > >> autoassigned). > >> >>> So > >> >>> >>> >> I'm > >> >>> >>> >> >>> not > >> >>> >>> >> >>> >> 100% > >> >>> >>> >> >>> >> >> >> sure > >> >>> >>> >> >>> >> >> >> >> > yet what's going on here but it seems > as > >> >>> though > >> >>> >>> the > >> >>> >>> >> above > >> >>> >>> >> >>> is > >> >>> >>> >> >>> >> >> not > >> >>> >>> >> >>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it > >> finds a > >> >>> vlan > >> >>> >>> >> number > >> >>> >>> >> >>> >> for > >> >>> >>> >> >>> >> >> >> >> > eth2.470, but when it enumerates the > IPs > >> for > >> >>> the > >> >>> >>> >> router, > >> >>> >>> >> >>> it > >> >>> >>> >> >>> >> >> then > >> >>> >>> >> >>> >> >> >> runs > >> >>> >>> >> >>> >> >> >> >> > ip.getVlanId() and doesn't find a nic > for > >> the > >> >>> >>> >> untagged IP > >> >>> >>> >> >>> >> and > >> >>> >>> >> >>> >> >> >> creates > >> >>> >>> >> >>> >> >> >> >> > one. > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > I realize this is perhaps an uncommon > case, > >> >>> but a > >> >>> >>> bug > >> >>> >>> >> >>> >> >> nonetheless. > >> >>> >>> >> >>> >> >> >> >> > I'll play with the code a bit and see > if I > >> can > >> >>> >>> come > >> >>> >>> >> up > >> >>> >>> >> >>> with > >> >>> >>> >> >>> >> a > >> >>> >>> >> >>> >> >> >> >> > solution. I'm thinking I can look at > the > >> nic's > >> >>> >>> >> broadcast > >> >>> >>> >> >>> URI > >> >>> >>> >> >>> >> >> and > >> >>> >>> >> >>> >> >> >> see > >> >>> >>> >> >>> >> >> >> >> > if it's supposed to be untagged, then > add > >> to > >> >>> >>> >> >>> >> vlanAllocatedToVM > >> >>> >>> >> >>> >> >> >> >> > appropriately, off the top of my head > >> >>> something > >> >>> >>> like: > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > String vlanId = > >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); > >> >>> >>> >> >>> >> >> >> >> > if (vlanId != null > && > >> >>> >>> >> >>> >> >> >> >> > >> >>> >>> > !nic.getBroadcastUri().toString().contains("untagged") > >> >>> >>> >> { > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> vlanAllocatedToVM.put(vlanId, > >> >>> >>> >> >>> >> nicPos); > >> >>> >>> >> >>> >> >> >> >> > } else { > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, > >> >>> >>> >> >>> >> >> >> nicPos); > >> >>> >>> >> >>> >> >> >> >> > } > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > > >> >>> >>> >> >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison > Su > >> >>> >>> >> >>> >> >> <edison...@citrix.com> > >> >>> >>> >> >>> >> >> >> >> wrote: > >> >>> >>> >> >>> >> >> >> >> >> Possible bug in in kvm code: > >> >>> >>> >> LibvirtComputingResource- > >> >>> >>> >> >>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, > >> which > >> >>> is > >> >>> >>> >> only > >> >>> >>> >> >>> place > >> >>> >>> >> >>> >> >> >> adding > >> >>> >>> >> >>> >> >> >> >> nic into router vm. > >> >>> >>> >> >>> >> >> >> >> >> Turn on agent log, then take a look > what > >> >>> happened. > >> >>> >>> >> >>> >> >> >> >> >> > >> >>> >>> >> >>> >> >> >> >> >>> -----Original Message----- > >> >>> >>> >> >>> >> >> >> >> >>> From: Marcus Sorensen > >> >>> >>> [mailto:shadow...@gmail.com] > >> >>> >>> >> >>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM > >> >>> >>> >> >>> >> >> >> >> >>> To: cloudstack- > d...@incubator.apache.org > >> >>> >>> >> >>> >> >> >> >> >>> Subject: VM router spawning multiple > >> public > >> >>> nics > >> >>> >>> >> >>> >> >> >> >> >>> > >> >>> >>> >> >>> >> >> >> >> >>> I've got two zones running the same > >> build of > >> >>> >>> >> cloudstack > >> >>> >>> >> >>> (a > >> >>> >>> >> >>> >> >> >> recent > >> >>> >>> >> >>> >> >> >> >> copy > >> >>> >>> >> >>> >> >> >> >> >>> of master). One of them creates > routers > >> that > >> >>> >>> turn > >> >>> >>> >> into > >> >>> >>> >> >>> >> ugly > >> >>> >>> >> >>> >> >> >> >> >>> multi-headed beasts, and by that I > mean > >> that > >> >>> any > >> >>> >>> >> time I > >> >>> >>> >> >>> >> >> create a > >> >>> >>> >> >>> >> >> >> >> port > >> >>> >>> >> >>> >> >> >> >> >>> forwarding or iptables rule for that > >> router > >> >>> I > >> >>> >>> get a > >> >>> >>> >> new > >> >>> >>> >> >>> >> >> public > >> >>> >>> >> >>> >> >> >> NIC > >> >>> >>> >> >>> >> >> >> >> >>> with an identical IP address, I have > an > >> >>> instance > >> >>> >>> >> with a > >> >>> >>> >> >>> >> few > >> >>> >>> >> >>> >> >> tens > >> >>> >>> >> >>> >> >> >> of > >> >>> >>> >> >>> >> >> >> >> >>> NICs. My guess is that some script > >> isn't > >> >>> >>> detecting > >> >>> >>> >> >>> that > >> >>> >>> >> >>> >> >> there's > >> >>> >>> >> >>> >> >> >> >> >>> already a NIC with the public IP on > it. > >> It > >> >>> >>> looks > >> >>> >>> >> fine > >> >>> >>> >> >>> in > >> >>> >>> >> >>> >> the > >> >>> >>> >> >>> >> >> >> >> >>> database, there is only one public > NIC > >> >>> defined > >> >>> >>> in > >> >>> >>> >> the > >> >>> >>> >> >>> nics > >> >>> >>> >> >>> >> >> table. > >> >>> >>> >> >>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if > >> anyone > >> >>> >>> knows > >> >>> >>> >> >>> where I > >> >>> >>> >> >>> >> >> >> should > >> >>> >>> >> >>> >> >> >> >> >>> begin the headstart would be > appreciated. > >> >>> >>> >> >>> >> >> >> >> >>> > >> >>> >>> >> >>> >> >> >> >> >>> Thanks > >> >>> >>> >> >> > >> >>> >>> >> > > >> >>> >>> >> >
