Ok, the SetPortForwardingRulesVpcCommand and SetStaticRouteCommand are working and have been added to the review request.
I'm going to generate a new e-mail on various non-platform-specific issues I've come across. I really haven't seen much on the current state of Vpc in general. On Fri, Aug 31, 2012 at 3:22 PM, Edison Su <edison...@citrix.com> wrote: > > >> -----Original Message----- >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> Sent: Friday, August 31, 2012 1:58 PM >> To: Edison Su >> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; >> Vijayendra Bhamidipati >> Subject: Re: VPC for KVM (was VM router spawning multiple public nics) >> >> Ok, I've gotten to most of the list and things seem to be working. >> Nics plug, Gateway IPs add, networks create, SNATs working, ACLs >> working. The ones I haven't gotten to are: >> >> *Site2SiteVpnCfgCommand >> >> and two which weren't on the original list that I just came across >> (will check for others): >> *SetPortForwardingRulesVpcCommand >> *CheckS2SVpnConnectionsCommand >> *SetStaticRouteCommand > > The commands related to site2site VPN can be finished later, as it needs more > complicated setup to test. > But I'd like to get SetPortForwardingRulesVpcCommand and > SetStaticRouteCommand working. > >> >> In addition, the following observations have been made and not yet >> fixed. If anyone has feedback on these (even a 'yeah, that doesn't >> work', or 'hmm, that should be working') so I can gauge whether I >> should dig into them or not I'd appreciate it. I don't want to break >> something that should be working or is working for someone else. >> >> * listNetworkACLs always returns an empty listnetworkaclsresponse >> * not getting default ACLs, not getting any with new tiers/vms, >> however setting ACLs via API works >> * password server is broken (saw the other thread where it's a dev >> binding issue... is someone actively looking at this?) >> * static route button in UI doesn't do anything when clicked (I would >> expect at least a dialogue or a fail message because the >> SetStaticRouteCommand is missing) > > That's ok, seems all of these "issues" are on the management server side. > Alena, could you help to take a look at, are they real issue? > >> >> >> On Fri, Aug 31, 2012 at 1:40 PM, Edison Su <edison...@citrix.com> wrote: >> > >> > >> >> -----Original Message----- >> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> Sent: Thursday, August 30, 2012 10:21 PM >> >> To: Edison Su >> >> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; >> >> Vijayendra Bhamidipati >> >> Subject: Re: VPC for KVM (was VM router spawning multiple public >> nics) >> >> >> >> I've made some good progress today, I'll probably have it all >> working >> >> tomorrow, but I have a question. Looking at the API command >> >> associateIpAddress and the functional spec, if I pass a vpcId it >> then >> >> calls associateIPToVpc. Unlike its brother, associateIPToNetwork, I >> >> don't clearly see where the ip is actually put on the router, I only >> >> see it marking the ip allocated in the database. The functional spec >> >> reads as though this IP should be on the VPC. Maybe I'm just not >> >> getting that bolded statement 'All ip addresses get allocated to the >> >> VPC', my first thought is that of course they would, any IP I >> allocate >> >> gets put on the router... I guess I just need to play with it a bit >> >> more. >> > >> > After reading the functional spec, my guess is that: >> > One VPC has multiple guest networks(one guest network is a tenant, >> e.g VLAN). On the guest network, there are a lot of VMs running on it. >> > All of these VMs will use ip address range allocated from guest >> network(createnetwork api), usually they are using internal ip address. >> > Regarding to these commands: >> > 1. AssociateIPAddrCmd >> > "in VPC setup as a result of associateIpAddress all ip addresses get >> allocated to VPC " means public ip address(when creating a zone, you'd >> specify public ip address range) are belong to a VPC, you can associate >> a public address to a guest network which attached to a VPC. >> > In the router, it will add a public ip address on an ethX(which is >> created on public.network.device on kvm host), add a routing >> table(routing guest network gateway to that public ip). So after that, >> user VMs created on that guest network should be able to access public >> network through that public ip. >> > 2. There are other commands(netwrok), which do some magic inside >> router vm, basically implement the features like, who can access this >> guest network, how to set up a portforwording rule for guest network >> etc. >> > >> >> >> >> Not in front of the code at the moment, I just thought of a few more >> >> quick questions that would help... What is the expected outcome of >> the >> >> default rules? no access between private networks, but public access >> >> for every network? >> > >> > By default, all incoming traffic to guest networks is blocked, but >> outgoing traffic should work. >> > >> >> >> >> The functional spec mentions public and private gateways, does >> public >> >> refer to giving someone say a /24 of public address space and then >> >> creating a route on our router to their VPC router, i.e. public >> >> networks instead of NATed ones? >> > >> > All of these gateways are referring to gateways inside router VM, not >> any physical router/gateway. >> > The public refer to the network created on public.network.device on >> your kvm host. >> > The private gateways referring to network created on >> guest.network.device on your kvm host. >> > In a VPC setup, the router can have multiple guest network(a.k. >> multiple private gateways) and multiple public networks. >> > You can call AssociateIPAddrCmd to bind a public ip to a guest >> network, then can implement portforwording etc. >> > >> > I am not VPC expert, my answer is not authoritative as it looks like:) >> Correct me if I am wrong. >> > >> >> >> >> On Thu, Aug 30, 2012 at 11:54 AM, Marcus Sorensen >> <shadow...@gmail.com> >> >> wrote: >> >> > Yes, thanks for clarifying. >> >> > >> >> > On Thu, Aug 30, 2012 at 11:53 AM, Edison Su <edison...@citrix.com> >> >> wrote: >> >> >> Oh, I see, there are two kinds of commands: >> >> >> One doesn't need to get information from libvirt, which just >> login >> >> into router vm and program rules. For these commands, we can put >> them >> >> into virtualRoutingResource. >> >> >> Another does need to get infor from libvirt, such ipassoc, which >> >> need to get nic infor from libvirt, and plug in vif if needed. For >> >> these commands, need to be implemented in libvirtcomputingresource. >> >> While if it needs to access/program router, then add put that code >> in >> >> virtualroutingresource. >> >> >> >> >> >> Does it make sense? >> >> >> >> >> >>> -----Original Message----- >> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> Sent: Thursday, August 30, 2012 10:47 AM >> >> >>> To: Edison Su >> >> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang; >> >> >>> Vijayendra Bhamidipati >> >> >>> Subject: Re: VPC for KVM (was VM router spawning multiple public >> >> nics) >> >> >>> >> >> >>> Ok, I think I get it. For example when I'm looking at >> >> IpAssocCommand >> >> >>> it calls _virtRouterResource.assignPublicIpAddress to actually >> run >> >> the >> >> >>> sh. So with SetupGuestNetworkCommand, I'll create that in >> >> >>> LibvirtComputingResource, and within that I'll make a call to >> >> >>> VirtualRoutingResource to do the work... is that right? >> >> >>> >> >> >>> On Thu, Aug 30, 2012 at 11:31 AM, Marcus Sorensen >> >> <shadow...@gmail.com> >> >> >>> wrote: >> >> >>> > Just want to clarify: >> >> >>> > >> >> >>> > "We can add implementation of >> >> >>> > >> >> >>> >> >> >> SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2 >> >> >>> SiteVpnCfgCommand >> >> >>> > in VirtualRoutingResource." >> >> >>> > >> >> >>> > So the Xen implementation of SetupGuestNetworkCommand is in >> >> >>> > CitrixResourceBase.java, and the VMware one is in >> >> VmwareResource.java, >> >> >>> > but you're saying I should put an implementation of these in >> >> >>> > VirtualRoutingResource.java for KVM? Sorry, still trying to >> piece >> >> >>> > together how everything relates. >> >> >>> > >> >> >>> > On Thu, Aug 30, 2012 at 10:42 AM, Edison Su >> >> <edison...@citrix.com> >> >> >>> wrote: >> >> >>> >> Yes, it is. >> >> >>> >> >> >> >>> >>> -----Original Message----- >> >> >>> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >>> Sent: Thursday, August 30, 2012 8:45 AM >> >> >>> >>> To: Edison Su >> >> >>> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven >> >> Yang; >> >> >>> >>> Vijayendra Bhamidipati >> >> >>> >>> Subject: VPC for KVM (was VM router spawning multiple public >> >> nics) >> >> >>> >>> >> >> >>> >>> I notice there's also an 'IpAssocVpcCommand', I'm assuming >> that >> >> >>> should >> >> >>> >>> be added to the list? >> >> >>> >>> >> >> >>> >>> On Wed, Aug 29, 2012 at 6:37 PM, Edison Su >> >> <edison...@citrix.com> >> >> >>> wrote: >> >> >>> >>> > You can find the VPC reference implementation from >> >> >>> >>> CitrixResourceBase.java, which is the implementation for >> >> Xenserver. >> >> >>> >>> Just take a look at how the VPC related commands are >> >> implemented. >> >> >>> >>> > Take SetNetworkACLCommand as an example: >> >> >>> >>> > The function execute(SetNetworkACLCommand cmd) in >> >> >>> citrixResourceBase: >> >> >>> >>> > 1. parse SetNetworkACLCommand >> >> >>> >>> > 2. call scripts/vm/hypervisor/xenserver/vmops, function >> >> >>> routerProxy >> >> >>> >>> > 3. routerproxy will call >> scripts/network/domr/router_proxy.sh >> >> >>> >>> > 4. router_proxy.sh will login into router vm, execute a >> shell >> >> >>> script >> >> >>> >>> inside router VM to program rules. >> >> >>> >>> > >> >> >>> >>> > In KVM, we can directly call router_proxy.sh or directly >> >> login >> >> >>> into >> >> >>> >>> router vm, we just need to prepare the parameters for this >> >> script. >> >> >>> >>> > The reference code is in VirtualRoutingResource.java, all >> the >> >> >>> network >> >> >>> >>> related command(extended from NetworkElementCommand) will be >> >> >>> redirected >> >> >>> >>> to VirtualRoutingResource. >> >> >>> >>> > We can add implementation of >> >> >>> >>> >> >> >>> >> >> >> SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2 >> >> >>> >>> SiteVpnCfgCommand in VirtualRoutingResource. >> >> >>> >>> > >> >> >>> >>> > >> >> >>> >>> >> -----Original Message----- >> >> >>> >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >>> >> Sent: Wednesday, August 29, 2012 4:34 PM >> >> >>> >>> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra >> >> Bhamidipati >> >> >>> >>> >> Subject: Re: VM router spawning multiple public nics >> >> >>> >>> >> >> >> >>> >>> >> Great, thanks. I've already got a working implementation >> of >> >> >>> >>> >> adding/removing nics from regular instances that I've >> been >> >> >>> playing >> >> >>> >>> >> with, so I'm getting vaguely familiar with the various >> data >> >> >>> types >> >> >>> >>> and >> >> >>> >>> >> things surrounding the networking. I don't know if this >> is >> >> quite >> >> >>> >>> >> within my reach just yet but I'll see how far I get. >> >> >>> >>> >> >> >> >>> >>> >> On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk >> >> >>> >>> >> <alena.prokharc...@citrix.com> wrote: >> >> >>> >>> >> > On 8/29/12 1:34 PM, "Edison Su" <edison...@citrix.com> >> >> wrote: >> >> >>> >>> >> > >> >> >>> >>> >> >>Hi Anthony & Alena, >> >> >>> >>> >> >> Could you help to provide information about VPC, how >> it >> >> >>> works, >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > Here is the functional spec on the feature: >> >> >>> >>> >> > >> >> >>> >>> >> > wiki.cloudstack.org/display/RelOps/Inter- >> >> >>> >>> VLAN+Routing+functional+spec >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > VpcVirtualNetworkApplianceManagerImpl is the manager >> >> >>> responsible >> >> >>> >>> for >> >> >>> >>> >> VPC >> >> >>> >>> >> > Virtual router operations (plug/unplug nics, etc) >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> >> which commands needed to implemented on the hypervisor >> >> side? >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > 1) PlugNicCommand/UnplugNicCommand - does Nic >> >> hotplug/unplug >> >> >>> >>> >> (currently >> >> >>> >>> >> > works for VR vm only). In VPC called when add nic for >> >> >>> Public/Guest >> >> >>> >>> >> > networks. >> >> >>> >>> >> > 2) SetupGuestNetworkCommand - sets up dhcp range, dns >> >> >>> information, >> >> >>> >>> >> > networkDomain information on the Nic to make it >> >> >>> >>> >> > 3) SetNetworkACLCommand - creates network ACL on the >> >> virtual >> >> >>> >>> router >> >> >>> >>> >> > 4) SetSourceNatCommand - used for setting source nat on >> >> the >> >> >>> Public >> >> >>> >>> IP >> >> >>> >>> >> on >> >> >>> >>> >> > the VPC VR. >> >> >>> >>> >> > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > Anthony/Kelven/Vijay did implementation for Xen/vmWare >> >> >>> resources, >> >> >>> >>> >> they can >> >> >>> >>> >> > help you answering all hypervisor related questions. If >> >> you >> >> >>> need >> >> >>> >>> more >> >> >>> >>> >> > details on business logic + Vpc VR management, I can >> help >> >> with >> >> >>> >>> that. >> >> >>> >>> >> > >> >> >>> >>> >> > >> >> >>> >>> >> > -Alena. >> >> >>> >>> >> > >> >> >>> >>> >> >> >> >> >>> >>> >> >>> -----Original Message----- >> >> >>> >>> >> >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> Sent: Wednesday, August 29, 2012 10:16 AM >> >> >>> >>> >> >>> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> >>> Subject: Re: VM router spawning multiple public nics >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> I'd be willing to give it a shot if someone could >> point >> >> me >> >> >>> in >> >> >>> >>> the >> >> >>> >>> >> >>> right direction and be available to answer questions. >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su >> >> >>> >>> <edison...@citrix.com> >> >> >>> >>> >> >>> wrote: >> >> >>> >>> >> >>> > Yah, KVM doesn't support VPC yet. Will you help to >> add >> >> VPC >> >> >>> >>> >> support on >> >> >>> >>> >> >>> KVM?:) Just implement a few VPC related commands... >> >> >>> >>> >> >>> > >> >> >>> >>> >> >>> >> -----Original Message----- >> >> >>> >>> >> >>> >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> >> Sent: Wednesday, August 29, 2012 6:49 AM >> >> >>> >>> >> >>> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> >>> >> Subject: Re: VM router spawning multiple public >> nics >> >> >>> >>> >> >>> >> >> >> >>> >>> >> >>> >> I can confirm that the patch has fixed my >> particular >> >> >>> issue. >> >> >>> >>> >> >>> >> >> >> >>> >>> >> >>> >> This is likely unrelated and I think it doesn't >> even >> >> use >> >> >>> the >> >> >>> >>> >> same >> >> >>> >>> >> >>> >> code, but I began to play with the VPC stuff a bit >> >> and >> >> >>> >>> noticed >> >> >>> >>> >> that >> >> >>> >>> >> >>> I >> >> >>> >>> >> >>> >> don't get any interfaces except for link local. >> I'd >> >> >>> probably >> >> >>> >>> >> chalk >> >> >>> >>> >> >>> >> that up to it not being ready for KVM, but I >> thought >> >> it >> >> >>> was >> >> >>> >>> >> worth a >> >> >>> >>> >> >>> >> mention. I'd be happy to try to help get it ready >> if >> >> >>> someone >> >> >>> >>> >> has >> >> >>> >>> >> >>> time >> >> >>> >>> >> >>> >> to nudge me in the right direction. >> >> >>> >>> >> >>> >> >> >> >>> >>> >> >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su >> >> >>> >>> >> <edison...@citrix.com> >> >> >>> >>> >> >>> wrote: >> >> >>> >>> >> >>> >> > >> >> >>> >>> >> >>> >> > >> >> >>> >>> >> >>> >> >> -----Original Message----- >> >> >>> >>> >> >>> >> >> From: Marcus Sorensen >> [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM >> >> >>> >>> >> >>> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> >>> >> >> Subject: Re: VM router spawning multiple public >> >> nics >> >> >>> >>> >> >>> >> >> >> >> >>> >>> >> >>> >> >> I thought about this solution myself, but >> below >> >> this >> >> >>> >>> portion >> >> >>> >>> >> of >> >> >>> >>> >> >>> >> code >> >> >>> >>> >> >>> >> >> it looks like it uses the hash map to determine >> >> which >> >> >>> nic >> >> >>> >>> >> number >> >> >>> >>> >> >>> to >> >> >>> >>> >> >>> >> >> add the IP to, so with multiple 'untagged' >> >> networks it >> >> >>> >>> would >> >> >>> >>> >> have >> >> >>> >>> >> >>> no >> >> >>> >>> >> >>> >> >> way of knowing which nicnum in the router >> >> corresponds >> >> >>> with >> >> >>> >>> >> the >> >> >>> >>> >> >>> >> correct >> >> >>> >>> >> >>> >> >> untagged vlan. >> >> >>> >>> >> >>> >> >> >> >> >>> >>> >> >>> >> >> nicNum = >> >> >>> >>> >> vlanAllocatedToVM.get(ip.getVlanId()); >> >> >>> >>> >> >>> >> >> networkUsage(routerIp, "addVif", >> >> "eth" >> >> >>> + >> >> >>> >>> >> nicNum); >> >> >>> >>> >> >>> >> >> result = >> >> >>> >>> >> >>> >> >> >> >> _virtRouterResource.assignPublicIpAddress(routerName, >> >> >>> >>> >> >>> >> >> routerIp, >> ip.getPublicIp(), >> >> >>> >>> >> ip.isAdd(), >> >> >>> >>> >> >>> >> >> ip.isFirstIP(), >> >> >>> >>> >> >>> >> >> ip.isSourceNat(), >> >> >>> ip.getVlanId(), >> >> >>> >>> >> >>> >> >> ip.getVlanGateway(), >> >> >>> >>> >> >>> >> >> ip.getVlanNetmask(), >> >> >>> >>> >> >>> ip.getVifMacAddress(), >> >> >>> >>> >> >>> >> >> ip.getGuestIp(), >> nicNum); >> >> >>> >>> >> >>> >> >> >> >> >>> >>> >> >>> >> >> if ip.getVlanId() returns untagged (as it does >> on >> >> >>> networks >> >> >>> >>> >> with >> >> >>> >>> >> >>> no >> >> >>> >>> >> >>> >> >> vlan id), and we tried to put multiple untagged >> >> keys >> >> >>> in >> >> >>> >>> >> >>> >> >> vlanAllocatedToVM (as with multiple untagged >> >> networks), >> >> >>> we >> >> >>> >>> >> get >> >> >>> >>> >> >>> the >> >> >>> >>> >> >>> >> >> wrong nicNum, no? >> >> >>> >>> >> >>> >> > >> >> >>> >>> >> >>> >> > In the ipassoc case, if there are multiple >> untagged >> >> >>> >>> networks, >> >> >>> >>> >> all >> >> >>> >>> >> >>> of >> >> >>> >>> >> >>> >> them are use the same >> >> >>> >>> >> >>> >> > Public bridge. Then multiple ip address will be >> >> added >> >> >>> on >> >> >>> >>> eth2 >> >> >>> >>> >> >>> inside >> >> >>> >>> >> >>> >> router VM. >> >> >>> >>> >> >>> >> > If it works physically, then it works. >> >> >>> >>> >> >>> >> > >> >> >>> >>> >> >>> >> >> >> >> >>> >>> >> >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su >> >> >>> >>> >> <edison...@citrix.com> >> >> >>> >>> >> >>> >> wrote: >> >> >>> >>> >> >>> >> >> > >> >> >>> >>> >> >>> >> >> > >> >> >>> >>> >> >>> >> >> >> -----Original Message----- >> >> >>> >>> >> >>> >> >> >> From: Marcus Sorensen >> >> [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM >> >> >>> >>> >> >>> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> >>> >> >> >> Subject: Re: VM router spawning multiple >> public >> >> >>> nics >> >> >>> >>> >> >>> >> >> >> >> >> >>> >>> >> >>> >> >> >> Yes, that looks like it would work for me, >> >> however >> >> >>> >>> that's >> >> >>> >>> >> not >> >> >>> >>> >> >>> >> >> >> something that would ever make it into >> master, >> >> >>> right? >> >> >>> >>> >> >>> Essentially >> >> >>> >>> >> >>> >> >> >> killing tagging for the public, private, and >> >> guest >> >> >>> >>> traffic >> >> >>> >>> >> >>> labels? >> >> >>> >>> >> >>> >> >> >> There's also still the issue of not being >> able >> >> to >> >> >>> >>> >> >>> differentiate >> >> >>> >>> >> >>> >> >> >> between multiple untagged networks, if we >> >> wanted to >> >> >>> add >> >> >>> >>> an >> >> >>> >>> >> IP >> >> >>> >>> >> >>> to >> >> >>> >>> >> >>> >> a >> >> >>> >>> >> >>> >> >> >> router it might not know which untagged >> >> interface >> >> >>> to >> >> >>> >>> apply >> >> >>> >>> >> it >> >> >>> >>> >> >>> to. >> >> >>> >>> >> >>> >> >> > >> >> >>> >>> >> >>> >> >> > Physically, all the "untagged" network will >> be >> >> >>> created >> >> >>> >>> on >> >> >>> >>> >> >>> >> >> public/guest/private bridge(the name we put in >> >> >>> >>> >> >>> >> >> private/public/guest.bridge.name in >> >> agent.properties"). >> >> >>> >>> >> >>> >> >> > Because, there is no way to create a new >> >> untagged >> >> >>> bridge >> >> >>> >>> by >> >> >>> >>> >> >>> agent >> >> >>> >>> >> >>> >> >> itself. Agent code only knows how to create a >> new >> >> >>> >>> tagged(vlan) >> >> >>> >>> >> >>> >> bridge. >> >> >>> >>> >> >>> >> >> > So the fix should be pushed into master. >> >> >>> >>> >> >>> >> >> > >> >> >>> >>> >> >>> >> >> >> >> >> >>> >>> >> >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su >> >> >>> >>> >> >>> <edison...@citrix.com> >> >> >>> >>> >> >>> >> >> wrote: >> >> >>> >>> >> >>> >> >> >> > >> >> >>> >>> >> >>> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> -----Original Message----- >> >> >>> >>> >> >>> >> >> >> >> From: Marcus Sorensen >> >> >>> [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM >> >> >>> >>> >> >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org >> >> >>> >>> >> >>> >> >> >> >> Subject: Re: VM router spawning multiple >> >> public >> >> >>> nics >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> >> >>> >> >> >> >> Thanks for pointing me in the right >> >> direction. >> >> >>> I've >> >> >>> >>> >> >>> reviewed >> >> >>> >>> >> >>> >> this >> >> >>> >>> >> >>> >> >> >> code >> >> >>> >>> >> >>> >> >> >> >> in a bit more detail, and it seems like >> it's >> >> >>> >>> >> accomplishing >> >> >>> >>> >> >>> the >> >> >>> >>> >> >>> >> >> >> >> following: >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> >> >>> >> >> >> >> 1. get all network interfaces currently >> >> >>> connected to >> >> >>> >>> >> the >> >> >>> >>> >> >>> >> running >> >> >>> >>> >> >>> >> >> VM >> >> >>> >>> >> >>> >> >> >> >> (a.k.a vnet devices via libvirt) >> >> >>> >>> >> >>> >> >> >> >> 2. find out which vlans these network >> >> interfaces >> >> >>> are >> >> >>> >>> >> >>> bridged >> >> >>> >>> >> >>> >> to, >> >> >>> >>> >> >>> >> >> >> store >> >> >>> >>> >> >>> >> >> >> >> this in a hash map of vlan ids and nics >> >> >>> >>> >> >>> >> >> >> >> 3. get all ip addresses to be added to >> the >> >> VM >> >> >>> >>> >> >>> >> >> >> >> 4. for each ip, get the configured vlan >> id >> >> for >> >> >>> the >> >> >>> >>> ip, >> >> >>> >>> >> >>> compare >> >> >>> >>> >> >>> >> it >> >> >>> >>> >> >>> >> >> to >> >> >>> >>> >> >>> >> >> >> >> the hash map of existing vlan ids and >> nics >> >> >>> >>> >> >>> >> >> >> >> 5. if the required vlan id is not found >> in >> >> the >> >> >>> hash >> >> >>> >>> map, >> >> >>> >>> >> >>> >> create a >> >> >>> >>> >> >>> >> >> >> new >> >> >>> >>> >> >>> >> >> >> >> nic >> >> >>> >>> >> >>> >> >> >> >> 6. assign the ip to the nic identified by >> >> the >> >> >>> vlan >> >> >>> >>> id >> >> >>> >>> >> key >> >> >>> >>> >> >>> in >> >> >>> >>> >> >>> >> the >> >> >>> >>> >> >>> >> >> >> hash >> >> >>> >>> >> >>> >> >> >> >> map >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> >> >>> >> >> >> >> In this case, we're getting a vlan id >> >> returned >> >> >>> in >> >> >>> >>> step >> >> >>> >>> >> 2 >> >> >>> >>> >> >>> for a >> >> >>> >>> >> >>> >> >> >> bridged >> >> >>> >>> >> >>> >> >> >> >> nic whose network is defined as untagged >> in >> >> the >> >> >>> >>> >> cloudstack >> >> >>> >>> >> >>> db, >> >> >>> >>> >> >>> >> >> >> >> therefore in step 5 we never match as >> >> already >> >> >>> having >> >> >>> >>> a >> >> >>> >>> >> nic >> >> >>> >>> >> >>> for >> >> >>> >>> >> >>> >> >> >> >> 'untagged'. I wrote a big long response >> >> >>> discussing >> >> >>> >>> this >> >> >>> >>> >> >>> issue, >> >> >>> >>> >> >>> >> >> but >> >> >>> >>> >> >>> >> >> >> as >> >> >>> >>> >> >>> >> >> >> >> I began to dig further I realized that >> aside >> >> >>> from my >> >> >>> >>> >> >>> >> particular >> >> >>> >>> >> >>> >> >> case, >> >> >>> >>> >> >>> >> >> >> >> untagged vlans in general are just broken >> >> (for >> >> >>> >>> example >> >> >>> >>> >> they >> >> >>> >>> >> >>> >> can't >> >> >>> >>> >> >>> >> >> be >> >> >>> >>> >> >>> >> >> >> >> dealt with uniquely in the current >> >> >>> IpAssocCommand >> >> >>> >>> code, >> >> >>> >>> >> >>> given >> >> >>> >>> >> >>> >> the >> >> >>> >>> >> >>> >> >> >> hash >> >> >>> >>> >> >>> >> >> >> >> map) and it would require more effort >> than I >> >> >>> have >> >> >>> >>> time >> >> >>> >>> >> for >> >> >>> >>> >> >>> now >> >> >>> >>> >> >>> >> to >> >> >>> >>> >> >>> >> >> >> make >> >> >>> >>> >> >>> >> >> >> >> things work. If the code were already in >> >> place >> >> >>> to >> >> >>> >>> >> >>> >> differentiate >> >> >>> >>> >> >>> >> >> >> >> between multiple untagged nics I think >> that >> >> >>> fixing >> >> >>> >>> my >> >> >>> >>> >> >>> problem >> >> >>> >>> >> >>> >> >> would >> >> >>> >>> >> >>> >> >> >> be >> >> >>> >>> >> >>> >> >> >> >> trivial, but since its not, I'll just >> find >> >> an >> >> >>> >>> >> alternative >> >> >>> >>> >> >>> >> >> solution. >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> >> >>> >> >> >> > >> >> >>> >>> >> >>> >> >> >> > The untagged network usually means >> "untagged", >> >> no >> >> >>> >>> vlan >> >> >>> >>> >> on >> >> >>> >>> >> >>> the >> >> >>> >>> >> >>> >> >> >> bridge... >> >> >>> >>> >> >>> >> >> >> > In your case, the untagged network >> actually >> >> has >> >> >>> >>> >> vlan(tagged) >> >> >>> >>> >> >>> on >> >> >>> >>> >> >>> >> >> the >> >> >>> >>> >> >>> >> >> >> bridge, thus getting things confused. >> >> >>> >>> >> >>> >> >> >> > Will this >> patch(http://pastebin.com/HJXzZwKp) >> >> >>> work >> >> >>> >>> for >> >> >>> >>> >> you? >> >> >>> >>> >> >>> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus >> >> >>> Sorensen >> >> >>> >>> >> >>> >> >> >> <shadow...@gmail.com> >> >> >>> >>> >> >>> >> >> >> >> wrote: >> >> >>> >>> >> >>> >> >> >> >> > ... >> >> >>> >>> >> >>> >> >> >> >> > Integer nicPos = 0; >> >> >>> >>> >> >>> >> >> >> >> > for (InterfaceDef nic : >> nics) >> >> { >> >> >>> >>> >> >>> >> >> >> >> > if >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) >> >> >>> >>> >> { >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> vlanAllocatedToVM.put("LinkLocal", >> >> >>> >>> >> >>> >> nicPos); >> >> >>> >>> >> >>> >> >> >> >> > } else { >> >> >>> >>> >> >>> >> >> >> >> > String vlanId = >> >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >>> >>> >> >>> >> >> >> >> > if (vlanId != null) >> { >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> vlanAllocatedToVM.put(vlanId, >> >> >>> >>> >> >>> >> nicPos); >> >> >>> >>> >> >>> >> >> >> >> > } else { >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >>> >>> >> >>> >> >> >> nicPos); >> >> >>> >>> >> >>> >> >> >> >> > } >> >> >>> >>> >> >>> >> >> >> >> > } >> >> >>> >>> >> >>> >> >> >> >> > nicPos++; >> >> >>> >>> >> >>> >> >> >> >> > } >> >> >>> >>> >> >>> >> >> >> >> > IpAddressTO[] ips = >> >> >>> >>> cmd.getIpAddresses(); >> >> >>> >>> >> >>> >> >> >> >> > int i = 0; >> >> >>> >>> >> >>> >> >> >> >> > String result = null; >> >> >>> >>> >> >>> >> >> >> >> > int nicNum = 0; >> >> >>> >>> >> >>> >> >> >> >> > for (IpAddressTO ip : ips) >> { >> >> >>> >>> >> >>> >> >> >> >> > if >> >> >>> >>> >> >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId())) >> >> >>> >>> >> >>> >> >> >> { >> >> >>> >>> >> >>> >> >> >> >> > /* plug a vif into >> >> router >> >> >>> */ >> >> >>> >>> >> >>> >> >> >> >> > VifHotPlug(conn, >> >> >>> routerName, >> >> >>> >>> >> >>> >> ip.getVlanId(), >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> ip.getVifMacAddress()); >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> vlanAllocatedToVM.put(ip.getVlanId(), >> >> >>> >>> >> >>> >> >> >> nicPos++); >> >> >>> >>> >> >>> >> >> >> >> > } >> >> >>> >>> >> >>> >> >> >> >> > ... >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > Looks like the getVlanIdFromBridge >> might >> >> be a >> >> >>> bit >> >> >>> >>> >> >>> misleading. >> >> >>> >>> >> >>> >> I >> >> >>> >>> >> >>> >> >> am >> >> >>> >>> >> >>> >> >> >> >> > running my guest public traffic on a >> >> >>> 'cloudbr470', >> >> >>> >>> >> which >> >> >>> >>> >> >>> is >> >> >>> >>> >> >>> >> a >> >> >>> >>> >> >>> >> >> >> bridge >> >> >>> >>> >> >>> >> >> >> >> > to eth2.470, yet I configured this >> network >> >> as >> >> >>> >>> >> 'untagged' >> >> >>> >>> >> >>> >> >> because I >> >> >>> >>> >> >>> >> >> >> >> > have a vlan 470 available on eth3 for >> >> >>> cloudstack >> >> >>> >>> to >> >> >>> >>> >> >>> >> autoassign >> >> >>> >>> >> >>> >> >> >> (eth3 >> >> >>> >>> >> >>> >> >> >> >> > is where all of my stuff will be >> >> autoassigned). >> >> >>> So >> >> >>> >>> >> I'm >> >> >>> >>> >> >>> not >> >> >>> >>> >> >>> >> 100% >> >> >>> >>> >> >>> >> >> >> sure >> >> >>> >>> >> >>> >> >> >> >> > yet what's going on here but it seems >> as >> >> >>> though >> >> >>> >>> the >> >> >>> >>> >> above >> >> >>> >>> >> >>> is >> >> >>> >>> >> >>> >> >> not >> >> >>> >>> >> >>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it >> >> finds a >> >> >>> vlan >> >> >>> >>> >> number >> >> >>> >>> >> >>> >> for >> >> >>> >>> >> >>> >> >> >> >> > eth2.470, but when it enumerates the >> IPs >> >> for >> >> >>> the >> >> >>> >>> >> router, >> >> >>> >>> >> >>> it >> >> >>> >>> >> >>> >> >> then >> >> >>> >>> >> >>> >> >> >> runs >> >> >>> >>> >> >>> >> >> >> >> > ip.getVlanId() and doesn't find a nic >> for >> >> the >> >> >>> >>> >> untagged IP >> >> >>> >>> >> >>> >> and >> >> >>> >>> >> >>> >> >> >> creates >> >> >>> >>> >> >>> >> >> >> >> > one. >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > I realize this is perhaps an uncommon >> case, >> >> >>> but a >> >> >>> >>> bug >> >> >>> >>> >> >>> >> >> nonetheless. >> >> >>> >>> >> >>> >> >> >> >> > I'll play with the code a bit and see >> if I >> >> can >> >> >>> >>> come >> >> >>> >>> >> up >> >> >>> >>> >> >>> with >> >> >>> >>> >> >>> >> a >> >> >>> >>> >> >>> >> >> >> >> > solution. I'm thinking I can look at >> the >> >> nic's >> >> >>> >>> >> broadcast >> >> >>> >>> >> >>> URI >> >> >>> >>> >> >>> >> >> and >> >> >>> >>> >> >>> >> >> >> see >> >> >>> >>> >> >>> >> >> >> >> > if it's supposed to be untagged, then >> add >> >> to >> >> >>> >>> >> >>> >> vlanAllocatedToVM >> >> >>> >>> >> >>> >> >> >> >> > appropriately, off the top of my head >> >> >>> something >> >> >>> >>> like: >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > String vlanId = >> >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName()); >> >> >>> >>> >> >>> >> >> >> >> > if (vlanId != null >> && >> >> >>> >>> >> >>> >> >> >> >> >> >> >>> >>> > !nic.getBroadcastUri().toString().contains("untagged") >> >> >>> >>> >> { >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> vlanAllocatedToVM.put(vlanId, >> >> >>> >>> >> >>> >> nicPos); >> >> >>> >>> >> >>> >> >> >> >> > } else { >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED, >> >> >>> >>> >> >>> >> >> >> nicPos); >> >> >>> >>> >> >>> >> >> >> >> > } >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > >> >> >>> >>> >> >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison >> Su >> >> >>> >>> >> >>> >> >> <edison...@citrix.com> >> >> >>> >>> >> >>> >> >> >> >> wrote: >> >> >>> >>> >> >>> >> >> >> >> >> Possible bug in in kvm code: >> >> >>> >>> >> LibvirtComputingResource- >> >> >>> >>> >> >>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, >> >> which >> >> >>> is >> >> >>> >>> >> only >> >> >>> >>> >> >>> place >> >> >>> >>> >> >>> >> >> >> adding >> >> >>> >>> >> >>> >> >> >> >> nic into router vm. >> >> >>> >>> >> >>> >> >> >> >> >> Turn on agent log, then take a look >> what >> >> >>> happened. >> >> >>> >>> >> >>> >> >> >> >> >> >> >> >>> >>> >> >>> >> >> >> >> >>> -----Original Message----- >> >> >>> >>> >> >>> >> >> >> >> >>> From: Marcus Sorensen >> >> >>> >>> [mailto:shadow...@gmail.com] >> >> >>> >>> >> >>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM >> >> >>> >>> >> >>> >> >> >> >> >>> To: cloudstack- >> d...@incubator.apache.org >> >> >>> >>> >> >>> >> >> >> >> >>> Subject: VM router spawning multiple >> >> public >> >> >>> nics >> >> >>> >>> >> >>> >> >> >> >> >>> >> >> >>> >>> >> >>> >> >> >> >> >>> I've got two zones running the same >> >> build of >> >> >>> >>> >> cloudstack >> >> >>> >>> >> >>> (a >> >> >>> >>> >> >>> >> >> >> recent >> >> >>> >>> >> >>> >> >> >> >> copy >> >> >>> >>> >> >>> >> >> >> >> >>> of master). One of them creates >> routers >> >> that >> >> >>> >>> turn >> >> >>> >>> >> into >> >> >>> >>> >> >>> >> ugly >> >> >>> >>> >> >>> >> >> >> >> >>> multi-headed beasts, and by that I >> mean >> >> that >> >> >>> any >> >> >>> >>> >> time I >> >> >>> >>> >> >>> >> >> create a >> >> >>> >>> >> >>> >> >> >> >> port >> >> >>> >>> >> >>> >> >> >> >> >>> forwarding or iptables rule for that >> >> router >> >> >>> I >> >> >>> >>> get a >> >> >>> >>> >> new >> >> >>> >>> >> >>> >> >> public >> >> >>> >>> >> >>> >> >> >> NIC >> >> >>> >>> >> >>> >> >> >> >> >>> with an identical IP address, I have >> an >> >> >>> instance >> >> >>> >>> >> with a >> >> >>> >>> >> >>> >> few >> >> >>> >>> >> >>> >> >> tens >> >> >>> >>> >> >>> >> >> >> of >> >> >>> >>> >> >>> >> >> >> >> >>> NICs. My guess is that some script >> >> isn't >> >> >>> >>> detecting >> >> >>> >>> >> >>> that >> >> >>> >>> >> >>> >> >> there's >> >> >>> >>> >> >>> >> >> >> >> >>> already a NIC with the public IP on >> it. >> >> It >> >> >>> >>> looks >> >> >>> >>> >> fine >> >> >>> >>> >> >>> in >> >> >>> >>> >> >>> >> the >> >> >>> >>> >> >>> >> >> >> >> >>> database, there is only one public >> NIC >> >> >>> defined >> >> >>> >>> in >> >> >>> >>> >> the >> >> >>> >>> >> >>> nics >> >> >>> >>> >> >>> >> >> table. >> >> >>> >>> >> >>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if >> >> anyone >> >> >>> >>> knows >> >> >>> >>> >> >>> where I >> >> >>> >>> >> >>> >> >> >> should >> >> >>> >>> >> >>> >> >> >> >> >>> begin the headstart would be >> appreciated. >> >> >>> >>> >> >>> >> >> >> >> >>> >> >> >>> >>> >> >>> >> >> >> >> >>> Thanks >> >> >>> >>> >> >> >> >> >>> >>> >> > >> >> >>> >>> >> >
