Andrija, nevertheless it sounds familiar. I will be back in the office on monday and ask around.
On Wed, May 28, 2014 at 9:23 PM, Andrija Panic <andrija.pa...@gmail.com> wrote: > Hi Daan, > > I don't think this is my issue, at least I don't make use of private > gateway - this is just simple as: create new VPC from scratch - Public IP > is not assigned to VR eth1 interface inside VR... > > I have filed the bug: https://issues.apache.org/jira/browse/CLOUDSTACK-6801 > > This same thing happened previously to Andrei Mikhailovsky: > http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand > it is not resolved > > Thanks, > > Andrija > > > On 28 May 2014 21:01, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > >> Andrija, >> >> this sound like something we seen as well. >> can you check if this is it : >> https://issues.apache.org/jira/browse/CLOUDSTACK-6485 >> >> thanks, >> Daan >> >> On Wed, May 28, 2014 at 3:30 PM, Andrija Panic <andrija.pa...@gmail.com> >> wrote: >> > Hi there, >> > >> > I'm having big time problems with Public IP missing from VPC VR's eth1, >> > after upgrade to ACS 4.3.1 - did not found this filed as bug so far...and >> > it worked all fine on ACS 4.2.1. >> > >> > No help so far from user mailing list... >> > >> > Below is a detailed explanation, and logs from inside VR, and from >> > management (all fine with management logs...) >> > >> > If anybody can help, I would very much appriciate this, since now I have >> > bunch fo VPC unoperational... >> > >> > Thanks >> > >> > ---------- Forwarded message ---------- >> > From: Andrija Panic <andrija.pa...@gmail.com> >> > Date: 28 May 2014 14:50 >> > Subject: Re: VPC's VR missing public NIC eth1 >> > To: us...@cloudstack.apache.org >> > >> > >> > and as I said eth1 is present: >> > >> > root@r-794-VM:~# cat /proc/net/dev >> > Inter-| Receive | >> Transmit >> > face |bytes packets errs drop fifo frame compressed multicast|bytes >> > packets errs drop fifo colls carrier compressed >> > eth3: 11484 131 0 0 0 0 0 0 >> 11590 >> > 131 0 0 0 0 0 0 >> > lo: 214 2 0 0 0 0 0 0 >> 214 >> > 2 0 0 0 0 0 0 >> > eth2: 32970 544 0 0 0 0 0 0 >> 2084 >> > 24 0 0 0 0 0 0 >> > eth1: 0 0 0 0 0 0 0 0 >> 0 >> > 0 0 0 0 0 0 0 >> > eth0: 150207 1319 0 0 0 0 0 0 >> 264232 >> > 1180 0 0 0 0 0 0 >> > >> > >> > On 28 May 2014 14:47, Andrija Panic <andrija.pa...@gmail.com> wrote: >> > >> >> Also, from /var/log/messages/ inside VR: >> >> >> >> This is a major show stopper - all our VPCs are unusable complete. >> >> Anybody... ? >> >> >> >> May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 0 seconds >> >> May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 1 seconds >> >> May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 2 seconds >> >> May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 3 seconds >> >> May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 4 seconds >> >> May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 5 seconds >> >> May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 6 seconds >> >> May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 7 seconds >> >> May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 8 seconds >> >> May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 9 seconds >> >> May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 10 seconds >> >> May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 11 seconds >> >> May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 12 seconds >> >> May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 13 seconds >> >> May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 14 seconds >> >> May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 15 seconds >> >> May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface >> >> ethnull to appear, 16 seconds >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull never >> >> appeared >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip 46.232.180.246 >> on >> >> interface ethnull >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing >> 46.232.180.246 >> >> on interface ethnull >> >> May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT >> >> 46.232.180.246 on interface ethnull >> >> May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT >> 46.232.180.246 >> >> on interface eth1 >> >> >> >> >> >> >> >> >> >> On 28 May 2014 12:59, Andrija Panic <andrija.pa...@gmail.com> wrote: >> >> >> >>> Defined eth1 manually inside /etc/network/interfaces inside VPC's VR. >> >>> iface eth1 inet static >> >>> address 46.232.180.246 >> >>> netmask 255.255.255.0 >> >>> >> >>> ifup eth1 >> >>> ip route add default via 46.232.180.1 >> >>> >> >>> so now VR works fine (have access to internet) >> >>> >> >>> But again, adding new IP to VR, and enabling static NAT is failing... >> >>> That is, geting new IP works fine (just associated with account) >> >>> But enabling static NAT fails, due to "resource unavailable" >> >>> >> >>> Here are management logs: >> >>> 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl] >> >>> (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static >> nat >> >>> rule due to >> >>> com.cloud.exception.ResourceUnavailableException: Resource >> [DataCenter:1] >> >>> is unreachable: Unable to apply static nat rules on router >> >>> at >> >>> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915) >> >>> at >> >>> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963) >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>> at >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >>> at >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> at java.lang.reflect.Method.invoke(Method.java:622) >> >>> at >> >>> >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> >>> at >> >>> >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> >>> at >> >>> >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> >>> at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source) >> >>> at >> >>> >> com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650) >> >>> at >> >>> >> com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762) >> >>> at >> >>> >> com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324) >> >>> at >> >>> >> com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602) >> >>> at >> >>> >> com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446) >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>> at >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >>> at >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> at java.lang.reflect.Method.invoke(Method.java:622) >> >>> at >> >>> >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> >>> at >> >>> >> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> >>> at >> >>> >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> >>> at >> >>> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> >>> at >> >>> >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> >>> at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source) >> >>> at >> >>> >> org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129) >> >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) >> >>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) >> >>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) >> >>> at >> >>> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) >> >>> at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) >> >>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) >> >>> at >> >>> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> >>> at >> >>> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> >>> at >> >>> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> >>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112) >> >>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) >> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> >>> at >> >>> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> >>> at >> >>> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> >>> at >> >>> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >> >>> at >> >>> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> >>> at >> >>> >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) >> >>> at >> >>> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> >>> at >> >>> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> >>> at >> >>> >> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) >> >>> at >> >>> >> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) >> >>> at >> >>> >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >> >>> at >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >>> at java.lang.Thread.run(Thread.java:701) >> >>> >> >>> >> >>> >> >>> >> >>> On 28 May 2014 00:58, Andrija Panic <andrija.pa...@gmail.com> wrote: >> >>> >> >>>> Hi Jayapal, >> >>>> >> >>>> eth1 seems present (lspci and virsh comfirmed), but not started inside >> >>>> VPC's VR - (VR used for Shared Network is fine)... >> >>>> I could confirm by virsh that is is plugged inside appropriate bridge >> >>>> breth1-500 (check management logs...) >> >>>> >> >>>> management log while createing new VPC (VR) - >> >>>> http://pastebin.com/s77nu5Ei >> >>>> The public IP is there, so command is fine for creating it I guess... >> >>>> >> >>>> VR's /var/log/cloud.log after rebooting VR from CS GUI >> >>>> Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config >> >>>> Tue May 27 22:46:58 UTC 2014 Detected that we are running inside kvm >> >>>> guest >> >>>> Tue May 27 22:46:59 UTC 2014 Patching cloud service >> >>>> Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml >> >>>> Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system vm >> >>>> Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP >> >>>> Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1 >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1 >> >>>> Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0 >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0 >> >>>> Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter >> >>>> Tue May 27 22:47:00 UTC 2014 disable rpfilter >> >>>> Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1 >> >>>> Tue May 27 22:47:00 UTC 2014 enable_fwding = 1 >> >>>> >> >>>> ifconfig (no eth1 shown) >> >>>> >> >>>> eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c >> >>>> inet addr:169.254.3.92 Bcast:169.254.255.255 >> Mask:255.255.0.0 >> >>>> >> >>>> eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10 >> >>>> inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 >> >>>> >> >>>> eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05 >> >>>> inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0 >> >>>> >> >>>> lo Link encap:Local Loopback >> >>>> inet addr:127.0.0.1 Mask:255.0.0.0 >> >>>> >> >>>> >> >>>> cat /etc/network/interfaces >> >>>> auto lo eth0 >> >>>> iface lo inet loopback >> >>>> iface eth0 inet static >> >>>> address 169.254.3.92 >> >>>> netmask 255.255.0.0 >> >>>> >> >>>> lspci - shows 4 ehternet addapters >> >>>> ethtool eth1 = no link detected >> >>>> virsh - confirmed that eth1 is plugged to correct bridge (breth1-500) >> as >> >>>> indicated by management logs, and shows good MAC address as shown in >> >>>> managemetn log on pastebin.. >> >>>> >> >>>> This is completely makeing VPCs unusable... >> >>>> :( >> >>>> >> >>>> Cheers >> >>>> >> >>>> >> >>>> On 27 May 2014 16:36, Jayapal Reddy Uradi < >> jayapalreddy.ur...@citrix.com >> >>>> > wrote: >> >>>> >> >>>>> Hi, >> >>>>> Can you please share management server and router logs in >> pastebin.comto understand the issue ? >> >>>>> >> >>>>> Thanks, >> >>>>> Jayapal >> >>>>> >> >>>>> On 27-May-2014, at 6:21 PM, Andrija Panic <andrija.pa...@gmail.com> >> >>>>> wrote: >> >>>>> >> >>>>> > Hi, >> >>>>> > >> >>>>> > after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC lost >> >>>>> their >> >>>>> > eth1 which is public NIC. VR got eth0(control nic) and eth2 and >> eth3 >> >>>>> (bith >> >>>>> > belonging to Tiers). From CS GUI, it is reported that the VR has >> eth1 >> >>>>> with >> >>>>> > Public network attached, but from inside (ssh to VR) there is no >> eth1 >> >>>>> with >> >>>>> > public IP... >> >>>>> > >> >>>>> > Even after destroying those VR, they are recreated again, but >> without >> >>>>> eth1. >> >>>>> > >> >>>>> > Anybody experienced same situtation ? >> >>>>> > >> >>>>> > Thanks, >> >>>>> > >> >>>>> > -- >> >>>>> > >> >>>>> > Andrija Panić >> >>>>> > -------------------------------------- >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> >> >>>> >> >>>> >> >> >> >> -- >> Daan >> > > > > -- > > Andrija Panić > -------------------------------------- > http://admintweets.com > -------------------------------------- -- Daan
