Thanks Daan, my problem is that I'm in production for 3rd day now, and restoring DB and downgrading back to 4.2.1 doesn't seem as option for me at the moment, since I would loose new acounts and single VMs, etc...
Thanks, Andrija On 28 May 2014 21:34, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Andrija, > > nevertheless it sounds familiar. I will be back in the office on > monday and ask around. > > On Wed, May 28, 2014 at 9:23 PM, Andrija Panic <andrija.pa...@gmail.com> > wrote: > > Hi Daan, > > > > I don't think this is my issue, at least I don't make use of private > > gateway - this is just simple as: create new VPC from scratch - Public > IP > > is not assigned to VR eth1 interface inside VR... > > > > I have filed the bug: > https://issues.apache.org/jira/browse/CLOUDSTACK-6801 > > > > This same thing happened previously to Andrei Mikhailovsky: > > > http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand > > it is not resolved > > > > Thanks, > > > > Andrija > > > > > > On 28 May 2014 21:01, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > > > >> Andrija, > >> > >> this sound like something we seen as well. > >> can you check if this is it : > >> https://issues.apache.org/jira/browse/CLOUDSTACK-6485 > >> > >> thanks, > >> Daan > >> > >> On Wed, May 28, 2014 at 3:30 PM, Andrija Panic <andrija.pa...@gmail.com > > > >> wrote: > >> > Hi there, > >> > > >> > I'm having big time problems with Public IP missing from VPC VR's > eth1, > >> > after upgrade to ACS 4.3.1 - did not found this filed as bug so > far...and > >> > it worked all fine on ACS 4.2.1. > >> > > >> > No help so far from user mailing list... > >> > > >> > Below is a detailed explanation, and logs from inside VR, and from > >> > management (all fine with management logs...) > >> > > >> > If anybody can help, I would very much appriciate this, since now I > have > >> > bunch fo VPC unoperational... > >> > > >> > Thanks > >> > > >> > ---------- Forwarded message ---------- > >> > From: Andrija Panic <andrija.pa...@gmail.com> > >> > Date: 28 May 2014 14:50 > >> > Subject: Re: VPC's VR missing public NIC eth1 > >> > To: us...@cloudstack.apache.org > >> > > >> > > >> > and as I said eth1 is present: > >> > > >> > root@r-794-VM:~# cat /proc/net/dev > >> > Inter-| Receive | > >> Transmit > >> > face |bytes packets errs drop fifo frame compressed > multicast|bytes > >> > packets errs drop fifo colls carrier compressed > >> > eth3: 11484 131 0 0 0 0 0 0 > >> 11590 > >> > 131 0 0 0 0 0 0 > >> > lo: 214 2 0 0 0 0 0 0 > >> 214 > >> > 2 0 0 0 0 0 0 > >> > eth2: 32970 544 0 0 0 0 0 0 > >> 2084 > >> > 24 0 0 0 0 0 0 > >> > eth1: 0 0 0 0 0 0 0 0 > >> 0 > >> > 0 0 0 0 0 0 0 > >> > eth0: 150207 1319 0 0 0 0 0 0 > >> 264232 > >> > 1180 0 0 0 0 0 0 > >> > > >> > > >> > On 28 May 2014 14:47, Andrija Panic <andrija.pa...@gmail.com> wrote: > >> > > >> >> Also, from /var/log/messages/ inside VR: > >> >> > >> >> This is a major show stopper - all our VPCs are unusable complete. > >> >> Anybody... ? > >> >> > >> >> May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 0 seconds > >> >> May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 1 seconds > >> >> May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 2 seconds > >> >> May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 3 seconds > >> >> May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 4 seconds > >> >> May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 5 seconds > >> >> May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 6 seconds > >> >> May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 7 seconds > >> >> May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 8 seconds > >> >> May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 9 seconds > >> >> May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 10 seconds > >> >> May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 11 seconds > >> >> May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 12 seconds > >> >> May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 13 seconds > >> >> May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 14 seconds > >> >> May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 15 seconds > >> >> May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface > >> >> ethnull to appear, 16 seconds > >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull > never > >> >> appeared > >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip > 46.232.180.246 > >> on > >> >> interface ethnull > >> >> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing > >> 46.232.180.246 > >> >> on interface ethnull > >> >> May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT > >> >> 46.232.180.246 on interface ethnull > >> >> May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT > >> 46.232.180.246 > >> >> on interface eth1 > >> >> > >> >> > >> >> > >> >> > >> >> On 28 May 2014 12:59, Andrija Panic <andrija.pa...@gmail.com> wrote: > >> >> > >> >>> Defined eth1 manually inside /etc/network/interfaces inside VPC's > VR. > >> >>> iface eth1 inet static > >> >>> address 46.232.180.246 > >> >>> netmask 255.255.255.0 > >> >>> > >> >>> ifup eth1 > >> >>> ip route add default via 46.232.180.1 > >> >>> > >> >>> so now VR works fine (have access to internet) > >> >>> > >> >>> But again, adding new IP to VR, and enabling static NAT is > failing... > >> >>> That is, geting new IP works fine (just associated with account) > >> >>> But enabling static NAT fails, due to "resource unavailable" > >> >>> > >> >>> Here are management logs: > >> >>> 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl] > >> >>> (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static > >> nat > >> >>> rule due to > >> >>> com.cloud.exception.ResourceUnavailableException: Resource > >> [DataCenter:1] > >> >>> is unreachable: Unable to apply static nat rules on router > >> >>> at > >> >>> > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915) > >> >>> at > >> >>> > >> > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963) > >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> >>> at > >> >>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> >>> at > >> >>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >>> at java.lang.reflect.Method.invoke(Method.java:622) > >> >>> at > >> >>> > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > >> >>> at > >> >>> > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > >> >>> at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source) > >> >>> at > >> >>> > >> > com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650) > >> >>> at > >> >>> > >> > com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762) > >> >>> at > >> >>> > >> > com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324) > >> >>> at > >> >>> > >> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602) > >> >>> at > >> >>> > >> > com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446) > >> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> >>> at > >> >>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> >>> at > >> >>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >>> at java.lang.reflect.Method.invoke(Method.java:622) > >> >>> at > >> >>> > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > >> >>> at > >> >>> > >> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > >> >>> at > >> >>> > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > >> >>> at > >> >>> > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > >> >>> at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source) > >> >>> at > >> >>> > >> > org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129) > >> >>> at > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > >> >>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) > >> >>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) > >> >>> at > >> >>> > com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) > >> >>> at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) > >> >>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) > >> >>> at > >> >>> > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> >>> at > >> >>> > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> >>> at > >> >>> > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> >>> at > com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112) > >> >>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) > >> >>> at > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > >> >>> at > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > >> >>> at > >> >>> > >> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > >> >>> at > >> >>> > >> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > >> >>> at > >> >>> > >> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > >> >>> at > >> >>> > >> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > >> >>> at > >> >>> > >> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > >> >>> at > >> >>> > >> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > >> >>> at > >> >>> > >> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > >> >>> at > >> >>> > >> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > >> >>> at > >> >>> > >> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > >> >>> at > >> >>> > >> > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > >> >>> at > >> >>> > >> > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) > >> >>> at > >> >>> > >> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) > >> >>> at > >> >>> > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > >> >>> at > >> >>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> >>> at java.lang.Thread.run(Thread.java:701) > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On 28 May 2014 00:58, Andrija Panic <andrija.pa...@gmail.com> > wrote: > >> >>> > >> >>>> Hi Jayapal, > >> >>>> > >> >>>> eth1 seems present (lspci and virsh comfirmed), but not started > inside > >> >>>> VPC's VR - (VR used for Shared Network is fine)... > >> >>>> I could confirm by virsh that is is plugged inside appropriate > bridge > >> >>>> breth1-500 (check management logs...) > >> >>>> > >> >>>> management log while createing new VPC (VR) - > >> >>>> http://pastebin.com/s77nu5Ei > >> >>>> The public IP is there, so command is fine for creating it I > guess... > >> >>>> > >> >>>> VR's /var/log/cloud.log after rebooting VR from CS GUI > >> >>>> Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config > >> >>>> Tue May 27 22:46:58 UTC 2014 Detected that we are running inside > kvm > >> >>>> guest > >> >>>> Tue May 27 22:46:59 UTC 2014 Patching cloud service > >> >>>> Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml > >> >>>> Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system > vm > >> >>>> Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP > >> >>>> Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC > >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1 > >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1 > >> >>>> Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0 > >> >>>> Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0 > >> >>>> Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter > >> >>>> Tue May 27 22:47:00 UTC 2014 disable rpfilter > >> >>>> Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1 > >> >>>> Tue May 27 22:47:00 UTC 2014 enable_fwding = 1 > >> >>>> > >> >>>> ifconfig (no eth1 shown) > >> >>>> > >> >>>> eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c > >> >>>> inet addr:169.254.3.92 Bcast:169.254.255.255 > >> Mask:255.255.0.0 > >> >>>> > >> >>>> eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10 > >> >>>> inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 > >> >>>> > >> >>>> eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05 > >> >>>> inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0 > >> >>>> > >> >>>> lo Link encap:Local Loopback > >> >>>> inet addr:127.0.0.1 Mask:255.0.0.0 > >> >>>> > >> >>>> > >> >>>> cat /etc/network/interfaces > >> >>>> auto lo eth0 > >> >>>> iface lo inet loopback > >> >>>> iface eth0 inet static > >> >>>> address 169.254.3.92 > >> >>>> netmask 255.255.0.0 > >> >>>> > >> >>>> lspci - shows 4 ehternet addapters > >> >>>> ethtool eth1 = no link detected > >> >>>> virsh - confirmed that eth1 is plugged to correct bridge > (breth1-500) > >> as > >> >>>> indicated by management logs, and shows good MAC address as shown > in > >> >>>> managemetn log on pastebin.. > >> >>>> > >> >>>> This is completely makeing VPCs unusable... > >> >>>> :( > >> >>>> > >> >>>> Cheers > >> >>>> > >> >>>> > >> >>>> On 27 May 2014 16:36, Jayapal Reddy Uradi < > >> jayapalreddy.ur...@citrix.com > >> >>>> > wrote: > >> >>>> > >> >>>>> Hi, > >> >>>>> Can you please share management server and router logs in > >> pastebin.comto understand the issue ? > >> >>>>> > >> >>>>> Thanks, > >> >>>>> Jayapal > >> >>>>> > >> >>>>> On 27-May-2014, at 6:21 PM, Andrija Panic < > andrija.pa...@gmail.com> > >> >>>>> wrote: > >> >>>>> > >> >>>>> > Hi, > >> >>>>> > > >> >>>>> > after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC > lost > >> >>>>> their > >> >>>>> > eth1 which is public NIC. VR got eth0(control nic) and eth2 and > >> eth3 > >> >>>>> (bith > >> >>>>> > belonging to Tiers). From CS GUI, it is reported that the VR has > >> eth1 > >> >>>>> with > >> >>>>> > Public network attached, but from inside (ssh to VR) there is no > >> eth1 > >> >>>>> with > >> >>>>> > public IP... > >> >>>>> > > >> >>>>> > Even after destroying those VR, they are recreated again, but > >> without > >> >>>>> eth1. > >> >>>>> > > >> >>>>> > Anybody experienced same situtation ? > >> >>>>> > > >> >>>>> > Thanks, > >> >>>>> > > >> >>>>> > -- > >> >>>>> > > >> >>>>> > Andrija Panić > >> >>>>> > -------------------------------------- > >> >>>>> > >> >>>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> > >> >>>> > >> >>>> > >> > >> > >> > >> -- > >> Daan > >> > > > > > > > > -- > > > > Andrija Panić > > -------------------------------------- > > http://admintweets.com > > -------------------------------------- > > > > -- > Daan > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
