Hi there,
I'm having big time problems with Public IP missing from VPC VR's eth1,
after upgrade to ACS 4.3.1 - did not found this filed as bug so far...and
it worked all fine on ACS 4.2.1.
No help so far from user mailing list...
Below is a detailed explanation, and logs from inside VR, and from
management (all fine with management logs...)
If anybody can help, I would very much appriciate this, since now I have
bunch fo VPC unoperational...
Thanks
---------- Forwarded message ----------
From: Andrija Panic <andrija.pa...@gmail.com>
Date: 28 May 2014 14:50
Subject: Re: VPC's VR missing public NIC eth1
To: us...@cloudstack.apache.org
and as I said eth1 is present:
root@r-794-VM:~# cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes
packets errs drop fifo colls carrier compressed
eth3: 11484 131 0 0 0 0 0 0 11590
131 0 0 0 0 0 0
lo: 214 2 0 0 0 0 0 0 214
2 0 0 0 0 0 0
eth2: 32970 544 0 0 0 0 0 0 2084
24 0 0 0 0 0 0
eth1: 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0
eth0: 150207 1319 0 0 0 0 0 0 264232
1180 0 0 0 0 0 0
On 28 May 2014 14:47, Andrija Panic <andrija.pa...@gmail.com> wrote:
> Also, from /var/log/messages/ inside VR:
>
> This is a major show stopper - all our VPCs are unusable complete.
> Anybody... ?
>
> May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 0 seconds
> May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 1 seconds
> May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 2 seconds
> May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 3 seconds
> May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 4 seconds
> May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 5 seconds
> May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 6 seconds
> May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 7 seconds
> May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 8 seconds
> May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 9 seconds
> May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 10 seconds
> May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 11 seconds
> May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 12 seconds
> May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 13 seconds
> May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 14 seconds
> May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 15 seconds
> May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
> ethnull to appear, 16 seconds
> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull never
> appeared
> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip 46.232.180.246 on
> interface ethnull
> May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing 46.232.180.246
> on interface ethnull
> May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT
> 46.232.180.246 on interface ethnull
> May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT 46.232.180.246
> on interface eth1
>
>
>
>
> On 28 May 2014 12:59, Andrija Panic <andrija.pa...@gmail.com> wrote:
>
>> Defined eth1 manually inside /etc/network/interfaces inside VPC's VR.
>> iface eth1 inet static
>> address 46.232.180.246
>> netmask 255.255.255.0
>>
>> ifup eth1
>> ip route add default via 46.232.180.1
>>
>> so now VR works fine (have access to internet)
>>
>> But again, adding new IP to VR, and enabling static NAT is failing...
>> That is, geting new IP works fine (just associated with account)
>> But enabling static NAT fails, due to "resource unavailable"
>>
>> Here are management logs:
>> 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl]
>> (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static nat
>> rule due to
>> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1]
>> is unreachable: Unable to apply static nat rules on router
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915)
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:622)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source)
>> at
>> com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650)
>> at
>> com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762)
>> at
>> com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324)
>> at
>> com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602)
>> at
>> com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:622)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531)
>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374)
>> at
>> com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323)
>> at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53)
>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112)
>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>> at
>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>> at
>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:701)
>>
>>
>>
>>
>> On 28 May 2014 00:58, Andrija Panic <andrija.pa...@gmail.com> wrote:
>>
>>> Hi Jayapal,
>>>
>>> eth1 seems present (lspci and virsh comfirmed), but not started inside
>>> VPC's VR - (VR used for Shared Network is fine)...
>>> I could confirm by virsh that is is plugged inside appropriate bridge
>>> breth1-500 (check management logs...)
>>>
>>> management log while createing new VPC (VR) -
>>> http://pastebin.com/s77nu5Ei
>>> The public IP is there, so command is fine for creating it I guess...
>>>
>>> VR's /var/log/cloud.log after rebooting VR from CS GUI
>>> Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config
>>> Tue May 27 22:46:58 UTC 2014 Detected that we are running inside kvm
>>> guest
>>> Tue May 27 22:46:59 UTC 2014 Patching cloud service
>>> Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml
>>> Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system vm
>>> Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP
>>> Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC
>>> Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1
>>> Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1
>>> Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0
>>> Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0
>>> Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter
>>> Tue May 27 22:47:00 UTC 2014 disable rpfilter
>>> Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1
>>> Tue May 27 22:47:00 UTC 2014 enable_fwding = 1
>>>
>>> ifconfig (no eth1 shown)
>>>
>>> eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c
>>> inet addr:169.254.3.92 Bcast:169.254.255.255 Mask:255.255.0.0
>>>
>>> eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10
>>> inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
>>>
>>> eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05
>>> inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0
>>>
>>> lo Link encap:Local Loopback
>>> inet addr:127.0.0.1 Mask:255.0.0.0
>>>
>>>
>>> cat /etc/network/interfaces
>>> auto lo eth0
>>> iface lo inet loopback
>>> iface eth0 inet static
>>> address 169.254.3.92
>>> netmask 255.255.0.0
>>>
>>> lspci - shows 4 ehternet addapters
>>> ethtool eth1 = no link detected
>>> virsh - confirmed that eth1 is plugged to correct bridge (breth1-500) as
>>> indicated by management logs, and shows good MAC address as shown in
>>> managemetn log on pastebin..
>>>
>>> This is completely makeing VPCs unusable...
>>> :(
>>>
>>> Cheers
>>>
>>>
>>> On 27 May 2014 16:36, Jayapal Reddy Uradi <jayapalreddy.ur...@citrix.com
>>> > wrote:
>>>
>>>> Hi,
>>>> Can you please share management server and router logs in pastebin.comto
>>>> understand the issue ?
>>>>
>>>> Thanks,
>>>> Jayapal
>>>>
>>>> On 27-May-2014, at 6:21 PM, Andrija Panic <andrija.pa...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC lost
>>>> their
>>>> > eth1 which is public NIC. VR got eth0(control nic) and eth2 and eth3
>>>> (bith
>>>> > belonging to Tiers). From CS GUI, it is reported that the VR has eth1
>>>> with
>>>> > Public network attached, but from inside (ssh to VR) there is no eth1
>>>> with
>>>> > public IP...
>>>> >
>>>> > Even after destroying those VR, they are recreated again, but without
>>>> eth1.
>>>> >
>>>> > Anybody experienced same situtation ?
>>>> >
>>>> > Thanks,
>>>> >
>>>> > --
>>>> >
>>>> > Andrija Panić
>>>> > --------------------------------------
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>>
