Hi Adrija,
From the logs, the public subnet is untagged.
I think this issue is coming for the untagged public vlan in 4.3.
1.
{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"e6b734d4-3302-4113-8ec7-5c205c90959a","ip":"46.232.180.248","netmask":"255.255.255.0","gateway":"46.232.180.1","mac":"06:5e:e8:00:00:27","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},
2.
3.
"instanceName":"r-779-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"46.232.180.248","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"untagged","vlanGateway":"46.232.180.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:5e:e8:00:00:27","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails":
From the logs VR logs, the ipassoc script got the interface id as null.
May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to
appear, 0 seconds
Thanks,
Jayapal
On 29-May-2014, at 1:08 AM, Andrija Panic
<[email protected]<mailto:[email protected]>> wrote:
Thanks Daan,
my problem is that I'm in production for 3rd day now, and restoring DB and
downgrading back to 4.2.1 doesn't seem as option for me at the moment,
since I would loose new acounts and single VMs, etc...
Thanks,
Andrija
On 28 May 2014 21:34, Daan Hoogland
<[email protected]<mailto:[email protected]>> wrote:
Andrija,
nevertheless it sounds familiar. I will be back in the office on
monday and ask around.
On Wed, May 28, 2014 at 9:23 PM, Andrija Panic
<[email protected]<mailto:[email protected]>>
wrote:
Hi Daan,
I don't think this is my issue, at least I don't make use of private
gateway - this is just simple as: create new VPC from scratch - Public
IP
is not assigned to VR eth1 interface inside VR...
I have filed the bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-6801
This same thing happened previously to Andrei Mikhailovsky:
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand
it is not resolved
Thanks,
Andrija
On 28 May 2014 21:01, Daan Hoogland <[email protected]> wrote:
Andrija,
this sound like something we seen as well.
can you check if this is it :
https://issues.apache.org/jira/browse/CLOUDSTACK-6485
thanks,
Daan
On Wed, May 28, 2014 at 3:30 PM, Andrija Panic <[email protected]
wrote:
Hi there,
I'm having big time problems with Public IP missing from VPC VR's
eth1,
after upgrade to ACS 4.3.1 - did not found this filed as bug so
far...and
it worked all fine on ACS 4.2.1.
No help so far from user mailing list...
Below is a detailed explanation, and logs from inside VR, and from
management (all fine with management logs...)
If anybody can help, I would very much appriciate this, since now I
have
bunch fo VPC unoperational...
Thanks
---------- Forwarded message ----------
From: Andrija Panic <[email protected]>
Date: 28 May 2014 14:50
Subject: Re: VPC's VR missing public NIC eth1
To: [email protected]
and as I said eth1 is present:
root@r-794-VM:~# cat /proc/net/dev
Inter-| Receive |
Transmit
face |bytes packets errs drop fifo frame compressed
multicast|bytes
packets errs drop fifo colls carrier compressed
eth3: 11484 131 0 0 0 0 0 0
11590
131 0 0 0 0 0 0
lo: 214 2 0 0 0 0 0 0
214
2 0 0 0 0 0 0
eth2: 32970 544 0 0 0 0 0 0
2084
24 0 0 0 0 0 0
eth1: 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0
eth0: 150207 1319 0 0 0 0 0 0
264232
1180 0 0 0 0 0 0
On 28 May 2014 14:47, Andrija Panic <[email protected]> wrote:
Also, from /var/log/messages/ inside VR:
This is a major show stopper - all our VPCs are unusable complete.
Anybody... ?
May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 0 seconds
May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 1 seconds
May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 2 seconds
May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 3 seconds
May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 4 seconds
May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 5 seconds
May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 6 seconds
May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 7 seconds
May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 8 seconds
May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 9 seconds
May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 10 seconds
May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 11 seconds
May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 12 seconds
May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 13 seconds
May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 14 seconds
May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 15 seconds
May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface
ethnull to appear, 16 seconds
May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull
never
appeared
May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip
46.232.180.246
on
interface ethnull
May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing
46.232.180.246
on interface ethnull
May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT
46.232.180.246 on interface ethnull
May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT
46.232.180.246
on interface eth1
On 28 May 2014 12:59, Andrija Panic <[email protected]> wrote:
Defined eth1 manually inside /etc/network/interfaces inside VPC's
VR.
iface eth1 inet static
address 46.232.180.246
netmask 255.255.255.0
ifup eth1
ip route add default via 46.232.180.1
so now VR works fine (have access to internet)
But again, adding new IP to VR, and enabling static NAT is
failing...
That is, geting new IP works fine (just associated with account)
But enabling static NAT fails, due to "resource unavailable"
Here are management logs:
2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl]
(catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static
nat
rule due to
com.cloud.exception.ResourceUnavailableException: Resource
[DataCenter:1]
is unreachable: Unable to apply static nat rules on router
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source)
at
com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650)
at
com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762)
at
com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324)
at
com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602)
at
com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source)
at
org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129)
at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323)
at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
On 28 May 2014 00:58, Andrija Panic <[email protected]>
wrote:
Hi Jayapal,
eth1 seems present (lspci and virsh comfirmed), but not started
inside
VPC's VR - (VR used for Shared Network is fine)...
I could confirm by virsh that is is plugged inside appropriate
bridge
breth1-500 (check management logs...)
management log while createing new VPC (VR) -
http://pastebin.com/s77nu5Ei
The public IP is there, so command is fine for creating it I
guess...
VR's /var/log/cloud.log after rebooting VR from CS GUI
Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config
Tue May 27 22:46:58 UTC 2014 Detected that we are running inside
kvm
guest
Tue May 27 22:46:59 UTC 2014 Patching cloud service
Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml
Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system
vm
Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP
Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC
Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1
Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1
Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0
Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0
Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter
Tue May 27 22:47:00 UTC 2014 disable rpfilter
Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1
Tue May 27 22:47:00 UTC 2014 enable_fwding = 1
ifconfig (no eth1 shown)
eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c
inet addr:169.254.3.92 Bcast:169.254.255.255
Mask:255.255.0.0
eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10
inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0
eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05
inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
cat /etc/network/interfaces
auto lo eth0
iface lo inet loopback
iface eth0 inet static
address 169.254.3.92
netmask 255.255.0.0
lspci - shows 4 ehternet addapters
ethtool eth1 = no link detected
virsh - confirmed that eth1 is plugged to correct bridge
(breth1-500)
as
indicated by management logs, and shows good MAC address as shown
in
managemetn log on pastebin..
This is completely makeing VPCs unusable...
:(
Cheers
On 27 May 2014 16:36, Jayapal Reddy Uradi <
[email protected]
wrote:
Hi,
Can you please share management server and router logs in
pastebin.comto understand the issue ?
Thanks,
Jayapal
On 27-May-2014, at 6:21 PM, Andrija Panic <
[email protected]>
wrote:
Hi,
after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC
lost
their
eth1 which is public NIC. VR got eth0(control nic) and eth2 and
eth3
(bith
belonging to Tiers). From CS GUI, it is reported that the VR has
eth1
with
Public network attached, but from inside (ssh to VR) there is no
eth1
with
public IP...
Even after destroying those VR, they are recreated again, but
without
eth1.
Anybody experienced same situtation ?
Thanks,
--
Andrija Panić
--------------------------------------
--
--
Daan
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
--
Daan
--
Andrija Panić
--------------------------------------
http://admintweets.com
--------------------------------------
