[
https://issues.apache.org/jira/browse/CLOUDSTACK-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13770450#comment-13770450
]
Paul Edwards commented on CLOUDSTACK-4690:
------------------------------------------
As I said above, there was no activity in the agent log. I was tailing it at
the time. Below I have pasted the entire log from the host where the vm was
running, as you can see there was no action at the period expected. Now it
maybe that I need to turn some logging on somewhere. If so please tell me where.
Thanks
P.
2013-09-17 06:59:14,628 INFO [cloud.agent.AgentShell] (main:null) Agent started
2013-09-17 06:59:14,630 INFO [cloud.agent.AgentShell] (main:null)
Implementation Version is 4.1.1
2013-09-17 06:59:14,632 INFO [cloud.agent.AgentShell] (main:null)
agent.properties found at /etc/cloudstack/agent/agent.properties
2013-09-17 06:59:14,634 INFO [cloud.agent.AgentShell] (main:null) Defaulting
to using properties file for storage
2013-09-17 06:59:14,635 INFO [cloud.agent.AgentShell] (main:null) Defaulting
to the constant time backoff algorithm
2013-09-17 06:59:14,704 INFO [cloud.agent.Agent] (main:null) id is 17
2013-09-17 06:59:14,724 INFO [resource.virtualnetwork.VirtualRoutingResource]
(main:null) VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm
2013-09-17 06:59:15,313 INFO [kvm.resource.LibvirtComputingResource]
(main:null) No libvirt.vif.driver specififed. Defaults to BridgeVifDriver.
2013-09-17 06:59:15,432 INFO [cloud.agent.Agent] (main:null) Agent [id = 17 :
type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host =
xxx.xxx.245.48 : port = 8250
2013-09-17 06:59:15,450 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to xxx.xxx.245.48:8250
2013-09-17 06:59:18,456 ERROR [utils.nio.NioConnection] (Agent-Selector:null)
Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
at java.lang.Thread.run(Thread.java:679)
2013-09-17 06:59:23,475 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to xxx.xxx.245.48:8250
2013-09-17 06:59:26,476 ERROR [utils.nio.NioConnection] (Agent-Selector:null)
Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
at java.lang.Thread.run(Thread.java:679)
2013-09-17 06:59:31,482 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to xxx.xxx.245.48:8250
2013-09-17 06:59:31,656 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL:
Handshake done
2013-09-17 06:59:31,911 INFO [cloud.serializer.GsonHelper]
(Agent-Handler-1:null) Default Builder inited.
2013-09-17 06:59:31,953 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
Proccess agent startup answer, agent id = 0
2013-09-17 06:59:31,953 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Set
agent id 0
2013-09-17 06:59:31,960 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
Startup Response Received: agent id = 0
2013-09-17 07:01:06,324 INFO [cloud.agent.Agent] (agentRequest-Handler-1:null)
Proccess agent ready command, agent id = 17
2013-09-17 07:01:06,325 INFO [cloud.agent.Agent] (agentRequest-Handler-1:null)
Set agent id 17
2013-09-17 07:01:06,326 INFO [cloud.agent.Agent] (agentRequest-Handler-1:null)
Ready command is processed: agent id = 17
2013-09-17 07:01:06,576 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null)
Proccess agent ready command, agent id = 17
2013-09-17 07:01:06,577 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null)
Set agent id 17
2013-09-17 07:01:06,585 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null)
Ready command is processed: agent id = 17
2013-09-17 07:39:57,937 WARN [kvm.resource.KVMHAMonitor] (Script-7:null)
Interrupting script.
2013-09-17 07:39:57,940 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
172.16.16.16 -p /cassandra2 -m /mnt/18c08b3e-251e-325e-b910-3063fa00756f -h
172.16.16.14 . Output is:
2013-09-17 07:39:57,940 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
write heartbeat failed: timeout, retry: 0
2013-09-17 07:40:57,949 WARN [kvm.resource.KVMHAMonitor] (Script-6:null)
Interrupting script.
2013-09-17 07:40:57,950 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
172.16.16.16 -p /cassandra2 -m /mnt/18c08b3e-251e-325e-b910-3063fa00756f -h
172.16.16.14 . Output is:
2013-09-17 07:40:57,950 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
write heartbeat failed: timeout, retry: 1
2013-09-17 07:41:57,958 WARN [kvm.resource.KVMHAMonitor] (Script-10:null)
Interrupting script.
2013-09-17 07:41:57,958 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
172.16.16.16 -p /cassandra2 -m /mnt/18c08b3e-251e-325e-b910-3063fa00756f -h
172.16.16.14 . Output is:
2013-09-17 07:41:57,959 WARN [kvm.resource.KVMHAMonitor] (Thread-42:null)
write heartbeat failed: timeout, retry: 2
> KVM Router - to many ethernet devices created
> ---------------------------------------------
>
> Key: CLOUDSTACK-4690
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4690
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
> Affects Versions: 4.1.1
> Environment: Centos 6.4
> Reporter: Paul Edwards
> Priority: Critical
> Attachments: management-server.log, router.log
>
>
> We have setup cloudstack with advanced networking. We have a network created
> 172.16.24.0/23, called news preprod. This has a number of vms created under
> it. When we initially set this up, we noticed that the router we created with
> 4 ethernet interfaces. This was unexpected, but didn't seem to be effecting
> the running of the router, so we didn't worry about it. The interfaces were:
> eth0 172.16.24.14, eth1 169.254.1.91, and both eth2 and eth2 were given the
> external ip (xxx.xxx.245.59). We investigated why we were getting 2
> externals, but couldn't figure it out. We also configured the router to have
> a redundant, with keepalived. They were working fine.
> Now I needed to add another external ip to the network. Did that, and
> restarted the network. I now have 8 (eight) eth interfaces. The new external
> ip is NOT one of them. Heres the ifconfig output:
> root@r-235-VM:~# ifconfig
> eth0 Link encap:Ethernet HWaddr 02:00:68:e1:00:1a
> inet addr:172.16.24.14 Bcast:172.16.25.255 Mask:255.255.254.0
> inet6 addr: fe80::68ff:fee1:1a/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:5485 errors:0 dropped:0 overruns:0 frame:0
> TX packets:21101 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:325618 (317.9 KiB) TX bytes:1581746 (1.5 MiB)
> eth1 Link encap:Ethernet HWaddr 0e:00:a9:fe:01:5b
> inet addr:169.254.1.91 Bcast:169.254.255.255 Mask:255.255.0.0
> inet6 addr: fe80::c00:a9ff:fefe:15b/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:22118 errors:0 dropped:0 overruns:0 frame:0
> TX packets:21148 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:3942785 (3.7 MiB) TX bytes:4287970 (4.0 MiB)
> eth2 Link encap:Ethernet HWaddr 06:16:06:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::416:6ff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:17331 errors:0 dropped:0 overruns:0 frame:0
> TX packets:3930 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:1063053 (1.0 MiB) TX bytes:234546 (229.0 KiB)
> eth3 Link encap:Ethernet HWaddr 06:d7:ec:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::4d7:ecff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13603 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:792907 (774.3 KiB) TX bytes:704 (704.0 B)
> eth4 Link encap:Ethernet HWaddr 06:8b:5a:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::48b:5aff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13319 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:779753 (761.4 KiB) TX bytes:704 (704.0 B)
> eth5 Link encap:Ethernet HWaddr 06:7e:68:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::47e:68ff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13080 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:762111 (744.2 KiB) TX bytes:704 (704.0 B)
> eth6 Link encap:Ethernet HWaddr 06:7e:68:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::47e:68ff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13086 errors:0 dropped:0 overruns:0 frame:0
> TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:762655 (744.7 KiB) TX bytes:774 (774.0 B)
> eth7 Link encap:Ethernet HWaddr 06:27:a8:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::427:a8ff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13064 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:761153 (743.3 KiB) TX bytes:704 (704.0 B)
> eth8 Link encap:Ethernet HWaddr 06:27:a8:00:01:03
> inet addr:xxx.xxx.245.59 Bcast:xxx.xxx.245.255 Mask:255.255.255.0
> inet6 addr: fe80::427:a8ff:fe00:103/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:13043 errors:0 dropped:0 overruns:0 frame:0
> TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:759287 (741.4 KiB) TX bytes:704 (704.0 B)
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:6 errors:0 dropped:0 overruns:0 frame:0
> TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:414 (414.0 B) TX bytes:414 (414.0 B)
> The cloudstack management browser still shows the router as only supposed to
> have 3 nics.
> The agent log on the physical host has nothing in it (I was tailing that log
> when I did this, as I would have thought that the agent has to create the
> devices for the vm to get them).
> I have attached the management log from just before to just after. (Time of
> change was 9:39am). I do not know why the router timestamp is not the same as
> the host.
> I have also attached the log of the router restart.
> The physical server is Centos 6.4, and all the vms that we have created are
> Centos 6.4. KVM is the hypervisor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
