A "brctl show" would also be good to have. On Apr 13, 2013 11:52 AM, "Marcus Sorensen" <[email protected]> wrote:
> If you do a "virsh list" on the agent there's a good chance you would see > a VM running, however the system will only wait so long for it to boot up > before shutting it down, so it will come and go. You can do "virsh > vncdisplay (vmname)" and it will tell you what port to vnc to on the host > in order to connect to the VM and see what state it is in. > > I see in the agent log that at one point it failed to start due to no > private bridge. Is cloudbr0 your private as defined in agent.properties? > > You can also open /etc/cloud/agent/log4j-cloud.xml and change every INFO > to DEBUG, restart the agent, and get more info. > On Apr 13, 2013 11:45 AM, "Maurice Lawler" <[email protected]> wrote: > >> Thank you. >> >> The FSCK was already completed during boot up, it was forced. However, >> how can I access the VM's when they are in starting state to see if they >> need a FSCK? >> >> Agent log is showing this presently. >> >> >> 2013-04-13 12:35:09,989 INFO [cloud.agent.Agent] >> (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill >> 2013-04-13 12:37:32,244 INFO [utils.component.ComponentLocator] >> (main:null) Unable to find components.xml >> 2013-04-13 12:37:32,285 INFO [utils.component.ComponentLocator] >> (main:null) Skipping configuration using components.xml >> 2013-04-13 12:37:32,285 INFO [cloud.agent.AgentShell] (main:null) >> Implementation Version is 4.0.1.20130201075054 >> 2013-04-13 12:37:32,286 INFO [cloud.agent.AgentShell] (main:null) >> agent.properties found at /etc/cloud/agent/agent.properties >> 2013-04-13 12:37:32,287 INFO [cloud.agent.AgentShell] (main:null) >> Defaulting to using properties file for storage >> 2013-04-13 12:37:32,289 INFO [cloud.agent.AgentShell] (main:null) >> Defaulting to the constant time backoff algorithm >> 2013-04-13 12:37:32,413 INFO [cloud.agent.Agent] (main:null) id is 1 >> 2013-04-13 12:37:32,418 ERROR [cloud.resource.ServerResourceBase] >> (main:null) Nics are not configured! >> 2013-04-13 12:37:32,420 ERROR [cloud.agent.AgentShell] (main:null) Unable >> to start agent: Private NIC is not configured >> 2013-04-13 12:42:30,653 INFO [utils.component.ComponentLocator] >> (main:null) Unable to find components.xml >> 2013-04-13 12:42:30,654 INFO [utils.component.ComponentLocator] >> (main:null) Skipping configuration using components.xml >> 2013-04-13 12:42:30,654 INFO [cloud.agent.AgentShell] (main:null) >> Implementation Version is 4.0.1.20130201075054 >> 2013-04-13 12:42:30,655 INFO [cloud.agent.AgentShell] (main:null) >> agent.properties found at /etc/cloud/agent/agent.properties >> 2013-04-13 12:42:30,656 INFO [cloud.agent.AgentShell] (main:null) >> Defaulting to using properties file for storage >> 2013-04-13 12:42:30,658 INFO [cloud.agent.AgentShell] (main:null) >> Defaulting to the constant time backoff algorithm >> 2013-04-13 12:42:30,721 INFO [cloud.agent.Agent] (main:null) id is 1 >> 2013-04-13 12:42:30,820 INFO >> [resource.virtualnetwork.VirtualRoutingResource] (main:null) >> VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm >> 2013-04-13 12:42:32,094 INFO [kvm.resource.LibvirtComputingResource] >> (main:null) No libvirt.vif.driver specififed. Defaults to BridgeVifDriver. >> 2013-04-13 12:42:32,147 INFO [cloud.agent.Agent] (main:null) Agent [id = >> 1 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : >> host = 96.31.67.232 : port = 8250 >> 2013-04-13 12:42:32,154 INFO [utils.nio.NioClient] (Agent-Selector:null) >> Connecting to myipaddress:8250 >> 2013-04-13 12:42:32,444 INFO [utils.nio.NioClient] (Agent-Selector:null) >> SSL: Handshake done >> 2013-04-13 12:42:32,599 INFO [cloud.serializer.GsonHelper] >> (Agent-Handler-1:null) Default Builder inited. >> 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> Proccess agent startup answer, agent id = 1 >> 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> Set agent id 1 >> 2013-04-13 12:42:32,808 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> Startup Response Received: agent id = 1 >> >> >> The management log says this: >> >> 2013-04-13 12:43:28,952 DEBUG [cloud.network.NetworkManagerImpl] >> (secstorage-1:null) Lock is released for network id 201 as a part of >> network implement >> 2013-04-13 12:43:28,969 DEBUG [db.Transaction.Transaction] >> (secstorage-1:null) Rolling back the transaction: Time = 1 Name = >> >> -SystemVmLoadScanner$1.run:71-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRunAndReset:351-FutureTask.runAndReset:178-ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201:165-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:267-ThreadPoolExecutor.runWorker:1146-ThreadPoolExecutor$Worker.run:615-Thread.run:679; >> called by >> -Transaction.rollback:887-DataCenterIpAddressDaoImpl.takeIpAddress:57-DatabaseCallback.intercept:34-DataCenterDaoImpl.allocatePrivateIpAddress:228-DatabaseCallback.intercept:34-PodBasedNetworkGuru.reserve:119-NetworkManagerImpl.prepareNic:2143-NetworkManagerImpl.prepare:2113-VirtualMachineManagerImpl.advanceStart:752-VirtualMachineManagerImpl.start:472-VirtualMachineManagerImpl.start:465-SecondaryStorageManagerImpl.startSecStorageVm:257 >> 2013-04-13 12:43:28,970 INFO [cloud.vm.VirtualMachineManagerImpl] >> (secstorage-1:null) Insufficient capacity >> com.cloud.exception.InsufficientAddressCapacityException: Unable to get a >> management ip addressScope=interface com.cloud.dc.Pod; id=1 >> at >> com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121) >> at >> com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2143) >> at >> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2113) >> at >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:752) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472) >> at >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465) >> at >> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257) >> at >> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684) >> at >> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310) >> at >> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119) >> at >> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) >> at >> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106) >> at >> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34) >> at >> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83) >> at >> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) >> at >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:679) >> 2013-04-13 12:43:28,973 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (secstorage-1:null) Cleaning up resources for the vm >> VM[SecondaryStorageVm|s-588-VM] in Starting state >> 2013-04-13 12:43:28,975 DEBUG [agent.transport.Request] >> (secstorage-1:null) Seq 1-751304715: Waiting for Seq 751304714 Scheduling: >> { Cmd , MgmtId: 219948120943996, via: 1, Ver: v1, Flags: 100111, >> [{"StopCommand":{"isProxy":false,"vmName":"s-588-VM","wait":0}}] } >> 2013-04-13 12:43:29,186 DEBUG >> [network.router.VirtualNetworkApplianceManagerImpl] >> (RouterStatusMonitor-1:null) Found 0 routers. >> 2013-04-13 12:43:37,927 DEBUG [agent.manager.AgentManagerImpl] >> (AgentManager-Handler-14:null) Ping from 1 >> 2013-04-13 12:43:43,240 DEBUG [cloud.server.StatsCollector] >> (StatsCollector-3:null) VmStatsCollector is running... >> 2013-04-13 12:43:43,323 DEBUG [cloud.server.StatsCollector] >> (StatsCollector-3:null) StorageCollector is running... >> 2013-04-13 12:43:43,327 DEBUG [cloud.server.StatsCollector] >> (StatsCollector-3:null) There is no secondary storage VM for secondary >> storage host nfs://96.31.67.232/secondary >> 2013-04-13 <http://96.31.67.232/secondary2013-04-13> 12:43:43,400 DEBUG >> [agent.transport.Request] (StatsCollector-3:null) Seq 1-751304716: >> Received: { Ans: , MgmtId: 219948120943996, via: 1, Ver: v1, Flags: 10, { >> GetStorageStatsAnswer } } >> 2013-04-13 12:43:43,936 DEBUG [cloud.server.StatsCollector] >> (StatsCollector-3:null) HostStatsCollector is running... >> 2013-04-13 12:43:44,545 DEBUG [agent.transport.Request] >> (StatsCollector-3:null) Seq 1-751304717: Received: { Ans: , MgmtId: >> 219948120943996, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } >> 2013-04-13 12:43:58,231 DEBUG [cloud.server.ManagementServerImpl] >> (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 CDT >> 2013 >> 2013-04-13 12:43:58,233 DEBUG [cloud.server.ManagementServerImpl] >> (EventChecker-1:null) Found 0 events to be purged >> 2013-04-13 12:43:58,235 DEBUG [cloud.server.ManagementServerImpl] >> (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 CDT >> 2013 >> 2013-04-13 12:43:58,238 DEBUG [cloud.server.ManagementServerImpl] >> (EventChecker-1:null) Found 0 events to be purged >> 2013-04-13 12:43:59,186 DEBUG >> [network.router.VirtualNetworkApplianceManagerImpl] >> (RouterStatusMonitor-1:null) Found 0 routers. >> [root@lunder agent]# >> >> >> >> >> On Apr 13, 2013, at 12:30 PM, Marcus Sorensen <[email protected]> >> wrote: >> >> > Well you've got something trying to start, because you have vnet >> > interfaces. You need to look at your agent logs to see why the system >> VMS >> > refuse to start. If the power went out it could be corruption, the >> system >> > VMS may be waiting for you to fsck. It sounds like maybe the system was >> put >> > into production without testing to make sure the host settings were >> > persistent and would survive a reboot? >> > >> > So 1) look at your agent logs. And 2) use vnc to look at whatever system >> > VMS are running and see what state they are in. They will probably >> > continually try to start and then shut down. >> > On Apr 13, 2013 11:24 AM, "Maurice Lawler" <[email protected]> >> wrote: >> > >> >> Greetings, >> >> >> >> I'm have a terrible way to go, nothing I have done will start my cloud. >> >> None of my system VM's will start, which in turn do not permit the >> regular >> >> OS VM's to start. I suffered from first a power outage, then I manually >> >> rebooted my server. Now, nothing is coming back online. >> >> >> >> I was previously told, having cloud0 first is the cause of this. Even >> when >> >> doing ifconfig cloud0 down, nothing seems to come back online. >> >> >> >> I have gone as far as stopping iptables / eatables along with >> >> stopping/starting the network and the management console. >> >> >> >> >> >> Checking the system VM's the continue to remain in a 'starting' status. >> >> >> >> [root@lunder ~]# service iptables status >> >> iptables: Firewall is not running. >> >> [root@lunder ~]# service ebtables status >> >> # Generated by ebtables-save v1.0 on Sat Apr 13 12:21:04 CDT 2013 >> >> *nat >> >> :PREROUTING ACCEPT >> >> :OUTPUT ACCEPT >> >> :POSTROUTING ACCEPT >> >> >> >> [root@lunder ~]# >> >> >> >> >> >> [root@lunder daoenix]# ifconfig >> >> cloud0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 >> >> inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 >> >> inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:658 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:0 >> >> RX bytes:0 (0.0 b) TX bytes:28068 (27.4 KiB) >> >> >> >> cloudbr0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C >> >> inet addr:myipaddress Bcast:9myipaddress Mask:255.255.255.224 >> >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:192832 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:11251 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:0 >> >> RX bytes:11481135 (10.9 MiB) TX bytes:25153331 (23.9 MiB) >> >> >> >> eth0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C >> >> inet6 addr: fe80::ca0a:a9ff:fe9e:2d7c/64 Scope:Link >> >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >> >> RX packets:199794 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:24157 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:1000 >> >> RX bytes:14647159 (13.9 MiB) TX bytes:25994485 (24.7 MiB) >> >> Memory:df6e0000-df700000 >> >> >> >> lo Link encap:Local Loopback >> >> inet addr:127.0.0.1 Mask:255.0.0.0 >> >> inet6 addr: ::1/128 Scope:Host >> >> UP LOOPBACK RUNNING MTU:16436 Metric:1 >> >> RX packets:7850808 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:7850808 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:0 >> >> RX bytes:1611132695 (1.5 GiB) TX bytes:1611132695 (1.5 GiB) >> >> >> >> virbr0 Link encap:Ethernet HWaddr 52:54:00:D9:D9:9A >> >> inet addr:192.168.122.1 Bcast:192.168.122.255 >> >> Mask:255.255.255.0 >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:0 >> >> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >> >> >> >> vnet0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 >> >> inet6 addr: fe80::fc00:a9ff:fefe:67/64 Scope:Link >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 >> >> collisions:0 txqueuelen:500 >> >> RX bytes:0 (0.0 b) TX bytes:5232 (5.1 KiB) >> >> >> >> vnet1 Link encap:Ethernet HWaddr FE:84:4C:00:00:01 >> >> inet6 addr: fe80::fc84:4cff:fe00:1/64 Scope:Link >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 >> >> collisions:0 txqueuelen:500 >> >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) >> >> >> >> vnet2 Link encap:Ethernet HWaddr FE:2C:BC:00:00:05 >> >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 >> >> collisions:0 txqueuelen:500 >> >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) >> >> >> >> [root@lunder daoenix]# >> >> >> >> >> >> >> >>
