If you do a "virsh list" on the agent there's a good chance you would see a VM running, however the system will only wait so long for it to boot up before shutting it down, so it will come and go. You can do "virsh vncdisplay (vmname)" and it will tell you what port to vnc to on the host in order to connect to the VM and see what state it is in.
I see in the agent log that at one point it failed to start due to no private bridge. Is cloudbr0 your private as defined in agent.properties? You can also open /etc/cloud/agent/log4j-cloud.xml and change every INFO to DEBUG, restart the agent, and get more info. On Apr 13, 2013 11:45 AM, "Maurice Lawler" <[email protected]> wrote: > Thank you. > > The FSCK was already completed during boot up, it was forced. However, how > can I access the VM's when they are in starting state to see if they need a > FSCK? > > Agent log is showing this presently. > > > 2013-04-13 12:35:09,989 INFO [cloud.agent.Agent] > (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill > 2013-04-13 12:37:32,244 INFO [utils.component.ComponentLocator] > (main:null) Unable to find components.xml > 2013-04-13 12:37:32,285 INFO [utils.component.ComponentLocator] > (main:null) Skipping configuration using components.xml > 2013-04-13 12:37:32,285 INFO [cloud.agent.AgentShell] (main:null) > Implementation Version is 4.0.1.20130201075054 > 2013-04-13 12:37:32,286 INFO [cloud.agent.AgentShell] (main:null) > agent.properties found at /etc/cloud/agent/agent.properties > 2013-04-13 12:37:32,287 INFO [cloud.agent.AgentShell] (main:null) > Defaulting to using properties file for storage > 2013-04-13 12:37:32,289 INFO [cloud.agent.AgentShell] (main:null) > Defaulting to the constant time backoff algorithm > 2013-04-13 12:37:32,413 INFO [cloud.agent.Agent] (main:null) id is 1 > 2013-04-13 12:37:32,418 ERROR [cloud.resource.ServerResourceBase] > (main:null) Nics are not configured! > 2013-04-13 12:37:32,420 ERROR [cloud.agent.AgentShell] (main:null) Unable > to start agent: Private NIC is not configured > 2013-04-13 12:42:30,653 INFO [utils.component.ComponentLocator] > (main:null) Unable to find components.xml > 2013-04-13 12:42:30,654 INFO [utils.component.ComponentLocator] > (main:null) Skipping configuration using components.xml > 2013-04-13 12:42:30,654 INFO [cloud.agent.AgentShell] (main:null) > Implementation Version is 4.0.1.20130201075054 > 2013-04-13 12:42:30,655 INFO [cloud.agent.AgentShell] (main:null) > agent.properties found at /etc/cloud/agent/agent.properties > 2013-04-13 12:42:30,656 INFO [cloud.agent.AgentShell] (main:null) > Defaulting to using properties file for storage > 2013-04-13 12:42:30,658 INFO [cloud.agent.AgentShell] (main:null) > Defaulting to the constant time backoff algorithm > 2013-04-13 12:42:30,721 INFO [cloud.agent.Agent] (main:null) id is 1 > 2013-04-13 12:42:30,820 INFO > [resource.virtualnetwork.VirtualRoutingResource] (main:null) > VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm > 2013-04-13 12:42:32,094 INFO [kvm.resource.LibvirtComputingResource] > (main:null) No libvirt.vif.driver specififed. Defaults to BridgeVifDriver. > 2013-04-13 12:42:32,147 INFO [cloud.agent.Agent] (main:null) Agent [id = > 1 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : > host = 96.31.67.232 : port = 8250 > 2013-04-13 12:42:32,154 INFO [utils.nio.NioClient] (Agent-Selector:null) > Connecting to myipaddress:8250 > 2013-04-13 12:42:32,444 INFO [utils.nio.NioClient] (Agent-Selector:null) > SSL: Handshake done > 2013-04-13 12:42:32,599 INFO [cloud.serializer.GsonHelper] > (Agent-Handler-1:null) Default Builder inited. > 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > Proccess agent startup answer, agent id = 1 > 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > Set agent id 1 > 2013-04-13 12:42:32,808 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > Startup Response Received: agent id = 1 > > > The management log says this: > > 2013-04-13 12:43:28,952 DEBUG [cloud.network.NetworkManagerImpl] > (secstorage-1:null) Lock is released for network id 201 as a part of > network implement > 2013-04-13 12:43:28,969 DEBUG [db.Transaction.Transaction] > (secstorage-1:null) Rolling back the transaction: Time = 1 Name = > > -SystemVmLoadScanner$1.run:71-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRunAndReset:351-FutureTask.runAndReset:178-ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201:165-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:267-ThreadPoolExecutor.runWorker:1146-ThreadPoolExecutor$Worker.run:615-Thread.run:679; > called by > -Transaction.rollback:887-DataCenterIpAddressDaoImpl.takeIpAddress:57-DatabaseCallback.intercept:34-DataCenterDaoImpl.allocatePrivateIpAddress:228-DatabaseCallback.intercept:34-PodBasedNetworkGuru.reserve:119-NetworkManagerImpl.prepareNic:2143-NetworkManagerImpl.prepare:2113-VirtualMachineManagerImpl.advanceStart:752-VirtualMachineManagerImpl.start:472-VirtualMachineManagerImpl.start:465-SecondaryStorageManagerImpl.startSecStorageVm:257 > 2013-04-13 12:43:28,970 INFO [cloud.vm.VirtualMachineManagerImpl] > (secstorage-1:null) Insufficient capacity > com.cloud.exception.InsufficientAddressCapacityException: Unable to get a > management ip addressScope=interface com.cloud.dc.Pod; id=1 > at > com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121) > at > com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2143) > at > com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2113) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:752) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83) > at > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > 2013-04-13 12:43:28,973 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (secstorage-1:null) Cleaning up resources for the vm > VM[SecondaryStorageVm|s-588-VM] in Starting state > 2013-04-13 12:43:28,975 DEBUG [agent.transport.Request] > (secstorage-1:null) Seq 1-751304715: Waiting for Seq 751304714 Scheduling: > { Cmd , MgmtId: 219948120943996, via: 1, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"s-588-VM","wait":0}}] } > 2013-04-13 12:43:29,186 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:null) Found 0 routers. > 2013-04-13 12:43:37,927 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-14:null) Ping from 1 > 2013-04-13 12:43:43,240 DEBUG [cloud.server.StatsCollector] > (StatsCollector-3:null) VmStatsCollector is running... > 2013-04-13 12:43:43,323 DEBUG [cloud.server.StatsCollector] > (StatsCollector-3:null) StorageCollector is running... > 2013-04-13 12:43:43,327 DEBUG [cloud.server.StatsCollector] > (StatsCollector-3:null) There is no secondary storage VM for secondary > storage host nfs://96.31.67.232/secondary > 2013-04-13 12:43:43,400 DEBUG [agent.transport.Request] > (StatsCollector-3:null) Seq 1-751304716: Received: { Ans: , MgmtId: > 219948120943996, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > 2013-04-13 12:43:43,936 DEBUG [cloud.server.StatsCollector] > (StatsCollector-3:null) HostStatsCollector is running... > 2013-04-13 12:43:44,545 DEBUG [agent.transport.Request] > (StatsCollector-3:null) Seq 1-751304717: Received: { Ans: , MgmtId: > 219948120943996, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } > 2013-04-13 12:43:58,231 DEBUG [cloud.server.ManagementServerImpl] > (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 CDT > 2013 > 2013-04-13 12:43:58,233 DEBUG [cloud.server.ManagementServerImpl] > (EventChecker-1:null) Found 0 events to be purged > 2013-04-13 12:43:58,235 DEBUG [cloud.server.ManagementServerImpl] > (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 CDT > 2013 > 2013-04-13 12:43:58,238 DEBUG [cloud.server.ManagementServerImpl] > (EventChecker-1:null) Found 0 events to be purged > 2013-04-13 12:43:59,186 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:null) Found 0 routers. > [root@lunder agent]# > > > > > On Apr 13, 2013, at 12:30 PM, Marcus Sorensen <[email protected]> wrote: > > > Well you've got something trying to start, because you have vnet > > interfaces. You need to look at your agent logs to see why the system VMS > > refuse to start. If the power went out it could be corruption, the system > > VMS may be waiting for you to fsck. It sounds like maybe the system was > put > > into production without testing to make sure the host settings were > > persistent and would survive a reboot? > > > > So 1) look at your agent logs. And 2) use vnc to look at whatever system > > VMS are running and see what state they are in. They will probably > > continually try to start and then shut down. > > On Apr 13, 2013 11:24 AM, "Maurice Lawler" <[email protected]> > wrote: > > > >> Greetings, > >> > >> I'm have a terrible way to go, nothing I have done will start my cloud. > >> None of my system VM's will start, which in turn do not permit the > regular > >> OS VM's to start. I suffered from first a power outage, then I manually > >> rebooted my server. Now, nothing is coming back online. > >> > >> I was previously told, having cloud0 first is the cause of this. Even > when > >> doing ifconfig cloud0 down, nothing seems to come back online. > >> > >> I have gone as far as stopping iptables / eatables along with > >> stopping/starting the network and the management console. > >> > >> > >> Checking the system VM's the continue to remain in a 'starting' status. > >> > >> [root@lunder ~]# service iptables status > >> iptables: Firewall is not running. > >> [root@lunder ~]# service ebtables status > >> # Generated by ebtables-save v1.0 on Sat Apr 13 12:21:04 CDT 2013 > >> *nat > >> :PREROUTING ACCEPT > >> :OUTPUT ACCEPT > >> :POSTROUTING ACCEPT > >> > >> [root@lunder ~]# > >> > >> > >> [root@lunder daoenix]# ifconfig > >> cloud0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 > >> inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 > >> inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:658 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:0 > >> RX bytes:0 (0.0 b) TX bytes:28068 (27.4 KiB) > >> > >> cloudbr0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C > >> inet addr:myipaddress Bcast:9myipaddress Mask:255.255.255.224 > >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:192832 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:11251 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:0 > >> RX bytes:11481135 (10.9 MiB) TX bytes:25153331 (23.9 MiB) > >> > >> eth0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C > >> inet6 addr: fe80::ca0a:a9ff:fe9e:2d7c/64 Scope:Link > >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > >> RX packets:199794 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:24157 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:1000 > >> RX bytes:14647159 (13.9 MiB) TX bytes:25994485 (24.7 MiB) > >> Memory:df6e0000-df700000 > >> > >> lo Link encap:Local Loopback > >> inet addr:127.0.0.1 Mask:255.0.0.0 > >> inet6 addr: ::1/128 Scope:Host > >> UP LOOPBACK RUNNING MTU:16436 Metric:1 > >> RX packets:7850808 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:7850808 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:0 > >> RX bytes:1611132695 (1.5 GiB) TX bytes:1611132695 (1.5 GiB) > >> > >> virbr0 Link encap:Ethernet HWaddr 52:54:00:D9:D9:9A > >> inet addr:192.168.122.1 Bcast:192.168.122.255 > >> Mask:255.255.255.0 > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:0 > >> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > >> > >> vnet0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 > >> inet6 addr: fe80::fc00:a9ff:fefe:67/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 > >> collisions:0 txqueuelen:500 > >> RX bytes:0 (0.0 b) TX bytes:5232 (5.1 KiB) > >> > >> vnet1 Link encap:Ethernet HWaddr FE:84:4C:00:00:01 > >> inet6 addr: fe80::fc84:4cff:fe00:1/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 > >> collisions:0 txqueuelen:500 > >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) > >> > >> vnet2 Link encap:Ethernet HWaddr FE:2C:BC:00:00:05 > >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 > >> collisions:0 txqueuelen:500 > >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) > >> > >> [root@lunder daoenix]# > >> > >> > >> > >
