Do you have a secondary storage defined? , does it work ? are the system vm templates in it? go to infrastructure and check the status of the system vms.
Thanks On Sat, Apr 13, 2013 at 12:57 PM, Marcus Sorensen <[email protected]>wrote: > A "brctl show" would also be good to have. > On Apr 13, 2013 11:52 AM, "Marcus Sorensen" <[email protected]> wrote: > > > If you do a "virsh list" on the agent there's a good chance you would see > > a VM running, however the system will only wait so long for it to boot up > > before shutting it down, so it will come and go. You can do "virsh > > vncdisplay (vmname)" and it will tell you what port to vnc to on the host > > in order to connect to the VM and see what state it is in. > > > > I see in the agent log that at one point it failed to start due to no > > private bridge. Is cloudbr0 your private as defined in agent.properties? > > > > You can also open /etc/cloud/agent/log4j-cloud.xml and change every INFO > > to DEBUG, restart the agent, and get more info. > > On Apr 13, 2013 11:45 AM, "Maurice Lawler" <[email protected]> > wrote: > > > >> Thank you. > >> > >> The FSCK was already completed during boot up, it was forced. However, > >> how can I access the VM's when they are in starting state to see if they > >> need a FSCK? > >> > >> Agent log is showing this presently. > >> > >> > >> 2013-04-13 12:35:09,989 INFO [cloud.agent.Agent] > >> (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill > >> 2013-04-13 12:37:32,244 INFO [utils.component.ComponentLocator] > >> (main:null) Unable to find components.xml > >> 2013-04-13 12:37:32,285 INFO [utils.component.ComponentLocator] > >> (main:null) Skipping configuration using components.xml > >> 2013-04-13 12:37:32,285 INFO [cloud.agent.AgentShell] (main:null) > >> Implementation Version is 4.0.1.20130201075054 > >> 2013-04-13 12:37:32,286 INFO [cloud.agent.AgentShell] (main:null) > >> agent.properties found at /etc/cloud/agent/agent.properties > >> 2013-04-13 12:37:32,287 INFO [cloud.agent.AgentShell] (main:null) > >> Defaulting to using properties file for storage > >> 2013-04-13 12:37:32,289 INFO [cloud.agent.AgentShell] (main:null) > >> Defaulting to the constant time backoff algorithm > >> 2013-04-13 12:37:32,413 INFO [cloud.agent.Agent] (main:null) id is 1 > >> 2013-04-13 12:37:32,418 ERROR [cloud.resource.ServerResourceBase] > >> (main:null) Nics are not configured! > >> 2013-04-13 12:37:32,420 ERROR [cloud.agent.AgentShell] (main:null) > Unable > >> to start agent: Private NIC is not configured > >> 2013-04-13 12:42:30,653 INFO [utils.component.ComponentLocator] > >> (main:null) Unable to find components.xml > >> 2013-04-13 12:42:30,654 INFO [utils.component.ComponentLocator] > >> (main:null) Skipping configuration using components.xml > >> 2013-04-13 12:42:30,654 INFO [cloud.agent.AgentShell] (main:null) > >> Implementation Version is 4.0.1.20130201075054 > >> 2013-04-13 12:42:30,655 INFO [cloud.agent.AgentShell] (main:null) > >> agent.properties found at /etc/cloud/agent/agent.properties > >> 2013-04-13 12:42:30,656 INFO [cloud.agent.AgentShell] (main:null) > >> Defaulting to using properties file for storage > >> 2013-04-13 12:42:30,658 INFO [cloud.agent.AgentShell] (main:null) > >> Defaulting to the constant time backoff algorithm > >> 2013-04-13 12:42:30,721 INFO [cloud.agent.Agent] (main:null) id is 1 > >> 2013-04-13 12:42:30,820 INFO > >> [resource.virtualnetwork.VirtualRoutingResource] (main:null) > >> VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm > >> 2013-04-13 12:42:32,094 INFO [kvm.resource.LibvirtComputingResource] > >> (main:null) No libvirt.vif.driver specififed. Defaults to > BridgeVifDriver. > >> 2013-04-13 12:42:32,147 INFO [cloud.agent.Agent] (main:null) Agent [id > = > >> 1 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : > >> host = 96.31.67.232 : port = 8250 > >> 2013-04-13 12:42:32,154 INFO [utils.nio.NioClient] > (Agent-Selector:null) > >> Connecting to myipaddress:8250 > >> 2013-04-13 12:42:32,444 INFO [utils.nio.NioClient] > (Agent-Selector:null) > >> SSL: Handshake done > >> 2013-04-13 12:42:32,599 INFO [cloud.serializer.GsonHelper] > >> (Agent-Handler-1:null) Default Builder inited. > >> 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > >> Proccess agent startup answer, agent id = 1 > >> 2013-04-13 12:42:32,803 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > >> Set agent id 1 > >> 2013-04-13 12:42:32,808 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > >> Startup Response Received: agent id = 1 > >> > >> > >> The management log says this: > >> > >> 2013-04-13 12:43:28,952 DEBUG [cloud.network.NetworkManagerImpl] > >> (secstorage-1:null) Lock is released for network id 201 as a part of > >> network implement > >> 2013-04-13 12:43:28,969 DEBUG [db.Transaction.Transaction] > >> (secstorage-1:null) Rolling back the transaction: Time = 1 Name = > >> > > -SystemVmLoadScanner$1.run:71-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRunAndReset:351-FutureTask.runAndReset:178-ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201:165-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:267-ThreadPoolExecutor.runWorker:1146-ThreadPoolExecutor$Worker.run:615-Thread.run:679; > >> called by > >> > -Transaction.rollback:887-DataCenterIpAddressDaoImpl.takeIpAddress:57-DatabaseCallback.intercept:34-DataCenterDaoImpl.allocatePrivateIpAddress:228-DatabaseCallback.intercept:34-PodBasedNetworkGuru.reserve:119-NetworkManagerImpl.prepareNic:2143-NetworkManagerImpl.prepare:2113-VirtualMachineManagerImpl.advanceStart:752-VirtualMachineManagerImpl.start:472-VirtualMachineManagerImpl.start:465-SecondaryStorageManagerImpl.startSecStorageVm:257 > >> 2013-04-13 12:43:28,970 INFO [cloud.vm.VirtualMachineManagerImpl] > >> (secstorage-1:null) Insufficient capacity > >> com.cloud.exception.InsufficientAddressCapacityException: Unable to get > a > >> management ip addressScope=interface com.cloud.dc.Pod; id=1 > >> at > >> > com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121) > >> at > >> > com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2143) > >> at > >> > com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2113) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:752) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472) > >> at > >> > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465) > >> at > >> > com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257) > >> at > >> > com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684) > >> at > >> > com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310) > >> at > >> > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119) > >> at > >> > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) > >> at > >> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106) > >> at > >> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34) > >> at > >> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83) > >> at > >> com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > >> at > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > >> at > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:679) > >> 2013-04-13 12:43:28,973 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> (secstorage-1:null) Cleaning up resources for the vm > >> VM[SecondaryStorageVm|s-588-VM] in Starting state > >> 2013-04-13 12:43:28,975 DEBUG [agent.transport.Request] > >> (secstorage-1:null) Seq 1-751304715: Waiting for Seq 751304714 > Scheduling: > >> { Cmd , MgmtId: 219948120943996, via: 1, Ver: v1, Flags: 100111, > >> [{"StopCommand":{"isProxy":false,"vmName":"s-588-VM","wait":0}}] } > >> 2013-04-13 12:43:29,186 DEBUG > >> [network.router.VirtualNetworkApplianceManagerImpl] > >> (RouterStatusMonitor-1:null) Found 0 routers. > >> 2013-04-13 12:43:37,927 DEBUG [agent.manager.AgentManagerImpl] > >> (AgentManager-Handler-14:null) Ping from 1 > >> 2013-04-13 12:43:43,240 DEBUG [cloud.server.StatsCollector] > >> (StatsCollector-3:null) VmStatsCollector is running... > >> 2013-04-13 12:43:43,323 DEBUG [cloud.server.StatsCollector] > >> (StatsCollector-3:null) StorageCollector is running... > >> 2013-04-13 12:43:43,327 DEBUG [cloud.server.StatsCollector] > >> (StatsCollector-3:null) There is no secondary storage VM for secondary > >> storage host nfs://96.31.67.232/secondary > >> 2013-04-13 <http://96.31.67.232/secondary2013-04-13> 12:43:43,400 DEBUG > >> [agent.transport.Request] (StatsCollector-3:null) Seq 1-751304716: > >> Received: { Ans: , MgmtId: 219948120943996, via: 1, Ver: v1, Flags: > 10, { > >> GetStorageStatsAnswer } } > >> 2013-04-13 12:43:43,936 DEBUG [cloud.server.StatsCollector] > >> (StatsCollector-3:null) HostStatsCollector is running... > >> 2013-04-13 12:43:44,545 DEBUG [agent.transport.Request] > >> (StatsCollector-3:null) Seq 1-751304717: Received: { Ans: , MgmtId: > >> 219948120943996, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } > >> 2013-04-13 12:43:58,231 DEBUG [cloud.server.ManagementServerImpl] > >> (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 > CDT > >> 2013 > >> 2013-04-13 12:43:58,233 DEBUG [cloud.server.ManagementServerImpl] > >> (EventChecker-1:null) Found 0 events to be purged > >> 2013-04-13 12:43:58,235 DEBUG [cloud.server.ManagementServerImpl] > >> (EventChecker-1:null) Deleting events older than: Fri Apr 12 12:43:58 > CDT > >> 2013 > >> 2013-04-13 12:43:58,238 DEBUG [cloud.server.ManagementServerImpl] > >> (EventChecker-1:null) Found 0 events to be purged > >> 2013-04-13 12:43:59,186 DEBUG > >> [network.router.VirtualNetworkApplianceManagerImpl] > >> (RouterStatusMonitor-1:null) Found 0 routers. > >> [root@lunder agent]# > >> > >> > >> > >> > >> On Apr 13, 2013, at 12:30 PM, Marcus Sorensen <[email protected]> > >> wrote: > >> > >> > Well you've got something trying to start, because you have vnet > >> > interfaces. You need to look at your agent logs to see why the system > >> VMS > >> > refuse to start. If the power went out it could be corruption, the > >> system > >> > VMS may be waiting for you to fsck. It sounds like maybe the system > was > >> put > >> > into production without testing to make sure the host settings were > >> > persistent and would survive a reboot? > >> > > >> > So 1) look at your agent logs. And 2) use vnc to look at whatever > system > >> > VMS are running and see what state they are in. They will probably > >> > continually try to start and then shut down. > >> > On Apr 13, 2013 11:24 AM, "Maurice Lawler" <[email protected]> > >> wrote: > >> > > >> >> Greetings, > >> >> > >> >> I'm have a terrible way to go, nothing I have done will start my > cloud. > >> >> None of my system VM's will start, which in turn do not permit the > >> regular > >> >> OS VM's to start. I suffered from first a power outage, then I > manually > >> >> rebooted my server. Now, nothing is coming back online. > >> >> > >> >> I was previously told, having cloud0 first is the cause of this. Even > >> when > >> >> doing ifconfig cloud0 down, nothing seems to come back online. > >> >> > >> >> I have gone as far as stopping iptables / eatables along with > >> >> stopping/starting the network and the management console. > >> >> > >> >> > >> >> Checking the system VM's the continue to remain in a 'starting' > status. > >> >> > >> >> [root@lunder ~]# service iptables status > >> >> iptables: Firewall is not running. > >> >> [root@lunder ~]# service ebtables status > >> >> # Generated by ebtables-save v1.0 on Sat Apr 13 12:21:04 CDT 2013 > >> >> *nat > >> >> :PREROUTING ACCEPT > >> >> :OUTPUT ACCEPT > >> >> :POSTROUTING ACCEPT > >> >> > >> >> [root@lunder ~]# > >> >> > >> >> > >> >> [root@lunder daoenix]# ifconfig > >> >> cloud0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 > >> >> inet addr:169.254.0.1 Bcast:169.254.255.255 > Mask:255.255.0.0 > >> >> inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:658 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:0 > >> >> RX bytes:0 (0.0 b) TX bytes:28068 (27.4 KiB) > >> >> > >> >> cloudbr0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C > >> >> inet addr:myipaddress Bcast:9myipaddress > Mask:255.255.255.224 > >> >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:192832 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:11251 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:0 > >> >> RX bytes:11481135 (10.9 MiB) TX bytes:25153331 (23.9 MiB) > >> >> > >> >> eth0 Link encap:Ethernet HWaddr C8:0A:A9:9E:2D:7C > >> >> inet6 addr: fe80::ca0a:a9ff:fe9e:2d7c/64 Scope:Link > >> >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > >> >> RX packets:199794 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:24157 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:1000 > >> >> RX bytes:14647159 (13.9 MiB) TX bytes:25994485 (24.7 MiB) > >> >> Memory:df6e0000-df700000 > >> >> > >> >> lo Link encap:Local Loopback > >> >> inet addr:127.0.0.1 Mask:255.0.0.0 > >> >> inet6 addr: ::1/128 Scope:Host > >> >> UP LOOPBACK RUNNING MTU:16436 Metric:1 > >> >> RX packets:7850808 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:7850808 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:0 > >> >> RX bytes:1611132695 (1.5 GiB) TX bytes:1611132695 (1.5 GiB) > >> >> > >> >> virbr0 Link encap:Ethernet HWaddr 52:54:00:D9:D9:9A > >> >> inet addr:192.168.122.1 Bcast:192.168.122.255 > >> >> Mask:255.255.255.0 > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:0 > >> >> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > >> >> > >> >> vnet0 Link encap:Ethernet HWaddr FE:00:A9:FE:00:67 > >> >> inet6 addr: fe80::fc00:a9ff:fefe:67/64 Scope:Link > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:116 errors:0 dropped:0 overruns:0 carrier:0 > >> >> collisions:0 txqueuelen:500 > >> >> RX bytes:0 (0.0 b) TX bytes:5232 (5.1 KiB) > >> >> > >> >> vnet1 Link encap:Ethernet HWaddr FE:84:4C:00:00:01 > >> >> inet6 addr: fe80::fc84:4cff:fe00:1/64 Scope:Link > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 > >> >> collisions:0 txqueuelen:500 > >> >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) > >> >> > >> >> vnet2 Link encap:Ethernet HWaddr FE:2C:BC:00:00:05 > >> >> inet6 addr: fe80::fc2c:bcff:fe00:5/64 Scope:Link > >> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > >> >> TX packets:256 errors:0 dropped:0 overruns:1 carrier:0 > >> >> collisions:0 txqueuelen:500 > >> >> RX bytes:0 (0.0 b) TX bytes:17849 (17.4 KiB) > >> >> > >> >> [root@lunder daoenix]# > >> >> > >> >> > >> >> > >> > >> >
