| Did you remove the Tags on each XenServer host prior to starting?
Management Controller Failure and Replacement
In setting up your cloud, you should have a backup routine for your controller. The most important item to back up is the MySQL databases that Cloudstack uses. A suitable backup script is attached to this page. In the even of a cloud management controller failure, the steps to replace the controller with a new one are:
These instructions assume your cluster is Xenserver - Contributors using other Hypervisor OSs, please contribute.
- Setup new management server hardware
- Install your OS
- Install Cloudstack, up to and including the "Install Database step"
- Import your database backup
- In Xencenter, connect to your Cloudstack host pool.
- On each host, remove the tags on Host > General Tab > Tags by editing the tags and un-checking each one.
- On the management controller, start Cloudstack
- service cloud-management start
- the new cloud management controller will connect to each host in the database and push out new tags and keys to each host in the pool.
After restoring the old database dump to new installation. CS is unable to
contact Xenserver hosts. I getting following errors in
mamangement-server.log,
2013-07-15 11:57:49,646 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(StatsCollector-1:null) Connection to peer opened: 130602634328, ip:
192.168.10.251
2013-07-15 11:57:49,652 DEBUG [agent.manager.ClusteredAgentAttache]
(StatsCollector-2:null) Seq 50-185008129: Forwarding null to 130602634328
2013-07-15 11:57:49,662 DEBUG [agent.manager.ClusteredAgentAttache]
(StatsCollector-1:null) Seq 39-1272840193: Forwarding null to 130602634328
2013-07-15 11:57:49,699 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-2:null) Seq 50-185008129: Routing from 81375086018793
2013-07-15 11:57:49,699 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-2:null) Seq 50-185008129: Link is closed
2013-07-15 11:57:49,699 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-3:null) Seq 39-1272840193: Routing from 81375086018793
2013-07-15 11:57:49,700 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-3:null) Seq 39-1272840193: Link is closed
2013-07-15 11:57:49,700 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentManager-Handler-3:null) Seq 39-1272840193: MgmtId 81375086018793:
Req: Resource [Host:39] is unreachable: Host 39: Link is closed
2013-07-15 11:57:49,861 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentManager-Handler-8:null) Seq 39--1: MgmtId 81375086018793: Req: Cancel
request received
2013-07-15 11:57:49,861 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-8:null) Seq 39-1272840194: Cancelling.
2013-07-15 11:57:49,861 DEBUG [agent.manager.AgentAttache]
(StatsCollector-2:null) Seq 39-1272840194: Waiting some more time because
this is the current command
2013-07-15 11:57:49,862 DEBUG [agent.manager.AgentAttache]
(StatsCollector-2:null) Seq 39-1272840194: Waiting some more time because
this is the current command
2013-07-15 11:57:49,862 INFO [utils.exception.CSExceptionErrorCode]
(StatsCollector-2:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2013-07-15 11:57:49,862 WARN [agent.manager.AgentAttache]
(StatsCollector-2:null) Seq 39-1272840194: Timed out on null
2013-07-15 11:57:49,862 DEBUG [agent.manager.AgentAttache]
(StatsCollector-2:null) Seq 39-1272840194: Cancelling.
2013-07-15 11:57:49,863 DEBUG [cloud.storage.StorageManagerImpl]
(StatsCollector-2:null) Unable to send storage pool command to
Pool[210|NetworkFilesystem] via 39
com.cloud.exception.OperationTimedoutException: Commands 1272840194 to Host
39 timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:429)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:511)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:464)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:2347)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:422)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:436)
at
com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:316)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-07-15 11:57:49,863 INFO [cloud.server.StatsCollector]
(StatsCollector-2:null) Unable to reach Pool[210|NetworkFilesystem]
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:210]
is unreachable: Unable to send command to the pool
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:2357)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:422)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:436)
at
com.cloud.server.StatsCollector$StorageCollector.run(StatsCollector.java:316)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Thanks
Leeno
On Tue, Jul 16, 2013 at 10:21 AM, Leeno Jose.P.A <[email protected]> wrote:
This is a dev box. We are planning a HA enabled environment for prod
setup. Thanks Geoff.
On Tue, Jul 16, 2013 at 12:11 AM, Geoff Higginbottom <
[email protected]> wrote:
Hi Leeno,
It theory that should work, but obviously you will lose all changes made
since the dump was taken. If any new VMs have been created, they will get
purged by the system etc.
I would highly recommend splitting the DB and the Management Server, and
if possible add a 2nd instance of each.
Regards
Geoff Higginbottom
D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
[email protected]
-----Original Message-----
From: Leeno Jose.P.A [mailto:[email protected]]
Sent: 15 July 2013 18:46
To: [email protected]
Subject: Re: Rebuilding management server
Hi Geoff,
1. I have only one management server.
2. Management server is not functioning now but 'cloud' database dump is
available in backup. CS version was 4.1.0 Hosts were Xenserver 6.1.0 3. DB
server was on same machine where management server installed.
Now I am planning to do a fresh install of CS 4.1.0 and restore cloud
database with old installation dump, which is available in backup. Will it
work?
Thanks
Leeno
On Mon, Jul 15, 2013 at 9:56 PM, Geoff Higginbottom <
[email protected]> wrote:
The Management Servers are 'Stateless' so as Chip points out, it's the
DB that stores all the info.
How you actually go about it depends on your current setup.
1. How many management servers do you currently have?
2. Are the original Management Server(s) still functioning, or are
they down?
3. Is DB on a separate server, or the same as the Management Server?
Regards
Geoff Higginbottom
D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
[email protected]
-----Original Message-----
From: Chip Childers [mailto:[email protected]]
Sent: 15 July 2013 15:50
To: [email protected]
Subject: Re: Rebuilding management server
On Mon, Jul 15, 2013 at 03:19:42PM +0530, Leeno Jose.P.A wrote:
Hi Users,
Has anyone tried to rebuild management server with Xenserver hosts?
If yes, could you please share experience?
--
Leeno Jose .P.A
I have not, but one of the most critical aspects of this is to ensure
that your database is retained.
This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
Any views or opinions expressed are solely those of the author and do
not necessarily represent those of Shape Blue Ltd or related
companies. If you are not the intended recipient of this email, you
must neither take any action based upon its contents, nor copy or show
it to anyone. Please contact the sender if you believe you have
received this email in error. Shape Blue Ltd is a company incorporated
in England & Wales. ShapeBlue Services India LLP is operated under
license from Shape Blue Ltd. ShapeBlue is a registered trademark.
--
Leeno Jose .P.A
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Shape Blue Ltd or related companies. If you are not the
intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender
if you believe you have received this email in error. Shape Blue Ltd is a
company incorporated in England & Wales. ShapeBlue Services India LLP is
operated under license from Shape Blue Ltd. ShapeBlue is a registered
trademark.
--
Leeno Jose .P.A
--
Leeno Jose .P.A
|
