Just an FYI, I had to manually scrub the database then re-add them via the CS UI. Things are coming back to life very slowly. Thanks again for your assistance! :)

Cheers,

Matthew


On 5/16/2012 10:00 AM, Matthew Hartmann wrote:
Well, I'm at the point I'm ready to remove Primary & Secondary Storage, however CloudStack refuses to remove them even though there are 0 hosts configured. CloudStack reports when trying to remove Primary Storage: "Failed to delete storage pool"

When trying to delete Secondary Storage CloudStack reports: "Can not delete this secondary storage due to there are still snapshots on it"

I disabled the Pod and Zone before attempting to remove Primary and Secondary Storage, as well as putting Primary Storage in Maintenance Mode.

Management Server log reports when trying to delete Secondary Storage:

2012-05-16 09:57:29,553 ERROR [cloud.api.ApiDispatcher] (catalina-exec-24:null) Exception while executing DeleteHostCmd: com.cloud.utils.exception.CloudRuntimeException: Can not delete this secondary storage due to there are still snapshots on it

Management Server log reports when trying to delete Primary Storage:

2012-05-16 09:58:53,600 WARN [cloud.storage.StorageManagerImpl] (catalina-exec-2:null) Cannot delete pool san-p1n1c1.iedc.cloud.tls.net as there are associated vols for this pool 2012-05-16 09:58:53,601 WARN [cloud.api.ApiDispatcher] (catalina-exec-2:null) class com.cloud.api.ServerApiException : Failed to delete storage pool

I suspect manual scrubbing of the database is in order?

Cheers,

Matthew


On 5/14/2012 5:17 PM, Geoff Higginbottom wrote:
Matthew,

Removing and then re-adding Pri Storage will not wipe the LUNs and the existing VHD files which are there, assuming the associated Guest Instances are still in the database should also remap.

How to avoid this in the future? It's pretty rare for a XenServer cluster to bomb out in this way, but a 'standby cluster' would not really help as that would have its own Storage Pool. However having at least two Primary Storage pools in the Cluster (eg two iSCSI LUNs) will allow you to migrate Guest Instances between them should you detect any problems.

Also having multiple PODs and Clusters and forcing the distribution of Guest Instances for a particular Account to be spread amongst them, rather than relying on the default 'Random' allocation policies reduces risks in the event of any failures.

If you do have data on the existing Primary Storage and want to be über safe (my preferred stance) then copy the VHDs off first, you can always re-import them back into CloudStack via Templates if you have to, but I assume as there are no clients on this system yet the LUNs should be empty.

Kind Regards

Geoff Higginbottom
CTO / Senior Consultant
ShapeBlue Ltd

geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com> | www.shapeblue.com<http://www.shapeblue.com/>

ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS


On 14 May 2012, at 22:02, "Matthew Hartmann"<mhartm...@tls.net<mailto:mhartm...@tls.net>> wrote:

On 5/14/2012 4:48 PM, Geoff Higginbottom wrote:
I had assumed you had Primary and Secondary storage on different IP ranges to your Management Server which does need to be able to access Secondary Storage but not Primary Storage.

That assumption is correct, Primary Storage is on a different subnet than Secondary Storage. Secondary Storage is sitting on the Management Network subnet.


I now note your Primary Storage is iSCSI, and that you had a failure of all Hosts.

When adding Storage and Hosts, you have to add Hosts, before you add Primary Storage, and you have to add Primary Storage before Secondary Storage.

This is all taken care of by the Add Zone Wizard, but if you have a failure of all Hosts in a Zone then you need to remove both Primary and Secondary Storage, then add a Host, then add Primary Storage, then add Secondary storage.

Templates, ISOs etc that are on Secondary will disappear from the GUI, but don't worry, they will be detected by the SSVM once it comes online, and they will then reappear in the GUI.

If this is a simple test system then it may be easier to simply drop the DB Schema, and recreate it from scratch, or create a new Zone and let the wizard take care of everything.

This actually was a system we had *just* placed into production. We luckily did not have any clients setup on this cloud yet. However, won't removing Primary Storage then re-adding it cause all data on the LUN's to be effectively wiped?

How would you propose I avoid something like this in the future? A standby cluster?

Thanks again for your help!

Cheers,

Matthew



On 14 May 2012, at 21:12, "Matthew Hartmann"<mhartm...@tls.net<mailto:mhartm...@tls.net>> wrote:


On 5/14/2012 3:35 PM, Geoff Higginbottom wrote:
Matthew,

If I understand correctly, your storage is on a different IP Range to your Management Server and Hosts etc, if this is the case you need a Storage Network, which also needs to be accessible from the Management Server. If no Storage Network is configured, CloudStack will use the Management Network when trying to access Storage. Odd, did something change? The Management Server has never needed to access the storage network before. I have tried using XenServer name labels for the Storage Network, but to no avail. Why would the Management Server need to access my iSCSI SAN (Primary Storage)? I have confirmed my my Management Server can still access Secondary Storage (NFS Server). All devices have an FQDN which can be resolved from the XenServer and Management Server.

Re http://bugs.cloudstack.org/browse/CS-14655 this is only required if you are using NIC Bonding and I do not believe you are Yes, we are using bonding but we are not using CSP. The bug report refers to bonding and XenServer 6.0.x but it is not clear whether the steps outlined should be taken regardless of whether or not CSP is being used in ones setup.

Cheers,

Matthew

Regards

Geoff


-----Original Message-----
From: Matthew Hartmann [mailto:mhartm...@tls.net]
Sent: 14 May 2012 20:27
To: cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>
Subject: Re: Two Issues with XenServer 6.0.2 and CS 3.0.1

Please see my responses inline, below:

On 5/14/2012 3:18 PM, Geoff Higginbottom wrote:
Matthew,

It looks like you have forgot to label your physical networks and map them within the Zone Setup process, this is vital if you want to use dedicated NICs for any of the four network types, ie Management, Guest, Public or Storage. This was a pre-existing zone that was operating well. The XenServer pool freaked out and unraveled quicker than the fall of Rome. I performed a clean install on my XenServer's and am simply trying to re-add them back into the Pod. I'm only using one Node at this moment just so I can get things going. When I initially setup my XenServer's and the pool, I made copious note should something like this happen. Now that it has unfortunately happened, configuring my XenServer's the same as they were initially when everything was functional doesn't seem to have helped.

Re the Cloud Supplemental Pack, this is only required if you are using
Security Groups with a  Basic Zone
Right, but is the patch required regardless of whether or not CSP is installed? I have experienced a similar issue and I am *not* using CSP.

Cheers,

Matthew

Regards

Geoff


-----Original Message-----
From: Matthew Hartmann [mailto:mhartm...@tls.net]
Sent: 14 May 2012 19:42
To: CloudStack user/admin discussions
Cc: CloudStack Devs
Subject: Fwd: Two Issues with XenServer 6.0.2 and CS 3.0.1


I forgot to include the following information that I found in the database after the host was added:

mysql>     select * from host where id=22 \G;
[...]
private_ip_address: 10.0.0.16
private_netmask: 255.255.254.0
private_mac_address: e8:9a:8f:22:9b:c4
storage_ip_address: 10.0.0.16
storage_netmask: 255.255.254.0
storage_mac_address: e8:9a:8f:22:9b:c4
storage_ip_address_2: 10.0.0.16
storage_mac_address_2: e8:9a:8f:22:9b:c4
storage_netmask_2: 255.255.254.0
[...]

That information is incorrect, as the storage_ip_address and
storage_ip_address_2 are two completely different subnets and CloudStack does not see these network devices.

Thanks!

Matthew

-------- Original Message --------
Subject:        Two Issues with XenServer 6.0.2 and CS 3.0.1
Date:   Mon, 14 May 2012 14:36:57 -0400
From:   Matthew Hartmann<mhartm...@tls.net<mailto:mhartm...@tls.net>>
Organization:   TLS.NET<http://TLS.NET>, Inc.
To:     CloudStack user/admin discussions
<cloudstack-us...@incubator.apache.org<mailto:cloudstack-us...@incubator.apache.org>> CC: CloudStack Devs<cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>>



First of all, it is not very clear whether or not this patch is required for XenServer with regard to the Cloud Supplemental Pack:
http://bugs.cloudstack.org/browse/CS-14655

Could someone shed some light on this?

Secondly, I'm having an issue with CloudStack trying to use my "cloud-private" network for Public, Guest, Management and Storage.

Here is what my CS 3.0.1 Management Server log is showing:

[java] INFO [xen.resource.CitrixResourceBase] (http-8080-6:) Private
Network is cloud-private for host 10.0.0.16 [java] INFO
[xen.resource.CitrixResourceBase] (http-8080-6:) Guest Network is
cloud-private for host 10.0.0.16 [java] INFO
[xen.resource.CitrixResourceBase] (http-8080-6:) Public Network is
cloud-private for host 10.0.0.16 [java] INFO
[xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 1 is
cloud-private for host 10.0.0.16 [java] INFO
[xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 2 is
cloud-private for host 10.0.0.16

On my XenServer 6.0.2 machine I have my "cloud-private" and "cloud-public" networks. In CS, I have my Storage network set to "Use default gateway". The XenServer has access to the Internet and can resolve the FQDN of all hosts, management servers and the SAN on the storage network. I can ping the IP's of the SAN on both Storage subnets, yet CloudStack reports after adding the host that it can't see the storage network; obviously because it is incorrectly discovering my networks.

I have verified in the config in my database for my networks:

mysql>     select * from physical_network_traffic_types;
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
| id | uuid                                 | physical_network_id |
traffic_type | xen_network_label | kvm_network_label |
vmware_network_label | simulator_network_label | ovm_network_label |
vlan |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
|  1 | 3a368a1d-5d33-4e11-80ea-02da222d3930 |                   1 |
Management   | cloud-private     | NULL              |
NULL | NULL | NULL | NULL |
|  2 | 3bf7b069-355e-4171-8ac5-c6f84777fc9d |                   2 |
Guest        | cloud-public      | NULL              |
NULL | NULL | NULL | NULL |
|  3 | bb19c965-d726-451e-98fe-e0363c3bbd19 |                   2 |
Public       | cloud-public      | NULL              |
NULL | NULL | NULL | NULL |
|  4 | 9df97cc6-f4e6-4d5f-b148-99f0dfc64dac |                   1 |
Storage      | NULL              | NULL              |
NULL | NULL | NULL | NULL | +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+
4 rows in set (0.00 sec)

Thoughts? Any assistance would be greatly appreciated!

Cheers,

Matthew


ShapeBlue provides a range of strategic and technical consulting services to help IT Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service providers and enterprises to deliver true, utility based, IaaS to the customer without reengineering existing physical, virtual or storage layers.

________________________________

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England& Wales. ShapeBlue provides a range of strategic and technical consulting services to help IT Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service providers and enterprises to deliver true, utility based, IaaS to the customer without reengineering existing physical, virtual or storage layers.

________________________________

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England& Wales.

ShapeBlue provides a range of strategic and technical consulting services to help IT Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service providers and enterprises to deliver true, utility based, IaaS to the customer without reengineering existing physical, virtual or storage layers.

________________________________

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England& Wales.





ShapeBlue provides a range of strategic and technical consulting services to help IT Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service providers and enterprises to deliver true, utility based, IaaS to the customer without reengineering existing physical, virtual or storage layers.

________________________________

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England& Wales.




Reply via email to