Hi Matthew, Glad you are back on track
Regards Geoff On 16 May 2012, at 17:25, "Matthew Hartmann" <mhartm...@tls.net> wrote: Just an FYI, I had to manually scrub the database then re-add them via the CS UI. Things are coming back to life very slowly. Thanks again for your assistance! :) Cheers, Matthew On 5/16/2012 10:00 AM, Matthew Hartmann wrote: > Well, I'm at the point I'm ready to remove Primary & Secondary Storage, > however CloudStack refuses to remove them even though there are 0 hosts > configured. CloudStack reports when trying to remove Primary Storage: "Failed > to delete storage pool" > > When trying to delete Secondary Storage CloudStack reports: "Can not delete > this secondary storage due to there are still snapshots on it" > > I disabled the Pod and Zone before attempting to remove Primary and Secondary > Storage, as well as putting Primary Storage in Maintenance Mode. > > Management Server log reports when trying to delete Secondary Storage: > > 2012-05-16 09:57:29,553 ERROR [cloud.api.ApiDispatcher] > (catalina-exec-24:null) Exception while executing DeleteHostCmd: > com.cloud.utils.exception.CloudRuntimeException: Can not delete this > secondary storage due to there are still snapshots on it > > Management Server log reports when trying to delete Primary Storage: > > 2012-05-16 09:58:53,600 WARN [cloud.storage.StorageManagerImpl] > (catalina-exec-2:null) Cannot delete pool san-p1n1c1.iedc.cloud.tls.net as > there are associated vols for this pool > 2012-05-16 09:58:53,601 WARN [cloud.api.ApiDispatcher] > (catalina-exec-2:null) class com.cloud.api.ServerApiException : Failed to > delete storage pool > > I suspect manual scrubbing of the database is in order? > > Cheers, > > Matthew > > > On 5/14/2012 5:17 PM, Geoff Higginbottom wrote: >> Matthew, >> >> Removing and then re-adding Pri Storage will not wipe the LUNs and the >> existing VHD files which are there, assuming the associated Guest Instances >> are still in the database should also remap. >> >> How to avoid this in the future? It's pretty rare for a XenServer cluster to >> bomb out in this way, but a 'standby cluster' would not really help as that >> would have its own Storage Pool. However having at least two Primary >> Storage pools in the Cluster (eg two iSCSI LUNs) will allow you to migrate >> Guest Instances between them should you detect any problems. >> >> Also having multiple PODs and Clusters and forcing the distribution of Guest >> Instances for a particular Account to be spread amongst them, rather than >> relying on the default 'Random' allocation policies reduces risks in the >> event of any failures. >> >> If you do have data on the existing Primary Storage and want to be über safe >> (my preferred stance) then copy the VHDs off first, you can always re-import >> them back into CloudStack via Templates if you have to, but I assume as >> there are no clients on this system yet the LUNs should be empty. >> >> Kind Regards >> >> Geoff Higginbottom >> CTO / Senior Consultant >> ShapeBlue Ltd >> >> geoff.higginbot...@shapeblue.com<mailto:geoff.higginbot...@shapeblue.com> | >> www.shapeblue.com<http://www.shapeblue.com/> >> >> ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS >> >> >> On 14 May 2012, at 22:02, "Matthew >> Hartmann"<mhartm...@tls.net<mailto:mhartm...@tls.net>> wrote: >> >> On 5/14/2012 4:48 PM, Geoff Higginbottom wrote: >> I had assumed you had Primary and Secondary storage on different IP ranges >> to your Management Server which does need to be able to access Secondary >> Storage but not Primary Storage. >> >> That assumption is correct, Primary Storage is on a different subnet than >> Secondary Storage. Secondary Storage is sitting on the Management Network >> subnet. >> >> >> I now note your Primary Storage is iSCSI, and that you had a failure of all >> Hosts. >> >> When adding Storage and Hosts, you have to add Hosts, before you add Primary >> Storage, and you have to add Primary Storage before Secondary Storage. >> >> This is all taken care of by the Add Zone Wizard, but if you have a failure >> of all Hosts in a Zone then you need to remove both Primary and Secondary >> Storage, then add a Host, then add Primary Storage, then add Secondary >> storage. >> >> Templates, ISOs etc that are on Secondary will disappear from the GUI, but >> don't worry, they will be detected by the SSVM once it comes online, and >> they will then reappear in the GUI. >> >> If this is a simple test system then it may be easier to simply drop the DB >> Schema, and recreate it from scratch, or create a new Zone and let the >> wizard take care of everything. >> >> This actually was a system we had *just* placed into production. We luckily >> did not have any clients setup on this cloud yet. However, won't removing >> Primary Storage then re-adding it cause all data on the LUN's to be >> effectively wiped? >> >> How would you propose I avoid something like this in the future? A standby >> cluster? >> >> Thanks again for your help! >> >> Cheers, >> >> Matthew >> >> >> >> On 14 May 2012, at 21:12, "Matthew >> Hartmann"<mhartm...@tls.net<mailto:mhartm...@tls.net>> wrote: >> >> >> On 5/14/2012 3:35 PM, Geoff Higginbottom wrote: >> Matthew, >> >> If I understand correctly, your storage is on a different IP Range to your >> Management Server and Hosts etc, if this is the case you need a Storage >> Network, which also needs to be accessible from the Management Server. If >> no Storage Network is configured, CloudStack will use the Management Network >> when trying to access Storage. >> Odd, did something change? The Management Server has never needed to access >> the storage network before. I have tried using XenServer name labels for the >> Storage Network, but to no avail. Why would the Management Server need to >> access my iSCSI SAN (Primary Storage)? I have confirmed my my Management >> Server can still access Secondary Storage (NFS Server). All devices have an >> FQDN which can be resolved from the XenServer and Management Server. >> >> Re http://bugs.cloudstack.org/browse/CS-14655 this is only required if you >> are using NIC Bonding and I do not believe you are >> Yes, we are using bonding but we are not using CSP. The bug report refers to >> bonding and XenServer 6.0.x but it is not clear whether the steps outlined >> should be taken regardless of whether or not CSP is being used in ones setup. >> >> Cheers, >> >> Matthew >> >> Regards >> >> Geoff >> >> >> -----Original Message----- >> From: Matthew Hartmann [mailto:mhartm...@tls.net] >> Sent: 14 May 2012 20:27 >> To: >> cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org> >> Subject: Re: Two Issues with XenServer 6.0.2 and CS 3.0.1 >> >> Please see my responses inline, below: >> >> On 5/14/2012 3:18 PM, Geoff Higginbottom wrote: >> Matthew, >> >> It looks like you have forgot to label your physical networks and map them >> within the Zone Setup process, this is vital if you want to use dedicated >> NICs for any of the four network types, ie Management, Guest, Public or >> Storage. >> This was a pre-existing zone that was operating well. The XenServer pool >> freaked out and unraveled quicker than the fall of Rome. I performed a clean >> install on my XenServer's and am simply trying to re-add them back into the >> Pod. I'm only using one Node at this moment just so I can get things going. >> When I initially setup my XenServer's and the pool, I made copious note >> should something like this happen. Now that it has unfortunately happened, >> configuring my XenServer's the same as they were initially when everything >> was functional doesn't seem to have helped. >> >> Re the Cloud Supplemental Pack, this is only required if you are using >> Security Groups with a Basic Zone >> Right, but is the patch required regardless of whether or not CSP is >> installed? I have experienced a similar issue and I am *not* using CSP. >> >> Cheers, >> >> Matthew >> >> Regards >> >> Geoff >> >> >> -----Original Message----- >> From: Matthew Hartmann [mailto:mhartm...@tls.net] >> Sent: 14 May 2012 19:42 >> To: CloudStack user/admin discussions >> Cc: CloudStack Devs >> Subject: Fwd: Two Issues with XenServer 6.0.2 and CS 3.0.1 >> >> >> I forgot to include the following information that I found in the database >> after the host was added: >> >> mysql> select * from host where id=22 \G; >> [...] >> private_ip_address: 10.0.0.16 >> private_netmask: 255.255.254.0 >> private_mac_address: e8:9a:8f:22:9b:c4 >> storage_ip_address: 10.0.0.16 >> storage_netmask: 255.255.254.0 >> storage_mac_address: e8:9a:8f:22:9b:c4 >> storage_ip_address_2: 10.0.0.16 >> storage_mac_address_2: e8:9a:8f:22:9b:c4 >> storage_netmask_2: 255.255.254.0 >> [...] >> >> That information is incorrect, as the storage_ip_address and >> storage_ip_address_2 are two completely different subnets and CloudStack >> does not see these network devices. >> >> Thanks! >> >> Matthew >> >> -------- Original Message -------- >> Subject: Two Issues with XenServer 6.0.2 and CS 3.0.1 >> Date: Mon, 14 May 2012 14:36:57 -0400 >> From: Matthew Hartmann<mhartm...@tls.net<mailto:mhartm...@tls.net>> >> Organization: TLS.NET<http://TLS.NET>, Inc. >> To: CloudStack user/admin discussions >> <cloudstack-us...@incubator.apache.org<mailto:cloudstack-us...@incubator.apache.org>> >> CC: CloudStack >> Devs<cloudstack-dev@incubator.apache.org<mailto:cloudstack-dev@incubator.apache.org>> >> >> >> >> First of all, it is not very clear whether or not this patch is required for >> XenServer with regard to the Cloud Supplemental Pack: >> http://bugs.cloudstack.org/browse/CS-14655 >> >> Could someone shed some light on this? >> >> Secondly, I'm having an issue with CloudStack trying to use my >> "cloud-private" network for Public, Guest, Management and Storage. >> >> Here is what my CS 3.0.1 Management Server log is showing: >> >> [java] INFO [xen.resource.CitrixResourceBase] (http-8080-6:) Private >> Network is cloud-private for host 10.0.0.16 [java] INFO >> [xen.resource.CitrixResourceBase] (http-8080-6:) Guest Network is >> cloud-private for host 10.0.0.16 [java] INFO >> [xen.resource.CitrixResourceBase] (http-8080-6:) Public Network is >> cloud-private for host 10.0.0.16 [java] INFO >> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 1 is >> cloud-private for host 10.0.0.16 [java] INFO >> [xen.resource.CitrixResourceBase] (http-8080-6:) Storage Network 2 is >> cloud-private for host 10.0.0.16 >> >> On my XenServer 6.0.2 machine I have my "cloud-private" and "cloud-public" >> networks. In CS, I have my Storage network set to "Use default gateway". The >> XenServer has access to the Internet and can resolve the FQDN of all hosts, >> management servers and the SAN on the storage network. I can ping the IP's >> of the SAN on both Storage subnets, yet CloudStack reports after adding the >> host that it can't see the storage network; obviously because it is >> incorrectly discovering my networks. >> >> I have verified in the config in my database for my networks: >> >> mysql> select * from physical_network_traffic_types; >> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+ >> | id | uuid | physical_network_id | >> traffic_type | xen_network_label | kvm_network_label | >> vmware_network_label | simulator_network_label | ovm_network_label | >> vlan | >> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+ >> | 1 | 3a368a1d-5d33-4e11-80ea-02da222d3930 | 1 | >> Management | cloud-private | NULL | >> NULL | NULL | NULL | NULL | >> | 2 | 3bf7b069-355e-4171-8ac5-c6f84777fc9d | 2 | >> Guest | cloud-public | NULL | >> NULL | NULL | NULL | NULL | >> | 3 | bb19c965-d726-451e-98fe-e0363c3bbd19 | 2 | >> Public | cloud-public | NULL | >> NULL | NULL | NULL | NULL | >> | 4 | 9df97cc6-f4e6-4d5f-b148-99f0dfc64dac | 1 | >> Storage | NULL | NULL | >> NULL | NULL | NULL | NULL | >> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+ >> 4 rows in set (0.00 sec) >> >> Thoughts? Any assistance would be greatly appreciated! >> >> Cheers, >> >> Matthew >> >> >> ShapeBlue provides a range of strategic and technical consulting services to >> help IT Service Providers and enterprises to build a true IaaS compute >> cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT >> Service providers and enterprises to deliver true, utility based, IaaS to >> the customer without reengineering existing physical, virtual or storage >> layers. >> >> ________________________________ >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd. If you are not the intended recipient of >> this email, you must neither take any action based upon its contents, nor >> copy or show it to anyone. Please contact the sender if you believe you have >> received this email in error. Shape Blue Ltd is a company incorporated in >> England& Wales. >> ShapeBlue provides a range of strategic and technical consulting services to >> help IT Service Providers and enterprises to build a true IaaS compute >> cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT >> Service providers and enterprises to deliver true, utility based, IaaS to >> the customer without reengineering existing physical, virtual or storage >> layers. >> >> ________________________________ >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd. If you are not the intended recipient of >> this email, you must neither take any action based upon its contents, nor >> copy or show it to anyone. Please contact the sender if you believe you have >> received this email in error. Shape Blue Ltd is a company incorporated in >> England& Wales. >> >> ShapeBlue provides a range of strategic and technical consulting services to >> help IT Service Providers and enterprises to build a true IaaS compute >> cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT >> Service providers and enterprises to deliver true, utility based, IaaS to >> the customer without reengineering existing physical, virtual or storage >> layers. >> >> ________________________________ >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd. If you are not the intended recipient of >> this email, you must neither take any action based upon its contents, nor >> copy or show it to anyone. Please contact the sender if you believe you have >> received this email in error. Shape Blue Ltd is a company incorporated in >> England& Wales. >> >> >> >> >> >> ShapeBlue provides a range of strategic and technical consulting services to >> help IT Service Providers and enterprises to build a true IaaS compute >> cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT >> Service providers and enterprises to deliver true, utility based, IaaS to >> the customer without reengineering existing physical, virtual or storage >> layers. >> >> ________________________________ >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd. If you are not the intended recipient of >> this email, you must neither take any action based upon its contents, nor >> copy or show it to anyone. Please contact the sender if you believe you have >> received this email in error. Shape Blue Ltd is a company incorporated in >> England& Wales. >> > > ShapeBlue provides a range of strategic and technical consulting services to help IT Service Providers and enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service providers and enterprises to deliver true, utility based, IaaS to the customer without reengineering existing physical, virtual or storage layers. ________________________________ This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales.
