Hi Kurt,
The results you have posted up are from your console proxy (v-2-VM), not your
SSVM (possibly typo?).
Can you post up the same results from your SSVM? If this isn’t working – i.e.
you can’t connect to it – then try to recreate it, make a note of the private
IP and connect to it as per the wiki article.
Regards,
Dag Sonstebo
On 17/11/2016, 02:51, "Kurt K" <[email protected]> wrote:
Hi Dag,
We have executed the *SSVM health check* systemvm-ssvm-check.sh script
on SSVM and its results are pasted below.
====
root@v-2-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
WARNING: cannot ping DNS server
route follows
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 XX.XX.XX.XX 0.0.0.0 UG 0 0 0 eth2
8.8.8.8 10.10.48.1 255.255.255.255 UGH 0 0 0 eth1
10.10.48.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
XX.XX.XX.XX 0.0.0.0 255.255.255.240 U 0 0 0 eth2
XX.XX.XX.XX 10.10.48.1 255.255.255.255 UGH 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
================================================
ERROR: DNS not resolving download.cloud.com
resolv.conf follows
nameserver 8.8.8.8
nameserver 8.8.8.8
nameserver 4.2.2.1
root@v-2-VM:~#
====
But we were unable to connect to in/out from SSVM. There is one private
IP and public IP on SSVM. We have checked the firewall and there was no
reject rules in it.
Any suggestions?
-Kurt
On 11/11/2016 06:22 PM, Dag Sonstebo wrote:
> Hi Kurt,
>
> My original comment stands – it seems you do not have a healthy SSVM at
the moment – until this has been sorted you will not be able to upload templates.
>
> The systemvm-ssvm-check.sh script lives on the SSVM itself – please see
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting
for how to access this. When you run the script it will do the basic checks for
connectivity and NFS access and hopefully give you an idea what is wrong. If you
can run this and post up the results we can advise further.
>
> Regards,
> Dag Sonstebo
>
> On 11/11/2016, 12:42, "Kurt K" <[email protected]> wrote:
>
> Hi Dag,
>
> We have recreated the SSVM, but still no luck :( . Regarding
> ssvm-check.sh check script, it was not presented on management
server.
>
> ===
> [root@manage1 /]# ls /usr/local/cloud/systemvm-ssvm-check.sh
> ls: cannot access /usr/local/cloud/systemvm-ssvm-check.sh: No such
file
> or directory
> ===
>
> While registering an ISO, we are getting the following errors.
>
> ===
> 2016-11-11 03:27:28,998 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
> (catalina-exec-7:ctx-23ce7527 ctx-47f4a89e) template 202 is not in
> store:1, type:Image
> 2016-11-11 03:27:29,118 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
> (catalina-exec-7:ctx-23ce7527 ctx-47f4a89e) template 202 is already
in
> store:1, type:Image
> 2016-11-11 03:27:29,257 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl]
> (catalina-exec-7:ctx-23ce7527 ctx-47f4a89e) Downloading template to
data
> store 1
> 2016-11-11 03:27:29,263 WARN [c.c.s.d.DownloadMonitorImpl]
> (catalina-exec-7:ctx-23ce7527 ctx-47f4a89e) There is no secondary
> storage VM for downloading template to image store
> nfs://162.x.x.x/export/secondary
> ===
>
> We had tried every possible workarounds, but none of them working.
Is
> this have any relation with NFS version on server? we are using nfs
4.
> Moreover I did some searches and some one recommend to set this
> parameter "secstorage.allowed.internal.sites" value. I have set
this to
> our IP range, still no luck.
>
>
> We have followed the general installation step on this doc
>
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.6/index.html
> . Are we miss anything on NFS server setup side? please let us know
.
> Also this time, we have excluded the NFS server and use management
> server as NFS itself. The same issue out there.
>
> Do you have any other specific step or any other instructions for
> configuring the cloudstack apart from cloudstack doc ?
>
> We were totally stuck here.
>
> -Kurt
>
> On 11/10/2016 09:09 PM, Dag Sonstebo wrote:
> > Sounds like your SSVM isn’t too healthy – I would first of all
delete it, let CS recreate the new one and see if the template starts downloading
thereafter.
> > If you still have problems afterwards do some troubleshooting on
your new SSVM – e.g. run the /usr/local/cloud/systemvm-ssvm-check.sh script and see
if all your connectivity is in place.
> >
> > Regards,
> > Dag Sonstebo
> >
> > On 10/11/2016, 13:46, "Kurt K" <[email protected]> wrote:
> >
> > Hi Dag,
> >
> > >> Can you clarify what the problem is?
> >
> > Please see the below URLs.
> > ===
> >
http://s1152.photobucket.com/user/vincemd12/media/CloudS1_zps1uz6qwjt.png.html
> >
http://s1152.photobucket.com/user/vincemd12/media/CloudS_zps59mfxgti.png.html
> > ===
> >
> > When we try to create the instance via management UI, there
were no
> > default templates detected on template selection option. The
above
> > pasted URLs described the same. 2 system VMs are also
running fine. At
> > the same time, the management server log captures the below
debug errors.
> >
> > ===
> > 2016-11-10 05:42:10,947 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-3:ctx-81097a24) There is no secondary
storage VM for
> > secondary storage host nfs://162.x.x.x/export/secondary
> > ===
> >
> > Any idea?
> >
> > Regards,
> > Kurt
> >
> >
> > On 11/10/2016 04:27 PM, Dag Sonstebo wrote:
> > > Hi Kurt,
> > >
> > > That looks all healthy, and the following shows your
system VM template + the standard CentOS 5.6 template being OK:
> > >
> > > | 3 | routing-3 | SystemVM Template (KVM) | SYSTEM |
2016-11-10 01:48:12| NULL | KVM | Active |
> > > | 4 | centos55-x86_64 | CentOS 5.5(64-bit) no GUI (KVM) |
BUILTIN |2016-11-10 01:48:12 | NULL | KVM | Active |
> > >
> > > In addition to that – if you have system VMs running you
must by definition have the system VM template downloaded OK.
> > >
> > > Can you clarify what the problem is?
> > >
> > > Regards,
> > > Dag Sonstebo
> > >
> > > On 10/11/2016, 10:35, "Kurt K" <[email protected]> wrote:
> > >
> > > Hi Dag,
> > >
> > > The mentioned details are pasted below.
> > >
> > > =====
> > > [root@nfs1 secondary]# tree .
> > > .
> > > └── template
> > > └── tmpl
> > > └── 1
> > > └── 3
> > > ├──
073ecae0-38a0-4a08-87c2-0f5c0f725955.qcow2
> > > └── template.properties
> > >
> > > 4 directories, 2 files
> > > ==
> > > mysql> SELECT
> > >
id,unique_name,name,type,created,removed,hypervisor_type,state FROM
> > > cloud.vm_template;
> > >
+-----+---------------------+---------------------------------------+---------+---------------------+---------+-----------------+----------+
> > > | id | unique_name | name | type | created | removed
| hypervisor_type |
> > > state |
> > >
+-----+---------------------+---------------------------------------+---------+---------------------+---------+-----------------+----------+
> > > | 1 | routing-1 | SystemVM Template (XenServer) |
SYSTEM | 2016-11-10
> > > 01:48:12 | NULL | XenServer | Active |
> > > | 2 | centos53-x86_64 | CentOS 5.3(64-bit) no GUI
(XenServer) | BUILTIN
> > > | 2016-11-10 01:48:12 | NULL | XenServer | Inactive |
> > > | 3 | routing-3 | SystemVM Template (KVM) | SYSTEM |
2016-11-10 01:48:12
> > > | NULL | KVM | Active |
> > > | 4 | centos55-x86_64 | CentOS 5.5(64-bit) no GUI
(KVM) | BUILTIN |
> > > 2016-11-10 01:48:12 | NULL | KVM | Active |
> > > | 5 | centos56-x86_64-xen | CentOS 5.6(64-bit) no GUI
(XenServer) |
> > > BUILTIN | 2016-11-10 01:48:12 | NULL | XenServer |
Active |
> > > | 6 | centos64-x64 | CentOS 6.4(64-bit) GUI (Hyperv)
| BUILTIN |
> > > 2016-11-10 01:48:13 | NULL | Hyperv | Active |
> > > | 7 | centos53-x64 | CentOS 5.3(64-bit) no GUI
(vSphere) | BUILTIN |
> > > 2016-11-10 01:48:13 | NULL | VMware | Active |
> > > | 8 | routing-8 | SystemVM Template (vSphere) |
SYSTEM | 2016-11-10
> > > 01:48:13 | NULL | VMware | Active |
> > > | 9 | routing-9 | SystemVM Template (HyperV) | SYSTEM
| 2016-11-10
> > > 01:48:13 | NULL | Hyperv | Active |
> > > | 10 | routing-10 | SystemVM Template (LXC) | SYSTEM
| 2016-11-10
> > > 01:50:37 | NULL | LXC | Active |
> > > | 200 | vmware-tools.iso | vmware-tools.iso | PERHOST
| 2016-11-10
> > > 09:53:02 | NULL | VMware | Active |
> > > | 201 | xs-tools.iso | xs-tools.iso | PERHOST |
2016-11-10 09:53:02 |
> > > NULL | XenServer | Active |
> > >
+-----+---------------------+---------------------------------------+---------+---------------------+---------+-----------------+----------+
> > > 12 rows in set (0.00 sec)
> > > =====
> > >
> > > From SystemVMs under the infrastructure, shows 2 VMs
running. One is
> > > console-proxy server and other one is secondary
storage VM.
> > >
> > > We have connected to the servers via internal IP.
> > >
> > > -Kurt
> > >
> > >
> > > On 11/10/2016 02:51 PM, Dag Sonstebo wrote:
> > > > Hi Kurt,
> > > >
> > > > Can you provide the following:
> > > >
> > > > - A file listing (e.g. a “tree” dump) of the
contents of nfs://162.219.249.166/export/secondary
> > > > - Output of the following from your DB: SELECT
id,unique_name,name,type,created,removed,hypervisor_type,state FROM cloud.vm_template;
> > > >
> > > > Going by the logs it could look like the system VM
template is downloaded OK, but your problem may be that system VMs are unable to start.
> > > >
> > > >
> > > > Regards,
> > > > Dag Sonstebo
> > > >
> > > > On 10/11/2016, 05:54, "Kurt K"
<[email protected]> wrote:
> > > >
> > > > Hi Dag,
> > > >
> > > > We were getting below logs from management
server.
> > > >
> > > >
> > > > ====
> > > > 2016-11-09 21:36:53,096 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-4:null) Seq
1-4251679523214458881: Processing: {
> > > > Ans: , MgmtId: 207381009036, via: 1, Ver: v1,
Flags: 10,
> > > >
[{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"162.x.x.x","localPath":"/mnt//380ea4ee-87df-39eb-92f4-730dfea4655a","hostPath":"/export/primary","poolType":"NetworkFilesystem","capacityBytes":979751665664,"availableBytes":977581113344},"templateInfo":{},"result":true,"wait":0}}]
> > > > }
> > > > 2016-11-09 21:36:53,451 INFO
[o.a.c.s.i.TemplateServiceImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e)
Downloading builtin template
> > > > centos55-x86_64 to data center: 1
> > > > 2016-11-09 21:36:53,461 DEBUG
[o.a.c.s.i.TemplateDataFactoryImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e) template
4 is already in store:1,
> > > > type:Image
> > > > 2016-11-09 21:36:53,571 DEBUG
[o.a.c.s.i.TemplateDataFactoryImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e) template
4 is already in store:1,
> > > > type:Image
> > > > 2016-11-09 21:36:53,709 DEBUG
[o.a.c.s.i.BaseImageStoreDriverImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e)
Downloading template to data store 1
> > > > 2016-11-09 21:36:53,720 WARN
[c.c.s.d.DownloadMonitorImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e) There is
no secondary storage VM
> > > > for downloading template to image store
> > > > nfs://162.219.249.166/export/secondary
> > > > 2016-11-09 21:36:53,725 DEBUG
[o.a.c.s.i.TemplateDataFactoryImpl]
> > > > (AgentConnectTaskPool-1:ctx-4e501d7e) template
4 is already in store:1,
> > > > type:Image
> > > > ====
> > > >
> > > > -Kurt
> > > >
> > > > On 11/10/2016 11:05 AM, Kurt K wrote:
> > > > > Hi Dag,
> > > > >
> > > > > We had mounted the NFS share on management
server before executed the
> > > > > script. Also there was no /secondary or
other local directory on our
> > > > > management server. All that presented is NFS
secondary share.
> > > > >
> > > > > Can we change any global settings or alter
cloud DB entries for the
> > > > > working of this template?
> > > > >
> > > > > -Kurt
> > > > >
> > > > >
> > > > > On 11/09/2016 07:54 PM, Dag Sonstebo wrote:
> > > > >> Hi Kurt,
> > > > >>
> > > > >> As Simon is hinting at it looks like you
didn’t mount the secondary
> > > > >> storage NFS share on your management server
before running:
> > > > >>
> > > > >>
"/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> > > > >>
> > > > >> -m /secondary -u
> > > > >>
http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-kvm.qcow2.bz2
> > > > >>
> > > > >> -h kvm -F"
> > > > >>
> > > > >> In other words the template downloaded to
the local folder /secondary
> > > > >> on the management server rather than to a
mounted NFS share.
> > > > >> If this is the case delete the contents of
your local /secondary
> > > > >> folder, mount the NFS share in the same
folder and re-run the script
> > > > >> above.
> > > > >>
> > > > >> With regards to downloading from the GUI –
keep in mind you have a
> > > > >> chicken-and-egg scenario – until you have
first downloaded the system
> > > > >> VM template with the above script
CloudStack can not create a SSVM,
> > > > >> hence it will not be able to download any
further templates.
> > > > >>
> > > > >> Regards,
> > > > >> Dag Sonstebo
> > > > >>
> > > > >> From: Kurt K <[email protected]>
> > > > >> Reply-To: "[email protected]"
<[email protected]>
> > > > >> Date: Wednesday, 9 November 2016 at 14:14
> > > > >> To: "[email protected]"
<[email protected]>
> > > > >> Subject: Re: Templates are not showing
> > > > >>
> > > > >> Hi Simon,
> > > > >>
> > > > >> Thanks for the response.
> > > > >>
> > > > >> >> Is your secondary storage on a
separate NFS server?
> > > > >>
> > > > >> Yeah, we have used separate NFS server.
> > > > >>
> > > > >>>> Have you added the secondary storage
mount to CloudStack Management?
> > > > >> We had tried this. But no luck.
> > > > >>
> > > > >> ===
> > > > >> 162.x.x.x:/export/secondary nfs 913G 2.0G
865G 1% /mnt/secondary
> > > > >> ===
> > > > >>
> > > > >> However, we have tried to download a CentOS
6.5 ISO directly using
> > > > >> register ISO option from Management UI.
There was no downloading
> > > > >> status shows and ready status as no. Please
see the below snippets
> > > > >> and attached screen shots.
> > > > >>
> > > > >> ===
> > > > >> [root@nfs1 3]# pwd;ll
> > > > >> /export/secondary/template/tmpl/1/3
> > > > >> total 315396
> > > > >> -rw-r--r--. 1 root root 322954240 Nov 8
23:45
> > > > >> 37ad47be-223d-43ff-ae2c-f10548274c95.qcow2
> > > > >> -rw-r--r--. 1 root root 295 Nov 9 04:30
template.properties
> > > > >> ===
> > > > >>
> > > > >> Any idea about this?
> > > > >>
> > > > >> -Kurt
> > > > >>
> > > > >>
> > > > >> On 11/08/2016 10:25 PM, Simon Weller wrote:
> > > > >>
> > > > >> Hi Kurt,
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> Is your secondary storage on a separate NFS
server?
> > > > >>
> > > > >>
> > > > >>
> > > > >> Have you added the secondary storage mount
to CloudStack Management?
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> If so, have you tried mounting your
secondary storage on a different
> > > > >> server to make sure the permissions are set
correctly?
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> - Si
> > > > >>
> > > > >>
> > > > >>
> > > > >> ________________________________
> > > > >>
> > > > >> From: Kurt K
<[email protected]><mailto:[email protected]>
> > > > >>
> > > > >> Sent: Tuesday, November 8, 2016 8:48 AM
> > > > >>
> > > > >> To:
[email protected]<mailto:[email protected]>
> > > > >>
> > > > >> Subject: Fwd: Templates are not showing
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> -------- Forwarded Message --------
> > > > >>
> > > > >> Subject: Templates are not showing
> > > > >>
> > > > >> Date: Tue, 8 Nov 2016 20:09:17 +0530
> > > > >> [email protected]
> > > > >> www.shapeblue.com
> > > > >> 53 Chandos Place, Covent Garden, London
WC2N 4HSUK
> > > > >> @shapeblue
> > > > >>
> > > > >>
> > > > >>
> > > > >> From: Kurt K
<[email protected]><mailto:[email protected]>
> > > > >>
> > > > >> To:
[email protected]<mailto:[email protected]>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> Hi There,
> > > > >>
> > > > >>
> > > > >>
> > > > >> We have started to build our own cloud
platform on our organization. Our
> > > > >>
> > > > >> deployment is like a small scale arch. We
have used 4 KVM hyper visor
> > > > >>
> > > > >> servers and 1 management server and 3
storage servers. We have
> > > > >>
> > > > >> successfully configured all those servers.
But while creating the
> > > > >>
> > > > >> instance, there were no templates shown on
management UI. We have
> > > > >>
> > > > >> downloaded the templates using this script
> > > > >>
> > > > >>
"/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> > > > >>
> > > > >>
> > > > >> -m /secondary -u
> > > > >>
> > > > >>
http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-kvm.qcow2.bz2
> > > > >>
> > > > >>
> > > > >> -h kvm -F", it was on secondary partition.
But didn't show up there on
> > > > >>
> > > > >> management UI. Is there any other thing
need to be done to show up
> > > > >>
> > > > >> templates on UI?
> > > > >>
> > > > >>
> > > > >>
> > > > >> Any help or suggestions from your end is
much appreciated.
> > > > >>
> > > > >>
> > > > >>
> > > > >> Regards,
> > > > >>
> > > > >> Kurt K
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> ~Kurt
> > > >
> > > >
> > > > --
> > > > *~Kurt *
> > > >
> > > >
> > > >
> > > > [email protected]
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > *~Kurt *
> > >
> > >
> > >
> > > [email protected]
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> >
> >
> > --
> > *~Kurt *
> >
> >
> >
> > [email protected]
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
>
>
> --
> *~Kurt *
>
>
>
>
>
>
--
*~Kurt *
[email protected]
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
