Cloudstack isnt in a state to launch vm's. you want to see it spin up system vm's first. Your deployvm command didnt even appear in the xensource log you provided... because the command didnt make it far in cloudstack (It'd be rejected pretty quick since it cant spin up the system vm's). In your management-server logs you see:
2013-10-30 12:58:34,923 DEBUG [cloud.storage.StorageManagerImpl] (StatsCollector-3:null) Unable to send storage pool command to Pool[202|NetworkFilesystem] via 1 com.cloud.exception.OperationTimedoutException: Commands 314507300 to Host 1 timed out after 3600 and 2013-10-30 12:58:48,425 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is not ready to launch secondary storage VM yet can you telnet to port 443 from the MS to the host? That OperationTimedoutException looks suspicious. On Wed, Oct 30, 2013 at 3:47 PM, Carlos Reategui <[email protected]>wrote: > forgot to include the list in my reply. > > > On Wed, Oct 30, 2013 at 3:44 PM, Carlos Reategui <[email protected]>wrote: > >> That was a cut and paste from a tail -f I did just before hitting the >> start VM button on the console. So not a log roll over. >> >> The management server logs are GMT-8. >> >> >> On Wed, Oct 30, 2013 at 3:36 PM, Ahmad Emneina <[email protected]>wrote: >> >>> i dont see any deploys in that xensource log... might have rolled over >>> as well. >>> >>> >>> On Wed, Oct 30, 2013 at 3:11 PM, Carlos Reategui <[email protected]>wrote: >>> >>>> Nothing that seems related to trying to start an instance in the >>>> xensource.log file (see below). The SMlog file is empty (rotated a short >>>> while ago). >>>> >>>> I have uploaded the new management log file to: >>>> http://reategui.com/cloudstack/management-server.log.new >>>> >>>> Here is the xensource.log section from after restart of the management >>>> server (from my pool master -- second node just showing hearbeat logs): >>>> [20131030T21:58:45.674Z|debug|srvengxen02|199 sr_scan|SR scanner >>>> D:a852c04f1e68|xapi] Automatically scanning SRs = [ >>>> OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d ] >>>> [20131030T21:58:45.676Z|debug|srvengxen02|17932||dummytaskhelper] task >>>> scan one D:75bf6896f47a created by task D:a852c04f1e68 >>>> [20131030T21:58:45.676Z|debug|srvengxen02|17932|scan one >>>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:58:45.679Z|debug|srvengxen02|17933 >>>> unix-RPC||dummytaskhelper] task dispatch:session.slave_login D:a72377292526 >>>> created by task D:75bf6896f47a >>>> [20131030T21:58:45.684Z| info|srvengxen02|17933 >>>> unix-RPC|session.slave_login D:f33a1c65be6f|xapi] Session.create >>>> trackid=f493c253a7b55c2571816cd0c9c90355 pool=true uname= >>>> is_local_superuser=true auth_user_sid= >>>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>>> [20131030T21:58:45.686Z|debug|srvengxen02|17933 >>>> unix-RPC|session.slave_login D:f33a1c65be6f|xapi] Attempting to open >>>> /var/xapi/xapi >>>> [20131030T21:58:45.689Z|debug|srvengxen02|17934 >>>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:02f721a999e5 >>>> created by task D:f33a1c65be6f >>>> [20131030T21:58:45.696Z|debug|srvengxen02|17932|scan one >>>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:58:45.698Z|debug|srvengxen02|17935 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.scan D:1926372044db created by >>>> task D:75bf6896f47a >>>> [20131030T21:58:45.707Z| info|srvengxen02|17935 >>>> unix-RPC|dispatch:SR.scan D:1926372044db|taskhelper] task SR.scan >>>> R:73f6dbba33aa (uuid:a3799647-e481-3697-74ea-49798804b477) created >>>> (trackid=f493c253a7b55c2571816cd0c9c90355) by task D:75bf6896f47a >>>> [20131030T21:58:45.707Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>>> R:73f6dbba33aa|xapi] SR.scan: SR = 'd340de31-8a2f-51b3-926d-5306e2b3405c >>>> (NFS ISO library)' >>>> [20131030T21:58:45.709Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>>> R:73f6dbba33aa|xapi] Marking SR for SR.scan >>>> (task=OpaqueRef:73f6dbba-33aa-a95c-178a-9f9d5ac10b6e) >>>> [20131030T21:58:45.713Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>>> R:73f6dbba33aa|sm] SM iso sr_scan >>>> sr=OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d >>>> [20131030T21:58:45.720Z| info|srvengxen02|17935 unix-RPC|sm_exec >>>> D:1161068a7230|xapi] Session.create >>>> trackid=4ee84749e687d05ce996d016e609f518 pool=false uname= >>>> is_local_superuser=true auth_user_sid= >>>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>>> [20131030T21:58:45.721Z|debug|srvengxen02|17935 unix-RPC|sm_exec >>>> D:1161068a7230|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:58:45.725Z|debug|srvengxen02|17936 >>>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:f2e326cdf62a >>>> created by task D:1161068a7230 >>>> [20131030T21:58:45.873Z|debug|srvengxen02|17937 >>>> unix-RPC||dummytaskhelper] task dispatch:host.get_other_config >>>> D:bc0c5e2a3024 created by task R:73f6dbba33aa >>>> [20131030T21:58:45.910Z|debug|srvengxen02|17938 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:ee7f022e0661 >>>> created by task R:73f6dbba33aa >>>> [20131030T21:58:45.920Z|debug|srvengxen02|17939 >>>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>>> D:b2a1cf32cca0 created by task R:73f6dbba33aa >>>> [20131030T21:58:45.991Z|debug|srvengxen02|17940 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:29146266030e >>>> created by task R:73f6dbba33aa >>>> [20131030T21:58:46.000Z|debug|srvengxen02|17941 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_virtual_allocation >>>> D:e82fe926a634 created by task R:73f6dbba33aa >>>> [20131030T21:58:46.013Z|debug|srvengxen02|17942 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_size >>>> D:ce753e786464 created by task R:73f6dbba33aa >>>> [20131030T21:58:46.025Z|debug|srvengxen02|17943 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_utilisation >>>> D:51d59ee2c652 created by task R:73f6dbba33aa >>>> [20131030T21:58:46.037Z|debug|srvengxen02|17944 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:ccbe694ba9b5 >>>> created by task R:73f6dbba33aa >>>> [20131030T21:58:46.047Z|debug|srvengxen02|17945 >>>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>>> D:1a049c6abb04 created by task R:73f6dbba33aa >>>> [20131030T21:58:46.126Z| info|srvengxen02|17935 unix-RPC|sm_exec >>>> D:1161068a7230|xapi] Session.destroy >>>> trackid=4ee84749e687d05ce996d016e609f518 >>>> [20131030T21:58:46.131Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>>> R:73f6dbba33aa|xapi] Unmarking SR after SR.scan >>>> (task=OpaqueRef:73f6dbba-33aa-a95c-178a-9f9d5ac10b6e) >>>> [20131030T21:58:46.146Z|debug|srvengxen02|17932|scan one >>>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:58:46.149Z|debug|srvengxen02|17946 >>>> unix-RPC||dummytaskhelper] task dispatch:session.logout D:beff7c62a646 >>>> created by task D:75bf6896f47a >>>> [20131030T21:58:46.154Z| info|srvengxen02|17946 unix-RPC|session.logout >>>> D:74779ffb1f36|xapi] Session.destroy >>>> trackid=f493c253a7b55c2571816cd0c9c90355 >>>> [20131030T21:58:46.157Z|debug|srvengxen02|17932|scan one >>>> D:75bf6896f47a|xapi] Scan of SR d340de31-8a2f-51b3-926d-5306e2b3405c >>>> complete. >>>> [20131030T21:59:15.705Z|debug|srvengxen02|199 sr_scan|SR scanner >>>> D:a852c04f1e68|xapi] Automatically scanning SRs = [ >>>> OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d ] >>>> [20131030T21:59:15.707Z|debug|srvengxen02|17949||dummytaskhelper] task >>>> scan one D:f838014d67c1 created by task D:a852c04f1e68 >>>> [20131030T21:59:15.707Z|debug|srvengxen02|17949|scan one >>>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:59:15.710Z|debug|srvengxen02|17950 >>>> unix-RPC||dummytaskhelper] task dispatch:session.slave_login D:d7f8041f0ea8 >>>> created by task D:f838014d67c1 >>>> [20131030T21:59:15.715Z| info|srvengxen02|17950 >>>> unix-RPC|session.slave_login D:9d5021ebb7bb|xapi] Session.create >>>> trackid=614665040ebf96f8d3688b561cae32fa pool=true uname= >>>> is_local_superuser=true auth_user_sid= >>>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>>> [20131030T21:59:15.717Z|debug|srvengxen02|17950 >>>> unix-RPC|session.slave_login D:9d5021ebb7bb|xapi] Attempting to open >>>> /var/xapi/xapi >>>> [20131030T21:59:15.720Z|debug|srvengxen02|17951 >>>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:9b961577759f >>>> created by task D:9d5021ebb7bb >>>> [20131030T21:59:15.726Z|debug|srvengxen02|17949|scan one >>>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:59:15.729Z|debug|srvengxen02|17952 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.scan D:34238362aa17 created by >>>> task D:f838014d67c1 >>>> [20131030T21:59:15.737Z| info|srvengxen02|17952 >>>> unix-RPC|dispatch:SR.scan D:34238362aa17|taskhelper] task SR.scan >>>> R:9993e27cadc8 (uuid:24efaaf1-1a7f-2f8b-aec5-d0bf386b4158) created >>>> (trackid=614665040ebf96f8d3688b561cae32fa) by task D:f838014d67c1 >>>> [20131030T21:59:15.737Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>>> R:9993e27cadc8|xapi] SR.scan: SR = 'd340de31-8a2f-51b3-926d-5306e2b3405c >>>> (NFS ISO library)' >>>> [20131030T21:59:15.739Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>>> R:9993e27cadc8|xapi] Marking SR for SR.scan >>>> (task=OpaqueRef:9993e27c-adc8-e77e-7813-995168649469) >>>> [20131030T21:59:15.743Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>>> R:9993e27cadc8|sm] SM iso sr_scan >>>> sr=OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d >>>> [20131030T21:59:15.750Z| info|srvengxen02|17952 unix-RPC|sm_exec >>>> D:f56ec69d7e1f|xapi] Session.create >>>> trackid=8d403a50eeb9a2bca0c314a3d1985803 pool=false uname= >>>> is_local_superuser=true auth_user_sid= >>>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>>> [20131030T21:59:15.751Z|debug|srvengxen02|17952 unix-RPC|sm_exec >>>> D:f56ec69d7e1f|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:59:15.754Z|debug|srvengxen02|17953 >>>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:fddc51c18b37 >>>> created by task D:f56ec69d7e1f >>>> [20131030T21:59:15.903Z|debug|srvengxen02|17954 >>>> unix-RPC||dummytaskhelper] task dispatch:host.get_other_config >>>> D:13fe3c6a32c9 created by task R:9993e27cadc8 >>>> [20131030T21:59:15.934Z|debug|srvengxen02|17955 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:c192d2117790 >>>> created by task R:9993e27cadc8 >>>> [20131030T21:59:15.944Z|debug|srvengxen02|17956 >>>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>>> D:ff0b31aa3a4f created by task R:9993e27cadc8 >>>> [20131030T21:59:16.012Z|debug|srvengxen02|17957 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:5d0800007279 >>>> created by task R:9993e27cadc8 >>>> [20131030T21:59:16.022Z|debug|srvengxen02|17958 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_virtual_allocation >>>> D:dec8e9bb21e3 created by task R:9993e27cadc8 >>>> [20131030T21:59:16.034Z|debug|srvengxen02|17959 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_size >>>> D:b205f6a468e2 created by task R:9993e27cadc8 >>>> [20131030T21:59:16.046Z|debug|srvengxen02|17960 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_utilisation >>>> D:dda160d61e63 created by task R:9993e27cadc8 >>>> [20131030T21:59:16.059Z|debug|srvengxen02|17961 >>>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:276d90611376 >>>> created by task R:9993e27cadc8 >>>> [20131030T21:59:16.069Z|debug|srvengxen02|17962 >>>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>>> D:7e73bcb1cb79 created by task R:9993e27cadc8 >>>> [20131030T21:59:16.150Z| info|srvengxen02|17952 unix-RPC|sm_exec >>>> D:f56ec69d7e1f|xapi] Session.destroy >>>> trackid=8d403a50eeb9a2bca0c314a3d1985803 >>>> [20131030T21:59:16.155Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>>> R:9993e27cadc8|xapi] Unmarking SR after SR.scan >>>> (task=OpaqueRef:9993e27c-adc8-e77e-7813-995168649469) >>>> [20131030T21:59:16.170Z|debug|srvengxen02|17949|scan one >>>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>>> [20131030T21:59:16.173Z|debug|srvengxen02|17963 >>>> unix-RPC||dummytaskhelper] task dispatch:session.logout D:c7be76666e35 >>>> created by task D:f838014d67c1 >>>> [20131030T21:59:16.178Z| info|srvengxen02|17963 unix-RPC|session.logout >>>> D:b075ca9e306b|xapi] Session.destroy >>>> trackid=614665040ebf96f8d3688b561cae32fa >>>> [20131030T21:59:16.181Z|debug|srvengxen02|17949|scan one >>>> D:f838014d67c1|xapi] Scan of SR d340de31-8a2f-51b3-926d-5306e2b3405c >>>> complete. >>>> >>>> >>>> >>>> On Wed, Oct 30, 2013 at 2:56 PM, Carlos Reategui >>>> <[email protected]>wrote: >>>> >>>>> Currently, Cloudstack is not showing an SSVM. It does have an CPVM >>>>> and a VR that are both stuck in an expunging state and have been for over >>>>> a >>>>> day. >>>>> >>>>> I'll clear the management log and restart the Management server and >>>>> try to start one of my instances that is currently in stopped state. >>>>> >>>>> I'll upload the logs once I'm done. >>>>> >>>>> Thank you for looking at these. >>>>> >>>>> >>>>> >>>>> On Wed, Oct 30, 2013 at 2:44 PM, Ahmad Emneina <[email protected]>wrote: >>>>> >>>>>> hrm... should work just fine from cloudstack. Do xensource.log or >>>>>> SMlog on >>>>>> the xenserver say anything specific when starting the vm's via >>>>>> cloudstack? >>>>>> >>>>>> >>>>>> On Wed, Oct 30, 2013 at 2:00 PM, Carlos Reategui <[email protected] >>>>>> >wrote: >>>>>> >>>>>> > Installed and ran fine using the same NFS SR as the rest of my CS >>>>>> root >>>>>> > disks are on. >>>>>> > >>>>>> > >>>>>> > On Wed, Oct 30, 2013 at 1:44 PM, Ahmad Emneina <[email protected]> >>>>>> wrote: >>>>>> > >>>>>> >> launch a vm independently via xen center. >>>>>> >> >>>>>> >> >>>>>> >> On Wed, Oct 30, 2013 at 1:36 PM, Carlos Reategui < >>>>>> [email protected]>wrote: >>>>>> >> >>>>>> >>> Should I try launching a vm independently? or should I try to >>>>>> start one >>>>>> >>> of the vhds that is in the primary storage? >>>>>> >>> >>>>>> >>> >>>>>> >>> On Wed, Oct 30, 2013 at 1:34 PM, Ahmad Emneina < >>>>>> [email protected]>wrote: >>>>>> >>> >>>>>> >>>> outside of cloudstack, can you deploy a vm on your host, to the >>>>>> desired >>>>>> >>>> storage pool. It looks like the hypervisor host cant connect to >>>>>> its >>>>>> >>>> storage >>>>>> >>>> server. >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On Wed, Oct 30, 2013 at 1:31 PM, Carlos Reategui < >>>>>> [email protected] >>>>>> >>>> >wrote: >>>>>> >>>> >>>>>> >>>> > Here is a link to the log file: >>>>>> >>>> > http://reategui.com/cloudstack/management-server.log >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > On Wed, Oct 30, 2013 at 12:22 PM, Ahmad Emneina < >>>>>> [email protected] >>>>>> >>>> >wrote: >>>>>> >>>> > >>>>>> >>>> >> can we get the full logs? There should be something simple >>>>>> blocking >>>>>> >>>> the >>>>>> >>>> >> reconnection of the management server to the hosts. I worked >>>>>> past >>>>>> >>>> this, >>>>>> >>>> >> this past weekend against 4.2. So i dont think your results >>>>>> will >>>>>> >>>> differ >>>>>> >>>> >> upgrading to 4.2... >>>>>> >>>> >> >>>>>> >>>> >> >>>>>> >>>> >> On Wed, Oct 30, 2013 at 12:14 PM, Carlos Reategui < >>>>>> >>>> [email protected] >>>>>> >>>> >> >wrote: >>>>>> >>>> >> >>>>>> >>>> >> > No replies to my other emails. I really need help getting >>>>>> my CS >>>>>> >>>> 4.1.1 >>>>>> >>>> >> > cluster back up. >>>>>> >>>> >> > >>>>>> >>>> >> > I basically have a CloudStack console that thinks >>>>>> everything is >>>>>> >>>> fine, >>>>>> >>>> >> but >>>>>> >>>> >> > looking at the management logs there seems to be a problem >>>>>> >>>> connecting to >>>>>> >>>> >> > the hosts. XenCenter does not seem to agree and thinks all >>>>>> is >>>>>> >>>> fine with >>>>>> >>>> >> > the hosts. Iptables is disabled on the hosts and the >>>>>> management >>>>>> >>>> server >>>>>> >>>> >> so >>>>>> >>>> >> > not a firewall issue. Primary storage is mounted on the >>>>>> hosts and >>>>>> >>>> I am >>>>>> >>>> >> > able to mount secondary storage. >>>>>> >>>> >> > >>>>>> >>>> >> > I believe I have the following options: >>>>>> >>>> >> > 1) Backup all my vhds, reinstall XenServer and CS, import >>>>>> the vhds >>>>>> >>>> as >>>>>> >>>> >> > templates and relaunch my 20+ VMs. I see this as a last >>>>>> resort >>>>>> >>>> option >>>>>> >>>> >> that >>>>>> >>>> >> > I would rather not have to do >>>>>> >>>> >> > 2) Remove my XS hosts from CS (assuming that wont get rid >>>>>> of my >>>>>> >>>> >> instances), >>>>>> >>>> >> > clear tags (or re-install XS), re-add XS hosts and hope for >>>>>> the >>>>>> >>>> best. >>>>>> >>>> >> > 3) Attempt to upgrade to 4.2 and hope my problems go away. >>>>>> >>>> >> > >>>>>> >>>> >> > Anyone have any thoughts on how to proceed? >>>>>> >>>> >> > >>>>>> >>>> >> > thanks >>>>>> >>>> >> > Carlos >>>>>> >>>> >> > >>>>>> >>>> >> >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> > >>>>>> >>>>> >>>>> >>>> >>> >> >
