forgot to include the list in my reply. On Wed, Oct 30, 2013 at 3:44 PM, Carlos Reategui <[email protected]>wrote:
> That was a cut and paste from a tail -f I did just before hitting the > start VM button on the console. So not a log roll over. > > The management server logs are GMT-8. > > > On Wed, Oct 30, 2013 at 3:36 PM, Ahmad Emneina <[email protected]> wrote: > >> i dont see any deploys in that xensource log... might have rolled over as >> well. >> >> >> On Wed, Oct 30, 2013 at 3:11 PM, Carlos Reategui <[email protected]>wrote: >> >>> Nothing that seems related to trying to start an instance in the >>> xensource.log file (see below). The SMlog file is empty (rotated a short >>> while ago). >>> >>> I have uploaded the new management log file to: >>> http://reategui.com/cloudstack/management-server.log.new >>> >>> Here is the xensource.log section from after restart of the management >>> server (from my pool master -- second node just showing hearbeat logs): >>> [20131030T21:58:45.674Z|debug|srvengxen02|199 sr_scan|SR scanner >>> D:a852c04f1e68|xapi] Automatically scanning SRs = [ >>> OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d ] >>> [20131030T21:58:45.676Z|debug|srvengxen02|17932||dummytaskhelper] task >>> scan one D:75bf6896f47a created by task D:a852c04f1e68 >>> [20131030T21:58:45.676Z|debug|srvengxen02|17932|scan one >>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:58:45.679Z|debug|srvengxen02|17933 >>> unix-RPC||dummytaskhelper] task dispatch:session.slave_login D:a72377292526 >>> created by task D:75bf6896f47a >>> [20131030T21:58:45.684Z| info|srvengxen02|17933 >>> unix-RPC|session.slave_login D:f33a1c65be6f|xapi] Session.create >>> trackid=f493c253a7b55c2571816cd0c9c90355 pool=true uname= >>> is_local_superuser=true auth_user_sid= >>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>> [20131030T21:58:45.686Z|debug|srvengxen02|17933 >>> unix-RPC|session.slave_login D:f33a1c65be6f|xapi] Attempting to open >>> /var/xapi/xapi >>> [20131030T21:58:45.689Z|debug|srvengxen02|17934 >>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:02f721a999e5 >>> created by task D:f33a1c65be6f >>> [20131030T21:58:45.696Z|debug|srvengxen02|17932|scan one >>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:58:45.698Z|debug|srvengxen02|17935 >>> unix-RPC||dummytaskhelper] task dispatch:SR.scan D:1926372044db created by >>> task D:75bf6896f47a >>> [20131030T21:58:45.707Z| info|srvengxen02|17935 >>> unix-RPC|dispatch:SR.scan D:1926372044db|taskhelper] task SR.scan >>> R:73f6dbba33aa (uuid:a3799647-e481-3697-74ea-49798804b477) created >>> (trackid=f493c253a7b55c2571816cd0c9c90355) by task D:75bf6896f47a >>> [20131030T21:58:45.707Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>> R:73f6dbba33aa|xapi] SR.scan: SR = 'd340de31-8a2f-51b3-926d-5306e2b3405c >>> (NFS ISO library)' >>> [20131030T21:58:45.709Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>> R:73f6dbba33aa|xapi] Marking SR for SR.scan >>> (task=OpaqueRef:73f6dbba-33aa-a95c-178a-9f9d5ac10b6e) >>> [20131030T21:58:45.713Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>> R:73f6dbba33aa|sm] SM iso sr_scan >>> sr=OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d >>> [20131030T21:58:45.720Z| info|srvengxen02|17935 unix-RPC|sm_exec >>> D:1161068a7230|xapi] Session.create >>> trackid=4ee84749e687d05ce996d016e609f518 pool=false uname= >>> is_local_superuser=true auth_user_sid= >>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>> [20131030T21:58:45.721Z|debug|srvengxen02|17935 unix-RPC|sm_exec >>> D:1161068a7230|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:58:45.725Z|debug|srvengxen02|17936 >>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:f2e326cdf62a >>> created by task D:1161068a7230 >>> [20131030T21:58:45.873Z|debug|srvengxen02|17937 >>> unix-RPC||dummytaskhelper] task dispatch:host.get_other_config >>> D:bc0c5e2a3024 created by task R:73f6dbba33aa >>> [20131030T21:58:45.910Z|debug|srvengxen02|17938 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:ee7f022e0661 >>> created by task R:73f6dbba33aa >>> [20131030T21:58:45.920Z|debug|srvengxen02|17939 >>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>> D:b2a1cf32cca0 created by task R:73f6dbba33aa >>> [20131030T21:58:45.991Z|debug|srvengxen02|17940 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:29146266030e >>> created by task R:73f6dbba33aa >>> [20131030T21:58:46.000Z|debug|srvengxen02|17941 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_virtual_allocation >>> D:e82fe926a634 created by task R:73f6dbba33aa >>> [20131030T21:58:46.013Z|debug|srvengxen02|17942 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_size >>> D:ce753e786464 created by task R:73f6dbba33aa >>> [20131030T21:58:46.025Z|debug|srvengxen02|17943 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_utilisation >>> D:51d59ee2c652 created by task R:73f6dbba33aa >>> [20131030T21:58:46.037Z|debug|srvengxen02|17944 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:ccbe694ba9b5 >>> created by task R:73f6dbba33aa >>> [20131030T21:58:46.047Z|debug|srvengxen02|17945 >>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>> D:1a049c6abb04 created by task R:73f6dbba33aa >>> [20131030T21:58:46.126Z| info|srvengxen02|17935 unix-RPC|sm_exec >>> D:1161068a7230|xapi] Session.destroy >>> trackid=4ee84749e687d05ce996d016e609f518 >>> [20131030T21:58:46.131Z|debug|srvengxen02|17935 unix-RPC|SR.scan >>> R:73f6dbba33aa|xapi] Unmarking SR after SR.scan >>> (task=OpaqueRef:73f6dbba-33aa-a95c-178a-9f9d5ac10b6e) >>> [20131030T21:58:46.146Z|debug|srvengxen02|17932|scan one >>> D:75bf6896f47a|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:58:46.149Z|debug|srvengxen02|17946 >>> unix-RPC||dummytaskhelper] task dispatch:session.logout D:beff7c62a646 >>> created by task D:75bf6896f47a >>> [20131030T21:58:46.154Z| info|srvengxen02|17946 unix-RPC|session.logout >>> D:74779ffb1f36|xapi] Session.destroy >>> trackid=f493c253a7b55c2571816cd0c9c90355 >>> [20131030T21:58:46.157Z|debug|srvengxen02|17932|scan one >>> D:75bf6896f47a|xapi] Scan of SR d340de31-8a2f-51b3-926d-5306e2b3405c >>> complete. >>> [20131030T21:59:15.705Z|debug|srvengxen02|199 sr_scan|SR scanner >>> D:a852c04f1e68|xapi] Automatically scanning SRs = [ >>> OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d ] >>> [20131030T21:59:15.707Z|debug|srvengxen02|17949||dummytaskhelper] task >>> scan one D:f838014d67c1 created by task D:a852c04f1e68 >>> [20131030T21:59:15.707Z|debug|srvengxen02|17949|scan one >>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:59:15.710Z|debug|srvengxen02|17950 >>> unix-RPC||dummytaskhelper] task dispatch:session.slave_login D:d7f8041f0ea8 >>> created by task D:f838014d67c1 >>> [20131030T21:59:15.715Z| info|srvengxen02|17950 >>> unix-RPC|session.slave_login D:9d5021ebb7bb|xapi] Session.create >>> trackid=614665040ebf96f8d3688b561cae32fa pool=true uname= >>> is_local_superuser=true auth_user_sid= >>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>> [20131030T21:59:15.717Z|debug|srvengxen02|17950 >>> unix-RPC|session.slave_login D:9d5021ebb7bb|xapi] Attempting to open >>> /var/xapi/xapi >>> [20131030T21:59:15.720Z|debug|srvengxen02|17951 >>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:9b961577759f >>> created by task D:9d5021ebb7bb >>> [20131030T21:59:15.726Z|debug|srvengxen02|17949|scan one >>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:59:15.729Z|debug|srvengxen02|17952 >>> unix-RPC||dummytaskhelper] task dispatch:SR.scan D:34238362aa17 created by >>> task D:f838014d67c1 >>> [20131030T21:59:15.737Z| info|srvengxen02|17952 >>> unix-RPC|dispatch:SR.scan D:34238362aa17|taskhelper] task SR.scan >>> R:9993e27cadc8 (uuid:24efaaf1-1a7f-2f8b-aec5-d0bf386b4158) created >>> (trackid=614665040ebf96f8d3688b561cae32fa) by task D:f838014d67c1 >>> [20131030T21:59:15.737Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>> R:9993e27cadc8|xapi] SR.scan: SR = 'd340de31-8a2f-51b3-926d-5306e2b3405c >>> (NFS ISO library)' >>> [20131030T21:59:15.739Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>> R:9993e27cadc8|xapi] Marking SR for SR.scan >>> (task=OpaqueRef:9993e27c-adc8-e77e-7813-995168649469) >>> [20131030T21:59:15.743Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>> R:9993e27cadc8|sm] SM iso sr_scan >>> sr=OpaqueRef:6fe82fcd-15b3-be18-6636-5380ee19de1d >>> [20131030T21:59:15.750Z| info|srvengxen02|17952 unix-RPC|sm_exec >>> D:f56ec69d7e1f|xapi] Session.create >>> trackid=8d403a50eeb9a2bca0c314a3d1985803 pool=false uname= >>> is_local_superuser=true auth_user_sid= >>> parent=trackid=9834f5af41c964e225f24279aefe4e49 >>> [20131030T21:59:15.751Z|debug|srvengxen02|17952 unix-RPC|sm_exec >>> D:f56ec69d7e1f|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:59:15.754Z|debug|srvengxen02|17953 >>> unix-RPC||dummytaskhelper] task dispatch:session.get_uuid D:fddc51c18b37 >>> created by task D:f56ec69d7e1f >>> [20131030T21:59:15.903Z|debug|srvengxen02|17954 >>> unix-RPC||dummytaskhelper] task dispatch:host.get_other_config >>> D:13fe3c6a32c9 created by task R:9993e27cadc8 >>> [20131030T21:59:15.934Z|debug|srvengxen02|17955 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:c192d2117790 >>> created by task R:9993e27cadc8 >>> [20131030T21:59:15.944Z|debug|srvengxen02|17956 >>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>> D:ff0b31aa3a4f created by task R:9993e27cadc8 >>> [20131030T21:59:16.012Z|debug|srvengxen02|17957 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:5d0800007279 >>> created by task R:9993e27cadc8 >>> [20131030T21:59:16.022Z|debug|srvengxen02|17958 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_virtual_allocation >>> D:dec8e9bb21e3 created by task R:9993e27cadc8 >>> [20131030T21:59:16.034Z|debug|srvengxen02|17959 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_size >>> D:b205f6a468e2 created by task R:9993e27cadc8 >>> [20131030T21:59:16.046Z|debug|srvengxen02|17960 >>> unix-RPC||dummytaskhelper] task dispatch:SR.set_physical_utilisation >>> D:dda160d61e63 created by task R:9993e27cadc8 >>> [20131030T21:59:16.059Z|debug|srvengxen02|17961 >>> unix-RPC||dummytaskhelper] task dispatch:SR.get_by_uuid D:276d90611376 >>> created by task R:9993e27cadc8 >>> [20131030T21:59:16.069Z|debug|srvengxen02|17962 >>> unix-RPC||dummytaskhelper] task dispatch:VDI.get_all_records_where >>> D:7e73bcb1cb79 created by task R:9993e27cadc8 >>> [20131030T21:59:16.150Z| info|srvengxen02|17952 unix-RPC|sm_exec >>> D:f56ec69d7e1f|xapi] Session.destroy >>> trackid=8d403a50eeb9a2bca0c314a3d1985803 >>> [20131030T21:59:16.155Z|debug|srvengxen02|17952 unix-RPC|SR.scan >>> R:9993e27cadc8|xapi] Unmarking SR after SR.scan >>> (task=OpaqueRef:9993e27c-adc8-e77e-7813-995168649469) >>> [20131030T21:59:16.170Z|debug|srvengxen02|17949|scan one >>> D:f838014d67c1|xapi] Attempting to open /var/xapi/xapi >>> [20131030T21:59:16.173Z|debug|srvengxen02|17963 >>> unix-RPC||dummytaskhelper] task dispatch:session.logout D:c7be76666e35 >>> created by task D:f838014d67c1 >>> [20131030T21:59:16.178Z| info|srvengxen02|17963 unix-RPC|session.logout >>> D:b075ca9e306b|xapi] Session.destroy >>> trackid=614665040ebf96f8d3688b561cae32fa >>> [20131030T21:59:16.181Z|debug|srvengxen02|17949|scan one >>> D:f838014d67c1|xapi] Scan of SR d340de31-8a2f-51b3-926d-5306e2b3405c >>> complete. >>> >>> >>> >>> On Wed, Oct 30, 2013 at 2:56 PM, Carlos Reategui <[email protected]>wrote: >>> >>>> Currently, Cloudstack is not showing an SSVM. It does have an CPVM and >>>> a VR that are both stuck in an expunging state and have been for over a >>>> day. >>>> >>>> I'll clear the management log and restart the Management server and try >>>> to start one of my instances that is currently in stopped state. >>>> >>>> I'll upload the logs once I'm done. >>>> >>>> Thank you for looking at these. >>>> >>>> >>>> >>>> On Wed, Oct 30, 2013 at 2:44 PM, Ahmad Emneina <[email protected]>wrote: >>>> >>>>> hrm... should work just fine from cloudstack. Do xensource.log or >>>>> SMlog on >>>>> the xenserver say anything specific when starting the vm's via >>>>> cloudstack? >>>>> >>>>> >>>>> On Wed, Oct 30, 2013 at 2:00 PM, Carlos Reategui <[email protected] >>>>> >wrote: >>>>> >>>>> > Installed and ran fine using the same NFS SR as the rest of my CS >>>>> root >>>>> > disks are on. >>>>> > >>>>> > >>>>> > On Wed, Oct 30, 2013 at 1:44 PM, Ahmad Emneina <[email protected]> >>>>> wrote: >>>>> > >>>>> >> launch a vm independently via xen center. >>>>> >> >>>>> >> >>>>> >> On Wed, Oct 30, 2013 at 1:36 PM, Carlos Reategui < >>>>> [email protected]>wrote: >>>>> >> >>>>> >>> Should I try launching a vm independently? or should I try to >>>>> start one >>>>> >>> of the vhds that is in the primary storage? >>>>> >>> >>>>> >>> >>>>> >>> On Wed, Oct 30, 2013 at 1:34 PM, Ahmad Emneina <[email protected] >>>>> >wrote: >>>>> >>> >>>>> >>>> outside of cloudstack, can you deploy a vm on your host, to the >>>>> desired >>>>> >>>> storage pool. It looks like the hypervisor host cant connect to >>>>> its >>>>> >>>> storage >>>>> >>>> server. >>>>> >>>> >>>>> >>>> >>>>> >>>> On Wed, Oct 30, 2013 at 1:31 PM, Carlos Reategui < >>>>> [email protected] >>>>> >>>> >wrote: >>>>> >>>> >>>>> >>>> > Here is a link to the log file: >>>>> >>>> > http://reategui.com/cloudstack/management-server.log >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > On Wed, Oct 30, 2013 at 12:22 PM, Ahmad Emneina < >>>>> [email protected] >>>>> >>>> >wrote: >>>>> >>>> > >>>>> >>>> >> can we get the full logs? There should be something simple >>>>> blocking >>>>> >>>> the >>>>> >>>> >> reconnection of the management server to the hosts. I worked >>>>> past >>>>> >>>> this, >>>>> >>>> >> this past weekend against 4.2. So i dont think your results >>>>> will >>>>> >>>> differ >>>>> >>>> >> upgrading to 4.2... >>>>> >>>> >> >>>>> >>>> >> >>>>> >>>> >> On Wed, Oct 30, 2013 at 12:14 PM, Carlos Reategui < >>>>> >>>> [email protected] >>>>> >>>> >> >wrote: >>>>> >>>> >> >>>>> >>>> >> > No replies to my other emails. I really need help getting >>>>> my CS >>>>> >>>> 4.1.1 >>>>> >>>> >> > cluster back up. >>>>> >>>> >> > >>>>> >>>> >> > I basically have a CloudStack console that thinks everything >>>>> is >>>>> >>>> fine, >>>>> >>>> >> but >>>>> >>>> >> > looking at the management logs there seems to be a problem >>>>> >>>> connecting to >>>>> >>>> >> > the hosts. XenCenter does not seem to agree and thinks all >>>>> is >>>>> >>>> fine with >>>>> >>>> >> > the hosts. Iptables is disabled on the hosts and the >>>>> management >>>>> >>>> server >>>>> >>>> >> so >>>>> >>>> >> > not a firewall issue. Primary storage is mounted on the >>>>> hosts and >>>>> >>>> I am >>>>> >>>> >> > able to mount secondary storage. >>>>> >>>> >> > >>>>> >>>> >> > I believe I have the following options: >>>>> >>>> >> > 1) Backup all my vhds, reinstall XenServer and CS, import >>>>> the vhds >>>>> >>>> as >>>>> >>>> >> > templates and relaunch my 20+ VMs. I see this as a last >>>>> resort >>>>> >>>> option >>>>> >>>> >> that >>>>> >>>> >> > I would rather not have to do >>>>> >>>> >> > 2) Remove my XS hosts from CS (assuming that wont get rid of >>>>> my >>>>> >>>> >> instances), >>>>> >>>> >> > clear tags (or re-install XS), re-add XS hosts and hope for >>>>> the >>>>> >>>> best. >>>>> >>>> >> > 3) Attempt to upgrade to 4.2 and hope my problems go away. >>>>> >>>> >> > >>>>> >>>> >> > Anyone have any thoughts on how to proceed? >>>>> >>>> >> > >>>>> >>>> >> > thanks >>>>> >>>> >> > Carlos >>>>> >>>> >> > >>>>> >>>> >> >>>>> >>>> > >>>>> >>>> > >>>>> >>>> >>>>> >>> >>>>> >>> >>>>> >> >>>>> > >>>>> >>>> >>>> >>> >> >
