I am checking SMlog now on all hosts. Jeremy
-----Original Message----- From: Rajani Karuturi [mailto:[email protected]] Sent: Friday, June 9, 2017 9:00 AM To: Users <[email protected]> Subject: Re: Recreating SystemVM's on xenserver log, did you check what is causing " HOST_CANNOT_ATTACH_NETWORK"? ~Rajani http://cloudplatform.accelerite.com/ On Fri, Jun 9, 2017 at 7:00 PM, Jeremy Peterson <[email protected]> wrote: > 08:28:43 select * from vm_instance where name like 's-%' limit > 10000 7481 row(s) returned 0.000 sec / 0.032 sec > > All vm's 'state' returned Destoryed outside of the current vm 7873 > which is in a Stopped state but that goes Destroyed and a new get created. > > Any other suggestions? > > Jeremy > > > -----Original Message----- > From: Jeremy Peterson [mailto:[email protected]] > Sent: Thursday, June 8, 2017 12:47 AM > To: [email protected] > Subject: Re: Recreating SystemVM's > > I'll make that change in the am. > > Today I put a host in maintence and rebooted because proxy and > secstore vm were constantly being created on that host and still no change. > > Let you know tomorrow. > > Jeremy > > > Sent from my Verizon, Samsung Galaxy smartphone > > > -------- Original message -------- > From: Rajani Karuturi <[email protected]> > Date: 6/8/17 12:07 AM (GMT-06:00) > To: Users <[email protected]> > Subject: Re: Recreating SystemVM's > > Did you check SMLog on xenserver? > unable to destroy task(com.xensource.xenapi.Task@256829a8) on > host(b34f086e-fabf-471e-9feb-8f54362d7d0f) due to You gave an invalid > object reference. The object may have recently been deleted. The > class parameter gives the type of reference given, and the handle > parameter echoes the bad value given. > > Looks like Destroy of SSVM failed. What state is SSVM in? mark it as > Destroyed in cloud DB and wait for cloudstack to create a new SSVM. > > ~Rajani > http://cloudplatform.accelerite.com/ > > On Thu, Jun 8, 2017 at 1:11 AM, Jeremy Peterson > <[email protected]> > wrote: > > > Probably agreed. > > > > But I ran toolstack restart on all hypervisors and v-3193 just tried > > to create and fail along with s-5398. > > > > The PIF error went away. But VM's are still recreating > > > > https://pastebin.com/4n4xBgMT > > > > New log from this afternoon. > > > > My catalina.out is over 4GB > > > > Jeremy > > > > > > -----Original Message----- > > From: Makrand [mailto:[email protected]] > > Sent: Wednesday, June 7, 2017 12:52 AM > > To: [email protected] > > Subject: Re: Recreating SystemVM's > > > > Hi there, > > > > Looks more like hypervisor issue. > > > > Just run *xe-toolstack-restart* on hosts where these VMs are trying > > to start or if you don't have too many hosts, better run on all > > members including master. most of i/o related issues squared off by > > toolstack bounce. > > > > -- > > Makrand > > > > > > On Wed, Jun 7, 2017 at 3:01 AM, Jeremy Peterson > > <[email protected]> > > wrote: > > > > > Ok so I pulled this from Sunday morning. > > > > > > https://pastebin.com/nCETw1sC > > > > > > > > > errorInfo: [HOST_CANNOT_ATTACH_NETWORK, > > > OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7, > > > OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2] > > > > > > XenServer error. > > > > > > Now this still gets me because all of the other VM's launched just > fine. > > > > > > Going into XenCenter I see an error at the bottom This PIF is a > > > bond slave and cannot be plugged. > > > > > > ??? > > > > > > If I go to networking on the hosts I see the storage vlans and > > > bonds are all there. > > > > > > I see my GUEST-PUB bond is there and LACP is setup correct. > > > > > > Any suggestions ? > > > > > > > > > Jeremy > > > > > > > > > -----Original Message----- > > > From: Jeremy Peterson [mailto:[email protected]] > > > Sent: Tuesday, June 6, 2017 9:23 AM > > > To: [email protected] > > > Subject: RE: Recreating SystemVM's > > > > > > Thank you all for those responses. > > > > > > I'll comb through my management-server.log and post a pastebin if > > > I'm scratching my head. > > > > > > Jeremy > > > > > > -----Original Message----- > > > From: Rajani Karuturi [mailto:[email protected]] > > > Sent: Tuesday, June 6, 2017 6:53 AM > > > To: [email protected] > > > Subject: Re: Recreating SystemVM's > > > > > > If the zone is enabled, cloudstack should recreate them automatically. > > > > > > ~ Rajani > > > > > > http://cloudplatform.accelerite.com/ > > > > > > On June 6, 2017 at 11:37 AM, Erik Weber ([email protected]) > > > wrote: > > > > > > CloudStack should recreate automatically, check the mgmt server > > > logs for hints of why it doesn't happen. > > > > > > -- > > > Erik > > > > > > tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson > > > <[email protected]>: > > > > > > I had an issue Sunday morning with cloudstack 4.9.0 and xenserver > 6.5.0. > > > My hosts stop sending LACP PDU's and caused a network drop to > > > iSCSI primary storage. > > > > > > So all my instances recovered via HA enabled. > > > > > > But my console proxy and secondary storage system VM's got stuck > > > in a boot state that would not power on. > > > > > > At this time they are expunged and gone. > > > > > > How do I tell cloudstack-management to recreate system VM's? > > > > > > I'm drawing a blank since deploying CS two years ago and just > > > keeping things running and adding hosts and more storage > > > everything has been so stable. > > > > > > Jeremy > > > > > >
