Help! I put the cluster into global maintenance, then powered off and then on all of the nodes I have powered off and powered on all the nodes. I have taken it out of global maintenance. No VM has started, including the hosted engine. This is very bad. I am going to look through logs to see why nothing has started. Help greatly appreciated.
Thanks, Cam On Fri, Jun 30, 2017 at 1:00 PM, cmc <[email protected]> wrote: > So I can run from any node: hosted-engine --set-maintenance > --mode=global. By 'agents', you mean the ovirt-ha-agent, right? This > shouldn't affect the running of any VMs, correct? Sorry for the > questions, just want to do it correctly and not make assumptions :) > > Cheers, > > C > > On Fri, Jun 30, 2017 at 12:12 PM, Martin Sivak <[email protected]> wrote: >> Hi, >> >>> Just to clarify: you mean the host_id in >>> /etc/ovirt-hosted-engine/hosted-engine.conf should match the spm_id, >>> correct? >> >> Exactly. >> >> Put the cluster to global maintenance first. Or kill all agents (has >> the same effect). >> >> Martin >> >> On Fri, Jun 30, 2017 at 12:47 PM, cmc <[email protected]> wrote: >>> Just to clarify: you mean the host_id in >>> /etc/ovirt-hosted-engine/hosted-engine.conf should match the spm_id, >>> correct? >>> >>> On Fri, Jun 30, 2017 at 9:47 AM, Martin Sivak <[email protected]> wrote: >>>> Hi, >>>> >>>> cleaning metadata won't help in this case. Try transferring the >>>> spm_ids you got from the engine to the proper hosted engine hosts so >>>> the hosted engine ids match the spm_ids. Then restart all hosted >>>> engine services. I would actually recommend restarting all hosts after >>>> this change, but I have no idea how many VMs you have running. >>>> >>>> Martin >>>> >>>> On Thu, Jun 29, 2017 at 8:27 PM, cmc <[email protected]> wrote: >>>>> Tried running a 'hosted-engine --clean-metadata" as per >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1350539, since >>>>> ovirt-ha-agent was not running anyway, but it fails with the following >>>>> error: >>>>> >>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Failed >>>>> to start monitoring domain >>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>>>> during domain acquisition >>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Traceback (most recent >>>>> call last): >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >>>>> line 191, in _run_agent >>>>> return action(he) >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >>>>> line 67, in action_clean >>>>> return he.clean(options.force_cleanup) >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>> line 345, in clean >>>>> self._initialize_domain_monitor() >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>> line 823, in _initialize_domain_monitor >>>>> raise Exception(msg) >>>>> Exception: Failed to start monitoring domain >>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>>>> during domain acquisition >>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Trying to restart agent >>>>> WARNING:ovirt_hosted_engine_ha.agent.agent.Agent:Restarting agent, >>>>> attempt '0' >>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Too many errors >>>>> occurred, giving up. Please review the log and consider filing a bug. >>>>> INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down >>>>> >>>>> On Thu, Jun 29, 2017 at 6:10 PM, cmc <[email protected]> wrote: >>>>>> Actually, it looks like sanlock problems: >>>>>> >>>>>> "SanlockInitializationError: Failed to initialize sanlock, the >>>>>> number of errors has exceeded the limit" >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 29, 2017 at 5:10 PM, cmc <[email protected]> wrote: >>>>>>> Sorry, I am mistaken, two hosts failed for the agent with the following >>>>>>> error: >>>>>>> >>>>>>> ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine >>>>>>> ERROR Failed to start monitoring domain >>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): timeout >>>>>>> during domain acquisition >>>>>>> ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine >>>>>>> ERROR Shutting down the agent because of 3 failures in a row! >>>>>>> >>>>>>> What could cause these timeouts? Some other service not running? >>>>>>> >>>>>>> On Thu, Jun 29, 2017 at 5:03 PM, cmc <[email protected]> wrote: >>>>>>>> Both services are up on all three hosts. The broke logs just report: >>>>>>>> >>>>>>>> Thread-6549::INFO::2017-06-29 >>>>>>>> 17:01:51,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup) >>>>>>>> Connection established >>>>>>>> Thread-6549::INFO::2017-06-29 >>>>>>>> 17:01:51,483::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) >>>>>>>> Connection closed >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Cam >>>>>>>> >>>>>>>> On Thu, Jun 29, 2017 at 4:00 PM, Martin Sivak <[email protected]> >>>>>>>> wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> please make sure that both ovirt-ha-agent and ovirt-ha-broker services >>>>>>>>> are restarted and up. The error says the agent can't talk to the >>>>>>>>> broker. Is there anything in the broker.log? >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> >>>>>>>>> Martin Sivak >>>>>>>>> >>>>>>>>> On Thu, Jun 29, 2017 at 4:42 PM, cmc <[email protected]> wrote: >>>>>>>>>> I've restarted those two services across all hosts, have taken the >>>>>>>>>> Hosted Engine host out of maintenance, and when I try to migrate the >>>>>>>>>> Hosted Engine over to another host, it reports that all three hosts >>>>>>>>>> 'did not satisfy internal filter HA because it is not a Hosted Engine >>>>>>>>>> host'. >>>>>>>>>> >>>>>>>>>> On the host that the Hosted Engine is currently on it reports in the >>>>>>>>>> agent.log: >>>>>>>>>> >>>>>>>>>> ovirt-ha-agent ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR >>>>>>>>>> Connection closed: Connection closed >>>>>>>>>> Jun 29 15:22:25 kvm-ldn-03 ovirt-ha-agent[12653]: ovirt-ha-agent >>>>>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Exception >>>>>>>>>> getting service path: Connection closed >>>>>>>>>> Jun 29 15:22:25 kvm-ldn-03 ovirt-ha-agent[12653]: ovirt-ha-agent >>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent >>>>>>>>>> call last): >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >>>>>>>>>> line 191, in _run_agent >>>>>>>>>> return >>>>>>>>>> action(he) >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", >>>>>>>>>> line 64, in action_proper >>>>>>>>>> return >>>>>>>>>> he.start_monitoring() >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>> line 411, in start_monitoring >>>>>>>>>> >>>>>>>>>> self._initialize_sanlock() >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>> line 691, in _initialize_sanlock >>>>>>>>>> >>>>>>>>>> constants.SERVICE_TYPE + constants.LOCKSPACE_EXTENSION) >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >>>>>>>>>> line 162, in get_service_path >>>>>>>>>> >>>>>>>>>> .format(str(e))) >>>>>>>>>> RequestError: >>>>>>>>>> Failed >>>>>>>>>> to get service path: Connection closed >>>>>>>>>> Jun 29 15:22:25 kvm-ldn-03 ovirt-ha-agent[12653]: ovirt-ha-agent >>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart >>>>>>>>>> agent >>>>>>>>>> >>>>>>>>>> On Thu, Jun 29, 2017 at 1:25 PM, Martin Sivak <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> yep, you have to restart the ovirt-ha-agent and ovirt-ha-broker >>>>>>>>>>> services. >>>>>>>>>>> >>>>>>>>>>> The scheduling message just means that the host has score 0 or is >>>>>>>>>>> not >>>>>>>>>>> reporting score at all. >>>>>>>>>>> >>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 29, 2017 at 1:33 PM, cmc <[email protected]> wrote: >>>>>>>>>>>> Thanks Martin, do I have to restart anything? When I try to use the >>>>>>>>>>>> 'migrate' operation, it complains that the other two hosts 'did not >>>>>>>>>>>> satisfy internal filter HA because it is not a Hosted Engine >>>>>>>>>>>> host..' >>>>>>>>>>>> (even though I reinstalled both these hosts with the 'deploy hosted >>>>>>>>>>>> engine' option, which suggests that something needs restarting. >>>>>>>>>>>> Should >>>>>>>>>>>> I worry about the sanlock errors, or will that be resolved by the >>>>>>>>>>>> change in host_id? >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> >>>>>>>>>>>> Cam >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:22 PM, Martin Sivak <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> Change the ids so they are distinct. I need to check if there is >>>>>>>>>>>>> a way >>>>>>>>>>>>> to read the SPM ids from the engine as using the same numbers >>>>>>>>>>>>> would be >>>>>>>>>>>>> the best. >>>>>>>>>>>>> >>>>>>>>>>>>> Martin >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:46 PM, cmc <[email protected]> wrote: >>>>>>>>>>>>>> Is there any way of recovering from this situation? I'd prefer >>>>>>>>>>>>>> to fix >>>>>>>>>>>>>> the issue rather than re-deploy, but if there is no recovery >>>>>>>>>>>>>> path, I >>>>>>>>>>>>>> could perhaps try re-deploying the hosted engine. In which case, >>>>>>>>>>>>>> would >>>>>>>>>>>>>> the best option be to take a backup of the Hosted Engine, and >>>>>>>>>>>>>> then >>>>>>>>>>>>>> shut it down, re-initialise the SAN partition (or use another >>>>>>>>>>>>>> partition) and retry the deployment? Would it be better to use >>>>>>>>>>>>>> the >>>>>>>>>>>>>> older backup from the bare metal engine that I originally used, >>>>>>>>>>>>>> or use >>>>>>>>>>>>>> a backup from the Hosted Engine? I'm not sure if any VMs have >>>>>>>>>>>>>> been >>>>>>>>>>>>>> added since switching to Hosted Engine. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unfortunately I have very little time left to get this working >>>>>>>>>>>>>> before >>>>>>>>>>>>>> I have to hand it over for eval (by end of Friday). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are some log snippets from the cluster that are current >>>>>>>>>>>>>> >>>>>>>>>>>>>> In /var/log/vdsm/vdsm.log on the host that has the Hosted Engine: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-06-29 10:50:15,071+0100 INFO (monitor/207221b) >>>>>>>>>>>>>> [storage.SANLock] >>>>>>>>>>>>>> Acquiring host id for domain >>>>>>>>>>>>>> 207221b2-959b-426b-b945-18e1adfed62f (id: >>>>>>>>>>>>>> 3) (clusterlock:282) >>>>>>>>>>>>>> 2017-06-29 10:50:15,072+0100 ERROR (monitor/207221b) >>>>>>>>>>>>>> [storage.Monitor] >>>>>>>>>>>>>> Error acquiring host id 3 for domain >>>>>>>>>>>>>> 207221b2-959b-426b-b945-18e1adfed62f (monitor:558) >>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>> File "/usr/share/vdsm/storage/monitor.py", line 555, in >>>>>>>>>>>>>> _acquireHostId >>>>>>>>>>>>>> self.domain.acquireHostId(self.hostId, async=True) >>>>>>>>>>>>>> File "/usr/share/vdsm/storage/sd.py", line 790, in >>>>>>>>>>>>>> acquireHostId >>>>>>>>>>>>>> self._manifest.acquireHostId(hostId, async) >>>>>>>>>>>>>> File "/usr/share/vdsm/storage/sd.py", line 449, in >>>>>>>>>>>>>> acquireHostId >>>>>>>>>>>>>> self._domainLock.acquireHostId(hostId, async) >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", >>>>>>>>>>>>>> line 297, in acquireHostId >>>>>>>>>>>>>> raise se.AcquireHostIdFailure(self._sdUUID, e) >>>>>>>>>>>>>> AcquireHostIdFailure: Cannot acquire host id: >>>>>>>>>>>>>> ('207221b2-959b-426b-b945-18e1adfed62f', SanlockException(22, >>>>>>>>>>>>>> 'Sanlock >>>>>>>>>>>>>> lockspace add failure', 'Invalid argument')) >>>>>>>>>>>>>> >>>>>>>>>>>>>> From /var/log/ovirt-hosted-engine-ha/agent.log on the same host: >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>>>> 13:30:50,592::hosted_engine::822::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) >>>>>>>>>>>>>> Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::WARNING::2017-06-19 >>>>>>>>>>>>>> 13:30:50,593::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Error while monitoring engine: Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::WARNING::2017-06-19 >>>>>>>>>>>>>> 13:30:50,593::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Unexpected error >>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>>>>>> line 443, in start_monitoring >>>>>>>>>>>>>> self._initialize_domain_monitor() >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>>>>>> line 823, in _initialize_domain_monitor >>>>>>>>>>>>>> raise Exception(msg) >>>>>>>>>>>>>> Exception: Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>>>> 13:30:50,593::hosted_engine::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Shutting down the agent because of 3 failures in a row! >>>>>>>>>>>>>> >>>>>>>>>>>>>> From sanlock.log: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-06-29 11:17:06+0100 1194149 [2530]: add_lockspace >>>>>>>>>>>>>> 207221b2-959b-426b-b945-18e1adfed62f:3:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>>>>>>>>>>>>> conflicts with name of list1 s5 >>>>>>>>>>>>>> 207221b2-959b-426b-b945-18e1adfed62f:1:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> From the two other hosts: >>>>>>>>>>>>>> >>>>>>>>>>>>>> host 2: >>>>>>>>>>>>>> >>>>>>>>>>>>>> vdsm.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-06-29 10:53:47,755+0100 ERROR (jsonrpc/4) >>>>>>>>>>>>>> [jsonrpc.JsonRpcServer] >>>>>>>>>>>>>> Internal server error (__init__:570) >>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", >>>>>>>>>>>>>> line >>>>>>>>>>>>>> 565, in _handle_request >>>>>>>>>>>>>> res = method(**params) >>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", >>>>>>>>>>>>>> line >>>>>>>>>>>>>> 202, in _dynamicMethod >>>>>>>>>>>>>> result = fn(*methodArgs) >>>>>>>>>>>>>> File "/usr/share/vdsm/API.py", line 1454, in >>>>>>>>>>>>>> getAllVmIoTunePolicies >>>>>>>>>>>>>> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() >>>>>>>>>>>>>> File "/usr/share/vdsm/clientIF.py", line 448, in >>>>>>>>>>>>>> getAllVmIoTunePolicies >>>>>>>>>>>>>> 'current_values': v.getIoTune()} >>>>>>>>>>>>>> File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune >>>>>>>>>>>>>> result = self.getIoTuneResponse() >>>>>>>>>>>>>> File "/usr/share/vdsm/virt/vm.py", line 2816, in >>>>>>>>>>>>>> getIoTuneResponse >>>>>>>>>>>>>> res = self._dom.blockIoTune( >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line >>>>>>>>>>>>>> 47, in __getattr__ >>>>>>>>>>>>>> % self.vmid) >>>>>>>>>>>>>> NotConnectedError: VM u'a79e6b0e-fff4-4cba-a02c-4c00be151300' >>>>>>>>>>>>>> was not >>>>>>>>>>>>>> started yet or was shut down >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/ovirt-hosted-engine-ha/agent.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:33,636::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>>>>>>>>>>>>> Found OVF_STORE: imgUUID:222610db-7880-4f4f-8559-a3635fd73555, >>>>>>>>>>>>>> volUUID:c6e0d29b-eabf-4a09-a330-df54cfdd73f1 >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:33,926::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>> Extracting Engine VM OVF from the OVF_STORE >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:33,938::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>> OVF_STORE volume path: >>>>>>>>>>>>>> /rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/images/222610db-7880-4f4f-8559-a3635fd73555/c6e0d29b-eabf-4a09-a330-df54cfdd73f1 >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:33,967::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>>>>>>>>>>>>> Found an OVF for HE VM, trying to convert >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:33,971::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>>>>>>>>>>>>> Got vm.conf from OVF_STORE >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:36,736::states::678::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) >>>>>>>>>>>>>> Score is 0 due to unexpected vm shutdown at Thu Jun 29 10:53:59 >>>>>>>>>>>>>> 2017 >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:36,736::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Current state EngineUnexpectedlyDown (score: 0) >>>>>>>>>>>>>> MainThread::INFO::2017-06-29 >>>>>>>>>>>>>> 10:56:46,772::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>>>>>>>>>>>>> Reloading vm.conf from the shared storage domain >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/messages: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jun 29 10:53:46 kvm-ldn-02 kernel: dd: sending ioctl 80306d02 to >>>>>>>>>>>>>> a partition! >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> host 1: >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/messages also in sanlock.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jun 29 11:01:02 kvm-ldn-01 sanlock[2400]: 2017-06-29 >>>>>>>>>>>>>> 11:01:02+0100 >>>>>>>>>>>>>> 678325 [9132]: s4531 delta_acquire host_id 1 busy1 1 2 1193177 >>>>>>>>>>>>>> 3d4ec963-8486-43a2-a7d9-afa82508f89f.kvm-ldn-03 >>>>>>>>>>>>>> Jun 29 11:01:03 kvm-ldn-01 sanlock[2400]: 2017-06-29 >>>>>>>>>>>>>> 11:01:03+0100 >>>>>>>>>>>>>> 678326 [24159]: s4531 add_lockspace fail result -262 >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/ovirt-hosted-engine-ha/agent.log: >>>>>>>>>>>>>> >>>>>>>>>>>>>> MainThread::ERROR::2017-06-27 >>>>>>>>>>>>>> 15:21:01,143::hosted_engine::822::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) >>>>>>>>>>>>>> Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::WARNING::2017-06-27 >>>>>>>>>>>>>> 15:21:01,144::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Error while monitoring engine: Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::WARNING::2017-06-27 >>>>>>>>>>>>>> 15:21:01,144::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Unexpected error >>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>>>>>> line 443, in start_monitoring >>>>>>>>>>>>>> self._initialize_domain_monitor() >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", >>>>>>>>>>>>>> line 823, in _initialize_domain_monitor >>>>>>>>>>>>>> raise Exception(msg) >>>>>>>>>>>>>> Exception: Failed to start monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f, host_id=1): >>>>>>>>>>>>>> timeout >>>>>>>>>>>>>> during domain acquisition >>>>>>>>>>>>>> MainThread::ERROR::2017-06-27 >>>>>>>>>>>>>> 15:21:01,144::hosted_engine::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>>>>>>>>>>>>> Shutting down the agent because of 3 failures in a row! >>>>>>>>>>>>>> MainThread::INFO::2017-06-27 >>>>>>>>>>>>>> 15:21:06,717::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) >>>>>>>>>>>>>> VDSM domain monitor status: PENDING >>>>>>>>>>>>>> MainThread::INFO::2017-06-27 >>>>>>>>>>>>>> 15:21:09,335::hosted_engine::776::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor) >>>>>>>>>>>>>> Failed to stop monitoring domain >>>>>>>>>>>>>> (sd_uuid=207221b2-959b-426b-b945-18e1adfed62f): Storage domain is >>>>>>>>>>>>>> member of pool: u'domain=207221b2-959b-426b-b945-18e1adfed62f' >>>>>>>>>>>>>> MainThread::INFO::2017-06-27 >>>>>>>>>>>>>> 15:21:09,339::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(run) >>>>>>>>>>>>>> Agent shutting down >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for any help, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cam >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 11:25 AM, cmc <[email protected]> wrote: >>>>>>>>>>>>>>> Hi Martin, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> yes, on two of the machines they have the same host_id. The >>>>>>>>>>>>>>> other has >>>>>>>>>>>>>>> a different host_id. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To update since yesterday: I reinstalled and deployed Hosted >>>>>>>>>>>>>>> Engine on >>>>>>>>>>>>>>> the other host (so all three hosts in the cluster now have it >>>>>>>>>>>>>>> installed). The second one I deployed said it was able to host >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> engine (unlike the first I reinstalled), so I tried putting the >>>>>>>>>>>>>>> host >>>>>>>>>>>>>>> with the Hosted Engine on it into maintenance to see if it would >>>>>>>>>>>>>>> migrate over. It managed to move all hosts but the Hosted >>>>>>>>>>>>>>> Engine. And >>>>>>>>>>>>>>> now the host that said it was able to host the engine says >>>>>>>>>>>>>>> 'unavailable due to HA score'. The host that it was trying to >>>>>>>>>>>>>>> move >>>>>>>>>>>>>>> from is now in 'preparing for maintenance' for the last 12 >>>>>>>>>>>>>>> hours. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The summary is: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kvm-ldn-01 - one of the original, pre-Hosted Engine hosts, >>>>>>>>>>>>>>> reinstalled >>>>>>>>>>>>>>> with 'Deploy Hosted Engine'. No icon saying it can host the >>>>>>>>>>>>>>> Hosted >>>>>>>>>>>>>>> Hngine, host_id of '2' in >>>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf. >>>>>>>>>>>>>>> 'add_lockspace' fails in sanlock.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kvm-ldn-02 - the other host that was pre-existing before Hosted >>>>>>>>>>>>>>> Engine >>>>>>>>>>>>>>> was created. Reinstalled with 'Deploy Hosted Engine'. Had an >>>>>>>>>>>>>>> icon >>>>>>>>>>>>>>> saying that it was able to host the Hosted Engine, but after >>>>>>>>>>>>>>> migration >>>>>>>>>>>>>>> was attempted when putting kvm-ldn-03 into maintenance, it >>>>>>>>>>>>>>> reports: >>>>>>>>>>>>>>> 'unavailable due to HA score'. It has a host_id of '1' in >>>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf. No errors in >>>>>>>>>>>>>>> sanlock.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kvm-ldn-03 - this was the host I deployed Hosted Engine on, >>>>>>>>>>>>>>> which was >>>>>>>>>>>>>>> not part of the original cluster. I restored the bare-metal >>>>>>>>>>>>>>> engine >>>>>>>>>>>>>>> backup in the Hosted Engine on this host when deploying it, >>>>>>>>>>>>>>> without >>>>>>>>>>>>>>> error. It currently has the Hosted Engine on it (as the only VM >>>>>>>>>>>>>>> after >>>>>>>>>>>>>>> I put that host into maintenance to test the HA of Hosted >>>>>>>>>>>>>>> Engine). >>>>>>>>>>>>>>> Sanlock log shows conflicts >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I will look through all the logs for any other errors. Please >>>>>>>>>>>>>>> let me >>>>>>>>>>>>>>> know if you need any logs or other clarification/information. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Campbell >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 9:25 AM, Martin Sivak >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> can you please check the contents of >>>>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf or >>>>>>>>>>>>>>>> /etc/ovirt-hosted-engine-ha/agent.conf (I am not sure which >>>>>>>>>>>>>>>> one it is >>>>>>>>>>>>>>>> right now) and search for host-id? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Make sure the IDs are different. If they are not, then there >>>>>>>>>>>>>>>> is a bug somewhere. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 6:26 PM, cmc <[email protected]> wrote: >>>>>>>>>>>>>>>>> I see this on the host it is trying to migrate in >>>>>>>>>>>>>>>>> /var/log/sanlock: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2017-06-27 17:10:40+0100 527703 [2407]: s3528 lockspace >>>>>>>>>>>>>>>>> 207221b2-959b-426b-b945-18e1adfed62f:1:/dev/207221b2-959b-426b-b945-18e1adfed62f/ids:0 >>>>>>>>>>>>>>>>> 2017-06-27 17:13:00+0100 527843 [27446]: s3528 delta_acquire >>>>>>>>>>>>>>>>> host_id 1 >>>>>>>>>>>>>>>>> busy1 1 2 1042692 >>>>>>>>>>>>>>>>> 3d4ec963-8486-43a2-a7d9-afa82508f89f.kvm-ldn-03 >>>>>>>>>>>>>>>>> 2017-06-27 17:13:01+0100 527844 [2407]: s3528 add_lockspace >>>>>>>>>>>>>>>>> fail result -262 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The sanlock service is running. Why would this occur? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> C >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:21 PM, cmc <[email protected]> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> Hi Martin, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the reply. I have done this, and the deployment >>>>>>>>>>>>>>>>>> completed >>>>>>>>>>>>>>>>>> without error. However, it still will not allow the Hosted >>>>>>>>>>>>>>>>>> Engine >>>>>>>>>>>>>>>>>> migrate to another host. The >>>>>>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf got created ok >>>>>>>>>>>>>>>>>> on the host >>>>>>>>>>>>>>>>>> I re-installed, but the ovirt-ha-broker.service, though it >>>>>>>>>>>>>>>>>> starts, >>>>>>>>>>>>>>>>>> reports: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------8<------------------- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jun 27 14:58:26 kvm-ldn-01 systemd[1]: Starting oVirt Hosted >>>>>>>>>>>>>>>>>> Engine >>>>>>>>>>>>>>>>>> High Availability Communications Broker... >>>>>>>>>>>>>>>>>> Jun 27 14:58:27 kvm-ldn-01 ovirt-ha-broker[6101]: >>>>>>>>>>>>>>>>>> ovirt-ha-broker >>>>>>>>>>>>>>>>>> ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker >>>>>>>>>>>>>>>>>> ERROR >>>>>>>>>>>>>>>>>> Failed to read metadata from >>>>>>>>>>>>>>>>>> /rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/ha_agent/hosted-engine.metadata >>>>>>>>>>>>>>>>>> Traceback >>>>>>>>>>>>>>>>>> (most >>>>>>>>>>>>>>>>>> recent call last): >>>>>>>>>>>>>>>>>> File >>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", >>>>>>>>>>>>>>>>>> line 129, in get_raw_stats_for_service_type >>>>>>>>>>>>>>>>>> f = >>>>>>>>>>>>>>>>>> os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) >>>>>>>>>>>>>>>>>> OSError: >>>>>>>>>>>>>>>>>> [Errno 2] >>>>>>>>>>>>>>>>>> No such file or directory: >>>>>>>>>>>>>>>>>> '/rhev/data-center/mnt/blockSD/207221b2-959b-426b-b945-18e1adfed62f/ha_agent/hosted-engine.metadata' >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------8<------------------- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I checked the path, and it exists. I can run 'less -f' on it >>>>>>>>>>>>>>>>>> fine. The >>>>>>>>>>>>>>>>>> perms are slightly different on the host that is running the >>>>>>>>>>>>>>>>>> VM vs the >>>>>>>>>>>>>>>>>> one that is reporting errors (600 vs 660), ownership is >>>>>>>>>>>>>>>>>> vdsm:qemu. Is >>>>>>>>>>>>>>>>>> this a san locking issue? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for any help, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cam >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 1:41 PM, Martin Sivak >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>> Should it be? It was not in the instructions for the >>>>>>>>>>>>>>>>>>>> migration from >>>>>>>>>>>>>>>>>>>> bare-metal to Hosted VM >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The hosted engine will only migrate to hosts that have the >>>>>>>>>>>>>>>>>>> services >>>>>>>>>>>>>>>>>>> running. Please put one other host to maintenance and >>>>>>>>>>>>>>>>>>> select Hosted >>>>>>>>>>>>>>>>>>> engine action: DEPLOY in the reinstall dialog. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Martin Sivak >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 1:23 PM, cmc <[email protected]> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> I changed the >>>>>>>>>>>>>>>>>>>> 'os.other.devices.display.protocols.value.3.6 = >>>>>>>>>>>>>>>>>>>> spice/qxl,vnc/cirrus,vnc/qxl' line to have the same >>>>>>>>>>>>>>>>>>>> display protocols >>>>>>>>>>>>>>>>>>>> as 4 and the hosted engine now appears in the list of VMs. >>>>>>>>>>>>>>>>>>>> I am >>>>>>>>>>>>>>>>>>>> guessing the compatibility version was causing it to use >>>>>>>>>>>>>>>>>>>> the 3.6 >>>>>>>>>>>>>>>>>>>> version. However, I am still unable to migrate the engine >>>>>>>>>>>>>>>>>>>> VM to >>>>>>>>>>>>>>>>>>>> another host. When I try putting the host it is currently >>>>>>>>>>>>>>>>>>>> on into >>>>>>>>>>>>>>>>>>>> maintenance, it reports: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Error while executing action: Cannot switch the Host(s) to >>>>>>>>>>>>>>>>>>>> Maintenance mode. >>>>>>>>>>>>>>>>>>>> There are no available hosts capable of running the engine >>>>>>>>>>>>>>>>>>>> VM. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Running 'hosted-engine --vm-status' still shows 'Engine >>>>>>>>>>>>>>>>>>>> status: >>>>>>>>>>>>>>>>>>>> unknown stale-data'. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The ovirt-ha-broker service is only running on one host. >>>>>>>>>>>>>>>>>>>> It was set to >>>>>>>>>>>>>>>>>>>> 'disabled' in systemd. It won't start as there is no >>>>>>>>>>>>>>>>>>>> /etc/ovirt-hosted-engine/hosted-engine.conf on the other >>>>>>>>>>>>>>>>>>>> two hosts. >>>>>>>>>>>>>>>>>>>> Should it be? It was not in the instructions for the >>>>>>>>>>>>>>>>>>>> migration from >>>>>>>>>>>>>>>>>>>> bare-metal to Hosted VM >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Cam >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Jun 22, 2017 at 1:07 PM, cmc <[email protected]> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> Hi Tomas, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> So in my >>>>>>>>>>>>>>>>>>>>> /usr/share/ovirt-engine/conf/osinfo-defaults.properties >>>>>>>>>>>>>>>>>>>>> on my >>>>>>>>>>>>>>>>>>>>> engine VM, I have: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> os.other.devices.display.protocols.value = >>>>>>>>>>>>>>>>>>>>> spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus >>>>>>>>>>>>>>>>>>>>> os.other.devices.display.protocols.value.3.6 = >>>>>>>>>>>>>>>>>>>>> spice/qxl,vnc/cirrus,vnc/qxl >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> That seems to match - I assume since this is 4.1, the 3.6 >>>>>>>>>>>>>>>>>>>>> should not apply >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Is there somewhere else I should be looking? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Cam >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 22, 2017 at 11:40 AM, Tomas Jelinek >>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 22, 2017 at 12:38 PM, Michal Skrivanek >>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> > On 22 Jun 2017, at 12:31, Martin Sivak >>>>>>>>>>>>>>>>>>>>>>> > <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > Tomas, what fields are needed in a VM to pass the >>>>>>>>>>>>>>>>>>>>>>> > check that causes >>>>>>>>>>>>>>>>>>>>>>> > the following error? >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> >>>>> WARN >>>>>>>>>>>>>>>>>>>>>>> >>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>>>>>>>>>>>>> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation >>>>>>>>>>>>>>>>>>>>>>> >>>>> of action >>>>>>>>>>>>>>>>>>>>>>> >>>>> 'ImportVm' >>>>>>>>>>>>>>>>>>>>>>> >>>>> failed for user SYSTEM. Reasons: >>>>>>>>>>>>>>>>>>>>>>> >>>>> VAR__ACTION__IMPORT >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> to match the OS and VM Display type;-) >>>>>>>>>>>>>>>>>>>>>>> Configuration is in osinfo….e.g. if that is import from >>>>>>>>>>>>>>>>>>>>>>> older releases on >>>>>>>>>>>>>>>>>>>>>>> Linux this is typically caused by the cahgen of cirrus >>>>>>>>>>>>>>>>>>>>>>> to vga for non-SPICE >>>>>>>>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> yep, the default supported combinations for 4.0+ is this: >>>>>>>>>>>>>>>>>>>>>> os.other.devices.display.protocols.value = >>>>>>>>>>>>>>>>>>>>>> spice/qxl,vnc/vga,vnc/qxl,vnc/cirrus >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > Thanks. >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > On Thu, Jun 22, 2017 at 12:19 PM, cmc >>>>>>>>>>>>>>>>>>>>>>> > <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >> Hi Martin, >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> >>> just as a random comment, do you still have the >>>>>>>>>>>>>>>>>>>>>>> >>> database backup from >>>>>>>>>>>>>>>>>>>>>>> >>> the bare metal -> VM attempt? It might be possible >>>>>>>>>>>>>>>>>>>>>>> >>> to just try again >>>>>>>>>>>>>>>>>>>>>>> >>> using it. Or in the worst case.. update the >>>>>>>>>>>>>>>>>>>>>>> >>> offending value there >>>>>>>>>>>>>>>>>>>>>>> >>> before restoring it to the new engine instance. >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> I still have the backup. I'd rather do the latter, >>>>>>>>>>>>>>>>>>>>>>> >> as re-running the >>>>>>>>>>>>>>>>>>>>>>> >> HE deployment is quite lengthy and involved (I have >>>>>>>>>>>>>>>>>>>>>>> >> to re-initialise >>>>>>>>>>>>>>>>>>>>>>> >> the FC storage each time). Do you know what the >>>>>>>>>>>>>>>>>>>>>>> >> offending value(s) >>>>>>>>>>>>>>>>>>>>>>> >> would be? Would it be in the Postgres DB or in a >>>>>>>>>>>>>>>>>>>>>>> >> config file >>>>>>>>>>>>>>>>>>>>>>> >> somewhere? >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> Cheers, >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> Cam >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >>> Regards >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> >>> Martin Sivak >>>>>>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>>>>>> >>> On Thu, Jun 22, 2017 at 11:39 AM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Yanir, >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks for the reply. >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> First of all, maybe a chain reaction of : >>>>>>>>>>>>>>>>>>>>>>> >>>>> WARN >>>>>>>>>>>>>>>>>>>>>>> >>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>>>>>>>>>>>>> >>>>> (org.ovirt.thread.pool-6-thread-23) [] Validation >>>>>>>>>>>>>>>>>>>>>>> >>>>> of action >>>>>>>>>>>>>>>>>>>>>>> >>>>> 'ImportVm' >>>>>>>>>>>>>>>>>>>>>>> >>>>> failed for user SYSTEM. Reasons: >>>>>>>>>>>>>>>>>>>>>>> >>>>> VAR__ACTION__IMPORT >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> ,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>>>>>>>>>>>>> >>>>> is causing the hosted engine vm not to be set up >>>>>>>>>>>>>>>>>>>>>>> >>>>> correctly and >>>>>>>>>>>>>>>>>>>>>>> >>>>> further >>>>>>>>>>>>>>>>>>>>>>> >>>>> actions were made when the hosted engine vm wasnt >>>>>>>>>>>>>>>>>>>>>>> >>>>> in a stable state. >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> As for now, are you trying to revert back to a >>>>>>>>>>>>>>>>>>>>>>> >>>>> previous/initial >>>>>>>>>>>>>>>>>>>>>>> >>>>> state ? >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>> I'm not trying to revert it to a previous state >>>>>>>>>>>>>>>>>>>>>>> >>>> for now. This was a >>>>>>>>>>>>>>>>>>>>>>> >>>> migration from a bare metal engine, and it didn't >>>>>>>>>>>>>>>>>>>>>>> >>>> report any error >>>>>>>>>>>>>>>>>>>>>>> >>>> during the migration. I'd had some problems on my >>>>>>>>>>>>>>>>>>>>>>> >>>> first attempts at >>>>>>>>>>>>>>>>>>>>>>> >>>> this migration, whereby it never completed (due to >>>>>>>>>>>>>>>>>>>>>>> >>>> a proxy issue) but >>>>>>>>>>>>>>>>>>>>>>> >>>> I managed to resolve this. Do you know of a way to >>>>>>>>>>>>>>>>>>>>>>> >>>> get the Hosted >>>>>>>>>>>>>>>>>>>>>>> >>>> Engine VM into a stable state, without rebuilding >>>>>>>>>>>>>>>>>>>>>>> >>>> the entire cluster >>>>>>>>>>>>>>>>>>>>>>> >>>> from scratch (since I have a lot of VMs on it)? >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks for any help. >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>> Regards, >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>> Cam >>>>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>> >>>>> Yanir >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> On Wed, Jun 21, 2017 at 4:32 PM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi Jenny/Martin, >>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>> Any idea what I can do here? The hosted engine >>>>>>>>>>>>>>>>>>>>>>> >>>>>> VM has no log on any >>>>>>>>>>>>>>>>>>>>>>> >>>>>> host in /var/log/libvirt/qemu, and I fear that >>>>>>>>>>>>>>>>>>>>>>> >>>>>> if I need to put the >>>>>>>>>>>>>>>>>>>>>>> >>>>>> host into maintenance, e.g., to upgrade it that >>>>>>>>>>>>>>>>>>>>>>> >>>>>> I created it on >>>>>>>>>>>>>>>>>>>>>>> >>>>>> (which >>>>>>>>>>>>>>>>>>>>>>> >>>>>> I think is hosting it), or if it fails for any >>>>>>>>>>>>>>>>>>>>>>> >>>>>> reason, it won't get >>>>>>>>>>>>>>>>>>>>>>> >>>>>> migrated to another host, and I will not be able >>>>>>>>>>>>>>>>>>>>>>> >>>>>> to manage the >>>>>>>>>>>>>>>>>>>>>>> >>>>>> cluster. It seems to be a very dangerous >>>>>>>>>>>>>>>>>>>>>>> >>>>>> position to be in. >>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>> Cam >>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Wed, Jun 21, 2017 at 11:48 AM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks Martin. The hosts are all part of the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> same cluster. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I get these errors in the engine.log on the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> engine: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z WARN >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Validation of action >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 'ImportVm' >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> failed for user SYST >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> EM. Reasons: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_VM_DISPLAY_TYPE_IS_NOT_SUPPORTED_BY_OS >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z INFO >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Lock >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> freed to object >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 'EngineLock:{exclusiveLocks='[a >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 79e6b0e-fff4-4cba-a02c-4c00be151300=<VM, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> HostedEngine>, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> HostedEngine=<VM_NAME, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> ACTION_TYPE_FAILED_NAME_ALREADY_USED>]', >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> sharedLocks= >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> '[a79e6b0e-fff4-4cba-a02c-4c00be151300=<REMOTE_VM, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> ACTION_TYPE_FAILED_VM_IS_BEING_IMPORTED$VmName >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> HostedEngine>]'}' >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 2017-06-19 03:28:05,030Z ERROR >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> [org.ovirt.engine.core.bll.HostedEngineImporter] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> (org.ovirt.thread.pool-6-thread-23) [] Failed >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> importing the Hosted >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Engine VM >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> The sanlock.log reports conflicts on that same >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> host, and a >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> different >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> error on the other hosts, not sure if they are >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> related. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And this in the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> /var/log/ovirt-hosted-engine-ha/agent log on the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> host >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> which I deployed the hosted engine VM on: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 13:09:49,743::ovf_store::124::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Unable to extract HEVM OVF >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> MainThread::ERROR::2017-06-19 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> 13:09:49,743::config::445::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Failed extracting VM OVF from the OVF_STORE >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> volume, falling back >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> initial vm.conf >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I've seen some of these issues reported in >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> bugzilla, but they were >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> older versions of oVirt (and appear to be >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> resolved). >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> I will install that package on the other two >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> hosts, for which I >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> will >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> put them in maintenance as vdsm is installed as >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> an upgrade. I >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> guess >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> restarting vdsm is a good idea after that? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Campbell >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, Jun 21, 2017 at 10:51 AM, Martin Sivak >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> you do not have to install it on all hosts. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> But you should have >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> than one and ideally all hosted engine enabled >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> nodes should >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> belong to >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> the same engine cluster. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Best regards >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Martin Sivak >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Jun 21, 2017 at 11:29 AM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Hi Jenny, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Does ovirt-hosted-engine-ha need to be >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> installed across all >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> hosts? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Could that be the reason it is failing to see >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> it properly? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Cam >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Mon, Jun 19, 2017 at 1:27 PM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Hi Jenny, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Logs are attached. I can see errors in >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> there, but am unsure how >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> they >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> arose. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Campbell >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Mon, Jun 19, 2017 at 12:29 PM, Evgenia >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Tokar >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> From the output it looks like the agent is >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> down, try starting >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> it by >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> running: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> systemctl start ovirt-ha-agent. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> The engine is supposed to see the hosted >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> engine storage domain >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> import it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> to the system, then it should import the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> hosted engine vm. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Can you attach the agent log from the host >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (/var/log/ovirt-hosted-engine-ha/agent.log) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> and the engine log from the engine vm >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (/var/log/ovirt-engine/engine.log)? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Jenny >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Mon, Jun 19, 2017 at 12:41 PM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Jenny, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> What version are you running? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> 4.1.2.2-1.el7.centos >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> For the hosted engine vm to be imported >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> and displayed in the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> engine, you >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> must first create a master storage domain. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> To provide a bit more detail: this was a >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> migration of a >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> bare-metal >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> engine in an existing cluster to a hosted >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> engine VM for that >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> cluster. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> As part of this migration, I built an >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> entirely new host and >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> ran >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> 'hosted-engine --deploy' (followed these >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> instructions: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> http://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_Metal_to_an_EL-Based_Self-Hosted_Environment/). >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I restored the backup from the engine and >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> it completed >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> without any >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> errors. I didn't see any instructions >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> regarding a master >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> storage >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> domain in the page above. The cluster has >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> two existing master >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> storage >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> domains, one is fibre channel, which is >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> up, and one ISO >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> domain, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> is currently offline. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> What do you mean the hosted engine >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> commands are failing? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> What >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> happens >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> you run hosted-engine --vm-status now? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Interestingly, whereas when I ran it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> before, it exited with >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> output >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> and a return code of '1', it now reports: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> --== Host 1 status ==-- >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> conf_on_shared_storage : True >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Status up-to-date : False >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hostname : >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> kvm-ldn-03.ldn.fscfc.co.uk >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Host ID : 1 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Engine status : >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> unknown stale-data >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Score : 0 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> stopped : True >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Local maintenance : False >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> crc32 : >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> 0217f07b >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> local_conf_timestamp : 2911 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Host timestamp : 2897 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Extra metadata (valid at timestamp): >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> metadata_parse_version=1 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> metadata_feature_version=1 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> timestamp=2897 (Thu Jun 15 16:22:54 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> 2017) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> host-id=1 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> score=0 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> vm_conf_refresh_time=2911 (Thu Jun >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> 15 16:23:08 2017) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> conf_on_shared_storage=True >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> maintenance=False >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> state=AgentStopped >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> stopped=True >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Yet I can login to the web GUI fine. I >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> guess it is not HA due >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> being >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> in an unknown state currently? Does the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> hosted-engine-ha rpm >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> need >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> be installed across all nodes in the >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> cluster, btw? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks for the help, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Cam >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jenny Tokar >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jun 15, 2017 at 6:32 PM, cmc >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I've migrated from a bare-metal engine >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> to a hosted engine. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> There >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> were >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> no errors during the install, however, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> the hosted engine >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> did not >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> started. I tried running: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> hosted-engine --status >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> on the host I deployed it on, and it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> returns nothing (exit >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> code >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> is 1 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> however). I could not ping it either. So >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I tried starting >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> it via >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> 'hosted-engine --vm-start' and it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> returned: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Virtual machine does not exist >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> But it then became available. I logged >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> into it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> successfully. It >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> is not >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> in the list of VMs however. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas why the hosted-engine commands >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> fail, and why it >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> is not >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> the list of virtual machines? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for any help, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Cam >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>>>>>> >>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> >>>>>> Users mailing list >>>>>>>>>>>>>>>>>>>>>>> >>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>> >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> > _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> > Users mailing list >>>>>>>>>>>>>>>>>>>>>>> > [email protected] >>>>>>>>>>>>>>>>>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
