Ah OK, my ignorance strikes again... I flushed all the rules on IPtables on
hosts -01 and -02, and now Gluster seems up and happy...
I saw messages after I flushed IPtables that the engine state was changing, and
eventually it landed on status: ReinitializeFSM-LocalMaintenance
...but I could not log into the admin website.
I then queried the engine VM state at the CLI, and saw it was thus:
[root@ovirt-node-01 ~]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True
Hostname : ovirt-node-01
Host ID : 1
Engine status : {"reason": "bad vm status", "health":
"bad", "vm": "up", "detail": "paused"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 538868a0
Host timestamp : 214954
--== Host 2 status ==--
Status up-to-date : True
Hostname : ovirt-node-02
Host ID : 2
Engine status : {"reason": "vm not running on this host",
"health": "bad", "vm": "down", "detail": "unknown"}
Score : 0
stopped : False
Local maintenance : True
crc32 : 419a0c6d
Host timestamp : 53528
So, I issued the command “hosted-engine --vm-shutdown” on host -01, and it
eventually came down (had to be force-killed per status emails) and then HA
restarted it :)
Looks like I’m back to being good now... Thanks everyone for the assist, and
talk to you soon, I’m sure ;)
-Will
From: Simone Tiraboschi [mailto:[email protected]]
Sent: Tuesday, December 22, 2015 10:23 AM
To: Will Dennis; users
Cc: Sahina Bose; Dan Kenigsberg
Subject: Re: [ovirt-users] Hosted Engine crash - state =
EngineUp-EngineUpBadHealth
hosted-engine-setup asks:
iptables was detected on your computer, do you wish setup to
configure it? (Yes, No)[Yes]:
You have just to say no here.
If you say no it's completely up to you to configure it opening the required
ports or everything disabling it if you don't care.
The issue with gluster ports is that hosted-engine-setup simply configure
iptables for what it knows you'll need and on 3.6 it's always assuming that the
gluster volume is served by external hosts.
From: Sahina Bose [mailto:[email protected]<mailto:[email protected]>]
Sent: Tuesday, December 22, 2015 9:19 AM
To: Will Dennis; Simone Tiraboschi; Dan Kenigsberg
Subject: Re: [ovirt-users] Hosted Engine crash - state =
EngineUp-EngineUpBadHealth
On 12/22/2015 07:47 PM, Sahina Bose wrote:
On 12/22/2015 07:28 PM, Will Dennis wrote:
See attached for requested log files
From gluster logs
[2015-12-22 00:40:53.501341] W [MSGID: 108001] [afr-common.c:3924:afr_notify]
0-engine-replicate-1: Client-quorum is not met
[2015-12-22 00:40:53.502288] W [socket.c:588:__socket_rwv] 0-engine-client-2:
readv on 138.15.200.93:49217<http://138.15.200.93:49217> failed (No data
available)
[2015-12-22 00:41:17.667302] W [fuse-bridge.c:2292:fuse_writev_cbk]
0-glusterfs-fuse: 3875597: WRITE => -1 (Read-only file system)
Could you check if the gluster ports are open on all nodes?
It's possible you ran into this ? -
https://bugzilla.redhat.com/show_bug.cgi?id=1288979
From: Sahina Bose [mailto:[email protected]]
Sent: Tuesday, December 22, 2015 4:59 AM
To: Simone Tiraboschi; Will Dennis; Dan Kenigsberg
Cc: users
Subject: Re: [ovirt-users] Hosted Engine crash - state =
EngineUp-EngineUpBadHealth
On 12/22/2015 02:38 PM, Simone Tiraboschi wrote:
On Tue, Dec 22, 2015 at 2:31 AM, Will Dennis
<[email protected]<mailto:[email protected]>> wrote:
OK, another problem :(
I was having the same problem with my second oVirt host that I had with my
first one, where when I ran “hosted-engine —deploy” on it, after it completed
successfully, then I was experiencing a ~50sec lag when SSH’ing into the node…
vpnp71:~ will$ time ssh root@ovirt-node-02 uptime
19:36:06 up 4 days, 8:31, 0 users, load average: 0.68, 0.70, 0.67
real 0m50.540s
user 0m0.025s
sys 0m0.008s
So, in the oVirt web admin console, I put the "ovirt-node-02” node into
Maintenance mode, then SSH’d to the server and rebooted it. Sure enough, after
the server came back up, SSH was fine (no delay), which again was the same
experience I had had with the first oVirt host. So, I went back to the web
console, and choose the “Confirm host has been rebooted” option, which I
thought would be the right action to take after a reboot. The system opened a
dialog box with a spinner, which never stopped spinning… So finally, I closed
the dialog box with the upper right (X) symbol, and then for this same host
choose “Activate” from the menu. It was then I noticed I had recieved a state
transition email notifying me that "EngineUp-EngineUpBadHealth” and sure
enough, the web UI was then unresponsive. I checked on the first oVirt host,
the VM with the name “HostedEngine” is still running, but obviously isn’t
working…
So, looks like I need to restart the HostedEngine VM or take whatever action is
needed to return oVirt to operation… Hate to keep asking this question, but
what’s the correct action at this point?
ovirt-ha-agent should always restart it for you after a few minutes but the
point is that the network configuration seams to be not that stable.
I know from another thread that you are trying to deploy hosted-engine over
GlusterFS in an hyperconverged way and this, as I said, is currently not
supported.
I think that it can also requires some specific configuration on network side.
For hyperconverged gluster+engine , it should work without any specific
configuration on network side. However if the network is flaky, it is possible
that there are errors with gluster volume access. Could you provide the
ovirt-ha-agent logs as well as gluster mount logs?
Adding Sahina and Dan here.
Thanks, again,
Will
_______________________________________________
Users mailing list
[email protected]<mailto:[email protected]>
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
