[
https://issues.apache.org/jira/browse/CLOUDSTACK-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242839#comment-15242839
]
Abhinandan Prateek commented on CLOUDSTACK-9350:
------------------------------------------------
The Marvin test cases cover testing host HA with nfs and secondary storage, nfs
only and local storage only zones.
This PR should be specifically tested with KVM zone. The test are generic and
will run on Xen setup too.
> Local storage hosts get HA tasks, cause issues
> -----------------------------------------------
>
> Key: CLOUDSTACK-9350
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9350
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Affects Versions: 4.5.1
> Reporter: Abhinandan Prateek
> Assignee: Abhinandan Prateek
>
> When a host hits its ping time out, for whatever reason, the investigators
> are triggered. The KVMInvestigator sends a CheckOnHostCommand to the target
> host, and then to all the remaining neighbor hosts in the cluster. The
> CheckOnHostCommand (and also FenceCommand, the code is nearly identical) is
> processed by the KVM agent and simply scans through all NFS primary storage
> looking for the host's heartbeat in the KVMHA directory. If no heartbeat file
> is found, it fails the check. In the case of clusters that are local-only,
> these hosts will always fail the check, whether it be the target host or a
> neighbor checking on the target. This triggers a host 'down' event, which
> triggers HA tasks. The HA tasks will attempt to stop any VMs on the host, and
> then if the VM's offering is HA-enabled it will try to restart the VM.
> Our recent issue was that a management server took extraordinarily long to
> rotate its logs and was slow to process some host pings. The
> CheckOnHostCommand was sent to a suspect host, which failed because it had no
> primary NFS. The neighbor checks also failed to check the suspect host's
> heartbeat for the same reason. Then the host was marked as down and all VMs
> were stopped. Multiply this by a few dozen hosts.
> The immediate fix, provided in the example, is a patch to KVMInvestigator
> which will only attempt investigation if the host's cluster has NFS storage,
> which is a requirement for the host to run the check, as described above. If
> there is none, the host state is determined to be disconnected rather than
> down. This means that the host will still end up in alert state and need
> manual investigation, but there will be no attempt to stop or HA the VMs.
> Additionally, the patch catches scenarios where a cluster might have both NFS
> and local storage and a host ends up in 'down' state. In this case, when the
> HA tasks are being created, if a VM is using local storage then the HA task
> generation is skipped. This VM can't be started anywhere else.
> We could also make the agent side more robust, in KVMHAChecker we may not
> want it to return 'false' if there were zero pools passed to check for HA
> heartbeat. Then again, maybe we do. We decided initially to patch just the
> server side, because it is easier to deploy.
> In the long run, I'd hope that the current HA work would supercede the
> current KVMInvestigator and take the cluster's ability to pass any defined
> checks into account before checking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
