jmsperu commented on code in PR #13090: URL: https://github.com/apache/cloudstack/pull/13090#discussion_r4333712006
########## scripts/vm/hypervisor/kvm/kvmha-fence.sh: ########## @@ -0,0 +1,71 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Shared fence-action helper for kvmheartbeat.sh, kvmspheartbeat.sh and kvmsmpheartbeat.sh. +# Sourced by those scripts; do not invoke directly. +# +# kvm.heartbeat.fence.action in agent.properties selects the action: +# hard-reboot (default), graceful-reboot, restart-agent, or log-only (only writes to +# syslog and takes no action). Any other value falls back to hard-reboot. +# +# Usage from caller: +# source "$(dirname "$0")/kvmha-fence.sh" +# fence_action "kvmheartbeat.sh" # script name passed for log tagging + +AGENT_PROPS="${AGENT_PROPS:-/etc/cloudstack/agent/agent.properties}" + +fence_action() { + local source_script="${1:-kvmha}" + local FENCE_ACTION="hard-reboot" + + if [ -r "$AGENT_PROPS" ]; then + local val + val=$(grep "^kvm.heartbeat.fence.action=" "$AGENT_PROPS" | tail -n 1 | cut -d= -f2- | tr -d '[:space:]') + [ -n "$val" ] && FENCE_ACTION="$val" + fi + + case "$FENCE_ACTION" in + log-only) Review Comment: Right, that was a real gap: once the heartbeat is stale the management server can declare the host Down and HA can start the same VMs elsewhere while they are still running here. Fixed in 0b61bd3b2c, safe by default: - `restart-agent` and `log-only` now stop every VM on the host (`virsh destroy`) before they act, so the host stays up but nothing keeps writing to the disks. If libvirt cannot be queried or any VM is still running afterwards, the fence falls back to `hard-reboot`. - An operator who really wants the VMs left running has to say so with a new agent property, `kvm.heartbeat.fence.keep.vms.running=true` (default false). The fence then logs a WARNING, and agent.properties / AgentProperties spell out that this is only safe when no VM on the host can be restarted elsewhere by HA. - `hard-reboot` and `graceful-reboot` are unchanged. New `KvmHaFenceScriptTest` runs the real script with stand-ins for virsh, systemctl, logger and the sysrq trigger (paths overridable through the environment, like `AGENT_PROPS`) and checks: VMs stopped before the agent restart, log-only stops VMs without rebooting, a VM that cannot be stopped forces hard-reboot, keep.vms.running is honoured with a warning, and default / unknown values still hard-reboot. Agent module 65/65 green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
