abh1sar commented on code in PR #14194:
URL: https://github.com/apache/cloudstack/pull/14194#discussion_r4309884879
##########
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java:
##########
@@ -636,6 +636,42 @@ public class AgentProperties{
*/
public static final Property<String> KVM_SCRIPTS_DIR = new
Property<>("kvm.scripts.dir", "scripts/vm/hypervisor/kvm");
+ /**
+ * Host-local parent directory for the NAS backup pull-mode scratch area:
the NBD socket and
+ * the libvirt-managed fleecing (copy-before-write) images used while
backing up a running VM
+ * on raw block-device storage such as LINSTOR/DRBD.
+ * <p>
+ * Must be on a real on-disk filesystem with room for the guest's write
churn during a backup,
+ * and must NOT be on the NAS share: copy-before-write runs inline with
guest writes, so a slow
+ * or remote location stalls the VM. Keep the path short — the resulting
socket path is subject
+ * to the kernel's 108-byte UNIX socket limit.
+ * <p>
+ * Data type: String.<br>
+ * Default value: <code>/var/tmp</code>
+ *
+ * @since 4.22.0
+ */
+ public static final Property<String> NAS_BACKUP_PULL_SCRATCH_DIR = new
Property<>("nas.backup.pull.scratch.dir", "/var/tmp");
+
+ /**
+ * Free space the NAS backup pull-mode scratch area ({@link
#NAS_BACKUP_PULL_SCRATCH_DIR}) must
+ * have before a backup starts, as a percentage of the VM's total disk
size. Otherwise the
+ * backup fails up front.
+ * <p>
+ * Copy-before-write saves every guest block that is overwritten during
the backup into the
+ * scratch area, at most once per block, so the total disk size is the
worst case. If the scratch
+ * area fills up anyway, the guest's writes fail and the VM is paused. 100
rules that out; the
Review Comment:
There could be multiple backups running on the agent, each will check the
scratch size independently and pass. Better to mention that also here.
##########
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java:
##########
@@ -636,6 +636,42 @@ public class AgentProperties{
*/
public static final Property<String> KVM_SCRIPTS_DIR = new
Property<>("kvm.scripts.dir", "scripts/vm/hypervisor/kvm");
+ /**
+ * Host-local parent directory for the NAS backup pull-mode scratch area:
the NBD socket and
+ * the libvirt-managed fleecing (copy-before-write) images used while
backing up a running VM
+ * on raw block-device storage such as LINSTOR/DRBD.
+ * <p>
+ * Must be on a real on-disk filesystem with room for the guest's write
churn during a backup,
+ * and must NOT be on the NAS share: copy-before-write runs inline with
guest writes, so a slow
+ * or remote location stalls the VM. Keep the path short — the resulting
socket path is subject
+ * to the kernel's 108-byte UNIX socket limit.
+ * <p>
+ * Data type: String.<br>
+ * Default value: <code>/var/tmp</code>
+ *
+ * @since 4.22.0
Review Comment:
```suggestion
* @since 4.22.2
```
##########
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java:
##########
@@ -636,6 +636,42 @@ public class AgentProperties{
*/
public static final Property<String> KVM_SCRIPTS_DIR = new
Property<>("kvm.scripts.dir", "scripts/vm/hypervisor/kvm");
+ /**
+ * Host-local parent directory for the NAS backup pull-mode scratch area:
the NBD socket and
+ * the libvirt-managed fleecing (copy-before-write) images used while
backing up a running VM
+ * on raw block-device storage such as LINSTOR/DRBD.
+ * <p>
+ * Must be on a real on-disk filesystem with room for the guest's write
churn during a backup,
+ * and must NOT be on the NAS share: copy-before-write runs inline with
guest writes, so a slow
+ * or remote location stalls the VM. Keep the path short — the resulting
socket path is subject
+ * to the kernel's 108-byte UNIX socket limit.
+ * <p>
+ * Data type: String.<br>
+ * Default value: <code>/var/tmp</code>
+ *
+ * @since 4.22.0
+ */
+ public static final Property<String> NAS_BACKUP_PULL_SCRATCH_DIR = new
Property<>("nas.backup.pull.scratch.dir", "/var/tmp");
+
+ /**
+ * Free space the NAS backup pull-mode scratch area ({@link
#NAS_BACKUP_PULL_SCRATCH_DIR}) must
+ * have before a backup starts, as a percentage of the VM's total disk
size. Otherwise the
+ * backup fails up front.
+ * <p>
+ * Copy-before-write saves every guest block that is overwritten during
the backup into the
+ * scratch area, at most once per block, so the total disk size is the
worst case. If the scratch
+ * area fills up anyway, the guest's writes fail and the VM is paused. 100
rules that out; the
+ * default assumes the guest overwrites at most a quarter of its disks
while a backup runs.
+ * 0 disables the check.
+ * <p>
+ * Data type: Integer.<br>
+ * Default value: <code>25</code>
+ *
+ * @since 4.22.0
Review Comment:
```suggestion
* @since 4.22.2
```
##########
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java:
##########
@@ -636,6 +636,42 @@ public class AgentProperties{
*/
public static final Property<String> KVM_SCRIPTS_DIR = new
Property<>("kvm.scripts.dir", "scripts/vm/hypervisor/kvm");
+ /**
+ * Host-local parent directory for the NAS backup pull-mode scratch area:
the NBD socket and
+ * the libvirt-managed fleecing (copy-before-write) images used while
backing up a running VM
+ * on raw block-device storage such as LINSTOR/DRBD.
+ * <p>
+ * Must be on a real on-disk filesystem with room for the guest's write
churn during a backup,
+ * and must NOT be on the NAS share: copy-before-write runs inline with
guest writes, so a slow
+ * or remote location stalls the VM. Keep the path short — the resulting
socket path is subject
+ * to the kernel's 108-byte UNIX socket limit.
+ * <p>
+ * Data type: String.<br>
+ * Default value: <code>/var/tmp</code>
+ *
+ * @since 4.22.0
+ */
+ public static final Property<String> NAS_BACKUP_PULL_SCRATCH_DIR = new
Property<>("nas.backup.pull.scratch.dir", "/var/tmp");
Review Comment:
Scratch sizing is ok, but multiple backups may be running and as mentioned
in the other comments incremental backups might be slow and there is more
chance of scratch getting full which will cause the VM to be silently paused.
Can we think about creating the scratch as a volume in the Linstor primary
storage itself?
##########
scripts/vm/hypervisor/kvm/nasbackup.sh:
##########
@@ -183,6 +191,260 @@ thaw_guest() {
return 1
}
+# ---------------------------------------------------------------------------
+# Content-based backup path (libvirt pull mode).
+#
+# Used for raw block-device disks such as LINSTOR/DRBD. Those cannot carry QEMU
+# persistent dirty bitmaps -- persistence is a qcow2-only feature -- so libvirt
+# checkpoints, and with them the push-mode incremental path, are unavailable.
+#
+# Instead libvirt is asked for a pull-mode backup: it starts an NBD server
exposing a
+# point-in-time view of each disk, held consistent by copy-before-write into
local
+# fleecing scratch images. The backup is then derived from that export:
+#
+# full qemu-img convert -> sparse standalone qcow2. Zero detection
happens
+# while streaming, so the fully allocated push-mode output and
its
+# separate re-convert pass are both avoided.
+# incremental a qcow2 overlay backed by the NBD export, then a SAFE rebase
onto the
+# parent backup. That merges exactly the clusters which differ
between
+# export and parent into the overlay, yielding a delta qcow2
backed by the
+# parent -- the same chain shape the push-mode incremental
produces, so
+# restore/delete are unchanged.
+#
+# Unlike a storage-snapshot based diff there is no trailing DRBD metadata to
clip: the
+# DRBD device, and therefore the NBD export, is already net-sized.
+#
+# Assumes mount_operation() already ran, so $dest and $mount_point are set.
+# ---------------------------------------------------------------------------
+
+# EXIT trap of the pull-mode path, armed once the scratch dir exists and
cleared once the
+# backup is complete. Any exit in between -- an explicit failure, an
unexpected error under
+# "set -e", or SIGINT/SIGTERM/SIGHUP -- has to end the pull job: unlike a push
job it never
+# finishes on its own, so it would keep the NBD server and the growing
fleecing images alive
+# and block every later backup or migration of the VM. cleanup() aborts the
job (via
+# BACKUP_JOB_ACTIVE) before removing the partial backup. SIGKILL cannot be
trapped, so the
+# agent wrapper aborts the job itself when the script times out.
+on_pull_backup_exit() {
+ local rc=$?
+ trap - EXIT INT TERM HUP
+ [[ $rc -ne 0 ]] || rc=1
+ # The fleecing images belong to the job: only remove them once it is gone.
+ if abort_backup_job; then
+ rm -rf "$PULL_SCRATCH_DIR"
+ fi
+ cleanup
+ exit $rc
+}
+
+backup_running_vm_pull() {
+ local effective_mode="$1"
+
+ # Scratch area for the NBD socket and libvirt's fleecing images. Never on
the NAS:
+ # copy-before-write runs inline with guest writes, so it wants fast local
storage with
+ # room for the write churn during the backup. 711 lets the qemu process
traverse in to
+ # the files libvirt creates as root.
+ local scratch_dir
+ scratch_dir=$(mktemp -d "$SCRATCH_PARENT/csbackup-scratch.XXXXXX") || {
+ echo "Failed to create scratch directory under $SCRATCH_PARENT"; cleanup;
exit 1;
+ }
+ chmod 711 "$scratch_dir"
+ PULL_SCRATCH_DIR="$scratch_dir"
+ trap on_pull_backup_exit EXIT
+ trap 'exit 129' HUP
+ trap 'exit 130' INT
+ trap 'exit 143' TERM
+ local nbd_sock="$scratch_dir/nbd.sock"
+ local backupxml="$scratch_dir/backup.xml"
+
+ # The kernel caps UNIX socket paths at 108 bytes. A long scratch dir would
otherwise
+ # fail deep inside qemu-img with an opaque error, so reject it up front.
+ if [[ ${#nbd_sock} -ge 108 ]]; then
+ echo "Pull-mode NBD socket path is too long (${#nbd_sock} bytes, limit
108): $nbd_sock"
+ echo "Set a shorter scratch dir via -S (agent.properties
nas.backup.pull.scratch.dir)"
+ exit 1
+ fi
+
+ # Snapshot the disk list once so the backup XML and the read-back loop stay
in sync.
+ local -a disks
+ mapfile -t disks < <(
+ virsh -c qemu:///system domblklist "$VM" --details 2>/dev/null | awk
'$2=="disk"{print $3, $4}'
+ )
+ if [[ ${#disks[@]} -eq 0 ]]; then
+ echo "No disks found for vm $VM"
+ exit 1
+ fi
+
+ # Copy-before-write saves each guest block overwritten during the backup
into the scratch
+ # area, at most once per block. If the scratch area fills up, the guest's
writes fail and
+ # QEMU pauses the VM, and nothing resumes it. Refuse to start unless the
scratch area can
+ # take the configured share of the worst case, the VM's total disk size.
+ if [[ ! "$SCRATCH_MIN_FREE_PERCENT" =~ ^[0-9]+$ ]]; then
+ echo "Invalid scratch minimum free percentage: $SCRATCH_MIN_FREE_PERCENT"
+ exit 1
+ fi
+ if [[ $SCRATCH_MIN_FREE_PERCENT -gt 0 ]]; then
+ local entry disk_bytes total_bytes=0 required_bytes available_bytes
+ for entry in "${disks[@]}"; do
+ disk_bytes=$(virsh -c qemu:///system domblkinfo "$VM" "${entry%% *}"
2>/dev/null | awk '/^Capacity:/ {print $2}')
+ if [[ ! "$disk_bytes" =~ ^[0-9]+$ ]]; then
+ echo "Failed to determine the size of disk ${entry%% *} of vm $VM"
+ exit 1
+ fi
+ total_bytes=$((total_bytes + disk_bytes))
+ done
+ required_bytes=$(((total_bytes * SCRATCH_MIN_FREE_PERCENT + 99) / 100))
+ available_bytes=$(df --output=avail -B1 "$scratch_dir" 2>/dev/null | tail
-n 1 | tr -d ' ')
+ if [[ ! "$available_bytes" =~ ^[0-9]+$ ]]; then
+ echo "Failed to determine the free space in $SCRATCH_PARENT"
+ exit 1
+ fi
+ if [[ $available_bytes -lt $required_bytes ]]; then
+ echo "Not enough free space in $SCRATCH_PARENT for the pull-mode backup
of vm $VM:" \
+ "$available_bytes bytes available, $required_bytes bytes required" \
+ "($SCRATCH_MIN_FREE_PERCENT% of the $total_bytes bytes of disk)."
+ echo "Free up or enlarge $SCRATCH_PARENT, set agent.properties
nas.backup.pull.scratch.dir to a" \
+ "larger filesystem, or lower nas.backup.pull.scratch.min.free.percent."
+ exit 1
+ fi
+ fi
+
+ # Resolve each disk's parent backup up front. PARENT_PATHS is comma
separated, one entry
+ # per volume in the same order as the disk list. Anything missing degrades
this run to a
+ # full backup rather than failing it, mirroring the push-mode fallback.
+ local -a parent_paths_arr=()
+ if [[ "$effective_mode" == "incremental" ]]; then
+ if [[ -z "$PARENT_PATHS" ]]; then
+ log -e "content incremental: no parent paths supplied -- falling back to
full"
+ echo "INCREMENTAL_FALLBACK=true"
+ effective_mode="full"
+ else
+ IFS=',' read -ra parent_paths_arr <<< "$PARENT_PATHS"
+ if [[ ${#parent_paths_arr[@]} -lt ${#disks[@]} ]]; then
+ log -e "content incremental: parent path list shorter than disk list
-- falling back to full"
+ echo "INCREMENTAL_FALLBACK=true"
+ effective_mode="full"
+ else
+ local pp
+ for pp in "${parent_paths_arr[@]}"; do
+ if [[ ! -f "$mount_point/$pp" ]]; then
+ log -e "content incremental: parent $mount_point/$pp missing on
NAS -- falling back to full"
+ echo "INCREMENTAL_FALLBACK=true"
+ effective_mode="full"
+ break
+ fi
+ done
+ fi
+ fi
+ fi
+
+ # Pull-mode backup XML: one NBD export and one fleecing scratch image per
disk.
+ {
+ echo "<domainbackup mode='pull'>"
+ echo "<server transport='unix' socket='$nbd_sock'/>"
+ echo "<disks>"
+ local entry
+ for entry in "${disks[@]}"; do
+ echo "<disk name='${entry%% *}' backup='yes' type='file'
backupmode='full'><scratch file='$scratch_dir/${entry%% *}.scratch'/></disk>"
+ done
+ echo "</disks></domainbackup>"
+ } > "$backupxml"
+
+ local thaw=0
+ local freeze_ok=0
+ if [[ ${QUIESCE} == "true" ]]; then
+ # Always thaw after a freeze attempt, see backup_running_vm().
+ thaw=1
+ if freeze_err=$(qemu_agent_command '{"execute":"guest-fsfreeze-freeze"}'
2>&1 > /dev/null); then
+ freeze_ok=1
+ else
+ log -e "Failed to freeze the filesystem for vm $VM, continuing without
quiescing: $freeze_err"
+ fi
+ fi
+
+ # backup-begin establishes the point-in-time and starts the NBD server.
+ local backup_begin=0
+ local backup_out
+ if backup_out=$(virsh -c qemu:///system backup-begin --domain "$VM"
--backupxml "$backupxml" 2>&1); then
+ backup_begin=1
+ BACKUP_JOB_ACTIVE=1
+ fi
+
+ if [[ $thaw -eq 1 ]] && ! thaw_guest "$freeze_ok"; then
+ exit 1
+ fi
+
+ if [[ $backup_begin -ne 1 ]]; then
+ echo "Failed to start pull-mode backup for $VM: $backup_out"
+ exit 1
+ fi
+
+ # Backup domain information
+ virsh -c qemu:///system dumpxml $VM > $dest/domain-config.xml 2>/dev/null
+ virsh -c qemu:///system dominfo $VM > $dest/dominfo.xml 2>/dev/null
+ virsh -c qemu:///system domiflist $VM > $dest/domiflist.xml 2>/dev/null
+ virsh -c qemu:///system domblklist $VM > $dest/domblklist.xml 2>/dev/null
+
+ local name="root"
+ local disk_idx=0
+ local entry disk fullpath volUuid output export_uri parent_abs parent_rel
+ for entry in "${disks[@]}"; do
+ disk="${entry%% *}"
+ fullpath="${entry#* }"
+ if [[ "$fullpath" == /dev/drbd/by-res/* ]]; then
+ volUuid=$(get_linstor_uuid_from_path "$fullpath")
+ elif [[ "$fullpath" == /dev/drbd[0-9]* ]]; then
+ if ! volUuid=$(get_linstor_uuid_from_device "$fullpath"); then
+ echo "Failed to resolve LINSTOR volume UUID for $fullpath"
+ exit 1
+ fi
+ else
+ volUuid="${fullpath##*/}"
+ fi
+ output="$dest/$name.$volUuid.qcow2"
+ # The export name defaults to the disk target (e.g. vda) and presents raw
bytes.
+ export_uri="nbd+unix:///$disk?socket=$nbd_sock"
+
+ if [[ "$effective_mode" == "incremental" ]]; then
+ parent_abs="$mount_point/${parent_paths_arr[$disk_idx]}"
+ parent_rel=$(realpath --relative-to="$dest" "$parent_abs")
+ # Overlay over the export, safe-rebase onto the parent to merge the
differing
+ # clusters in, then a metadata-only rebase to store the parent as a
relative path
+ # so the chain survives being mounted at a different mount point.
+ if ! qemu-img create -f qcow2 -b "$export_uri" -F raw "$output" >>
"$logFile" 2> >(cat >&2) \
+ || ! qemu-img rebase -b "$parent_abs" -F qcow2 "$output" >>
"$logFile" 2> >(cat >&2) \
Review Comment:
This will read the whole disk through NBD and also the whole parent chain
from the NAS, because the safe rebase compares every cluster. So it saves NAS
space, but it costs more I/O and time. Few things to be cautious about here:
1. The longer window also means more copy-before-write churn into scratch.
2. Backup timeout being reached for larger disks.
3. NAS backup performance impact (as each incremental will read the whole
disk from NAS) - which will further delay the backup process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
