[pve-devel] applied-series: [PATCH proxmox-apt 1/3] upgrade to edition 2021

2022-02-03 Thread Wolfgang Bumiller
applied series, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server 2/4] api: clone: fork before locking

2022-02-03 Thread Fabian Ebner
Am 31.01.22 um 13:34 schrieb Fabian Grünbichler: > On January 27, 2022 3:01 pm, Fabian Ebner wrote: >> using the familiar early+repeated checks pattern from other API calls. >> Only intended functional changes are with regard to locking/forking. > > two questions: > - the FW config cloning happens

[pve-devel] [PATCH manager] api: apt: list_updates: fix dependency detection for virtual package

2022-02-03 Thread Fabian Ebner
A virtual package does not have SelectedState Install, but the dependency will still be satisfied if a package providing it has. Fixes a bug, wrongly showing that postfix will be installed, when a different mail-transport-agent is installed and a pve-manager update is available: https://forum.prox

[pve-devel] [PATCH v4 guest-common 1/3] migrate: handle migration_network with remote migration

2022-02-03 Thread Fabian Grünbichler
remote migration always has an explicit endpoint from the start which gets used for everything. Signed-off-by: Fabian Grünbichler --- src/PVE/AbstractMigrate.pm | 37 + 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/src/PVE/AbstractMigrate.pm

[pve-devel] [PATCH v4 guest-common 2/3] add tunnel helper module

2022-02-03 Thread Fabian Grünbichler
lifted from PVE::QemuMigrate, abstracting away use-case specific data. Signed-off-by: Fabian Grünbichler --- Notes: v4: - remove remaining 'migration' mentions - fix wrong 'log' call - set 'log' during WS tunnel setup already so it's available for error handling - switch 'lo

[pve-devel] [PATCH v4 proxmox-websocket-tunnel 1/4] initial commit

2022-02-03 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- .gitignore| 1 + .cargo/config | 5 + Cargo.toml| 11 +++ 3 files changed, 17 insertions(+) create mode 100644 .gitignore create mode 100644 .cargo/config create mode 100644 Cargo.toml diff --git a/.gitignore b/.gitignore new file mo

[pve-devel] [PATCH v4 guest-common 3/3] add storage tunnel module

2022-02-03 Thread Fabian Grünbichler
encapsulating storage-related tunnel methods, currently - source-side storage-migrate helper - target-side disk-import handler - target-side query-disk-import handler - target-side bwlimit handler to be extended further with replication-related handlers and helpers. Signed-off-by: Fabian Grünbich

[pve-devel] [PATCH v4 proxmox-websocket-tunnel 3/4] add fingerprint validation

2022-02-03 Thread Fabian Grünbichler
in case we have no explicit fingerprint, we use openssl's regular "PEER" verification. if we have a fingerprint, we ignore openssl verification results altogether and just verify the fingerprint of the presented leaf certificate, skipping the rest of the certificate chain (depth != 0). Signed-off-

[pve-devel] [PATCH v4 qemu-server 01/11] refactor map_storage to map_id

2022-02-03 Thread Fabian Grünbichler
since we are going to reuse the same mechanism/code for network bridge mapping. Signed-off-by: Fabian Grünbichler --- PVE/QemuMigrate.pm | 6 +++--- PVE/QemuServer.pm | 6 -- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm index c9bc39d

[pve-devel] [PATCH v4 qemu-server 02/11] schema: use pve-bridge-id

2022-02-03 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- Notes: requires pve-common with pve-bridge-id PVE/QemuServer.pm | 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index db3a0b13..f5beb1b4 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer

[pve-devel] [PATCH v4 qemu-server 03/11] parse_config: optional strict mode

2022-02-03 Thread Fabian Grünbichler
when passing a config from one cluster to another, we want to be strict when parsing - it's better to fail the migration early and upgrade the target node instead of failing the migration later (when significant work for transferring disks and/or state has already been done) or not at all, but sile

[pve-devel] [PATCH v4 qemu-server 05/11] nbd alloc helper: allow passing in explicit format

2022-02-03 Thread Fabian Grünbichler
and make $volname optional, to support remote migration usage without code duplication. Signed-off-by: Fabian Grünbichler --- Notes: v2: new requires - pve-storage with UNIX import support - pve-access-control with tunnel ticket support - pve-http-server with websocket f

[pve-devel] [PATCH v4 guest-common 0/22] remote migration

2022-02-03 Thread Fabian Grünbichler
this series adds remote migration for VMs. both live and offline migration including NBD and storage-migrated disks should work. groundwork for extending to pve-container and pvesr already laid. new in v4: lots of small fixes, improved bwlimit handling, `qm` command (thanks Fabian Ebner and Domin

[pve-devel] [PATCH v4 qemu-server 08/11] migrate: refactor remote VM/tunnel start

2022-02-03 Thread Fabian Grünbichler
no semantic changes intended, except for: - no longer passing the main migration UNIX socket to SSH twice for forwarding - dropping the 'unix:' prefix in start_remote_tunnel's timeout error message Signed-off-by: Fabian Grünbichler --- PVE/QemuMigrate.pm | 158 ---

[pve-devel] [PATCH v4 qemu-server 10/11] api: add remote migrate endpoint

2022-02-03 Thread Fabian Grünbichler
entry point for the remote migration on the source side, mainly preparing the API client that gets passed to the actual migration code and doing some parameter parsing. querying of the remote sides resources (like available storages, free VMIDs, lookup of endpoint details for specific nodes, ...)

[pve-devel] [PATCH v4 proxmox-websocket-tunnel 2/4] add tunnel implementation

2022-02-03 Thread Fabian Grünbichler
the websocket tunnel helper accepts control commands (encoded as single-line JSON) on stdin, and prints responses on stdout. the following commands are available: - "connect" a 'control' tunnel via a websocket - "forward" a local unix socket to a remote socket via a websocket -- if requested, this

[pve-devel] [PATCH v4 proxmox-websocket-tunnel 4/4] add packaging

2022-02-03 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- Notes: v3: rebased/regenerated Makefile | 44 + debian/changelog | 5 debian/control | 58 debian/copyright | 16 debian/debcargo.t

[pve-devel] [PATCH v4 storage 1/4] volname_for_storage: parse volname before calling

2022-02-03 Thread Fabian Grünbichler
to allow reusing this with remote migration, where parsing of the source volid has to happen on the source node, but this call has to happen on the target node. Signed-off-by: Fabian Grünbichler --- PVE/Storage.pm | 16 +--- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git

[pve-devel] [PATCH v4 storage 2/4] storage_migrate: pull out snapshot decision

2022-02-03 Thread Fabian Grünbichler
into new top-level helper for re-use with remote migration. Signed-off-by: Fabian Grünbichler --- Notes: v4: - correctly use source storage for decision - fold fixup into correct patch PVE/Storage.pm | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git

[pve-devel] [PATCH v4 storage 3/4] storage_migrate: pull out import/export_prepare

2022-02-03 Thread Fabian Grünbichler
for re-use with remote migration, where import and export happen on different clusters connected via a websocket instead of SSH tunnel. Signed-off-by: Fabian Grünbichler --- Notes: v4: - unify array refs - small fixups new in v3 PVE/Storage.pm | 113

[pve-devel] [PATCH v4 qemu-server 04/11] update_vm: allow simultaneous setting of boot-order and dev

2022-02-03 Thread Fabian Grünbichler
else this fails if we check 'boot' before the device was put into the config or pending section. Signed-off-by: Fabian Grünbichler --- Notes: as happens when doing a remote migration and the full config is passed through update_vm_api PVE/API2/Qemu.pm | 2 +- 1 file changed, 1 inserti

[pve-devel] [PATCH v4 storage 4/4] add volume_import/export_start helpers

2022-02-03 Thread Fabian Grünbichler
exposing the two halves of a storage migration for usage across cluster boundaries. Signed-off-by: Fabian Grünbichler --- Notes: v4: - add log parameter - unify array-refs new in v3 PVE/Storage.pm | 66 ++ 1 file changed, 66

[pve-devel] [PATCH v4 qemu-server 06/11] migrate: move tunnel-helpers to pve-guest-common

2022-02-03 Thread Fabian Grünbichler
besides the log calls these don't need any parts of the migration state, so let's make them generic and re-use them for container migration and replication in the future. Signed-off-by: Fabian Grünbichler --- Notes: v4: switch log to use two-parameter logging sub like migration new in v3

[pve-devel] [PATCH v4 qemu-server 07/11] mtunnel: add API endpoints

2022-02-03 Thread Fabian Grünbichler
the following two endpoints are used for migration on the remote side POST /nodes/NODE/qemu/VMID/mtunnel which creates and locks an empty VM config, and spawns the main qmtunnel worker which binds to a VM-specific UNIX socket. this worker handles JSON-encoded migration commands coming in via thi

[pve-devel] [PATCH v4 qemu-server 09/11] migrate: add remote migration handling

2022-02-03 Thread Fabian Grünbichler
remote migration uses a websocket connection to a task worker running on the target node instead of commands via SSH to control the migration. this websocket tunnel is started earlier than the SSH tunnel, and allows adding UNIX-socket forwarding over additional websocket connections on-demand. the

[pve-devel] [PATCH v4 qemu-server 11/11] qm: add remote_migrate command

2022-02-03 Thread Fabian Grünbichler
which wraps the remote_migrate_vm API endpoint, but does the precondition checks that can be done up front itself. this now just leaves the FP retrieval and target node name lookup to the sync part of the API endpoint, which should be do-able in <30s .. Signed-off-by: Fabian Grünbichler --- new

Re: [pve-devel] [PATCH qemu-server 2/4] api: clone: fork before locking

2022-02-03 Thread Fabian Grünbichler
On February 3, 2022 10:31 am, Fabian Ebner wrote: > Am 31.01.22 um 13:34 schrieb Fabian Grünbichler: >> On January 27, 2022 3:01 pm, Fabian Ebner wrote: >>> using the familiar early+repeated checks pattern from other API calls. >>> Only intended functional changes are with regard to locking/forking