[pve-devel] [PATCH v2 qemu-server 1/3] drive-mirror: add support for incremental sync

2020-03-17 Thread Fabian Grünbichler
by re-using a dirty bitmap that represents changes since the divergence of source and target volume. requires a qemu that supports incremental drive-mirroring, and will die otherwise. Signed-off-by: Fabian Grünbichler --- Notes: v1-v2: - use newer Qemu patches picked up by me and submitt

[pve-devel] [PATCH v2 qemu/qemu-server 0/4] live-migration with replicated disks

2020-03-17 Thread Fabian Grünbichler
I recently picked up and finished some work-in-progress patches for adding bitmap support to drive-mirror (it got added to backup block jobs in 4.0, with plenty of fixes in 4.1 and 4.2) and submitted them upstream. IMHO this is in a shape now where we can include it, but I'd also be fine with hidi

[pve-devel] [PATCH v2 qemu-server 2/3] migrate: add replication info to disk overview

2020-03-17 Thread Fabian Grünbichler
to make migration logs a bit easier to grasp with a quick glance. Signed-off-by: Fabian Grünbichler --- Notes: unchanged since v1 PVE/QemuMigrate.pm | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm index 44e4c57..d87476

[pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Fabian Grünbichler
with incremental drive-mirror and dirty-bitmap tracking. 1.) get replicated disks that are currently referenced by running VM 2.) add a block-dirty-bitmap to each of them 3.) replicate ALL replicated disks 4.) pass bitmaps from 2) to drive-mirror for disks from 1) 5.) skip replicated disks when cl

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else systemd decides to regress and suddenly break network stack in CT after an upgrade[1]. This mou

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else systemd decides to regress and suddenly break net

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Stefan Reiter
Casually looked over the patches, looking good to me so far - I'll give them some testing later. Two things inline. On 17/03/2020 08:55, Fabian Grünbichler wrote: with incremental drive-mirror and dirty-bitmap tracking. 1.) get replicated disks that are currently referenced by running VM 2.)

Re: [pve-devel] [PATCH container 1/2] update_lxc_config: set in-CT network devices to up

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 7:35 AM, Thomas Lamprecht wrote: Else some newer system do not see the interface as up and refuse to manage it.. Where do you run into that actually? I can get my arch container to behave normally with just adding `lxc.mount.auto = sys:mixed` to /etc/pve/lxc/$vmid.conf atm. without

[pve-devel] [PATCH v2 manager 2/2] gui/cluster: add structured peerLinks to join info

2020-03-17 Thread Stefan Reiter
Instead of the old 'ring_addr' property (which is kept for compatibility), we also encode the link numbers into the new peerLinks structure. This allows us to display which IP is assigned to which link on the cluster in the join dialog, helping a user identify which link should receive which interf

[pve-devel] [PATCH v2 manager 1/2] gui/cluster: add CorosyncLinkEdit component to support up to 8 links

2020-03-17 Thread Stefan Reiter
CorosyncLinkEdit is a Panel that contains between one and 8 CorosyncLinkSelectors. These can be added or removed with according buttons. Values submitted to the API are calculated by each ProxmoxNetworkSelector itself. This works because ExtJS searches recursively through all child components for

[pve-devel] [PATCH v2 0/2] Support all 8 corosync3 links in GUI

2020-03-17 Thread Stefan Reiter
Since it's been a while, here's a rebased version the two still missing patches of my previous RFC[0]. Except for a slight rewording of the message in patch 2 it's exactly the same, so consider this a RESEND more than anything. I think the series might have gotten lost a bit after parts where appl

Re: [pve-devel] [PATCH container 1/2] update_lxc_config: set in-CT network devices to up

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 11:48 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >> Else some newer system do not see the interface as up and refuse to >> manage it.. > > Where do you run into that actually? I can get my arch container to behave > normally with just adding `lxc.mount.a

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >> CONTAINER_INTERFACE[0] is omething systemd people call their API and >> we need to adapt to it a bit, even if it means doing stupid >> unnecessary things, as else systemd decides to regress and suddenly >

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 11:21 AM, Stefan Reiter wrote: >> +$local_volumes->{$opt} = $conf->{${opt}}; > > Does $conf->{${opt}} have too many brackets or is this another arcane perl  > syntax I've yet to discover? (iow. why not just $conf->{$opt} ?) It's not that arcane, you surely used it sometimes

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Stefan Reiter
On 17/03/2020 12:40, Thomas Lamprecht wrote: On 3/17/20 11:21 AM, Stefan Reiter wrote: +$local_volumes->{$opt} = $conf->{${opt}}; Does $conf->{${opt}} have too many brackets or is this another arcane perl syntax  I've yet to discover? (iow. why not just $conf->{$opt} ?) It's not

[pve-devel] [PATCH qemu-server] Add unused description to drivedesc_hash

2020-03-17 Thread Fabian Ebner
Moved code so that initialization of drivedesc_hash stays a single block. Avoid auto-vivication in parse_drive. Signed-off-by: Fabian Ebner --- Follow-up suggested by Fabian G. I checked the call-sites for is_valid_drivename and in most cases it won't benefit from including unused drives, often

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Fabian Grünbichler
On March 17, 2020 12:40 pm, Thomas Lamprecht wrote: > On 3/17/20 11:21 AM, Stefan Reiter wrote: >>> +$local_volumes->{$opt} = $conf->{${opt}}; >> >> Does $conf->{${opt}} have too many brackets or is this another arcane perl  >> syntax I've yet to discover? (iow. why not just $conf->{$o

[pve-devel] [PATCH V3] Revision of the pvesr documentation

2020-03-17 Thread Wolfgang Link
Improvement of grammar and punctuation. Clarify the HA limitations. Remove future tense in some sentences. It is not good to use it in technical/scientific papers. Rewrite some sentences to improve understanding. --- pvesr.adoc | 112 ++--- 1 file ch

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else sys

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: > On 3/17/20 12:31 PM, Thomas Lamprecht wrote: >> On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: >>> On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, e

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 2:24 PM, Thomas Lamprecht wrote: On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 2:33 PM, Wolfgang Bumiller wrote: > On 3/17/20 2:24 PM, Thomas Lamprecht wrote: >> On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: >>> On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >> CON

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 2:40 PM, Thomas Lamprecht wrote: On 3/17/20 2:33 PM, Wolfgang Bumiller wrote: On 3/17/20 2:24 PM, Thomas Lamprecht wrote: On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Tho

Re: [pve-devel] [PATCH v3 manager 1/1] backup: move logic to include guests into method

2020-03-17 Thread Fabian Grünbichler
On March 16, 2020 4:44 pm, Aaron Lauterer wrote: > This extracts the logic which guests are to be included in a backup job > into its own method 'get_included_guests'. This makes it possible to > develop other features around backup jobs. > > Logic which was spread out accross the API2/VZDump.pm f

Re: [pve-devel] [PATCH v3 qemu-server] vzdump: move include logic for volumes to method

2020-03-17 Thread Fabian Grünbichler
On March 16, 2020 4:44 pm, Aaron Lauterer wrote: > Move the logic which volumes are included in the backup job to its own > method and adapt the VZDump code accordingly. This makes it possible to > develop other features around backup jobs. > > Signed-off-by: Aaron Lauterer > --- > > v2 -> v3: r

[pve-devel] [PATCH v2 qemu-server 0/4] fix secure live migration with local

2020-03-17 Thread Mira Limbeck
Currently NBD storage migration always uses unencrypted TCP. The following 4 patches add support for unix sockets that are forwarded over SSH. For backwards compatibility this requires some kind of communication from the source node to the target node, because the NBD server can only be started wit

[pve-devel] [PATCH v2 qemu-server 4/4] add unix socket support for NBD storage migration

2020-03-17 Thread Mira Limbeck
The reuse of the tunnel, which we're opening to communicate with the target node and to forward the unix socket for the state migration, for the NBD unix socket requires adding support for an array of sockets to forward, not just a single one. We also have to change the $sock_addr variable to an ar

[pve-devel] [PATCH v2 qemu-server 3/4] parse nbd_protocol_version if available

2020-03-17 Thread Mira Limbeck
For secure live migration with local disks via NBD over a unix socket, we have to somehow communicate from the source node to the target node if it supports it. This is because there can only be one NBD server with exactly one socket bound. The source node passes that information via STDIN. Suppor

[pve-devel] [PATCH v2 qemu-server 2/4] add NBD server unix socket support in vm_start

2020-03-17 Thread Mira Limbeck
As the NBD server spawned by qemu can only listen on a single socket, we're dependent on a version being passed to vm_start that indicates which protocol can be used, TCP or Unix, by the source node. The change in socket type (TCP to Unix) comes with a different URI. For unix sockets it has the fo

[pve-devel] [PATCH v2 qemu-server 1/4] add qemu_drive_mirror_monitor completion modes

2020-03-17 Thread Mira Limbeck
With Qemu 4.2 we encountered a problem with unix sockets and SSH socket forwarding for drive-mirror. It seems the socket gets reopened again and again after it closes for some reason. This can be worked around by specifying 'block-job-cancel' instead of 'block-job-complete' when we're not intereste

[pve-devel] [zsync] fix: check for incremental sync snapshot.

2020-03-17 Thread Wolfgang Link
For an incremental sync you need the last_snap on both sides. --- pve-zsync | 13 - 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/pve-zsync b/pve-zsync index ea3178e..893baf0 100755 --- a/pve-zsync +++ b/pve-zsync @@ -931,6 +931,7 @@ sub snapshot_destroy { } } +