Re: [pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 18:52, Stoiko Ivanov wrote: > On Tue, 22 Jun 2021 17:15:08 +0200 > Thomas Lamprecht wrote: > >> On 22.06.21 17:10, Stoiko Ivanov wrote: >>> I had a patch for untainting the individual values in >>> PVE::Storage::Plugin::volume_size_info but then went with this patch, >> >> I'd rathe

Re: [pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Stoiko Ivanov
On Tue, 22 Jun 2021 17:15:08 +0200 Thomas Lamprecht wrote: > On 22.06.21 17:10, Stoiko Ivanov wrote: > > I had a patch for untainting the individual values in > > PVE::Storage::Plugin::volume_size_info but then went with this patch, > > I'd rather have that patch, especially for back-porting t

[pve-devel] [PATCH storage] plugins: untaint volume_size_info retuns

2021-06-22 Thread Stoiko Ivanov
the size returned by volume_size_info is used for creating the new destination image in PVE::QemuServer::clone_disk (and probably elsewhere). In certain cases the return values are tainted - they are obtained by a run_command call and depending on the format and length of the parsed output can stil

[pve-devel] applied: [PATCH manager 2/2] postinst: remove outdated calls

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 12:16, Fabian Grünbichler wrote: > any system upgrading to 7.x was either installed with >= 6.4 in the > first place, or upgraded to >= 6.4 and thus fixed those issues already. > > Signed-off-by: Fabian Grünbichler > --- > debian/postinst | 10 -- > 1 file changed, 10 deletio

Re: [pve-devel] [PATCH manager 1/2] postinst: set custom LVM settings

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 12:16, Fabian Grünbichler wrote: > now that we no longer ship our own LVM packages, set the relevant > filtering options here if they are missing. > > for an upgrade from PVE 6.x, the following two scenarios are likely: > > A: user edited config provided by our old lvm2 package. it li

Re: [pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 17:10, Stoiko Ivanov wrote: > I had a patch for untainting the individual values in > PVE::Storage::Plugin::volume_size_info but then went with this patch, I'd rather have that patch, especially for back-porting to stable. I mean, else we can probably just turn of the taint mode comple

Re: [pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Stoiko Ivanov
On Tue, 22 Jun 2021 16:41:50 +0200 Thomas Lamprecht wrote: > On 22.06.21 16:28, Stoiko Ivanov wrote: > > The performance improvements added in > > cb9db10c1a9855cf40ff13e81f9dd97d6a9b2698 changed the output handling > > to not remove the taintedness (see perlsec (1)) of the complete output > > an

Re: [pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 16:28, Stoiko Ivanov wrote: > The performance improvements added in > cb9db10c1a9855cf40ff13e81f9dd97d6a9b2698 changed the output handling > to not remove the taintedness (see perlsec (1)) of the complete output > anymore. > > This results in a few bugs which show up every now and then

[pve-devel] [PATCH common] run_command: untaint end of buffer

2021-06-22 Thread Stoiko Ivanov
The performance improvements added in cb9db10c1a9855cf40ff13e81f9dd97d6a9b2698 changed the output handling to not remove the taintedness (see perlsec (1)) of the complete output anymore. This results in a few bugs which show up every now and then, and are usually quite tedious to hunt down - usual

[pve-devel] [PATCH v4 qemu-server] vm_start: check if storages of volumes support correct content-type

2021-06-22 Thread Lorenz Stechauner
Signed-off-by: Lorenz Stechauner --- changes to v3: * dropped already applied patch * moved check to cfg2cmd (with helper) * 'images' content-type is not anymore hard-coded vm state files are not included anymore and efi disks have type 'images' - therefore they should work PVE/QemuServer.pm |

[pve-devel] [PATCH v2 container 3/3] special case btrfs+quotas to use subvolumes

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- New in v2. src/PVE/LXC.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm index 0a8a532..fc06842 100644 --- a/src/PVE/LXC.pm +++ b/src/PVE/LXC.pm @@ -1893,7 +1893,7 @@ sub alloc_disk { eval { my

[pve-devel] [PATCH v2 container 2/3] enable btrfs support via subvolumes

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- Changes to v1: Rebased & Comment fixup. Note that I have not yet opted to switch these tests to `volume_has_feature` based ones. There are still some unresolved questions there. src/PVE/LXC/Migrate.pm | 9 + 1 file changed, 5 insertions(+), 4 delet

[pve-devel] [PATCH v2 storage 1/5] add BTRFS storage plugin

2021-06-22 Thread Wolfgang Bumiller
This is mostly the same as a directory storage, with 2 major differences: * 'subvol' volumes are actual btrfs subvolumes and therefore allow snapshots * 'raw' files are placed *into* a subvolume and therefore also allow snapshots, the raw file for volume `btrstore:100/vm-100-disk-1.raw` can

[pve-devel] [PATCH v2 storage 2/5] bump storage API: update import/export methods

2021-06-22 Thread Wolfgang Bumiller
Bumps APIVER to 9 and resets APIAGE to zero. The import methods (volume_import, volume_import_formats): These additionally get the '$snapshot' parameter which is already present on the export side as an informational piece to know which of the snapshots is the *current* one. This parameter is ins

[pve-devel] [PATCH v2 storage 3/5] btrfs: add 'btrfs' import/export format

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- Changes to v1: * import api parameters reordered du to diff in patch 2/5 PVE/CLI/pvesm.pm | 2 +- PVE/Storage.pm | 2 +- PVE/Storage/BTRFSPlugin.pm | 248 +++-- 3 files changed, 240 insertions(+), 12

[pve-devel] [PATCH v2 storage 5/5] btrfs: support quota-based subvols optionally

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- New in this version. Disables `btrfs-send/recv` *for now*... In my local tests, while merely enabling quotas on the file system does make the fs measurably slower, it was still faster than the ext4-raw image (and even faster than ZFS subvolumes), and while I h

[pve-devel] [PATCH v2 qemu-server] allow migrating raw btrfs volumes

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- No changes to v1. Also: see comment on the container side. PVE/QemuMigrate.pm | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm index 5f37890..a2dd11c 100644 --- a/PVE/QemuMigrate.pm +++ b/PVE/Q

[pve-devel] [PATCH v2 container 1/3] migration: fix snapshots boolean accounting

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- No changes to v1. src/PVE/LXC/Migrate.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm index 3cd895d..ce1f7dd 100644 --- a/src/PVE/LXC/Migrate.pm +++ b/src/PVE/LXC/Migrate.pm @@ -146,7 +146

[pve-devel] [PATCH v2 storage 4/5] btrfs: make NOCOW optional

2021-06-22 Thread Wolfgang Bumiller
Signed-off-by: Wolfgang Bumiller --- New in this version. It's a performance knob... PVE/Storage/BTRFSPlugin.pm | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/PVE/Storage/BTRFSPlugin.pm b/PVE/Storage/BTRFSPlugin.pm index 072dfe0..1fe5db0 100644 --- a/PVE/Sto

[pve-devel] [PATCH v2 multiple] btrfs, file system for the brave

2021-06-22 Thread Wolfgang Bumiller
Changes to v1: * Storage API gets a hard bump: (ver=9, age=0), due to the import method signature changes. * Added `nocow` file storage option as a performance knob. This causes raw files to be marked as `NOCOW` (`chattr +C`), which does 2 things: a) Disables checksumming: b

[pve-devel] applied: [PATCH widget-toolkit] panel/JournalView: fix flickering in journal livemode

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 11:57, Dominik Csapak wrote: > it seems that something changed in extjs 7 which does not quite > restore the correct scroll position when the identical content is set > on a component. this means that sometimes, we update the text > with the identical one, but the scroll position is now

[pve-devel] [PATCH docs] pbs: add information about master key support

2021-06-22 Thread Fabian Grünbichler
Signed-off-by: Fabian Grünbichler --- pve-storage-pbs.adoc | 19 +++ 1 file changed, 19 insertions(+) diff --git a/pve-storage-pbs.adoc b/pve-storage-pbs.adoc index c22f5b3..a3d7da1 100644 --- a/pve-storage-pbs.adoc +++ b/pve-storage-pbs.adoc @@ -57,6 +57,13 @@ restricted to the

Re: [pve-devel] [PATCH manager 1/2] postinst: set custom LVM settings

2021-06-22 Thread Fabian Grünbichler
On June 22, 2021 12:16 pm, Fabian Grünbichler wrote: > now that we no longer ship our own LVM packages, set the relevant > filtering options here if they are missing. > > for an upgrade from PVE 6.x, the following two scenarios are likely: > > A: user edited config provided by our old lvm2 packag

[pve-devel] [PATCH manager 2/2] postinst: remove outdated calls

2021-06-22 Thread Fabian Grünbichler
any system upgrading to 7.x was either installed with >= 6.4 in the first place, or upgraded to >= 6.4 and thus fixed those issues already. Signed-off-by: Fabian Grünbichler --- debian/postinst | 10 -- 1 file changed, 10 deletions(-) diff --git a/debian/postinst b/debian/postinst index

[pve-devel] [PATCH manager 1/2] postinst: set custom LVM settings

2021-06-22 Thread Fabian Grünbichler
now that we no longer ship our own LVM packages, set the relevant filtering options here if they are missing. for an upgrade from PVE 6.x, the following two scenarios are likely: A: user edited config provided by our old lvm2 package. it likely contains our (or a modified) global_filter, but the

[pve-devel] [PATCH widget-toolkit] panel/JournalView: fix flickering in journal livemode

2021-06-22 Thread Dominik Csapak
it seems that something changed in extjs 7 which does not quite restore the correct scroll position when the identical content is set on a component. this means that sometimes, we update the text with the identical one, but the scroll position is now off, only to scroll back to the bottom this cau

[pve-devel] applied: [PATCH manager] ui: dc: backup: fix job detail search

2021-06-22 Thread Thomas Lamprecht
On 22.06.21 10:07, Aaron Lauterer wrote: > 'for...in array' returns the id in the array but not the value, > 'for...of array' returns the values. > > Another issue that I ran into was if the property did not exist. Having something like "another issue fixed with additional changes" is always a go

[pve-devel] [PATCH v10 manager 4/4] fix #1710: ui: storage: add download from url button

2021-06-22 Thread Lorenz Stechauner
uses the common function PVE::Tools::download_file_from_url to download a iso image or container template. note: Only users with permissions `Sys.Audit` and `Sys.Modify` on `/` are permitted to use the api endpoints due to security reasons. (it is possible to download files from internal networks

[pve-devel] [PATCH v10 manager 2/4] api: nodes: add query_url_metadata method

2021-06-22 Thread Lorenz Stechauner
metadata is gained using a HEAD request. Due to the ability of this api endpoint to request files on internal networks (which would not be visible/accessible from outside) it is restricted to users with permissions `Sys.Audit` and `Sys.Modify` on `/`. Users with these permissions are able to alter

[pve-devel] [PATCH v10 storage 2/3] status: factoring out normalize_content_filename

2021-06-22 Thread Lorenz Stechauner
Signed-off-by: Lorenz Stechauner --- PVE/API2/Storage/Status.pm | 6 +- PVE/Storage.pm | 12 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm index 7069244..11ad60f 100644 --- a/PVE/API2/Storage/S

[pve-devel] [PATCH-SERIES v10 manager/storage] fix #1710: add download from url button

2021-06-22 Thread Lorenz Stechauner
changes to v9: * dropped already applied paches * split storage-patch into multiple patches * added download-url api endpoint to index * updated description of filename parameter (download-url) * added two new patches to factor out regex'es (manager+storage) pve-storage: Lorenz Stechauner (3):

[pve-devel] [PATCH v10 storage 3/3] status: add download_url method

2021-06-22 Thread Lorenz Stechauner
uses common function PVE::Tools::download_file_from_url to download iso files. Only users with permissions `Sys.Audit` and `Sys.Modify` on `/` are permitted to perform this action. This restriction is due to the fact, that the download function is able to download files from internal networks (whi

[pve-devel] [PATCH v10 storage 1/3] factoring out regex'es for backup and vztmpl

2021-06-22 Thread Lorenz Stechauner
uniformly stores these regex definitions in PVE::Storage. One test had to be adapted because it tested obsolete code. Namely: it expects vztmpl to only end with .tar.gz, but the new regex also includes .tar.xz, there is nothing against allowing .tar.xz files as vztmpl files. Signed-off-by: Lorenz

[pve-devel] [PATCH v10 manager 3/4] ui: Utils: change download task format

2021-06-22 Thread Lorenz Stechauner
Signed-off-by: Lorenz Stechauner --- www/manager6/Utils.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js index 3415c9eb..6f208954 100644 --- a/www/manager6/Utils.js +++ b/www/manager6/Utils.js @@ -1776,7 +1776,7 @@ Ext.define('PV

[pve-devel] [PATCH v10 manager 1/4] aplinfo: factoring out regex for vztmpl

2021-06-22 Thread Lorenz Stechauner
Signed-off-by: Lorenz Stechauner --- PVE/APLInfo.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/PVE/APLInfo.pm b/PVE/APLInfo.pm index 31522ae5..fa991a1b 100644 --- a/PVE/APLInfo.pm +++ b/PVE/APLInfo.pm @@ -8,6 +8,7 @@ use LWP::UserAgent; use POSIX qw(strftime); use P

[pve-devel] [PATCH manager] ui: dc: backup: fix job detail search

2021-06-22 Thread Aaron Lauterer
'for...in array' returns the id in the array but not the value, 'for...of array' returns the values. Another issue that I ran into was if the property did not exist. Checking if the property evaluates to false will catch situations where the property does not exist or is null. All other situation

Re: [pve-devel] [PATCH v2 guest-common 2/2] fix 3111: replicate guest on rollback if there are replication jobs for it

2021-06-22 Thread Fabian Grünbichler
On June 9, 2021 11:18 am, Fabian Ebner wrote: > so that there will be a valid replication snapshot again. > > Otherwise, replication will be broken after a rollback if the last > (non-replication) snapshot is removed before replication can run again. I still see issues with these two patches appl

Re: [pve-devel] [PATCH qemu-server 1/2] use KillMode 'process' for systemd scope

2021-06-22 Thread Stefan Reiter
On 22/06/2021 08:02, Thomas Lamprecht wrote: On 21.06.21 18:35, Stefan Reiter wrote: KillMode 'none' is deprecated, and systemd loudly complains about that in the journal. To avoid the warning, but keep the behaviour the same, use KillMode 'process'. This mode does two things differently, which

Re: [pve-devel] [PATCH container] fix #3367: skip bind mounts when converting to template

2021-06-22 Thread Fabian Grünbichler
On June 21, 2021 10:41 am, Wolfgang Bumiller wrote: > On Tue, Apr 06, 2021 at 08:26:50AM +0200, Fabian Ebner wrote: >> It turns out that we do not yet allow cloning from container templates with >> bind mounts. So in a sense container templates with bind mounts are >> currently misconfigured, and t

Re: [pve-devel] [PATCH/RFC cluster/common/... many] add cluster-wide hardware device mapping

2021-06-22 Thread Dominik Csapak
i just noticed, it seems i forgot to add some hunks/stashes? usb does not work properly there is one qemu patch missing and two would have to be adapted i can send a v2 for the whole series, only for qemu-server or only for the patches, depending on what is preferred the whole diff is rather sm