Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"
Le 31/07/22 à 18:49, DERUMIER, Alexandre a écrit :
> Le 31/07/22 à 18:19, Dietmar Maurer a écrit :
>>> This is really a blocker for me,I can't use pbs because I'm using nvme
>>> is production, and a 7200k hdd backup in a remote 200km site with 5ms
>>> latency.
>> Why don't you use a local(fast) PBS instance, then sync to the slow remote?
>>
> Hi Dietmar.
>
> Can I use a small local fast PBS instance without need to keep the full
> datastore chunks ?
>
> I have 300TB nvme in production, I don't want to buy 300TB nvme for backup.
>
> I known that I can keep more retentions on remote storage slow, but what
> about the local fast pbs ?
>
>
Maybe also, currently if you pbs server is crashing/shutdown/halt/...
when a backup is running,
the vm writes are totally frozen.
Or if network problem occur, it can hang the vm too.
That's why I think than a local cache (could be optionnal) could be a
great improvment.
I found doc about fleecing,
technally, it's just exposing a new blockdev inside qemu, like a virtual
frozen snapshot.
So I think it could work with proxmox backup code too.
https://www.mail-archive.com/qemu-devel@nongnu.org/msg876056.html
create the fleecing device
qmp: transaction [
block-dirty-bitmap-add {node: disk0, name: bitmap0, persistent: true}
blockdev-add* {node-name: tmp-protocol, driver: file, filename:
temp.qcow2}
blockdev-add {node-name: tmp, driver: qcow2, file: tmp-protocol}
blockdev-add {node-name: cbw, driver: copy-before-write, file: disk0,
target: tmp}
blockdev-replace** {parent-type: qdev, qdev-id: sda, new-child: cbw}
blockdev-add {node-name: acc, driver: snapshot-access, file: cbw}
]
launch qemu backup (push model) --> should use proxmox backup code
here instead
# Add target node. Here is qcow2 added, but it may be nbd node or
something else
blockdev-add {node-name: target-protocol, driver: file, filename:
target.qcow2}
blockdev-add {node-name: target, driver: qcow2, file: target-protocol}
# Start backup
blockdev-backup {device: acc, target: target, ...}
]
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"
> Can I use a small local fast PBS instance without need to keep the full > datastore chunks ? > > I have 300TB nvme in production, I don't want to buy 300TB nvme for backup. no, sorry. Do you want to use Ceph as temp backup storage, or simply an additional (node local) nvme? ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH access-control 2/2] PVE/RPCEnvironment: add helper for checking hw permissions
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> like check_vm_perm, etc.
>
> Signed-off-by: Dominik Csapak
> ---
> src/PVE/RPCEnvironment.pm | 8
> 1 file changed, 8 insertions(+)
>
> diff --git a/src/PVE/RPCEnvironment.pm b/src/PVE/RPCEnvironment.pm
> index 7c37c6e..c1b712d 100644
> --- a/src/PVE/RPCEnvironment.pm
> +++ b/src/PVE/RPCEnvironment.pm
> @@ -356,6 +356,14 @@ sub check_vm_perm {
> return $self->check_full($user, "/vms/$vmid", $privs, $any, $noerr);
> };
>
> +sub check_hw_perm {
> +my ($self, $user, $id, $privs, $any, $noerr) = @_;
> +
> +my $cfg = $self->{user_cfg};
> +
> +return $self->check_full($user, "/hardware/$id", $privs, $any, $noerr);
> +}
is this really needed (here?)?
I mean, yes,
$rpcenv->check_hw_perm('foo@bar', "hardware_id", ['Hardware.Use'], 0, 0)
is a (tiny) bit shorter than
$rpcenv->check_full('foo@bar', "/hardware/hardware_id", ['Hardware.Use'], 0, 0)
but ;)
note that check_vm has a special job and is not just a wrapper for
checking $ID against /$PREFIX/$ID, it is specifically for checking guest
ACLs while honoring pool ACLs for the special case of "VM is currently
being created and not formally part of the pool yet"..
similary, check_perm_modify serves the purpose of containing all the
"modify $path" -> "actual privilege" mappings in a single place.
the rest of the check_foo subs are low-level building blocks/helpers.
> +
> sub is_group_member {
> my ($self, $group, $user) = @_;
>
> --
> 2.30.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH common 1/1] add PVE/HardwareMap
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> this adds functionality for the hardwaremap config (as json)
> the format of the config is like this:
>
> {
> usb => {
> name => {
> nodename1 => { /* mapping object */ },
> nodename2 => { /* mapping object */ }
> }
> },
> pci => {
> /* same as above */
> },
> digest => ""
> }
kind of missing an argument for why
- this is a json file instead of a regular section config (the two
stored types share most of their structure?)
- this is a cluster-wide file containing a map per node instead of
being one config file per node?
from what I can piece together from the rest of the series ;)
- cannot be a cluster-wide section config since values differ
(potentially) on each node (so a simple `nodes` filter is not enough)
- we don't want ID foo to be type USB on one node and type PCI on
another
- we want to check all nodes for migration precondition checks
the first one is a given obviously. the type mismatch wouldn't actually
cause any problems although it could be confusing possibly. the
migration check could just check all (relevant) nodes.
not saying I'm opposed to the "one json file" approach per se, but it
would be nice to weigh the pros and cons before deviating from our usual
approach to config files.
>
> it also adds some helpers for the api schema & asserting that the
> device mappings are valid (by checking the saved properties
> against the ones found on the current available devices)
>
> Signed-off-by: Dominik Csapak
> ---
> src/Makefile | 1 +
> src/PVE/HardwareMap.pm | 363 +
> 2 files changed, 364 insertions(+)
> create mode 100644 src/PVE/HardwareMap.pm
>
> diff --git a/src/Makefile b/src/Makefile
> index 13de6c6..8527704 100644
> --- a/src/Makefile
> +++ b/src/Makefile
> @@ -17,6 +17,7 @@ LIB_SOURCES = \
> Daemon.pm \
> Exception.pm \
> Format.pm \
> + HardwareMap.pm \
> INotify.pm \
> JSONSchema.pm \
> LDAP.pm \
> diff --git a/src/PVE/HardwareMap.pm b/src/PVE/HardwareMap.pm
> new file mode 100644
> index 000..1b94abc
> --- /dev/null
> +++ b/src/PVE/HardwareMap.pm
> @@ -0,0 +1,363 @@
> +package PVE::HardwareMap;
> +
> +use strict;
> +use warnings;
> +
> +use Digest::SHA;
> +use JSON;
> +use Storable qw(dclone);
> +
> +use PVE::Cluster qw(cfs_register_file cfs_read_file cfs_write_file
> cfs_lock_file);
> +use PVE::INotify;
> +use PVE::JSONSchema qw(get_standard_option);
> +use PVE::SysFSTools;
> +
> +use base qw(Exporter);
> +
> +our @EXPORT_OK = qw(find_device_on_current_node);
> +
> +my $FILENAME = "nodes/hardware-map.conf";
> +cfs_register_file($FILENAME, \&read_hardware_map, \&write_hardware_map);
> +
> +# a mapping format per type
> +my $format = {
> +usb => {
> + vendor => {
> + description => "The vendor ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + },
> + device => {
> + description => "The device ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + },
> + 'subsystem-vendor' => {
> + description => "The subsystem vendor ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + optional => 1,
> + },
> + 'subsystem-device' => {
> + description => "The subsystem device ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + optional => 1,
> + },
> + path => {
> + description => "The path to the usb device.",
> + type => 'string',
> + optional => 1,
> + pattern => qr/^(\d+)\-(\d+(\.\d+)*)$/,
> + },
> + comment => {
> + description => "Description.",
> + type => 'string',
> + optional => 1,
> + maxLength => 4096,
> + },
> +},
> +pci => {
> + vendor => {
> + description => "The vendor ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + },
> + device => {
> + description => "The device ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + },
> + 'subsystem-vendor' => {
> + description => "The subsystem vendor ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + optional => 1,
> + },
> + 'subsystem-device' => {
> + description => "The subsystem device ID",
> + type => 'string',
> + pattern => qr/^(:?0x)?[0-9A-Fa-f]{4}$/,
> + optional => 1,
> + },
> + path => {
> + description => "The path to the device. If the function is omitted,
> the whole device is mapped. In that case use the attrubes of the first
> device.",
> + type => 'string',
> + pattern =>
> "(?:[a-f0-9]{4,}:)?[a-f0-9]{2}:[a-f0-9]{2}(?:\.[a-f0-9])?",
> + }
Re: [pve-devel] [PATCH qemu-server 2/7] PVE/QemuServer: allow mapped pci deviced in config
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> and get the correct pci device during parsing
>
> Signed-off-by: Dominik Csapak
> ---
> PVE/QemuServer/PCI.pm | 20 ++--
> 1 file changed, 18 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer/PCI.pm b/PVE/QemuServer/PCI.pm
> index 23fe508..1ff4bce 100644
> --- a/PVE/QemuServer/PCI.pm
> +++ b/PVE/QemuServer/PCI.pm
> @@ -4,6 +4,7 @@ use warnings;
> use strict;
>
> use PVE::JSONSchema;
> +use PVE::HardwareMap;
> use PVE::SysFSTools;
> use PVE::Tools;
>
> @@ -23,8 +24,8 @@ my $hostpci_fmt = {
> host => {
> default_key => 1,
> type => 'string',
> - pattern => qr/$PCIRE(;$PCIRE)*/,
> - format_description => 'HOSTPCIID[;HOSTPCIID2...]',
> + pattern => qr/(:?$PCIRE(;$PCIRE)*)|(:?$PVE::JSONSchema::CONFIGID_RE)/,
> + format_description => 'HOSTPCIID[;HOSTPCIID2...] or configured mapping
> id',
> description => < Host PCI device pass through. The PCI ID of a host's PCI device or a list
> of PCI virtual functions of the host. HOSTPCIID syntax is:
> @@ -32,6 +33,8 @@ of PCI virtual functions of the host. HOSTPCIID syntax is:
> 'bus:dev.func' (hexadecimal numbers)
>
> You can us the 'lspci' command to list existing PCI devices.
> +
> +Alternatively use the ID of a mapped pci device.
> EODESCR
> },
> rombar => {
> @@ -383,6 +386,19 @@ sub parse_hostpci {
>
> my $res = PVE::JSONSchema::parse_property_string($hostpci_fmt, $value);
>
> +if ($res->{host} !~ m/:/) {
> + # we have no ordinary pci id, must be a mapping
> + my $device = PVE::HardwareMap::find_device_on_current_node('pci',
> $res->{host});
> + die "PCI device mapping not found for '$res->{host}'\n" if
> !defined($device);
> + eval {
> + PVE::HardwareMap::assert_device_valid('pci', $device);
> + };
> + if (my $err = $@) {
> + die "PCI device mapping invalid (hardware probably changed):
> $err\n";
here we die, with USB we warn.. OTOH this gets called way less than the
USB counter part, since we mostly just parse the property string without
the extra checks for PCI.. might make sense to unify the approach taken.
> + }
> + $res->{host} = $device->{path};
> +}
> +
> my @idlist = split(/;/, $res->{host});
> delete $res->{host};
> foreach my $id (@idlist) {
> --
> 2.30.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 1/7] PVE/QemuServer: allow mapped usb devices in config
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> Signed-off-by: Dominik Csapak
> ---
> PVE/QemuServer.pm | 2 ++
> PVE/QemuServer/USB.pm | 21 -
> 2 files changed, 22 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 7d9cf22..a6ca80d 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -1069,6 +1069,8 @@ The Host USB device or port or the value 'spice'.
> HOSTUSBDEVICE syntax is:
>
> You can use the 'lsusb -t' command to list existing usb devices.
>
> +Alternatively, you can used an ID of a mapped usb device.
> +
> NOTE: This option allows direct access to host hardware. So it is no longer
> possible to migrate such
> machines - use with special care.
>
> diff --git a/PVE/QemuServer/USB.pm b/PVE/QemuServer/USB.pm
> index 3c8da2c..279078a 100644
> --- a/PVE/QemuServer/USB.pm
> +++ b/PVE/QemuServer/USB.pm
> @@ -4,6 +4,7 @@ use strict;
> use warnings;
> use PVE::QemuServer::PCI qw(print_pci_addr);
> use PVE::JSONSchema;
> +use PVE::HardwareMap;
> use base 'Exporter';
>
> our @EXPORT_OK = qw(
> @@ -27,7 +28,25 @@ sub parse_usb_device {
> } elsif ($value =~ m/^spice$/i) {
> $res->{spice} = 1;
> } else {
> - return;
> + # we have no ordinary usb device, must be a mapping
> + my $device = PVE::HardwareMap::find_device_on_current_node('usb',
> $value);
> + return undef if !defined($device);
> + eval {
> + PVE::HardwareMap::assert_device_valid('usb', $device);
> + };
> + if (my $err = $@) {
> + warn "USB Mapping invalid (hardware probably changed): $err\n";
> + return;
> + }
nit: this would also trigger on deleting such an invalid mapping, which is a
bit strange?
> +
> + if ($device->{path}) {
> + $res = parse_usb_device($device->{path});
> + } else {
> + $res->{vendorid} = $device->{vendor};
> + $res->{productid} = $device->{device};
> + }
> +
> + $res->{mapped} = 1;
> }
>
> return $res;
> --
> 2.30.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 3/7] PVE/API2/Qemu: add permission checks for mapped usb devices
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> Signed-off-by: Dominik Csapak
> ---
> PVE/API2/Qemu.pm | 39 ---
> 1 file changed, 36 insertions(+), 3 deletions(-)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index 99b426e..aa7ddea 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -26,6 +26,7 @@ use PVE::QemuServer::Drive;
> use PVE::QemuServer::ImportDisk;
> use PVE::QemuServer::Monitor qw(mon_cmd);
> use PVE::QemuServer::Machine;
> +use PVE::QemuServer::USB qw(parse_usb_device);
> use PVE::QemuMigrate;
> use PVE::RPCEnvironment;
> use PVE::AccessControl;
> @@ -567,8 +568,12 @@ my $check_vm_create_usb_perm = sub {
>
> foreach my $opt (keys %{$param}) {
> next if $opt !~ m/^usb\d+$/;
> + my $device = parse_usb_device($param->{$opt});
>
> - if ($param->{$opt} =~ m/spice/) {
> + if ($device->{spice}) {
> + $rpcenv->check_vm_perm($authuser, $vmid, $pool,
> ['VM.Config.HWType']);
> + } elsif ($device->{mapped}) {
> + $rpcenv->check_hw_perm($authuser, $device->{host},
> ['Hardware.Use']);
maybe I am overlooking something, but where does $device->{host} come
from?
parse_usb_device (for a mapped USB device) looks up device in the
hardware map, asserts it's valid (for the local node), and then either
returns
{
vendorid => $map->{vendor},
productid => $map->{device},
mapped => 1,
}
or the result of parse_usb_device($map->{path}), with 'mapped' set.
since the lookup in the map doesn't set a 'host' member, wouldn't
$device->{host} always be undef for mapped devices? maybe this was
wrongly copied from the PCI code, where the hostpci property string has
a 'host' property (that with this series, also possibly contains a
mapping entry ID)? or is this supposed to parse the property string, and
use the host property from there?
> $rpcenv->check_vm_perm($authuser, $vmid, $pool,
> ['VM.Config.HWType']);
> } else {
> die "only root can set '$opt' config for real devices\n";
> @@ -1552,7 +1557,12 @@ my $update_vm_api = sub {
> PVE::QemuConfig->add_to_pending_delete($conf, $opt, $force);
> PVE::QemuConfig->write_config($vmid, $conf);
> } elsif ($opt =~ m/^usb\d+$/) {
> - if ($val =~ m/spice/) {
> + my $device = PVE::QemuServer::USB::parse_usb_device($val);
> + my $host = parse_usb_device($device->{host});
> + if ($host->{spice}) {
> + $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> + } elsif ($host->{mapped}) {
> + $rpcenv->check_hw_perm($authuser, $device->{host},
> ['Hardware.Use']);
same question here..
> $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> } elsif ($authuser ne 'root@pam') {
> die "only root can delete '$opt' config for real
> devices\n";
> @@ -1613,7 +1623,30 @@ my $update_vm_api = sub {
> }
> $conf->{pending}->{$opt} = $param->{$opt};
> } elsif ($opt =~ m/^usb\d+/) {
> - if ((!defined($conf->{$opt}) || $conf->{$opt} =~ m/spice/)
> && $param->{$opt} =~ m/spice/) {
> + my $olddevice;
> + my $oldhost;
> + if (defined($conf->{$opt})) {
> + $olddevice =
> PVE::QemuServer::USB::parse_usb_device($conf->{$opt});
> + $oldhost = parse_usb_device($olddevice->{host});
and here
> + }
> + if (defined($oldhost)) {
> + if ($oldhost->{spice}) {
> + $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> + } elsif ($oldhost->{mapped}) {
> + $rpcenv->check_hw_perm($authuser,
> $olddevice->{host}, ['Hardware.Use']);
and here
> + $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> + } elsif ($authuser ne 'root@pam') {
> + die "only root can modify '$opt' config for real
> devices\n";
> + }
> + }
> +
> + my $newdevice =
> PVE::QemuServer::USB::parse_usb_device($param->{$opt});
> + my $newhost = parse_usb_device($newdevice->{host});
and here
> +
> + if ($newhost->{spice}) {
> + $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> + } elsif ($newhost->{mapped}) {
> + $rpcenv->check_hw_perm($authuser, $newdevice->{host},
> ['Hardware.Use']);
and here
> $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> } elsif ($authuser ne 'root@pam') {
> die "on
Re: [pve-devel] [PATCH qemu-server 4/7] PVE/API2/Qemu: add permission checks for mapped pci devices
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> Signed-off-by: Dominik Csapak
> ---
> PVE/API2/Qemu.pm | 54 ++--
> 1 file changed, 52 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index aa7ddea..a8029c2 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -26,6 +26,7 @@ use PVE::QemuServer::Drive;
> use PVE::QemuServer::ImportDisk;
> use PVE::QemuServer::Monitor qw(mon_cmd);
> use PVE::QemuServer::Machine;
> +use PVE::QemuServer::PCI;
> use PVE::QemuServer::USB qw(parse_usb_device);
> use PVE::QemuMigrate;
> use PVE::RPCEnvironment;
> @@ -583,6 +584,26 @@ my $check_vm_create_usb_perm = sub {
> return 1;
> };
>
> +my $check_vm_create_hostpci_perm = sub {
> +my ($rpcenv, $authuser, $vmid, $pool, $param) = @_;
> +
> +return 1 if $authuser eq 'root@pam';
> +
> +foreach my $opt (keys %{$param}) {
> + next if $opt !~ m/^hostpci\d+$/;
> +
> + my $device = PVE::JSONSchema::parse_property_string('pve-qm-hostpci',
> $param->{$opt});
> + if ($device->{host} !~ m/:/) {
> + $rpcenv->check_hw_perm($authuser, $device->{host},
> ['Hardware.Use']);
> + $rpcenv->check_vm_perm($authuser, $vmid, $pool,
> ['VM.Config.HWType']);
> + } else {
> + die "only root can set '$opt' config for non-mapped devices\n";
> + }
> +}
> +
> +return 1;
> +};
> +
> my $check_vm_modify_config_perm = sub {
> my ($rpcenv, $authuser, $vmid, $pool, $key_list) = @_;
>
> @@ -593,7 +614,7 @@ my $check_vm_modify_config_perm = sub {
> # else, as there the permission can be value dependend
> next if PVE::QemuServer::is_valid_drivename($opt);
> next if $opt eq 'cdrom';
> - next if $opt =~ m/^(?:unused|serial|usb)\d+$/;
> + next if $opt =~ m/^(?:unused|serial|usb|hostpci)\d+$/;
>
>
> if ($cpuoptions->{$opt} || $opt =~ m/^numa\d+$/) {
> @@ -621,7 +642,7 @@ my $check_vm_modify_config_perm = sub {
> # also needs privileges on the storage, that will be checked later
> $rpcenv->check_vm_perm($authuser, $vmid, $pool, ['VM.Config.Disk',
> 'VM.PowerMgmt' ]);
> } else {
> - # catches hostpci\d+, args, lock, etc.
> + # catches args, lock, etc.
> # new options will be checked here
> die "only root can set '$opt' config\n";
> }
> @@ -856,6 +877,7 @@ __PACKAGE__->register_method({
>
> &$check_vm_create_serial_perm($rpcenv, $authuser, $vmid, $pool,
> $param);
> &$check_vm_create_usb_perm($rpcenv, $authuser, $vmid, $pool,
> $param);
> + &$check_vm_create_hostpci_perm($rpcenv, $authuser, $vmid, $pool,
> $param);
>
> &$check_cpu_model_access($rpcenv, $authuser, $param);
>
> @@ -1569,6 +1591,16 @@ my $update_vm_api = sub {
> }
> PVE::QemuConfig->add_to_pending_delete($conf, $opt, $force);
> PVE::QemuConfig->write_config($vmid, $conf);
> + } elsif ($opt =~ m/^hostpci\d+$/) {
> + my $olddevice =
> PVE::JSONSchema::parse_property_string('pve-qm-hostpci', $val);
> + if ($olddevice->{host} !~ m/:/) {
> + $rpcenv->check_hw_perm($authuser, $olddevice->{host},
> ['Hardware.Use']);
> + $rpcenv->check_vm_perm($authuser, $vmid, undef,
> ['VM.Config.HWType']);
> + } elsif ($authuser ne 'root@pam') {
> + die "only root can set '$opt' config for non-mapped
> devices\n";
> + }
> + PVE::QemuConfig->add_to_pending_delete($conf, $opt, $force);
> + PVE::QemuConfig->write_config($vmid, $conf);
> } else {
> PVE::QemuConfig->add_to_pending_delete($conf, $opt, $force);
> PVE::QemuConfig->write_config($vmid, $conf);
> @@ -1651,6 +1683,24 @@ my $update_vm_api = sub {
> } elsif ($authuser ne 'root@pam') {
> die "only root can modify '$opt' config for real
> devices\n";
> }
> +
> + $conf->{pending}->{$opt} = $param->{$opt};
> + } elsif ($opt =~ m/^hostpci\d+$/) {
> + my $olddevice;
> + if (defined($conf->{$opt})) {
> + $olddevice =
> PVE::JSONSchema::parse_property_string('pve-qm-hostpci', $conf->{$opt});
> + }
> + my $newdevice =
> PVE::JSONSchema::parse_property_string('pve-qm-hostpci', $param->{$opt});
> + if ((!defined($olddevice) || $olddevice->{host} !~ m/:/) &&
> $newdevice->{host} !~ m/:/) {
> + if (defined($olddevice)) {
> + $rpcenv->check_hw_perm($authuser,
> $olddevice->{host}, ['Hardware.Use']);
> + }
> + $rpcenv->check_hw_perm($authuser, $newdevice->{host},
> ['Hardware.Use']);
> + $rpcenv->che
Re: [pve-devel] [PATCH qemu-server 5/7] PVE/QemuServer: extend 'check_local_resources' for mapped resources
On July 19, 2022 1:46 pm, Dominik Csapak wrote:
> by adding them to their own list, saving the nodes where
> they are not allowed, and return those on 'wantarray'
so this seems to be the main place where we benefit from a cluster-wide
HW map - but it could just as well be a loop over the node specific
config files, possibly after reducing the node list via means of
storages..
and, for the call in migrate where we know the target node already we
could pass it here to reduce the amount of work we have to do ;)
>
> Signed-off-by: Dominik Csapak
> ---
> PVE/QemuServer.pm | 36 ++--
> 1 file changed, 34 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index a6ca80d..ea7e213 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -2617,6 +2617,21 @@ sub check_local_resources {
> my ($conf, $noerr) = @_;
>
> my @loc_res = ();
> +my $mapped_res = [];
> +
> +my $nodelist = PVE::Cluster::get_nodelist();
> +my $hw_map = PVE::HardwareMap::config();
> +
> +my $not_allowed_nodes = { map { $_ => [] } @$nodelist };
> +
> +my $add_not_allowed_nodes = sub {
> + my ($type, $key, $id) = @_;
> + for my $node (@$nodelist) {
> + if (!defined($id) || !defined($hw_map->{$type}->{$id}->{$node})) {
> + push @{$not_allowed_nodes->{$node}}, $key;
> + }
> + }
> +};
>
> push @loc_res, "hostusb" if $conf->{hostusb}; # old syntax
> push @loc_res, "hostpci" if $conf->{hostpci}; # old syntax
> @@ -2624,7 +2639,24 @@ sub check_local_resources {
> push @loc_res, "ivshmem" if $conf->{ivshmem};
>
> foreach my $k (keys %$conf) {
> - next if $k =~ m/^usb/ && ($conf->{$k} =~ m/^spice(?![^,])/);
> + if ($k =~ m/^usb/) {
> + my $entry = parse_property_string($usb_fmt, $conf->{$k});
> + my $usb = PVE::QemuServer::USB::parse_usb_device($entry->{host});
> + next if $usb->{spice};
> + if ($usb->{mapped}) {
> + $add_not_allowed_nodes->('usb', $k, $entry->{host});
> + push @$mapped_res, $k;
> + next;
> + }
> + }
> + if ($k =~ m/^hostpci/) {
> + my $entry = parse_property_string('pve-qm-hostpci', $conf->{$k});
> + if ($entry->{host} !~ m/:/) {
> + $add_not_allowed_nodes->('pci', $k, $entry->{host});
> + push @$mapped_res, $k;
> + next;
> + }
> + }
> # sockets are safe: they will recreated be on the target side
> post-migrate
> next if $k =~ m/^serial/ && ($conf->{$k} eq 'socket');
> push @loc_res, $k if $k =~ m/^(usb|hostpci|serial|parallel)\d+$/;
> @@ -2632,7 +2664,7 @@ sub check_local_resources {
>
> die "VM uses local resources\n" if scalar @loc_res && !$noerr;
>
> -return \@loc_res;
> +return wantarray ? (\@loc_res, $mapped_res, $not_allowed_nodes) :
> \@loc_res;
> }
>
> # check if used storages are available on all nodes (use by migrate)
> --
> 2.30.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH V3 pve-manager 1/2] fix #2822: add lvm, lvmthin & zfs storage for all cluster nodes
Hello :)
On 7/27/22 12:19, Fiona Ebner wrote:
Am 19.07.22 um 13:57 schrieb Stefan Hrdlicka:
This adds a dropdown box for LVM, LVMThin & ZFS storage options where a
cluster node needs to be chosen. As default the current node is
selected. It restricts the the storage to be only availabe on the
selected node.
Signed-off-by: Stefan Hrdlicka
---
www/manager6/storage/Base.js| 40 +
www/manager6/storage/IScsiEdit.js | 54 ++---
www/manager6/storage/LVMEdit.js | 27 +--
www/manager6/storage/LvmThinEdit.js | 42 +-
www/manager6/storage/ZFSPoolEdit.js | 32 ++---
5 files changed, 166 insertions(+), 29 deletions(-)
diff --git a/www/manager6/storage/Base.js b/www/manager6/storage/Base.js
index 7f6d7a09..28c5c9d0 100644
--- a/www/manager6/storage/Base.js
+++ b/www/manager6/storage/Base.js
@@ -36,6 +36,7 @@ Ext.define('PVE.panel.StorageBase', {
{
xtype: 'pveNodeSelector',
name: 'nodes',
+ reference: 'storageNodeRestriction',
disabled: me.storageId === 'local',
fieldLabel: gettext('Nodes'),
emptyText: gettext('All') + ' (' + gettext('No restrictions')
+')',
@@ -76,6 +77,45 @@ Ext.define('PVE.panel.StorageBase', {
},
});
+Ext.define('PVE.panel.StorageBaseWithNodeSelector', {
It's not a panel and not a StorageBaseWithNodeSelector? How about
PVE.form.ScanNodeSelector or StorageScanNodeSelector? If you go for the
latter, please adapt the xtype too to make it match.
+extend: 'PVE.form.NodeSelector',
+xtype: 'pveScanNodeSelector',
+
+name: 'node',
This is a too generic name. Again, how about storageScanNode?
+itemId: 'pveScanNodeSelector',
+fieldLabel: gettext('Scan node'),
+allowBlank: true,
I'd really like to see an emptyText with "localhost" or similar here, so
that it's clear what's being scanned if nothing is selected.
Alternatively, the local node should be explicitly preselected (but
without pre-selecting a node restriction to keep current default behavior).
+disallowedNodes: undefined,
+onlineValidator: true,
+autoSelect: false,
+submitValue: false,
+autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext('Scan for available storages on the selected
node'),
+},
+});
+
+Ext.define('PVE.storage.ComboBoxSetStoreNode', {
+extend: 'Ext.form.field.ComboBox',
+config: {
+ apiBaseUrl: '/api2/json/nodes/',
+ apiStoragePath: '',
Very minor style nit: this class doesn't really depend on anything being
a storage, so we could also call this 'apiSuffix' or something. Then the
name would still work for any future (not storage-related) re-use of the
class.
+},
+
+setNodeName: function(value) {
+ let me = this;
+ if (value === null || value === '') {
+ value = Proxmox.NodeName;
+ }
Could also use
value ||= Proxmox.NodeName;
+
+ let store = me.getStore();
+ let proxy = store.getProxy();
Style nit: no need for these single-use variables
+ proxy.setUrl(me.apiBaseUrl + value + me.apiStoragePath);
+ this.clearValue();
Style nit: can use 'me' again.
+},
+
+});
+
Ext.define('PVE.storage.BaseEdit', {
extend: 'Proxmox.window.Edit',
diff --git a/www/manager6/storage/IScsiEdit.js b/www/manager6/storage/IScsiEdit.js
index 2f35f882..272c42d9 100644
--- a/www/manager6/storage/IScsiEdit.js
+++ b/www/manager6/storage/IScsiEdit.js
@@ -1,5 +1,5 @@
Ext.define('PVE.storage.IScsiScan', {
-extend: 'Ext.form.field.ComboBox',
+extend: 'PVE.storage.ComboBoxSetStoreNode',
alias: 'widget.pveIScsiScan',
queryParam: 'portal',
@@ -10,6 +10,9 @@ Ext.define('PVE.storage.IScsiScan', {
loadingText: gettext('Scanning...'),
width: 350,
},
+config: {
+ apiStoragePath: '/scan/iscsi',
+},
doRawQuery: function() {
// do nothing
},
@@ -42,7 +45,7 @@ Ext.define('PVE.storage.IScsiScan', {
fields: ['target', 'portal'],
proxy: {
type: 'proxmox',
- url: `/api2/json/nodes/${me.nodename}/scan/iscsi`,
+ url: me.apiBaseUrl + me.nodename + me.apiStoragePath,
Style nit: please keep using template string syntax
Same for the other ones below
Exactly this one:
url: `/api2/json/nodes/${me.nodename}/scan/iscsi` or do you mean or
would something like this be ok as well:
url: `${me.apiBaseUrl}${me.nodename}${me.apiSuffix}`
},
});
store.sort('target', 'ASC');
@@ -77,8 +80,40 @@ Ext.define('PVE.storage.IScsiInputPanel', {
initComponent: function() {
var me = this;
- me.column1 = [
- {
+ me.column1 = [];
+ let target = null;
+ if (me.isCreate) {
+ target = Ext.createWidget('pveIScsiScan', {
+ readOnly: !me.isCreate,
+
Re: [pve-devel] qemu 7.0 : fleecing backup (aka: local temp write cache"
Le 1/08/22 à 10:31, Dietmar Maurer a écrit : >> Can I use a small local fast PBS instance without need to keep the full >> datastore chunks ? >> >> I have 300TB nvme in production, I don't want to buy 300TB nvme for backup. > no, sorry. > > Do you want to use Ceph as temp backup storage, or simply an additional (node > local) nvme? > Both methods are ok for me, I just need a fast storage somewhere on the primary datacenter. If I can reuse my ceph cluster, why not, don't need to buy new local nvme. local nvme could avoid to add load to the ceph cluster, but it's really marginal. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
