[pve-devel] [PATCH qemu-server] qemu_img_convert: add missing newline for progress output
which was accidentally removed by b5e9d97bdf8a63a542f8cbb3c1d0821ee731f796.
Signed-off-by: Fabian Ebner
---
PVE/QemuServer.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 8b9c40d..e768072 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -6958,7 +6958,7 @@ sub qemu_img_convert {
my $total_h = render_bytes($size, 1);
my $transferred_h = render_bytes($transferred, 1);
- print "transferred $transferred_h of $total_h ($percent%)";
+ print "transferred $transferred_h of $total_h ($percent%)\n";
}
};
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager] ui: set file name for spice console download in chrome
When the virt-viewer file is downloaded we already set
a file name in Android, so the file type may be
recognized. Also doing this in Chrome (and Chromium
based browsers) allows users to "alyways open files of
this type". So the browser automatically opens the
console window without user interaction.
Signed-off-by: Lorenz Stechauner
---
www/manager6/Utils.js | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index f502950f..35d49630 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -1358,11 +1358,10 @@ Ext.define('PVE.Utils', {
css: 'display:none;visibility:hidden;height:0px;',
});
- // Note: we need to tell android the correct file name extension
+ // Note: we need to tell Android and Chrome the correct file name
extension
// but we do not set 'download' tag for other environments, because
// It can have strange side effects (additional user prompt on
firefox)
- var andriod = !!navigator.userAgent.match(/Android/i);
- if (andriod) {
+ if (navigator.userAgent.match(/Android|Chrome/i)) {
link.download = name;
}
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager v2] ui: Utils: change default Console order for vms
On 22.04.21 08:39, Dominik Csapak wrote: > we want to use spice for vms more than xtermjs if both are available > (since spice must be chosen as display in that case) > so the resulting order of preference for vms is: > spice > xtermjs > novnc > > since all methods work for containers always, there we use > xtermjs by default, or what is chosen in the datacenter option > > Signed-off-by: Dominik Csapak > --- > changes from v1: > * fixed the default so that it is only 'vv' for vms, not other > types of consoles (shell, upgrade,..) > > www/manager6/Utils.js | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > applied, thanks! But wondering if form VMs noVNC should be the next in line, as long as the display property of the VM is not set to a serialX console. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] fix #3389: Skip CSRF token check also for API tokens on file upload
On 21.04.21 13:59, Lorenz Stechauner wrote: > On file upload, the check for CSRF tokens was already skipped > when performing user authentication.This now happens for API > tokens also. > > Signed-off-by: Lorenz Stechauner > --- > PVE/HTTPServer.pm | 34 +- > 1 file changed, 17 insertions(+), 17 deletions(-) > > applied, thanks! `git show -w` really helped reviewing that one ;-) ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] fix #3389: Skip CSRF token check also for API tokens on file upload
On April 21, 2021 1:59 pm, Lorenz Stechauner wrote:
> On file upload, the check for CSRF tokens was already skipped
> when performing user authentication.This now happens for API
> tokens also.
>
> Signed-off-by: Lorenz Stechauner
> ---
> PVE/HTTPServer.pm | 34 +-
> 1 file changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/PVE/HTTPServer.pm b/PVE/HTTPServer.pm
> index 64976c7c..63b8583e 100755
> --- a/PVE/HTTPServer.pm
> +++ b/PVE/HTTPServer.pm
> @@ -79,8 +79,8 @@ sub auth_handler {
>
> if ($require_auth) {
> if ($api_token) {
> + # returns tokenid actually
> $username = PVE::AccessControl::verify_token($api_token);
> - $rpcenv->set_user($username); #actually tokenid in this case
> } else {
> die "No ticket\n" if !$ticket;
>
> @@ -94,25 +94,25 @@ sub auth_handler {
> die "No ticket\n"
> if ($rel_uri ne '/access/tfa' || $method ne 'POST');
> }
> + }
>
> - $rpcenv->set_user($username);
> -
> - if ($method eq 'POST' && $rel_uri =~
> m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
> - my ($node, $storeid) = ($1, $2);
> - # we disable CSRF checks if $isUpload is set,
> - # to improve security we check user upload permission here
> - my $perm = { check => ['perm', "/storage/$storeid",
> ['Datastore.AllocateTemplate']] };
> - $rpcenv->check_api2_permissions($perm, $username, {});
> - $isUpload = 1;
> - }
> + $rpcenv->set_user($username);
>
> - # we skip CSRF check for file upload, because it is
> - # difficult to pass CSRF HTTP headers with native html forms,
> - # and it should not be necessary at all.
> - my $euid = $>;
> - PVE::AccessControl::verify_csrf_prevention_token($username, $token)
> - if !$isUpload && ($euid != 0) && ($method ne 'GET');
> + if ($method eq 'POST' && $rel_uri =~
> m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
> + my ($node, $storeid) = ($1, $2);
> + # we disable CSRF checks if $isUpload is set,
> + # to improve security we check user upload permission here
> + my $perm = { check => ['perm', "/storage/$storeid",
> ['Datastore.AllocateTemplate']] };
> + $rpcenv->check_api2_permissions($perm, $username, {});
> + $isUpload = 1;
> }
> +
> + # we skip CSRF check for file upload, because it is
> + # difficult to pass CSRF HTTP headers with native html forms,
> + # and it should not be necessary at all.
> + my $euid = $>;
> + PVE::AccessControl::verify_csrf_prevention_token($username, $token)
> + if !$isUpload && ($euid != 0) && ($method ne 'GET');
this verify_csrf_prevention_token call was never triggered for API
tokens before, on purpose - one of the design goals of API tokens was to
provide stateless API access without requiring round-trips like our
regular, ticket-based sessions do. CSRF-prevention also does not make
much sense outside of the browser context (whereas API tokens are for
automated access by non-browser clients).
with this change we'd now suddenly require CSRF tokens for API tokens as
well, and IIRC API tokens can't even get one of those (and even if they
could, none of the existing clients would know about that and could no
longer do any non-GET requests with tokens).
I think (but I haven't tested it!) that
- setting $isUpload for the API token case as well
- leaving the CSRF check limited to regular (non-API-token) authentication
should give us the desired effect without any side-effects. the actual
upload handling is in PVE::APIServer::AnyEvent and just seems to check
for $isUpload in the result of auth_handler.
> }
>
> return {
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied-series: [PATCH v3 pve-network 0/6] evpn && bgp improvements
On 21.04.21 23:49, Alexandre Derumier wrote: > - fix broken evpn tests since last commit > https://lists.proxmox.com/pipermail/pve-devel/2021-April/047521.html > > - add ebgp-multihop option > (replace > https://lists.proxmox.com/pipermail/pve-devel/2021-April/047547.html) > > > Changelog v2: > > - move mac address option from vnet to evpn zone(this is only need for evpn > anycast gateway) > - readd lost ip-forward,ip6-forward,arp-accept since subnet implementation > - fix ipv6 snat > - tests : add ipv6 and ipv4v6 tests > - tests: add an hetzner routed setup with mutiple /32 && a full /29 > > Changelog v3: > - increase controllerid max characters > > *** BLURB HERE *** > > Alexandre Derumier (6): > tests: fix evpn vrf > bgp: add ebgp_multihop option > zones: evpn: move vnet mac option to evpn zone plugin > zones: evpn: fix arp-accept && ip-forward + ipv6 snat > zones: simple: fix ip-forward && ipv6 snat > controllers: increase controllerid to 64 characters max applied series, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] fix #3389: Skip CSRF token check also for API tokens on file upload
On 22.04.21 10:05, Fabian Grünbichler wrote:
> On April 21, 2021 1:59 pm, Lorenz Stechauner wrote:
>> On file upload, the check for CSRF tokens was already skipped
>> when performing user authentication.This now happens for API
>> tokens also.
>>
>> Signed-off-by: Lorenz Stechauner
>> ---
>> PVE/HTTPServer.pm | 34 +-
>> 1 file changed, 17 insertions(+), 17 deletions(-)
>>
>> diff --git a/PVE/HTTPServer.pm b/PVE/HTTPServer.pm
>> index 64976c7c..63b8583e 100755
>> --- a/PVE/HTTPServer.pm
>> +++ b/PVE/HTTPServer.pm
>> @@ -79,8 +79,8 @@ sub auth_handler {
>>
>> if ($require_auth) {
>> if ($api_token) {
>> +# returns tokenid actually
>> $username = PVE::AccessControl::verify_token($api_token);
>> -$rpcenv->set_user($username); #actually tokenid in this case
>> } else {
>> die "No ticket\n" if !$ticket;
>>
>> @@ -94,25 +94,25 @@ sub auth_handler {
>> die "No ticket\n"
>> if ($rel_uri ne '/access/tfa' || $method ne 'POST');
>> }
>> +}
>>
>> -$rpcenv->set_user($username);
>> -
>> -if ($method eq 'POST' && $rel_uri =~
>> m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
>> -my ($node, $storeid) = ($1, $2);
>> -# we disable CSRF checks if $isUpload is set,
>> -# to improve security we check user upload permission here
>> -my $perm = { check => ['perm', "/storage/$storeid",
>> ['Datastore.AllocateTemplate']] };
>> -$rpcenv->check_api2_permissions($perm, $username, {});
>> -$isUpload = 1;
>> -}
>> +$rpcenv->set_user($username);
>>
>> -# we skip CSRF check for file upload, because it is
>> -# difficult to pass CSRF HTTP headers with native html forms,
>> -# and it should not be necessary at all.
>> -my $euid = $>;
>> -PVE::AccessControl::verify_csrf_prevention_token($username, $token)
>> -if !$isUpload && ($euid != 0) && ($method ne 'GET');
>> +if ($method eq 'POST' && $rel_uri =~
>> m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
>> +my ($node, $storeid) = ($1, $2);
>> +# we disable CSRF checks if $isUpload is set,
>> +# to improve security we check user upload permission here
>> +my $perm = { check => ['perm', "/storage/$storeid",
>> ['Datastore.AllocateTemplate']] };
>> +$rpcenv->check_api2_permissions($perm, $username, {});
>> +$isUpload = 1;
>> }
>> +
>> +# we skip CSRF check for file upload, because it is
>> +# difficult to pass CSRF HTTP headers with native html forms,
>> +# and it should not be necessary at all.
>> +my $euid = $>;
>> +PVE::AccessControl::verify_csrf_prevention_token($username, $token)
>> +if !$isUpload && ($euid != 0) && ($method ne 'GET');
>
> this verify_csrf_prevention_token call was never triggered for API
> tokens before, on purpose - one of the design goals of API tokens was to
> provide stateless API access without requiring round-trips like our
> regular, ticket-based sessions do. CSRF-prevention also does not make
> much sense outside of the browser context (whereas API tokens are for
> automated access by non-browser clients).
>
> with this change we'd now suddenly require CSRF tokens for API tokens as
> well, and IIRC API tokens can't even get one of those (and even if they
> could, none of the existing clients would know about that and could no
> longer do any non-GET requests with tokens).
>
> I think (but I haven't tested it!) that
> - setting $isUpload for the API token case as well
> - leaving the CSRF check limited to regular (non-API-token) authentication
>
> should give us the desired effect without any side-effects. the actual
> upload handling is in PVE::APIServer::AnyEvent and just seems to check
> for $isUpload in the result of auth_handler.
>
you're right, good catch! @Lorenz, care to send out a patch bringing it
back it shape?
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] applied: [PATCH manager v2] ui: Utils: change default Console order for vms
On 4/22/21 10:02, Thomas Lamprecht wrote: On 22.04.21 08:39, Dominik Csapak wrote: we want to use spice for vms more than xtermjs if both are available (since spice must be chosen as display in that case) so the resulting order of preference for vms is: spice xtermjs novnc since all methods work for containers always, there we use xtermjs by default, or what is chosen in the datacenter option Signed-off-by: Dominik Csapak --- changes from v1: * fixed the default so that it is only 'vv' for vms, not other types of consoles (shell, upgrade,..) www/manager6/Utils.js | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) applied, thanks! But wondering if form VMs noVNC should be the next in line, as long as the display property of the VM is not set to a serialX console. could be nice, but would require adaptation in the status api call (we only return if a serial terminal is there, not the vm config) ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH docs 1/2] vzdump: add section about live-restore
Signed-off-by: Stefan Reiter --- As discussed off-list, the vzdump section in general seems a bit misorganized, and is certainly missing some references to PBS. Dylan agreed to take a look at it, thanks again for that! vzdump.adoc | 28 1 file changed, 28 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 9453684..0577a97 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -337,6 +337,34 @@ per configured storage, this can be done with: # pvesm set STORAGEID --bwlimit restore=KIBs +Live-Restore + + +Restoring a large backup can take a long time, in which a guest is still +unavailable. For VM backups stored on a Proxmox Backup Server, this wait +time can be mitigated using the live-restore option. + +Enabling live-restore via either the checkbox in the GUI or the `--live-restore` +property for `qmrestore` causes the VM to start immediately as the restore +begins. Data is copied in the background, chunks that the VM is actively +accessing are prioritized. + +Note that this comes with two caveats: + +* During live-restore, the VM will operate with limited disk read speeds, as + data has to be loaded from the backup server (once loaded it is immediately + available on the destination storage however, so accessing data twice only + incurs the penalty the first time). Write speeds are largely unaffected. +* If the live-restore fails for any reason, the VM will be left in an + unspecified state - that is, not all data might have been copied from the + backup, and it is _most likely_ not possible to keep any data that was written + during the failed restore operation. + +This mode of operation is especially useful for large VMs, where only a small +amount of data is required for initial operation, e.g. web servers - once the OS +and necessary services are started, the VM is already operational, while the +background task keeps copying seldomly used data. + [[vzdump_configuration]] Configuration - -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH docs 2/2] vzdump: add section about single file restore
Signed-off-by: Stefan Reiter --- vzdump.adoc | 24 1 file changed, 24 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 0577a97..c6063ad 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -365,6 +365,30 @@ amount of data is required for initial operation, e.g. web servers - once the OS and necessary services are started, the VM is already operational, while the background task keeps copying seldomly used data. +Single File Restore +~~~ + +The 'File Restore' button in the 'Backups' tab of the storage GUI can be used to +open a file browser directly on the data contained in a backup. This feature +is only available for backups on a Proxmox Backup Server. + +For containers, the first layer of the file tree shows all included 'pxar' +archives, which can be opened and browsed freely. For VMs, the first layer shows +contained drive images, which can be opened to reveal a list of supported +storage technologies found on the drive. In the most basic case, this will be an +entry called 'part', representing a partition table, containing entries for each +partition found on the drive. Note that for VMs, not all data might be +accessible (unsupported guest file systems, storage technologies, etc...). + +Files and directories can be downloaded using the 'Download' button, the latter +being compressed into a zip archive on the fly. + +To enable secure access to VM images, which might contain untrusted data, a +temporary VM (not visible as a guest) is started. This does not mean that data +downloaded from such an archive is inherently safe, but avoids exposing the +hypervisor system to danger. The VM will stop itself after a timeout, the entire +process happens transparently from a users point of view. + [[vzdump_configuration]] Configuration - -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 qemu-server] fix #3369: auto-start vm after failed stopmode backup
On April 20, 2021 3:14 pm, Dylan Whyte wrote:
> Fixes an issue in which a VM/CT fails to automatically restart after a
> failed stop-mode backup.
>
> Also fixes a minor typo in a comment
>
> Signed-off-by: Dylan Whyte
Reviewed-by: Fabian Grünbichler
> ---
>
> Note:
>
> v1->v2:
> - Fix the issue from within PVE::VZDump::QemuServer, rather than adding
> tedious sleep call and state checking in PVE::VZDump.
>
> PVE/VZDump/QemuServer.pm | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/VZDump/QemuServer.pm b/PVE/VZDump/QemuServer.pm
> index 8920ac1f..42a60fc7 100644
> --- a/PVE/VZDump/QemuServer.pm
> +++ b/PVE/VZDump/QemuServer.pm
> @@ -551,6 +551,7 @@ sub archive_pbs {
> if ($err) {
> $self->logerr($err);
> $self->mon_backup_cancel($vmid);
> + $self->resume_vm_after_job_start($task, $vmid);
> }
> $self->restore_vm_power_state($vmid);
>
> @@ -729,6 +730,7 @@ sub archive_vma {
> if ($err) {
> $self->logerr($err);
> $self->mon_backup_cancel($vmid);
> + $self->resume_vm_after_job_start($task, $vmid);
> }
>
> $self->restore_vm_power_state($vmid);
> @@ -815,7 +817,7 @@ sub enforce_vm_running_for_backup {
> die $@ if $@;
> }
>
> -# resume VM againe once we got in a clear state (stop mode backup of running
> VM)
> +# resume VM again once in a clear state (stop mode backup of running VM)
> sub resume_vm_after_job_start {
> my ($self, $task, $vmid) = @_;
>
> --
> 2.20.1
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager] http-server: fix for api token csrf token check
Do not check any csrf tokens for auth with api tokens.
Signed-off-by: Lorenz Stechauner
---
PVE/HTTPServer.pm | 12 +++-
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/PVE/HTTPServer.pm b/PVE/HTTPServer.pm
index bfb753eb..7a3bf72b 100755
--- a/PVE/HTTPServer.pm
+++ b/PVE/HTTPServer.pm
@@ -104,11 +104,13 @@ sub auth_handler {
$isUpload = 1;
}
- # we skip CSRF check for file upload, because it is difficult to pass
CSRF HTTP headers
- # with native html forms, and it should not be necessary at all.
- my $euid = $>;
- PVE::AccessControl::verify_csrf_prevention_token($username, $token)
- if !$isUpload && ($euid != 0) && ($method ne 'GET');
+ if (!$api_token) {
+ # we skip CSRF check for file upload, because it is difficult to
pass CSRF HTTP headers
+ # with native html forms, and it should not be necessary at all.
+ my $euid = $>;
+ PVE::AccessControl::verify_csrf_prevention_token($username, $token)
+ if !$isUpload && ($euid != 0) && ($method ne 'GET');
+ }
}
return {
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] http-server: fix for api token csrf token check
On 22.04.21 10:46, Lorenz Stechauner wrote: > Do not check any csrf tokens for auth with api tokens. > > Signed-off-by: Lorenz Stechauner > --- > PVE/HTTPServer.pm | 12 +++- > 1 file changed, 7 insertions(+), 5 deletions(-) > > applied, thanks! Added an actual comment explaining this and the why in a followup commit... ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 qemu-server] fix #3369: auto-start vm after failed stopmode backup
On 20.04.21 15:14, Dylan Whyte wrote: > Fixes an issue in which a VM/CT fails to automatically restart after a > failed stop-mode backup. > > Also fixes a minor typo in a comment > > Signed-off-by: Dylan Whyte > --- > > Note: > > v1->v2: > - Fix the issue from within PVE::VZDump::QemuServer, rather than adding > tedious sleep call and state checking in PVE::VZDump. > > PVE/VZDump/QemuServer.pm | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > applied, with Fabians R-b tag, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH qemu-server] qemu_img_convert: add missing newline for progress output
On 22.04.21 08:57, Fabian Ebner wrote: > which was accidentally removed by b5e9d97bdf8a63a542f8cbb3c1d0821ee731f796. > > Signed-off-by: Fabian Ebner > --- > PVE/QemuServer.pm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] ui: Parser: respect mtu in {parse, print}QemuNetwork
On 3/23/21 13:27, Dominik Csapak wrote:
Signed-off-by: Dominik Csapak
---
www/manager6/Parser.js | 5 +
1 file changed, 5 insertions(+)
diff --git a/www/manager6/Parser.js b/www/manager6/Parser.js
index b83a5d46..1a5fe1c2 100644
--- a/www/manager6/Parser.js
+++ b/www/manager6/Parser.js
@@ -142,6 +142,8 @@ Ext.define('PVE.Parser', {
res.queues = match_res[1];
} else if ((match_res =
p.match(/^trunks=(\d+(?:-\d+)?(?:;\d+(?:-\d+)?)*)$/)) !== null) {
res.trunks = match_res[1];
+ } else if ((match_res = p.match(/^mtu=(\d+)$/)) !== null) {
+ res.mtu = match_res[1];
} else {
errors = true;
return false; // break
@@ -181,6 +183,9 @@ Ext.define('PVE.Parser', {
if (net.trunks) {
netstr += ",trunks=" + net.trunks;
}
+ if (net.mtu) {
+ netstr += ",mtu=" + net.mtu;
+ }
return netstr;
},
any comment?
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] ui: Parser: respect mtu in {parse, print}QemuNetwork
On 23.03.21 13:27, Dominik Csapak wrote: > Signed-off-by: Dominik Csapak > --- > www/manager6/Parser.js | 5 + > 1 file changed, 5 insertions(+) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH docs 1/2] vzdump: add section about live-restore
Hi, I just made some very minor suggestions inline, but overall it's good (i had to search extra-hard for issues) :) Reviewed-by: Dylan Whyte On 4/22/21 10:25 AM, Stefan Reiter wrote: Signed-off-by: Stefan Reiter --- As discussed off-list, the vzdump section in general seems a bit misorganized, and is certainly missing some references to PBS. Dylan agreed to take a look at it, thanks again for that! vzdump.adoc | 28 1 file changed, 28 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 9453684..0577a97 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -337,6 +337,34 @@ per configured storage, this can be done with: # pvesm set STORAGEID --bwlimit restore=KIBs +Live-Restore + + +Restoring a large backup can take a long time, in which a guest is still +unavailable. For VM backups stored on a Proxmox Backup Server, this wait +time can be mitigated using the live-restore option. + +Enabling live-restore via either the checkbox in the GUI or the `--live-restore` +property for `qmrestore` causes the VM to start immediately as the restore s/property for/argument of/ s/immediately/as soon/ +begins. Data is copied in the background, chunks that the VM is actively s/, chunks/, and chunks/ or to reformulate: "Data is copied in the background, prioritizing chunks that the VM is actively accessing." +accessing are prioritized. + +Note that this comes with two caveats: + +* During live-restore, the VM will operate with limited disk read speeds, as + data has to be loaded from the backup server (once loaded it is immediately s/once loaded/once loaded,/ + available on the destination storage however, so accessing data twice only + incurs the penalty the first time). Write speeds are largely unaffected. +* If the live-restore fails for any reason, the VM will be left in an + unspecified state - that is, not all data might have been copied from the s/unspecified/undefined/ + backup, and it is _most likely_ not possible to keep any data that was written + during the failed restore operation. + +This mode of operation is especially useful for large VMs, where only a small +amount of data is required for initial operation, e.g. web servers - once the OS +and necessary services are started, the VM is already operational, while the s/are started/have been started/ s/already operational/operational/ +background task keeps copying seldomly used data. s/keeps/continues/ + [[vzdump_configuration]] Configuration - ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH docs 2/2] vzdump: add section about single file restore
Hi, Have even less notes for this one. Just some little things sounded off to me. Reviewed-by: Dylan Whyte On 4/22/21 10:25 AM, Stefan Reiter wrote: Signed-off-by: Stefan Reiter --- vzdump.adoc | 24 1 file changed, 24 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 0577a97..c6063ad 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -365,6 +365,30 @@ amount of data is required for initial operation, e.g. web servers - once the OS and necessary services are started, the VM is already operational, while the background task keeps copying seldomly used data. +Single File Restore +~~~ + +The 'File Restore' button in the 'Backups' tab of the storage GUI can be used to +open a file browser directly on the data contained in a backup. This feature +is only available for backups on a Proxmox Backup Server. + +For containers, the first layer of the file tree shows all included 'pxar' +archives, which can be opened and browsed freely. For VMs, the first layer shows +contained drive images, which can be opened to reveal a list of supported +storage technologies found on the drive. In the most basic case, this will be an +entry called 'part', representing a partition table, containing entries for each s/containing/which contains/ +partition found on the drive. Note that for VMs, not all data might be +accessible (unsupported guest file systems, storage technologies, etc...). + +Files and directories can be downloaded using the 'Download' button, the latter +being compressed into a zip archive on the fly. + +To enable secure access to VM images, which might contain untrusted data, a +temporary VM (not visible as a guest) is started. This does not mean that data +downloaded from such an archive is inherently safe, but avoids exposing the s/avoids/it avoids/ +hypervisor system to danger. The VM will stop itself after a timeout, the entire s/, the entire/. This entire/ +process happens transparently from a users point of view. s/users/user's/ + [[vzdump_configuration]] Configuration - ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH 00/10] Single-file-restore GUI for PBS snapshots
On Wed, Apr 21, 2021 at 01:15:29PM +0200, Stefan Reiter wrote: > Implements the necessary API for allowing single file restore via the PVE web > GUI. Restoring a couple of small text files and folders with textfiles from containers worked really well for me. For VMs I could also restore some text files & folders, but I didn't work as smooth. I tried both with a single .fidx/root.pxar so far. * This > proxmox-file-restore failed: Error: mounting 'drive-scsi0.img.fidx/part/1' > failed: all mounts failed or no supported file system (500) is correct but it would be nice if we could try to say something like "We found LVM stuff which is currently unsupported". Currently it looks more like bug than "expected error" to me. Maybe we could improve this (at some point in the future) so that the error does not block the whole panel and we can continue to select another part/number. Because when two out of three parts block the whole thing, then remembering the bad one(s), clicking X, clicking File restore->fidx->part->some number is not ideal IMO. * Sometimes the Download button remains enabled when errors (like the previous) happen and then you get a couple of 0 byte files * On the respective PBS sometimes this > 2021-04-22T11:31:01+02:00: starting new backup reader datastore 'test': > "/test" > 2021-04-22T11:31:01+02:00: protocol upgrade done > 2021-04-22T11:31:01+02:00: GET /download > 2021-04-22T11:31:01+02:00: download > "/test/vm/100/2021-04-22T09:29:17Z/index.json.blob" > 2021-04-22T11:31:01+02:00: GET /download > 2021-04-22T11:31:01+02:00: download > "/test/vm/100/2021-04-22T09:29:17Z/drive-scsi0.img.fidx" > 2021-04-22T11:31:01+02:00: register chunks in 'drive-scsi0.img.fidx' as > downloadable. > 2021-04-22T11:31:01+02:00: GET /chunk > 2021-04-22T11:31:01+02:00: download chunk > "/test/.chunks/394a/394a4b4c316868a466fbece11881752a10b31a782d2740ab414c3f48b5fb4e58" > 2021-04-22T11:31:01+02:00: GET /chunk > 2021-04-22T11:31:01+02:00: download chunk > "/test/.chunks/d628/d62885ad37c7a44a2fa9fdc558e85e89ecf4c9754617fab7e1f491fa3da65d38" appears and stays there forever and when I clicked Stop > 2021-04-22T11:47:00+02:00: TASK ERROR: task aborted it seemed to accept but the circle remained spinning forever again (OK, I only checked for a couple of minutes). * Sometimes the PBS also logged > 2021-04-22T11:42:40+02:00: starting new backup reader datastore 'test': > "/test" > 2021-04-22T11:42:40+02:00: protocol upgrade done > 2021-04-22T11:42:40+02:00: GET /download > 2021-04-22T11:42:40+02:00: download > "/test/ct/102/2021-04-22T09:39:23Z/index.json.blob" > 2021-04-22T11:42:40+02:00: GET /download > 2021-04-22T11:42:40+02:00: download > "/test/ct/102/2021-04-22T09:39:23Z/root.pxar.didx" > 2021-04-22T11:42:40+02:00: register chunks in 'root.pxar.didx' as > downloadable. > 2021-04-22T11:42:40+02:00: GET /chunk > 2021-04-22T11:42:40+02:00: download chunk > "/test/.chunks/ff63/ff63c49eae6b77e2933ea08af0dcf3786e74fc454ac2462ba780f0726c070023" > 2021-04-22T11:42:40+02:00: GET /chunk > 2021-04-22T11:42:40+02:00: download chunk > "/test/.chunks/261e/261ec247951ab57a51df87c593ae2aaf4f76c671bb94e13245f53ebdf25cfd45" > 2021-04-22T11:42:40+02:00: GET /chunk > 2021-04-22T11:42:40+02:00: download chunk > "/test/.chunks/3a60/3a60c4eb21123c01a63e981d8f550ba0f99b5e90b97340d384da9f079fa767ed" > 2021-04-22T11:42:40+02:00: TASK ERROR: connection error: Transport endpoint > is not connected (os error 107) but it looked to me like it didn't really matter? I could still download my text files. Note that when it started to log such messages, there were actually quite a lot of those entries. * A few times I got in the PVE GUI when I clicked on ...img.fidx > malformed JSON string, neither tag, array, object, number, string or atom, at > character offset 0 (before "VM 'qemu_root\\x40pa...") at > /usr/share/perl5/PVE/PBSClient.pm line 220. (500) and I couldn't find out yet why and later it just didn't appear again. * This is helpful > executable not found '/usr/bin/proxmox-file-restore'! Proxmox backup client > or file restore not installed? (500) but the precise package name would make it even more helpful for me. > proxmox-backup-file-restore Today I also tried around with existing options of restoring data with the backup client (CLI). And even with the previous concerns, I think that in comparison to those CLI possibilities this new feature makes exploring & restoring really, really convenient. Tested-by: Dominic Jäger ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 3/7] proxmox-boot-tool: sort and remove duplicates on clean
This is mostly in preparation for renaming pve-efiboot-uuids into
proxmox-boot-uuids, but can help in general (since each duplicate uuid
causes excessive disk i/o upon kernel upgrades).
Signed-off-by: Stoiko Ivanov
---
bin/proxmox-boot-tool | 4
1 file changed, 4 insertions(+)
diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
index 7b06d57..df80c73 100755
--- a/bin/proxmox-boot-tool
+++ b/bin/proxmox-boot-tool
@@ -195,6 +195,10 @@ clean() {
if [ -e "$ESP_LIST".tmp ]; then
mv "$ESP_LIST".tmp "$ESP_LIST"
fi
+
+ echo "Sorting and removing duplicate ESPs.."
+ sort -uo "$ESP_LIST".tmp "$ESP_LIST"
+ mv "$ESP_LIST".tmp "$ESP_LIST"
}
refresh() {
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta/pve-installer] boot ZFS on legacy BIOS systems from vfat
the pve-kernel-meta patches were prepared on top of the pve-kernel-5.4 branch
rfc->v1:
* Thanks to Fabian's great feedback this version handles a few cases I did not
think of for the RFC (e.g. an update to grub which runs grub-install and
removes the core.img/stage0 pointing to the ESP (instead of the zfs pool)
* pve-kernel-helper now adds a diversion (dpkg-divert(1)) to grub-install,
which should prevent users/grub-upgrades accidentally overwriting grub
with a config which tries to boot from the zpool
* both `update-grub` and the config it generates adds a banner/warning if
proxmox-boot is used (notifying the user where the correct place for editing
is)
* the renaming from pve-efiboot-tool to proxmox-boot-tool was carried further
to also include all hooks and snippets
* a first version of a patch for pve-installer was added (and very roughly
tested)
original cover-letter for the RFC:
This patchset has been long overdue, and complements the solution to booting
ZFS on UEFI systems using systemd-boot.
With the upgrade of ZFS 2.0.0 (and it's support for ZSTD compression), quite
a few users found out that their systems were still booted with legacy bios
boot and were consequently rendered unbootable with enabling zstd
compression on (a dataset on rpool).
The solution is inspired by our community-forum, especially @avw, and seems
rather lightweight (patch 2/2 is best viewed with '-w').
My first approach was to generate a working grub-config ourselves, but
given that grub has a few years of handling special cases - bind-mounting
the ESP on /boot and running 'update-grub' seems like a less painful way.
* patch 1/2 renames pve-efiboot-tool to proxmox-boot-tool (which seems more
appropriate by now)
* patch 2/2 adds support for installing grub appropriately on the ESPs
and running the kernel sync-logic in a way that update-grub feels fine
with
Sending as RFC, because this is a proof-of-concept and missing quite a few
things.
What works:
* installing this version on a root ZFS RAID-Z@ PVE (based on an old pre 6.2
install)
* reformatting all 4 ESPs `proxmox-boot-tool format /dev/sda2 --force)
* initializing them
* rebooting into 5.4.106 and zfs 2.0.0
* upgrading the pool, setting compression=zstd, writing a file, rebooting
(successfully)
* rebooting into an old 5.3 kernel - and getting greeted by busy-box instead
of grub-rescue
What's missing (at least):
* support in the installer
* the renaming is not quite through (the kernel-hooks are still containing
pve/efi in their name)
* testing the removal part of the$kernel-sync
pve-kernel-meta:
Stoiko Ivanov (7):
proxmox-boot-tool: rename from pve-efiboot-tool
proxmox-boot-tool: add status command
proxmox-boot-tool: sort and remove duplicates on clean
proxmox-boot: rename uuid list file
proxmox-boot-tool: handle legacy boot zfs installs
proxmox-boot: add grub.cfg header snippet
proxmox-boot: add grub-install wrapper
Makefile | 2 +-
bin/Makefile | 3 +-
bin/grub-install-wrapper | 12 +++
bin/{pve-efiboot-tool => proxmox-boot-tool} | 52 +---
debian/pve-kernel-helper.install | 5 +-
debian/pve-kernel-helper.links| 1 +
debian/pve-kernel-helper.postinst | 10 +++
debian/pve-kernel-helper.postrm | 14
debian/pve-kernel-helper.preinst | 9 +++
proxmox-boot/000_proxmox_boot_header | 22 +
{efiboot => proxmox-boot}/Makefile| 8 +-
{efiboot => proxmox-boot}/functions | 2 +-
.../proxmox-auto-removal | 0
.../proxmox-boot-sync | 2 +-
.../zz-proxmox-boot | 81 +--
15 files changed, 182 insertions(+), 41 deletions(-)
create mode 100755 bin/grub-install-wrapper
rename bin/{pve-efiboot-tool => proxmox-boot-tool} (88%)
create mode 100644 debian/pve-kernel-helper.links
create mode 100644 debian/pve-kernel-helper.postinst
create mode 100644 debian/pve-kernel-helper.postrm
create mode 100644 debian/pve-kernel-helper.preinst
create mode 100755 proxmox-boot/000_proxmox_boot_header
rename {efiboot => proxmox-boot}/Makefile (72%)
rename {efiboot => proxmox-boot}/functions (98%)
rename efiboot/pve-auto-removal => proxmox-boot/proxmox-auto-removal (100%)
rename efiboot/pve-efiboot-sync => proxmox-boot/proxmox-boot-sync (84%)
rename efiboot/zz-pve-efiboot => proxmox-boot/zz-proxmox-boot (69%)
pve-installer:
Stoiko Ivanov (1):
always boot zfs with proxmox-boot-tool
proxinstall | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 1/1] always boot zfs with proxmox-boot-tool
proxmox-boot-tool now supports configuring grub to boot from the ESPs
the installer creates on all (bootable) disks - use this approach
for zfs installs.
Signed-off-by: Stoiko Ivanov
---
best viewed with `git show -w`
proxinstall | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/proxinstall b/proxinstall
index 03f6ec7..3395fa2 100755
--- a/proxinstall
+++ b/proxinstall
@@ -1224,11 +1224,11 @@ my sub chroot_chmod {
die "chroot: unable to change permission mode for '$path'\n";
}
-sub prepare_systemd_boot_esp {
+sub prepare_proxmox_boot_esp {
my ($espdev, $targetdir) = @_;
-syscmd("chroot $targetdir pve-efiboot-tool init $espdev") == 0 ||
- die "unable to init ESP and install systemd-boot loader on '$espdev'\n";
+syscmd("chroot $targetdir proxmox-boot-tool init $espdev") == 0 ||
+ die "unable to init ESP and install proxmox-boot loader on '$espdev'\n";
}
sub prepare_grub_efi_boot_esp {
@@ -1820,19 +1820,19 @@ _EOD
foreach my $di (@$bootdevinfo) {
my $dev = $di->{devname};
- if (!$native_4k_disk_bootable) {
- eval {
- syscmd("chroot $targetdir /usr/sbin/grub-install
--target i386-pc --no-floppy --bootloader-id='proxmox' $dev") == 0 ||
- die "unable to install the i386-pc boot
loader on '$dev'\n";
- };
- push @$bootloader_err_list, $@ if $@;
- }
+ if ($use_zfs) {
+ prepare_proxmox_boot_esp($di->{esp}, $targetdir);
+ } else {
+ if (!$native_4k_disk_bootable) {
+ eval {
+ syscmd("chroot $targetdir
/usr/sbin/grub-install --target i386-pc --no-floppy --bootloader-id='proxmox'
$dev") == 0 ||
+ die "unable to install the i386-pc boot
loader on '$dev'\n";
+ };
+ push @$bootloader_err_list, $@ if $@;
+ }
eval {
if (my $esp = $di->{esp}) {
- if ($use_zfs) {
- prepare_systemd_boot_esp($esp, $targetdir);
- } else {
prepare_grub_efi_boot_esp($dev, $esp,
$targetdir);
}
}
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 7/7] proxmox-boot: add grub-install wrapper
if a (legacy) system is booted with proxmox-boot-tool, running
`grub-install` without being aware of the fact can render the system
unbootable (e.g. when letting the early stage point to an incompatible
zpool instead of the ESP).
To prevent this we add a dpkg-diversion [0], which simply checks if
`proxmox-boot-tool status` indicates that proxmox-boot is used and
errors out in that case, and runs the actual grub-install else.
Signed-off-by: Stoiko Ivanov
---
bin/Makefile | 1 +
bin/grub-install-wrapper | 12
bin/proxmox-boot-tool| 2 +-
debian/pve-kernel-helper.install | 1 +
debian/pve-kernel-helper.postrm | 14 ++
debian/pve-kernel-helper.preinst | 9 +
6 files changed, 38 insertions(+), 1 deletion(-)
create mode 100755 bin/grub-install-wrapper
create mode 100644 debian/pve-kernel-helper.postrm
create mode 100644 debian/pve-kernel-helper.preinst
diff --git a/bin/Makefile b/bin/Makefile
index b78fa42..2e18342 100644
--- a/bin/Makefile
+++ b/bin/Makefile
@@ -6,6 +6,7 @@ all:
install:
install -d ${SBINDIR}
install -m 0755 proxmox-boot-tool ${SBINDIR}/
+ install -m 0755 grub-install-wrapper ${SBINDIR}/grub-install
.PHONY: clean distclean
distclean:
diff --git a/bin/grub-install-wrapper b/bin/grub-install-wrapper
new file mode 100755
index 000..d9ef9cc
--- /dev/null
+++ b/bin/grub-install-wrapper
@@ -0,0 +1,12 @@
+#! /bin/sh
+set -e
+
+. /usr/share/pve-kernel-helper/scripts/functions
+
+if proxmox-boot-tool status; then
+ warn "grub-install is disabled because this system is booted via
proxmox-boot-tool, if you really need to run it, run
/usr/sbin/grub-install.real"
+ exit 1
+else
+ grub-install.real "$@"
+fi
+
diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
index 8baa577..bf9af7c 100755
--- a/bin/proxmox-boot-tool
+++ b/bin/proxmox-boot-tool
@@ -161,7 +161,7 @@ init() {
mv "$esp_mp/$PMX_LOADER_CONF.tmp" "$esp_mp/$PMX_LOADER_CONF"
else
echo "Installing grub i386-pc target.."
- grub-install \
+ grub-install.real \
--boot-directory $esp_mp \
--target i386-pc \
--no-floppy \
diff --git a/debian/pve-kernel-helper.install b/debian/pve-kernel-helper.install
index aae9494..e410ce7 100644
--- a/debian/pve-kernel-helper.install
+++ b/debian/pve-kernel-helper.install
@@ -2,4 +2,5 @@ etc/kernel/postinst.d/*
etc/kernel/postrm.d/*
etc/initramfs/post-update.d/proxmox-boot-sync
usr/sbin/proxmox-boot-tool
+usr/sbin/grub-install
usr/share/pve-kernel-helper/scripts/functions
diff --git a/debian/pve-kernel-helper.postrm b/debian/pve-kernel-helper.postrm
new file mode 100644
index 000..98e5550
--- /dev/null
+++ b/debian/pve-kernel-helper.postrm
@@ -0,0 +1,14 @@
+#! /bin/sh
+
+set -e
+
+if [ remove = "$1" ] || [ abort-install = "$1" ] || [ disappear = "$1" ]; then
+ dpkg-divert --package pve-kernel-helper --remove --rename \
+ --divert /usr/sbin/grub-install.real /usr/sbin/grub-install
+fi
+
+if [ abort-upgrade = "$1" ] && dpkg --compare-versions "$2" lt 6.3-9; then
+ dpkg-divert --package pve-kernel-helper --remove --rename \
+ --divert /usr/sbin/grub-install.real /usr/sbin/grub-install
+fi
+
diff --git a/debian/pve-kernel-helper.preinst b/debian/pve-kernel-helper.preinst
new file mode 100644
index 000..d0a8f67
--- /dev/null
+++ b/debian/pve-kernel-helper.preinst
@@ -0,0 +1,9 @@
+#! /bin/sh
+
+set -e
+
+if [ upgrade != "$1" ] || dpkg --compare-versions "$2" lt 6.3-9; then
+ dpkg-divert --package pve-kernel-helper --add --rename \
+ --divert /usr/sbin/grub-install.real /usr/sbin/grub-install
+fi
+
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 5/7] proxmox-boot-tool: handle legacy boot zfs installs
This patch adds support for booting non-uefi/legacy/bios-boot ZFS
installs, by using proxmox-boot-tool to copy the kernels to the ESP
and then generate a fitting grub config for booting from the vfat ESP:
* grub is installed onto the ESP and the MBR points to the ESP
* after copying/deleting the kernels proxmox-boot-tool bindmounts the
ESP on /boot (inside the new mount namespace)
* grub-update then manages to generate a fitting config.
Some paths/sanity-checks needed adaptation (to differentiate between
EFI boot and not (based on the existence of /sys/firmware/efi)
The arguments for grub-install are taken from the pve-installer.
The approach is inspired by @avw in our community-forum [0].
[0]
https://forum.proxmox.com/threads/zfs-error-no-such-device-error-unknown-filesystem-entering-rescue-mode.75122/post-374799
Signed-off-by: Stoiko Ivanov
---
best viewed with `git show -w`
bin/proxmox-boot-tool| 26
proxmox-boot/zz-proxmox-boot | 81 +---
2 files changed, 75 insertions(+), 32 deletions(-)
diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
index df80c73..8baa577 100755
--- a/bin/proxmox-boot-tool
+++ b/bin/proxmox-boot-tool
@@ -150,14 +150,24 @@ init() {
echo "Mounting '$part' on '$esp_mp'."
mount -t vfat "$part" "$esp_mp"
- echo "Installing systemd-boot.."
- mkdir -p "$esp_mp/$PMX_ESP_DIR"
- bootctl --path "$esp_mp" install
-
- echo "Configuring systemd-boot.."
- echo "timeout 3" > "$esp_mp/$PMX_LOADER_CONF.tmp"
- echo "default proxmox-*" >> "$esp_mp/$PMX_LOADER_CONF.tmp"
- mv "$esp_mp/$PMX_LOADER_CONF.tmp" "$esp_mp/$PMX_LOADER_CONF"
+ if [ -d /sys/firmware/efi ]; then
+ echo "Installing systemd-boot.."
+ mkdir -p "$esp_mp/$PMX_ESP_DIR"
+ bootctl --path "$esp_mp" install
+
+ echo "Configuring systemd-boot.."
+ echo "timeout 3" > "$esp_mp/$PMX_LOADER_CONF.tmp"
+ echo "default proxmox-*" >> "$esp_mp/$PMX_LOADER_CONF.tmp"
+ mv "$esp_mp/$PMX_LOADER_CONF.tmp" "$esp_mp/$PMX_LOADER_CONF"
+ else
+ echo "Installing grub i386-pc target.."
+ grub-install \
+ --boot-directory $esp_mp \
+ --target i386-pc \
+ --no-floppy \
+ --bootloader-id='proxmox' \
+ "/dev/$PKNAME"
+ fi
echo "Unmounting '$part'."
umount "$part"
diff --git a/proxmox-boot/zz-proxmox-boot b/proxmox-boot/zz-proxmox-boot
index 1c4ad73..019f3ad 100755
--- a/proxmox-boot/zz-proxmox-boot
+++ b/proxmox-boot/zz-proxmox-boot
@@ -76,18 +76,30 @@ update_esp_func() {
{ warn "creation of mountpoint ${mountpoint} failed -
skipping"; return; }
mount "${path}" "${mountpoint}" || \
{ warn "mount of ${path} failed - skipping"; return; }
- if [ ! -f "${mountpoint}/$PMX_LOADER_CONF" ]; then
- warn "${path} contains no loader.conf - skipping"
- return
- fi
- if [ ! -d "${mountpoint}/$PMX_ESP_DIR" ]; then
- warn "${path}/$PMX_ESP_DIR does not exist- skipping"
+ if [ -d /sys/firmware/efi ]; then
+ if [ ! -f "${mountpoint}/$PMX_LOADER_CONF" ]; then
+ warn "${path} contains no loader.conf - skipping"
+ return
+ fi
+ if [ ! -d "${mountpoint}/$PMX_ESP_DIR" ]; then
+ warn "${path}/$PMX_ESP_DIR does not exist- skipping"
+ return
+ fi
+ elif [ ! -d "${mountpoint}/grub" ]; then
+ warn "${path} contains no grub directory - skipping"
return
fi
-
warn "Copying and configuring kernels on ${path}"
copy_and_config_kernels "${mountpoint}"
- remove_old_kernels "${mountpoint}"
+ if [ -d /sys/firmware/efi ]; then
+ remove_old_kernels_efi "${mountpoint}"
+ else
+ remove_old_kernels_legacy "${mountpoint}"
+ mount --bind "${mountpoint}" "/boot"
+ update-grub
+ umount /boot
+
+ fi
umount "${mountpoint}" || \
{ warn "umount of ${path} failed - failure"; exit 0; }
@@ -113,26 +125,33 @@ copy_and_config_kernels() {
continue
fi
- warn " Copying kernel and creating boot-entry for ${kver}"
- KERNEL_ESP_DIR="${PMX_ESP_DIR}/${kver}"
- KERNEL_LIVE_DIR="${esp}/${KERNEL_ESP_DIR}"
- mkdir -p "${KERNEL_LIVE_DIR}"
- cp -u --preserve=timestamps "${linux_image}"
"${KERNEL_LIVE_DIR}/"
- cp -u --preserve=timestamps "${initrd}" "${KERNEL_LIVE_DIR}/"
-
- # create loader entry
- cat > "${esp}/loader/entries/proxmox-${kver}.conf" <<- EOF
-
[pve-devel] [PATCH pve-kernel-meta 2/7] proxmox-boot-tool: add status command
currently simply checking if $ESP_LIST exists, and indicating via
the exit status if proxmox-boot-tool is used for booting the system.
Signed-off-by: Stoiko Ivanov
---
bin/proxmox-boot-tool | 20
1 file changed, 20 insertions(+)
diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
index 2d625a6..7b06d57 100755
--- a/bin/proxmox-boot-tool
+++ b/bin/proxmox-boot-tool
@@ -283,6 +283,7 @@ usage() {
warn " $0 refresh [--hook ]"
warn " $0 kernel "
warn " $0 kernel list"
+ warn " $0 status"
warn " $0 help"
}
@@ -312,6 +313,20 @@ help() {
echo ""
echo "list kernel versions currently selected for inclusion on
ESPs."
echo ""
+ echo "USAGE: $0 status [--verbose]"
+ echo ""
+ echo "Exit with 0 if any ESP is configured, else with 2."
+ echo ""
+}
+
+status() {
+ verbose="$1"
+ if [ ! -e ${ESP_LIST} ]; then
+ if [ -n "$verbose" ]; then
+ warn "E: $ESP_LIST does not exist."
+ fi
+ exit 2
+ fi
}
if [ -z "$1" ]; then
@@ -390,6 +405,11 @@ case "$1" in
;;
esac
;;
+ 'status')
+ shift
+ status "$1"
+ exit 0
+ ;;
'help')
shift
help
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 1/7] proxmox-boot-tool: rename from pve-efiboot-tool
We will be using the mechanics also for ZFS systems booting with BIOS
legacy boot, and the tool is used also in PMG and PBS.
A symlink is kept in place for compatibility reasons
Signed-off-by: Stoiko Ivanov
---
Makefile | 2 +-
bin/Makefile | 2 +-
bin/{pve-efiboot-tool => proxmox-boot-tool} | 2 +-
debian/pve-kernel-helper.install | 4 ++--
debian/pve-kernel-helper.links| 1 +
{efiboot => proxmox-boot}/Makefile| 4 ++--
{efiboot => proxmox-boot}/functions | 0
efiboot/pve-auto-removal => proxmox-boot/proxmox-auto-removal | 0
efiboot/pve-efiboot-sync => proxmox-boot/proxmox-boot-sync| 2 +-
efiboot/zz-pve-efiboot => proxmox-boot/zz-proxmox-boot| 0
10 files changed, 9 insertions(+), 8 deletions(-)
rename bin/{pve-efiboot-tool => proxmox-boot-tool} (99%)
create mode 100644 debian/pve-kernel-helper.links
rename {efiboot => proxmox-boot}/Makefile (87%)
rename {efiboot => proxmox-boot}/functions (100%)
rename efiboot/pve-auto-removal => proxmox-boot/proxmox-auto-removal (100%)
rename efiboot/pve-efiboot-sync => proxmox-boot/proxmox-boot-sync (84%)
rename efiboot/zz-pve-efiboot => proxmox-boot/zz-proxmox-boot (100%)
diff --git a/Makefile b/Makefile
index 0b62b3e..90d5989 100644
--- a/Makefile
+++ b/Makefile
@@ -13,7 +13,7 @@ BUILD_DIR=build
DEBS=${KERNEL_DEB} ${HEADERS_DEB} ${HELPER_DEB}
-SUBDIRS = efiboot bin
+SUBDIRS = proxmox-boot bin
.PHONY: all
all: ${SUBDIRS}
diff --git a/bin/Makefile b/bin/Makefile
index 058c86f..b78fa42 100644
--- a/bin/Makefile
+++ b/bin/Makefile
@@ -5,7 +5,7 @@ all:
install:
install -d ${SBINDIR}
- install -m 0755 pve-efiboot-tool ${SBINDIR}/
+ install -m 0755 proxmox-boot-tool ${SBINDIR}/
.PHONY: clean distclean
distclean:
diff --git a/bin/pve-efiboot-tool b/bin/proxmox-boot-tool
similarity index 99%
rename from bin/pve-efiboot-tool
rename to bin/proxmox-boot-tool
index f57a752..2d625a6 100755
--- a/bin/pve-efiboot-tool
+++ b/bin/proxmox-boot-tool
@@ -199,7 +199,7 @@ clean() {
refresh() {
hook=$1
- hookscripts='pve-auto-removal zz-pve-efiboot'
+ hookscripts='proxmox-auto-removal zz-proxmox-boot'
if [ -n "$hook" ]; then
if echo "$hookscripts" | grep -sqE
"(^|[[:space:]]+)$hook([[:space:]]+|$)"; then
diff --git a/debian/pve-kernel-helper.install b/debian/pve-kernel-helper.install
index 6f7f713..aae9494 100644
--- a/debian/pve-kernel-helper.install
+++ b/debian/pve-kernel-helper.install
@@ -1,5 +1,5 @@
etc/kernel/postinst.d/*
etc/kernel/postrm.d/*
-etc/initramfs/post-update.d/pve-efiboot-sync
-usr/sbin/pve-efiboot-tool
+etc/initramfs/post-update.d/proxmox-boot-sync
+usr/sbin/proxmox-boot-tool
usr/share/pve-kernel-helper/scripts/functions
diff --git a/debian/pve-kernel-helper.links b/debian/pve-kernel-helper.links
new file mode 100644
index 000..70bf372
--- /dev/null
+++ b/debian/pve-kernel-helper.links
@@ -0,0 +1 @@
+/usr/sbin/proxmox-boot-tool /usr/sbin/pve-efiboot-tool
diff --git a/efiboot/Makefile b/proxmox-boot/Makefile
similarity index 87%
rename from efiboot/Makefile
rename to proxmox-boot/Makefile
index fc9e333..3a36cb7 100644
--- a/efiboot/Makefile
+++ b/proxmox-boot/Makefile
@@ -1,5 +1,5 @@
-KERNEL_HOOKSCRIPTS = pve-auto-removal zz-pve-efiboot
-INITRAMFS_HOOKSCRIPTS = pve-efiboot-sync
+KERNEL_HOOKSCRIPTS = proxmox-auto-removal zz-proxmox-boot
+INITRAMFS_HOOKSCRIPTS = proxmox-boot-sync
SHARE_FILES = functions
POSTINSTHOOKDIR = ${DESTDIR}/etc/kernel/postinst.d
diff --git a/efiboot/functions b/proxmox-boot/functions
similarity index 100%
rename from efiboot/functions
rename to proxmox-boot/functions
diff --git a/efiboot/pve-auto-removal b/proxmox-boot/proxmox-auto-removal
similarity index 100%
rename from efiboot/pve-auto-removal
rename to proxmox-boot/proxmox-auto-removal
diff --git a/efiboot/pve-efiboot-sync b/proxmox-boot/proxmox-boot-sync
similarity index 84%
rename from efiboot/pve-efiboot-sync
rename to proxmox-boot/proxmox-boot-sync
index c3ccf8e..5bdd72e 100644
--- a/efiboot/pve-efiboot-sync
+++ b/proxmox-boot/proxmox-boot-sync
@@ -7,5 +7,5 @@ set -e
# this variable will be set to 1 and we do nothing, since our pve-kernel
# hooks will update the ESPs all at once anyway.
if [ -z "$INITRAMFS_TOOLS_KERNEL_HOOK" ]; then
- /usr/sbin/pve-efiboot-tool refresh --hook 'zz-pve-efiboot'
+ /usr/sbin/proxmox-boot-tool refresh --hook 'zz-proxmox-boot'
fi
diff --git a/efiboot/zz-pve-efiboot b/proxmox-boot/zz-proxmox-boot
similarity index 100%
rename from efiboot/zz-pve-efiboot
rename to proxmox-boot/zz-proxmox-boot
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 4/7] proxmox-boot: rename uuid list file
in order to be consistent with the renaming of pve-efiboot-tool to proxmox-boot-tool. Sending as separate patch, since it changes and removes a file in '/etc', which could be considered part of the external 'api' of proxmox-boot-tool Signed-off-by: Stoiko Ivanov --- debian/pve-kernel-helper.postinst | 10 ++ proxmox-boot/functions| 2 +- 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 debian/pve-kernel-helper.postinst diff --git a/debian/pve-kernel-helper.postinst b/debian/pve-kernel-helper.postinst new file mode 100644 index 000..5299d26 --- /dev/null +++ b/debian/pve-kernel-helper.postinst @@ -0,0 +1,10 @@ +#! /bin/sh + +set -e + +if [ -e /etc/kernel/pve-efiboot-uuids ]; then + echo "$0: legacy ESP list /etc/kernel/pve-efiboot-uuids found merging and cleaning." 1>&2 + cat /etc/kernel/pve-efiboot-uuids >> /etc/kernel/proxmox-boot-uuids + proxmox-boot-tool clean + rm -f /etc/kernel/pve-efiboot-uuids +fi diff --git a/proxmox-boot/functions b/proxmox-boot/functions index 72fe15d..04bdc51 100755 --- a/proxmox-boot/functions +++ b/proxmox-boot/functions @@ -1,7 +1,7 @@ #! /bin/sh set -e -ESP_LIST="/etc/kernel/pve-efiboot-uuids" +ESP_LIST="/etc/kernel/proxmox-boot-uuids" ESPTYPE='c12a7328-f81f-11d2-ba4b-00a0c93ec93b' MANUAL_KERNEL_LIST="/etc/kernel/pve-efiboot-manual-kernels" -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel-meta 6/7] proxmox-boot: add grub.cfg header snippet
If the system seems to be booted using proxmox-boot, write a header at
the beginning of the grub.cfg generated when running `update-grub`
Additionally print a warning in case the script is run interactively.
This is determined by checking for DPKG_VERSION, which is set when
running as post-inst task (after a kernel install/removal)
and for PVE_EFIBOOT_UNSHARED, which is set by proxmox-boot-tool when
running `proxmox-boot-tool refresh.`
Signed-off-by: Stoiko Ivanov
---
proxmox-boot/000_proxmox_boot_header | 22 ++
proxmox-boot/Makefile| 4
2 files changed, 26 insertions(+)
create mode 100755 proxmox-boot/000_proxmox_boot_header
diff --git a/proxmox-boot/000_proxmox_boot_header
b/proxmox-boot/000_proxmox_boot_header
new file mode 100755
index 000..57dae2c
--- /dev/null
+++ b/proxmox-boot/000_proxmox_boot_header
@@ -0,0 +1,22 @@
+#! /bin/sh
+set -e
+
+. /usr/share/pve-kernel-helper/scripts/functions
+
+if proxmox-boot-tool status; then
+ cat <<- EOF
+ #
+ # This system is booted via proxmox-boot-tool! The grub-config used when
+ # booting from the disks configured with proxmox-boot-tool resides on
the vfat
+ # partitions with UUIDs listed in ${ESP_LIST}.
+ # /boot/grub/grub.cfg is NOT read when booting from those disk!
+ EOF
+
+ if [ -z "$DPKG_VERSION" ] && [ -z "$PVE_EFIBOOT_UNSHARED" ]; then
+ warn "W: This system is booted via proxmox-boot-tool:"
+ warn "W: Running update-grub does not update the correct
config!"
+ warn "W: Run 'proxmox-boot-tool refresh' instead."
+ warn ""
+ fi
+fi
+
diff --git a/proxmox-boot/Makefile b/proxmox-boot/Makefile
index 3a36cb7..effd726 100644
--- a/proxmox-boot/Makefile
+++ b/proxmox-boot/Makefile
@@ -1,11 +1,13 @@
KERNEL_HOOKSCRIPTS = proxmox-auto-removal zz-proxmox-boot
INITRAMFS_HOOKSCRIPTS = proxmox-boot-sync
SHARE_FILES = functions
+GRUB_CFG_SNIPPET = 000_proxmox_boot_header
POSTINSTHOOKDIR = ${DESTDIR}/etc/kernel/postinst.d
POSTRMHOOKDIR = ${DESTDIR}/etc/kernel/postrm.d
POSTINITRAMFSHOOKDIR = ${DESTDIR}/etc/initramfs/post-update.d
SHARE_SCRIPTDIR = ${DESTDIR}/usr/share/pve-kernel-helper/scripts
+GRUB_CFG_DIR = ${DESTDIR}/etc/grub.d
.PHONY: all
all:
@@ -19,6 +21,8 @@ install:
install -m 0755 ${INITRAMFS_HOOKSCRIPTS} ${POSTINITRAMFSHOOKDIR}
install -d ${SHARE_SCRIPTDIR}
install -m 0755 ${SHARE_FILES} ${SHARE_SCRIPTDIR}
+ install -d ${GRUB_CFG_DIR}
+ install -m 0755 ${GRUB_CFG_SNIPPET} ${GRUB_CFG_DIR}
.PHONY: clean distclean
distclean:
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH 00/10] Single-file-restore GUI for PBS snapshots
On 22/04/2021 12:33, Dominic Jäger wrote: On Wed, Apr 21, 2021 at 01:15:29PM +0200, Stefan Reiter wrote: Implements the necessary API for allowing single file restore via the PVE web GUI. Restoring a couple of small text files and folders with textfiles from containers worked really well for me. For VMs I could also restore some text files & folders, but I didn't work as smooth. I tried both with a single .fidx/root.pxar so far. * This proxmox-file-restore failed: Error: mounting 'drive-scsi0.img.fidx/part/1' failed: all mounts failed or no supported file system (500) is correct but it would be nice if we could try to say something like "We found LVM stuff which is currently unsupported". Currently it looks more like bug than "expected error" to me. I mean, the real fix here is to support those types of storage ;) I think "no supported file system" explains the situation reasonably well, but open to suggestions of course. Maybe we could improve this (at some point in the future) so that the error does not block the whole panel and we can continue to select another part/number. Because when two out of three parts block the whole thing, then remembering the bad one(s), clicking X, clicking File restore->fidx->part->some number is not ideal IMO. argh, I already have a fix for that, but apparently I forgot to include my widget-toolkit patches in the series - thanks for noticing! * Sometimes the Download button remains enabled when errors (like the previous) happen and then you get a couple of 0 byte files yup, should probably disable that as well on error * On the respective PBS sometimes this 2021-04-22T11:31:01+02:00: starting new backup reader datastore 'test': "/test" 2021-04-22T11:31:01+02:00: protocol upgrade done 2021-04-22T11:31:01+02:00: GET /download 2021-04-22T11:31:01+02:00: download "/test/vm/100/2021-04-22T09:29:17Z/index.json.blob" 2021-04-22T11:31:01+02:00: GET /download 2021-04-22T11:31:01+02:00: download "/test/vm/100/2021-04-22T09:29:17Z/drive-scsi0.img.fidx" 2021-04-22T11:31:01+02:00: register chunks in 'drive-scsi0.img.fidx' as downloadable. 2021-04-22T11:31:01+02:00: GET /chunk 2021-04-22T11:31:01+02:00: download chunk "/test/.chunks/394a/394a4b4c316868a466fbece11881752a10b31a782d2740ab414c3f48b5fb4e58" 2021-04-22T11:31:01+02:00: GET /chunk 2021-04-22T11:31:01+02:00: download chunk "/test/.chunks/d628/d62885ad37c7a44a2fa9fdc558e85e89ecf4c9754617fab7e1f491fa3da65d38" appears and stays there forever and when I clicked Stop 2021-04-22T11:47:00+02:00: TASK ERROR: task aborted it seemed to accept but the circle remained spinning forever again (OK, I only checked for a couple of minutes). When you clicked "stop" where? On PBS side? That would be an error in the server code somewhere then. But yes, the task stays active - that is due to the nature of the beast, the VM used to restore keeps the channel open for disk access as long as it runs, once it times out the task should disappear too. * Sometimes the PBS also logged 2021-04-22T11:42:40+02:00: starting new backup reader datastore 'test': "/test" 2021-04-22T11:42:40+02:00: protocol upgrade done 2021-04-22T11:42:40+02:00: GET /download 2021-04-22T11:42:40+02:00: download "/test/ct/102/2021-04-22T09:39:23Z/index.json.blob" 2021-04-22T11:42:40+02:00: GET /download 2021-04-22T11:42:40+02:00: download "/test/ct/102/2021-04-22T09:39:23Z/root.pxar.didx" 2021-04-22T11:42:40+02:00: register chunks in 'root.pxar.didx' as downloadable. 2021-04-22T11:42:40+02:00: GET /chunk 2021-04-22T11:42:40+02:00: download chunk "/test/.chunks/ff63/ff63c49eae6b77e2933ea08af0dcf3786e74fc454ac2462ba780f0726c070023" 2021-04-22T11:42:40+02:00: GET /chunk 2021-04-22T11:42:40+02:00: download chunk "/test/.chunks/261e/261ec247951ab57a51df87c593ae2aaf4f76c671bb94e13245f53ebdf25cfd45" 2021-04-22T11:42:40+02:00: GET /chunk 2021-04-22T11:42:40+02:00: download chunk "/test/.chunks/3a60/3a60c4eb21123c01a63e981d8f550ba0f99b5e90b97340d384da9f079fa767ed" 2021-04-22T11:42:40+02:00: TASK ERROR: connection error: Transport endpoint is not connected (os error 107) but it looked to me like it didn't really matter? I could still download my text files. Note that when it started to log such messages, there were actually quite a lot of those entries. That is an interesting one... does it happen in the middle of the task log? It almost looks like the VM stopped and ended the connection. Maybe some intermittant network stuff? * A few times I got in the PVE GUI when I clicked on ...img.fidx malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "VM 'qemu_root\\x40pa...") at /usr/share/perl5/PVE/PBSClient.pm line 220. (500) and I couldn't find out yet why and later it just didn't appear again. Fixed with 6ea6324bd6cc in proxmox-backup. * This is helpful executable not found '/usr/bin/proxmox-file-restore'! Proxmox backup client or file restore not installed? (
[pve-devel] [PATCH v2 docs 2/2] vzdump: add section about single file restore
Signed-off-by: Stefan Reiter Reviewed-by: Dylan Whyte --- v2: * Incorporate Dylan's review + R-b vzdump.adoc | 24 1 file changed, 24 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index b8ea081..0937fe2 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -365,6 +365,30 @@ amount of data is required for initial operation, e.g. web servers - once the OS and necessary services have been started, the VM is operational, while the background task continues copying seldomly used data. +Single File Restore +~~~ + +The 'File Restore' button in the 'Backups' tab of the storage GUI can be used to +open a file browser directly on the data contained in a backup. This feature +is only available for backups on a Proxmox Backup Server. + +For containers, the first layer of the file tree shows all included 'pxar' +archives, which can be opened and browsed freely. For VMs, the first layer shows +contained drive images, which can be opened to reveal a list of supported +storage technologies found on the drive. In the most basic case, this will be an +entry called 'part', representing a partition table, which contains entries for +each partition found on the drive. Note that for VMs, not all data might be +accessible (unsupported guest file systems, storage technologies, etc...). + +Files and directories can be downloaded using the 'Download' button, the latter +being compressed into a zip archive on the fly. + +To enable secure access to VM images, which might contain untrusted data, a +temporary VM (not visible as a guest) is started. This does not mean that data +downloaded from such an archive is inherently safe, but it avoids exposing the +hypervisor system to danger. The VM will stop itself after a timeout. This +entire process happens transparently from a user's point of view. + [[vzdump_configuration]] Configuration - -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 docs 1/2] vzdump: add section about live-restore
Signed-off-by: Stefan Reiter Reviewed-by: Dylan Whyte --- v2: * Incorporate Dylan's review + R-b Agreed with all suggestions, for both patches :) vzdump.adoc | 28 1 file changed, 28 insertions(+) diff --git a/vzdump.adoc b/vzdump.adoc index 9453684..b8ea081 100644 --- a/vzdump.adoc +++ b/vzdump.adoc @@ -337,6 +337,34 @@ per configured storage, this can be done with: # pvesm set STORAGEID --bwlimit restore=KIBs +Live-Restore + + +Restoring a large backup can take a long time, in which a guest is still +unavailable. For VM backups stored on a Proxmox Backup Server, this wait +time can be mitigated using the live-restore option. + +Enabling live-restore via either the checkbox in the GUI or the `--live-restore` +argument of `qmrestore` causes the VM to start as soon as the restore +begins. Data is copied in the background, prioritizing chunks that the VM is +actively accessing. + +Note that this comes with two caveats: + +* During live-restore, the VM will operate with limited disk read speeds, as + data has to be loaded from the backup server (once loaded, it is immediately + available on the destination storage however, so accessing data twice only + incurs the penalty the first time). Write speeds are largely unaffected. +* If the live-restore fails for any reason, the VM will be left in an + undefined state - that is, not all data might have been copied from the + backup, and it is _most likely_ not possible to keep any data that was written + during the failed restore operation. + +This mode of operation is especially useful for large VMs, where only a small +amount of data is required for initial operation, e.g. web servers - once the OS +and necessary services have been started, the VM is operational, while the +background task continues copying seldomly used data. + [[vzdump_configuration]] Configuration - -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH firewall] fix #967: source: dest: limit length
iptables-restore has a buffer limit of 1024 for paramters [0].
If users end up adding a long list of IPs in the source or dest field
they might reach this limit. The result is that the rule will not be
applied and pve-firewall will show some error in the syslog which will
be "hidden" for most users.
Enforcing a smaller limit ourselves should help to avoid any such
situation. 512 characters should help to not run into any problems that
stem from differences in what counts as character. If people need longer
lists, using IP sets are the better approach anyway.
[0] http://git.netfilter.org/iptables/tree/iptables/xshared.c?h=v1.8.7#n469
Signed-off-by: Aaron Lauterer
---
src/PVE/Firewall.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/PVE/Firewall.pm b/src/PVE/Firewall.pm
index 92ea33d..50be187 100644
--- a/src/PVE/Firewall.pm
+++ b/src/PVE/Firewall.pm
@@ -1449,11 +1449,13 @@ my $rule_properties = {
description => "Restrict packet source address. $addr_list_descr",
type => 'string', format => 'pve-fw-addr-spec',
optional => 1,
+ maxLength => 512,
},
dest => {
description => "Restrict packet destination address. $addr_list_descr",
type => 'string', format => 'pve-fw-addr-spec',
optional => 1,
+ maxLength => 512,
},
proto => {
description => "IP protocol. You can use protocol names ('tcp'/'udp')
or simple numbers, as defined in '/etc/protocols'.",
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager] ui: firewall: rule: maxlength for source and dest
Limiting the length of the source and dest paramters helps to avoid
problems with iptables-restore which would not apply a rule if a
parameter is larger than the parameter buffer (1024)[0]. As the API is
already limiting this, we should also reflect that in the GUI and give
people a hint that IP sets are most likely the better approach.
[0] http://git.netfilter.org/iptables/tree/iptables/xshared.c?h=v1.8.7#n469
Signed-off-by: Aaron Lauterer
---
This one "needs" the firewall patch 'fix #967: source: dest: limit length'
As always when it comes to messages, someone might have a better idea
how to phrase the maxLengthText.
www/manager6/grid/FirewallRules.js | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/www/manager6/grid/FirewallRules.js
b/www/manager6/grid/FirewallRules.js
index 424bdfcb..f32a1ea1 100644
--- a/www/manager6/grid/FirewallRules.js
+++ b/www/manager6/grid/FirewallRules.js
@@ -135,7 +135,8 @@ Ext.define('PVE.FirewallRulePanel', {
base_url: me.list_refs_url,
value: '',
fieldLabel: gettext('Source'),
-
+ maxLength: 512,
+ maxLengthText: gettext('Too long, consider using IP sets.'),
},
{
xtype: 'pveIPRefSelector',
@@ -145,6 +146,8 @@ Ext.define('PVE.FirewallRulePanel', {
base_url: me.list_refs_url,
value: '',
fieldLabel: gettext('Destination'),
+ maxLength: 512,
+ maxLengthText: gettext('Too long, consider using IP sets.'),
},
);
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH pve-kernel-meta 1/7] proxmox-boot-tool: rename from pve-efiboot-tool
On April 22, 2021 1:17 pm, Stoiko Ivanov wrote:
> We will be using the mechanics also for ZFS systems booting with BIOS
> legacy boot, and the tool is used also in PMG and PBS.
>
> A symlink is kept in place for compatibility reasons
>
> Signed-off-by: Stoiko Ivanov
> ---
> Makefile | 2 +-
> bin/Makefile | 2 +-
> bin/{pve-efiboot-tool => proxmox-boot-tool} | 2 +-
> debian/pve-kernel-helper.install | 4 ++--
> debian/pve-kernel-helper.links| 1 +
> {efiboot => proxmox-boot}/Makefile| 4 ++--
> {efiboot => proxmox-boot}/functions | 0
> efiboot/pve-auto-removal => proxmox-boot/proxmox-auto-removal | 0
> efiboot/pve-efiboot-sync => proxmox-boot/proxmox-boot-sync| 2 +-
> efiboot/zz-pve-efiboot => proxmox-boot/zz-proxmox-boot| 0
> 10 files changed, 9 insertions(+), 8 deletions(-)
> rename bin/{pve-efiboot-tool => proxmox-boot-tool} (99%)
> create mode 100644 debian/pve-kernel-helper.links
> rename {efiboot => proxmox-boot}/Makefile (87%)
> rename {efiboot => proxmox-boot}/functions (100%)
> rename efiboot/pve-auto-removal => proxmox-boot/proxmox-auto-removal (100%)
> rename efiboot/pve-efiboot-sync => proxmox-boot/proxmox-boot-sync (84%)
> rename efiboot/zz-pve-efiboot => proxmox-boot/zz-proxmox-boot (100%)
these three files are installed as five conffiles, so they need special
care when being renamed/moved, see `man dpkg-maintscript-helper`.
alternatively we could also just unconditionally remove them on
upgrades I guess.
otherwise the old ones will stay around on upgrades, and also be
called..
>
> diff --git a/Makefile b/Makefile
> index 0b62b3e..90d5989 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -13,7 +13,7 @@ BUILD_DIR=build
>
> DEBS=${KERNEL_DEB} ${HEADERS_DEB} ${HELPER_DEB}
>
> -SUBDIRS = efiboot bin
> +SUBDIRS = proxmox-boot bin
>
> .PHONY: all
> all: ${SUBDIRS}
> diff --git a/bin/Makefile b/bin/Makefile
> index 058c86f..b78fa42 100644
> --- a/bin/Makefile
> +++ b/bin/Makefile
> @@ -5,7 +5,7 @@ all:
>
> install:
> install -d ${SBINDIR}
> - install -m 0755 pve-efiboot-tool ${SBINDIR}/
> + install -m 0755 proxmox-boot-tool ${SBINDIR}/
>
> .PHONY: clean distclean
> distclean:
> diff --git a/bin/pve-efiboot-tool b/bin/proxmox-boot-tool
> similarity index 99%
> rename from bin/pve-efiboot-tool
> rename to bin/proxmox-boot-tool
> index f57a752..2d625a6 100755
> --- a/bin/pve-efiboot-tool
> +++ b/bin/proxmox-boot-tool
> @@ -199,7 +199,7 @@ clean() {
>
> refresh() {
> hook=$1
> - hookscripts='pve-auto-removal zz-pve-efiboot'
> + hookscripts='proxmox-auto-removal zz-proxmox-boot'
>
> if [ -n "$hook" ]; then
> if echo "$hookscripts" | grep -sqE
> "(^|[[:space:]]+)$hook([[:space:]]+|$)"; then
> diff --git a/debian/pve-kernel-helper.install
> b/debian/pve-kernel-helper.install
> index 6f7f713..aae9494 100644
> --- a/debian/pve-kernel-helper.install
> +++ b/debian/pve-kernel-helper.install
> @@ -1,5 +1,5 @@
> etc/kernel/postinst.d/*
> etc/kernel/postrm.d/*
> -etc/initramfs/post-update.d/pve-efiboot-sync
> -usr/sbin/pve-efiboot-tool
> +etc/initramfs/post-update.d/proxmox-boot-sync
> +usr/sbin/proxmox-boot-tool
> usr/share/pve-kernel-helper/scripts/functions
> diff --git a/debian/pve-kernel-helper.links b/debian/pve-kernel-helper.links
> new file mode 100644
> index 000..70bf372
> --- /dev/null
> +++ b/debian/pve-kernel-helper.links
> @@ -0,0 +1 @@
> +/usr/sbin/proxmox-boot-tool /usr/sbin/pve-efiboot-tool
> diff --git a/efiboot/Makefile b/proxmox-boot/Makefile
> similarity index 87%
> rename from efiboot/Makefile
> rename to proxmox-boot/Makefile
> index fc9e333..3a36cb7 100644
> --- a/efiboot/Makefile
> +++ b/proxmox-boot/Makefile
> @@ -1,5 +1,5 @@
> -KERNEL_HOOKSCRIPTS = pve-auto-removal zz-pve-efiboot
> -INITRAMFS_HOOKSCRIPTS = pve-efiboot-sync
> +KERNEL_HOOKSCRIPTS = proxmox-auto-removal zz-proxmox-boot
> +INITRAMFS_HOOKSCRIPTS = proxmox-boot-sync
> SHARE_FILES = functions
>
> POSTINSTHOOKDIR = ${DESTDIR}/etc/kernel/postinst.d
> diff --git a/efiboot/functions b/proxmox-boot/functions
> similarity index 100%
> rename from efiboot/functions
> rename to proxmox-boot/functions
> diff --git a/efiboot/pve-auto-removal b/proxmox-boot/proxmox-auto-removal
> similarity index 100%
> rename from efiboot/pve-auto-removal
> rename to proxmox-boot/proxmox-auto-removal
> diff --git a/efiboot/pve-efiboot-sync b/proxmox-boot/proxmox-boot-sync
> similarity index 84%
> rename from efiboot/pve-efiboot-sync
> rename to proxmox-boot/proxmox-boot-sync
> index c3ccf8e..5bdd72e 100644
> --- a/efiboot/pve-efiboot-sync
> +++ b/proxmox-boot/proxmox-boot-sync
> @@ -7,5 +7,5 @@ set -e
> # this variable will be set to 1 and we do nothing, since our pve-kernel
> # hooks will update the ESPs all at
Re: [pve-devel] [PATCH pve-kernel-meta 2/7] proxmox-boot-tool: add status command
On April 22, 2021 1:17 pm, Stoiko Ivanov wrote:
> currently simply checking if $ESP_LIST exists, and indicating via
> the exit status if proxmox-boot-tool is used for booting the system.
>
> Signed-off-by: Stoiko Ivanov
> ---
> bin/proxmox-boot-tool | 20
> 1 file changed, 20 insertions(+)
>
> diff --git a/bin/proxmox-boot-tool b/bin/proxmox-boot-tool
> index 2d625a6..7b06d57 100755
> --- a/bin/proxmox-boot-tool
> +++ b/bin/proxmox-boot-tool
> @@ -283,6 +283,7 @@ usage() {
> warn " $0 refresh [--hook ]"
> warn " $0 kernel "
> warn " $0 kernel list"
> + warn " $0 status"
> warn " $0 help"
> }
>
> @@ -312,6 +313,20 @@ help() {
> echo ""
> echo "list kernel versions currently selected for inclusion on
> ESPs."
> echo ""
> + echo "USAGE: $0 status [--verbose]"
> + echo ""
> + echo "Exit with 0 if any ESP is configured, else with 2."
> + echo ""
> +}
> +
> +status() {
> + verbose="$1"
> + if [ ! -e ${ESP_LIST} ]; then
> + if [ -n "$verbose" ]; then
> + warn "E: $ESP_LIST does not exist."
this is a bit strange (a command called status that has a verbose flag
that does not output anything if everything is okay). maybe we could
list the ESPs? or even, if --verbose is given, mount them and print
THEIR status? ANY output would be great as a start ;)
> + fi
> + exit 2
> + fi
> }
>
> if [ -z "$1" ]; then
> @@ -390,6 +405,11 @@ case "$1" in
> ;;
> esac
> ;;
> + 'status')
> + shift
> + status "$1"
> + exit 0
> + ;;
> 'help')
> shift
> help
> --
> 2.20.1
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 proxmox-backup 03/13] file-restore: support encrypted VM backups
Signed-off-by: Stefan Reiter
---
new in v2
src/bin/proxmox-file-restore.rs | 22 +---
src/bin/proxmox_file_restore/block_driver.rs | 1 +
src/bin/proxmox_file_restore/qemu_helper.rs | 9 ++--
3 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/src/bin/proxmox-file-restore.rs b/src/bin/proxmox-file-restore.rs
index 2726eeb7..3d750152 100644
--- a/src/bin/proxmox-file-restore.rs
+++ b/src/bin/proxmox-file-restore.rs
@@ -30,7 +30,7 @@ pub mod proxmox_client_tools;
use proxmox_client_tools::{
complete_group_or_snapshot, complete_repository, connect,
extract_repository_from_value,
key_source::{
-crypto_parameters, format_key_source, get_encryption_key_password,
KEYFD_SCHEMA,
+crypto_parameters_keep_fd, format_key_source,
get_encryption_key_password, KEYFD_SCHEMA,
KEYFILE_SCHEMA,
},
REPO_URL_SCHEMA,
@@ -76,6 +76,18 @@ fn parse_path(path: String, base64: bool) ->
Result {
}
}
+fn keyfile_path(param: &Value) -> Option {
+if let Some(Value::String(keyfile)) = param.get("keyfile") {
+return Some(keyfile.to_owned());
+}
+
+if let Some(Value::Number(keyfd)) = param.get("keyfd") {
+return Some(format!("/dev/fd/{}", keyfd));
+}
+
+None
+}
+
#[api(
input: {
properties: {
@@ -138,7 +150,8 @@ async fn list(
let snapshot: BackupDir = snapshot.parse()?;
let path = parse_path(path, base64)?;
-let crypto = crypto_parameters(¶m)?;
+let keyfile = keyfile_path(¶m);
+let crypto = crypto_parameters_keep_fd(¶m)?;
let crypt_config = match crypto.enc_key {
None => None,
Some(ref key) => {
@@ -210,6 +223,7 @@ async fn list(
manifest,
repo,
snapshot,
+keyfile,
};
let driver: Option = match param.get("driver") {
Some(drv) => Some(serde_json::from_value(drv.clone())?),
@@ -309,7 +323,8 @@ async fn extract(
None => Some(std::env::current_dir()?),
};
-let crypto = crypto_parameters(¶m)?;
+let keyfile = keyfile_path(¶m);
+let crypto = crypto_parameters_keep_fd(¶m)?;
let crypt_config = match crypto.enc_key {
None => None,
Some(ref key) => {
@@ -360,6 +375,7 @@ async fn extract(
manifest,
repo,
snapshot,
+keyfile,
};
let driver: Option = match param.get("driver") {
Some(drv) => Some(serde_json::from_value(drv.clone())?),
diff --git a/src/bin/proxmox_file_restore/block_driver.rs
b/src/bin/proxmox_file_restore/block_driver.rs
index 924503a7..ba9794e3 100644
--- a/src/bin/proxmox_file_restore/block_driver.rs
+++ b/src/bin/proxmox_file_restore/block_driver.rs
@@ -21,6 +21,7 @@ pub struct SnapRestoreDetails {
pub repo: BackupRepository,
pub snapshot: BackupDir,
pub manifest: BackupManifest,
+pub keyfile: Option,
}
/// Return value of a BlockRestoreDriver.status() call, 'id' must be valid for
.stop(id)
diff --git a/src/bin/proxmox_file_restore/qemu_helper.rs
b/src/bin/proxmox_file_restore/qemu_helper.rs
index 7fd2f1f8..0f3a7feb 100644
--- a/src/bin/proxmox_file_restore/qemu_helper.rs
+++ b/src/bin/proxmox_file_restore/qemu_helper.rs
@@ -190,9 +190,14 @@ pub async fn start_vm(
continue;
}
drives.push("-drive".to_owned());
+let keyfile = if let Some(ref keyfile) = details.keyfile {
+format!(",,keyfile={}", keyfile)
+} else {
+"".to_owned()
+};
drives.push(format!(
-
"file=pbs:repository={},,snapshot={},,archive={},read-only=on,if=none,id=drive{}",
-details.repo, details.snapshot, file, id
+
"file=pbs:repository={},,snapshot={},,archive={}{},read-only=on,if=none,id=drive{}",
+details.repo, details.snapshot, file, keyfile, id
));
drives.push("-device".to_owned());
// drive serial is used by VM to map .fidx files to /dev paths
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 common 04/13] PBSClient: adapt error message to include full package names
More helpful for a user to know what they're missing.
Suggested-by: Dominic Jäger
Signed-off-by: Stefan Reiter
---
new in v2
src/PVE/PBSClient.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
index 857cff0..16f7765 100644
--- a/src/PVE/PBSClient.pm
+++ b/src/PVE/PBSClient.pm
@@ -141,7 +141,7 @@ my sub do_raw_client_cmd {
my $client_exe = (delete $opts{binary}) || 'proxmox-backup-client';
$client_exe = "/usr/bin/$client_exe";
-die "executable not found '$client_exe'! Proxmox backup client or file
restore not installed?\n"
+die "executable not found '$client_exe'! proxmox-backup-client or
proxmox-backup-file-restore not installed?\n"
if ! -x $client_exe;
my $scfg = $self->{scfg};
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 common 05/13] PBSClient: add file_restore_list command
Signed-off-by: Stefan Reiter
---
v2:
* one line per param
src/PVE/PBSClient.pm | 11 +++
1 file changed, 11 insertions(+)
diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
index 16f7765..8e4deca 100644
--- a/src/PVE/PBSClient.pm
+++ b/src/PVE/PBSClient.pm
@@ -335,4 +335,15 @@ sub status {
return ($total, $free, $used, $active);
};
+sub file_restore_list {
+my ($self, $snapshot, $filepath, $base64) = @_;
+return run_client_cmd(
+ $self,
+ "list",
+ [ $snapshot, $filepath, "--base64", $base64 ? 1 : 0 ],
+ 0,
+ "proxmox-file-restore",
+);
+}
+
1;
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 proxmox-widget-toolkit 13/13] FileBrowser: show errors in messagebox and allow expand 'all'
If an error is received upon expanding a node, chances are the rest of
the tree is still valid (i.e. opening a partition fails because it
doesn't contain a supported filesystem). Only show an error box for the
user, but don't mask the component in that case. Additionally, disable
the download button.
Also support an archive set to 'all' to expand all children, useful for
initializing a file-restore VM on initial load.
Signed-off-by: Stefan Reiter
---
new in v2
src/window/FileBrowser.js | 20 +---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/src/window/FileBrowser.js b/src/window/FileBrowser.js
index d138d1b..99a7a85 100644
--- a/src/window/FileBrowser.js
+++ b/src/window/FileBrowser.js
@@ -123,6 +123,16 @@ Ext.define("Proxmox.window.FileBrowser", {
me.lookup('downloadBtn').setDisabled(!canDownload);
},
+ errorHandler: function(error, msg) {
+ let me = this;
+ me.lookup('downloadBtn').setDisabled(true);
+ if (me.initialLoadDone) {
+ Ext.Msg.alert(gettext('Error'), msg);
+ return true;
+ }
+ return false;
+ },
+
init: function(view) {
let me = this;
let tree = me.lookup('tree');
@@ -134,13 +144,16 @@ Ext.define("Proxmox.window.FileBrowser", {
let store = tree.getStore();
let proxy = store.getProxy();
- Proxmox.Utils.monStoreErrors(tree, store, true);
+ let errorCallback = (error, msg) => me.errorHandler(error, msg);
+ Proxmox.Utils.monStoreErrors(tree, store, true, errorCallback);
proxy.setUrl(view.listURL);
proxy.setExtraParams(view.extraParams);
- store.load(() => {
+ store.load((rec, op, success) => {
let root = store.getRoot();
root.expand(); // always expand invisible root node
- if (view.archive) {
+ if (view.archive === 'all') {
+ root.expandChildren(false);
+ } else if (view.archive) {
let child = root.findChild('text', view.archive);
if (child) {
child.expand();
@@ -152,6 +165,7 @@ Ext.define("Proxmox.window.FileBrowser", {
} else if (root.childNodes.length === 1) {
root.firstChild.expand();
}
+ me.initialLoadDone = success;
});
},
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 http-server 09/13] allow stream download from path and over pvedaemon-proxy
Allow specifying a filepath for stream=1 instead of either a path or fh
with stream=1.
With this in place, we can also just return the path to the proxy in
case we want to stream a response back, and let it read from the file
itself. This way, the pvedaemon is cut out of the transfer pipe.
Signed-off-by: Stefan Reiter
---
v2:
* unchanged
PVE/APIServer/AnyEvent.pm | 40 +++
1 file changed, 36 insertions(+), 4 deletions(-)
diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm
index f3db0e3..0654bd4 100644
--- a/PVE/APIServer/AnyEvent.pm
+++ b/PVE/APIServer/AnyEvent.pm
@@ -410,9 +410,33 @@ sub send_file_start {
my $mime;
if (ref($download) eq 'HASH') {
- $fh = $download->{fh};
$mime = $download->{'content-type'};
+ if ($download->{path} && $download->{stream} &&
+ $reqstate->{request}->header('PVEDisableProxy'))
+ {
+ # avoid double stream from a file, let the proxy handle it
+ die "internal error: file proxy streaming only available
for pvedaemon\n"
+ if !$self->{trusted_env};
+ my $header = HTTP::Headers->new(
+ pvestreamfile => $download->{path},
+ Content_Type => $mime,
+ );
+ # we need some data so Content-Length gets set correctly and
+ # the proxy doesn't wait for more data - place a canary
+ my $resp = HTTP::Response->new(200, "OK", $header, "error
canary");
+ $self->response($reqstate, $resp);
+ return;
+ }
+
+ if (!($fh = $download->{fh})) {
+ my $path = $download->{path};
+ die "internal error: {download} returned but neither fh not
path given\n"
+ if !$path;
+ sysopen($fh, "$path", O_NONBLOCK | O_RDONLY)
+ or die "open stream path '$path' for reading failed:
$!\n";
+ }
+
if ($download->{stream}) {
my $header = HTTP::Headers->new(Content_Type => $mime);
my $resp = HTTP::Response->new(200, "OK", $header);
@@ -750,6 +774,7 @@ sub proxy_request {
eval {
my $code = delete $hdr->{Status};
my $msg = delete $hdr->{Reason};
+ my $stream = delete $hdr->{pvestreamfile};
delete $hdr->{URL};
delete $hdr->{HTTPVersion};
my $header = HTTP::Headers->new(%$hdr);
@@ -757,9 +782,16 @@ sub proxy_request {
$location =~ s|^http://localhost:85||;
$header->header(Location => $location);
}
- my $resp = HTTP::Response->new($code, $msg, $header, $body);
- # Note: disable compression, because body is already
compressed
- $self->response($reqstate, $resp, undef, 1);
+ if ($stream) {
+ sysopen(my $fh, "$stream", O_NONBLOCK | O_RDONLY)
+ or die "open stream path '$stream' for forwarding
failed: $!\n";
+ my $resp = HTTP::Response->new($code, $msg, $header,
undef);
+ $self->response($reqstate, $resp, undef, 1, 0, $fh);
+ } else {
+ my $resp = HTTP::Response->new($code, $msg, $header,
$body);
+ # Note: disable compression, because body is already
compressed
+ $self->response($reqstate, $resp, undef, 1);
+ }
};
warn $@ if $@;
});
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 common 06/13] PBSClient: add file_restore_extract function
*_prepare creates a fifo for streaming data back to clients directly,
filefile_restore_extract blocks and should be called from a background
worker - while it is running outcoming data can be read from the FIFO.
Signed-off-by: Stefan Reiter
---
v2:
* don't use "startcmd", just pass "opts" directly
* better error handling
src/PVE/PBSClient.pm | 56 +++-
1 file changed, 55 insertions(+), 1 deletion(-)
diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
index 8e4deca..e96f20d 100644
--- a/src/PVE/PBSClient.pm
+++ b/src/PVE/PBSClient.pm
@@ -5,9 +5,10 @@ use strict;
use warnings;
use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
+use File::Temp qw(tempdir);
use IO::File;
use JSON;
-use POSIX qw(strftime ENOENT);
+use POSIX qw(mkfifo strftime ENOENT);
use PVE::JSONSchema qw(get_standard_option);
use PVE::Tools qw(run_command file_set_contents file_get_contents
file_read_firstline $IPV6RE);
@@ -346,4 +347,57 @@ sub file_restore_list {
);
}
+# call sync from API, returns a fifo path for streaming data to clients,
+# pass it to file_restore_extract to start transfering data
+sub file_restore_extract_prepare {
+my ($self) = @_;
+
+my $tmpdir = tempdir();
+mkfifo("$tmpdir/fifo", 0600)
+ or die "creating file download fifo '$tmpdir/fifo' failed: $!\n";
+
+# allow reading data for proxy user
+my $wwwid = getpwnam('www-data') ||
+ die "getpwnam failed";
+chown $wwwid, -1, "$tmpdir"
+ or die "changing permission on fifo dir '$tmpdir' failed: $!\n";
+chown $wwwid, -1, "$tmpdir/fifo"
+ or die "changing permission on fifo '$tmpdir/fifo' failed: $!\n";
+
+return "$tmpdir/fifo";
+}
+
+# this blocks while data is transfered, call this from a background worker
+sub file_restore_extract {
+my ($self, $output_file, $snapshot, $filepath, $base64) = @_;
+
+my $ret = eval {
+ local $SIG{ALRM} = sub { die "got timeout\n" };
+ alarm(30);
+ sysopen(my $fh, "$output_file", O_WRONLY)
+ or die "open target '$output_file' for writing failed: $!\n";
+ alarm(0);
+
+ my $fn = fileno($fh);
+ my $errfunc = sub { print $_[0], "\n"; };
+
+ return run_raw_client_cmd(
+ $self,
+"extract",
+ [ $snapshot, $filepath, "-", "--base64", $base64 ? 1 : 0 ],
+ binary => "proxmox-file-restore",
+ errfunc => $errfunc,
+ output => ">&$fn",
+ );
+};
+my $err = $@;
+
+unlink($output_file);
+$output_file =~ s/fifo$//;
+rmdir($output_file) if -d $output_file;
+
+die "file restore task failed: $err" if $err;
+return $ret;
+}
+
1;
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 http-server 08/13] support streaming data form fh to client
Use an explicit AnyEvent::Handle similar to websocket proxying.
Needs some special care to make sure we apply backpressure correctly to
avoid caching too much data. Note that because of AnyEvent restrictions,
specifying a "fh" to point to a file or a packet-based socket may result
in unwanted behaviour[0].
[0]: https://metacpan.org/pod/AnyEvent::Handle#DESCRIPTION
Signed-off-by: Stefan Reiter
---
v2:
* move to extra sub "response_stream"
* adapt comments
PVE/APIServer/AnyEvent.pm | 105 --
1 file changed, 101 insertions(+), 4 deletions(-)
diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm
index 60a2a1c..f3db0e3 100644
--- a/PVE/APIServer/AnyEvent.pm
+++ b/PVE/APIServer/AnyEvent.pm
@@ -188,8 +188,93 @@ sub finish_response {
}
}
+sub response_stream {
+my ($self, $reqstate, $stream_fh) = @_;
+
+# disable timeout, we don't know how big the data is
+$reqstate->{hdl}->timeout(0);
+
+my $buf_size = 4*1024*1024;
+
+my $on_read;
+$on_read = sub {
+ my ($hdl) = @_;
+ my $reqhdl = $reqstate->{hdl};
+ return if !$reqhdl;
+
+ my $wbuf_len = length($reqhdl->{wbuf});
+ my $rbuf_len = length($hdl->{rbuf});
+ # TODO: Take into account $reqhdl->{wbuf_max} ? Right now
+ # that's unbounded, so just assume $buf_size
+ my $to_read = $buf_size - $wbuf_len;
+ $to_read = $rbuf_len if $rbuf_len < $to_read;
+ if ($to_read > 0) {
+ my $data = substr($hdl->{rbuf}, 0, $to_read, '');
+ $reqhdl->push_write($data);
+ $rbuf_len -= $to_read;
+ } elsif ($hdl->{_eof}) {
+ # workaround: AnyEvent gives us a fake EPIPE if we don't consume
+ # any data when called at EOF, so unregister ourselves - data is
+ # flushed by on_eof anyway
+ # see:
https://sources.debian.org/src/libanyevent-perl/7.170-2/lib/AnyEvent/Handle.pm/#L1329
+ $hdl->on_read();
+ return;
+ }
+
+ # apply backpressure so we don't accept any more data into
+ # buffer if the client isn't downloading fast enough
+ # note: read_size can double upon read, and we also need to
+ # account for one more read after start_read, so *4
+ if ($rbuf_len + $hdl->{read_size}*4 > $buf_size) {
+ # stop reading until write buffer is empty
+ $hdl->on_read();
+ my $prev_on_drain = $reqhdl->{on_drain};
+ $reqhdl->on_drain(sub {
+ my ($wrhdl) = @_;
+ # on_drain called because write buffer is empty, continue
reading
+ $hdl->on_read($on_read);
+ if ($prev_on_drain) {
+ $wrhdl->on_drain($prev_on_drain);
+ $prev_on_drain->($wrhdl);
+ }
+ });
+ }
+};
+
+$reqstate->{proxyhdl} = AnyEvent::Handle->new(
+ fh => $stream_fh,
+ rbuf_max => $buf_size,
+ timeout => 0,
+ on_read => $on_read,
+ on_eof => sub {
+ my ($hdl) = @_;
+ eval {
+ if (my $reqhdl = $reqstate->{hdl}) {
+ $self->log_aborted_request($reqstate);
+ # write out any remaining data
+ $reqhdl->push_write($hdl->{rbuf}) if length($hdl->{rbuf}) >
0;
+ $hdl->{rbuf} = "";
+ $reqhdl->push_shutdown();
+ $self->finish_response($reqstate);
+ }
+ };
+ if (my $err = $@) { syslog('err', "$err"); }
+ $on_read = undef;
+ },
+ on_error => sub {
+ my ($hdl, $fatal, $message) = @_;
+ eval {
+ $self->log_aborted_request($reqstate, $message);
+ $self->client_do_disconnect($reqstate);
+ };
+ if (my $err = $@) { syslog('err', "$err"); }
+ $on_read = undef;
+ },
+);
+}
+
sub response {
-my ($self, $reqstate, $resp, $mtime, $nocomp, $delay) = @_;
+my ($self, $reqstate, $resp, $mtime, $nocomp, $delay, $stream_fh) = @_;
#print "$$: send response: " . Dumper($resp);
@@ -231,7 +316,7 @@ sub response {
$resp->header('Server' => "pve-api-daemon/3.0");
my $content_length;
-if ($content) {
+if ($content && !$stream_fh) {
$content_length = length($content);
@@ -258,11 +343,16 @@ sub response {
#print "SEND(without content) $res\n" if $self->{debug};
$res .= "\015\012";
-$res .= $content if $content;
+$res .= $content if $content && !$stream_fh;
$self->log_request($reqstate, $reqstate->{request});
-if ($delay && $delay > 0) {
+if ($stream_fh) {
+ # write headers and preamble...
+ $reqstate->{hdl}->push_write($res);
+ # ...then stream data via an AnyEvent::Handle
+ $self->response_stream($reqstate, $stream_fh);
+} elsif ($delay && $delay > 0) {
my $w; $w = AnyEvent->timer(after => $delay, cb => sub {
[pve-devel] [PATCH v2 proxmox-backup 01/13] file-restore: don't force PBS_FINGERPRINT env var
It is valid to not set it, in case the server has a valid certificate.
Signed-off-by: Stefan Reiter
---
new in v2
src/bin/proxmox_file_restore/qemu_helper.rs | 3 ---
1 file changed, 3 deletions(-)
diff --git a/src/bin/proxmox_file_restore/qemu_helper.rs
b/src/bin/proxmox_file_restore/qemu_helper.rs
index 22563263..7fd2f1f8 100644
--- a/src/bin/proxmox_file_restore/qemu_helper.rs
+++ b/src/bin/proxmox_file_restore/qemu_helper.rs
@@ -127,9 +127,6 @@ pub async fn start_vm(
if let Err(_) = std::env::var("PBS_PASSWORD") {
bail!("environment variable PBS_PASSWORD has to be set for QEMU VM
restore");
}
-if let Err(_) = std::env::var("PBS_FINGERPRINT") {
-bail!("environment variable PBS_FINGERPRINT has to be set for QEMU VM
restore");
-}
let pid;
let (pid_fd, pid_path) = make_tmp_file("/tmp/file-restore-qemu.pid.tmp",
CreateOptions::new())?;
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 proxmox-backup 02/13] client-tools: add crypto_parameters_keep_fd
same functionality as crypto_parameters, except it keeps the file
descriptor passed as "keyfd" open (and seeks to the beginning after
reading), if one is given.
Signed-off-by: Stefan Reiter
---
new in v2
src/bin/proxmox_client_tools/key_source.rs | 17 +++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/src/bin/proxmox_client_tools/key_source.rs
b/src/bin/proxmox_client_tools/key_source.rs
index 0ad06bb0..fee00723 100644
--- a/src/bin/proxmox_client_tools/key_source.rs
+++ b/src/bin/proxmox_client_tools/key_source.rs
@@ -86,6 +86,14 @@ pub struct CryptoParams {
}
pub fn crypto_parameters(param: &Value) -> Result {
+do_crypto_parameters(param, false)
+}
+
+pub fn crypto_parameters_keep_fd(param: &Value) -> Result
{
+do_crypto_parameters(param, true)
+}
+
+fn do_crypto_parameters(param: &Value, keep_keyfd_open: bool) ->
Result {
let keyfile = match param.get("keyfile") {
Some(Value::String(keyfile)) => Some(keyfile),
Some(_) => bail!("bad --keyfile parameter type"),
@@ -135,11 +143,16 @@ pub fn crypto_parameters(param: &Value) ->
Result {
file_get_contents(keyfile)?,
)),
(None, Some(fd)) => {
-let input = unsafe { std::fs::File::from_raw_fd(fd) };
+let mut input = unsafe { std::fs::File::from_raw_fd(fd) };
let mut data = Vec::new();
-let _len: usize = { input }.read_to_end(&mut data).map_err(|err| {
+let _len: usize = input.read_to_end(&mut data).map_err(|err| {
format_err!("error reading encryption key from fd {}: {}", fd,
err)
})?;
+if keep_keyfd_open {
+// don't close fd if requested, and try to reset seek position
+std::mem::forget(input);
+unsafe { libc::lseek(fd, 0, libc::SEEK_SET); }
+}
Some(KeyWithSource::from_fd(data))
}
};
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 storage 10/13] add FileRestore API for PBS
Includes list and restore calls.
Requires VM.Backup and Datastore.Audit permissions, for the accessed
VM/CT and containing datastore respectively.
Signed-off-by: Stefan Reiter
---
Requires updated pve-common, pve-http-server.
v2:
* use same permissions as regular backup access, check with check_volume_access
Note: I didn't add the "snapshot -> volume ID" change for now, but AFAICT this
should be a drop-in replacement (maybe aside from the parameter name). The
$real_volume_id sub in Content.pm should handle that just fine? Well, it would
need the "backup/" prefix to become a volid, but again, should be fine.
Feel free to adapt this though if you like the other way better, no hard
feelings tbh...
PVE/API2/Storage/FileRestore.pm | 160
PVE/API2/Storage/Makefile | 2 +-
PVE/API2/Storage/Status.pm | 6 ++
3 files changed, 167 insertions(+), 1 deletion(-)
create mode 100644 PVE/API2/Storage/FileRestore.pm
diff --git a/PVE/API2/Storage/FileRestore.pm b/PVE/API2/Storage/FileRestore.pm
new file mode 100644
index 000..68f81eb
--- /dev/null
+++ b/PVE/API2/Storage/FileRestore.pm
@@ -0,0 +1,160 @@
+package PVE::API2::Storage::FileRestore;
+
+use strict;
+use warnings;
+
+use MIME::Base64;
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::PBSClient;
+use PVE::Storage;
+use PVE::Tools qw(extract_param);
+
+use PVE::RESTHandler;
+use base qw(PVE::RESTHandler);
+
+__PACKAGE__->register_method ({
+name => 'list',
+path => 'list',
+method => 'GET',
+proxyto => 'node',
+permissions => {
+ description => "You need read access for the volume.",
+ user => 'all',
+},
+description => "List files and directories for single file restore under
the given path.",
+parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ storage => get_standard_option('pve-storage-id'),
+ snapshot => {
+ description => "Backup snapshot identifier.",
+ type => 'string',
+ },
+ filepath => {
+ description => 'base64-path to the directory or file being
listed, or "/".',
+ type => 'string',
+ },
+ },
+},
+returns => {
+ type => 'array',
+ items => {
+ type => "object",
+ properties => {
+ filepath => {
+ description => "base64 path of the current entry",
+ type => 'string',
+ },
+ type => {
+ description => "Entry type.",
+ type => 'string',
+ },
+ text => {
+ description => "Entry display text.",
+ type => 'string',
+ },
+ leaf => {
+ description => "If this entry is a leaf in the directory
graph.",
+ type => 'any', # JSON::PP::Boolean gets passed through
+ },
+ size => {
+ description => "Entry file size.",
+ type => 'integer',
+ optional => 1,
+ },
+ mtime => {
+ description => "Entry last-modified time (unix timestamp).",
+ type => 'integer',
+ optional => 1,
+ },
+ },
+ },
+},
+protected => 1,
+code => sub {
+ my ($param) = @_;
+
+ my $rpcenv = PVE::RPCEnvironment::get();
+ my $user = $rpcenv->get_user();
+
+ my $path = extract_param($param, 'filepath') || "/";
+ my $base64 = $path ne "/";
+ my $snap = extract_param($param, 'snapshot');
+ my $storeid = extract_param($param, 'storage');
+ my $cfg = PVE::Storage::config();
+ my $scfg = PVE::Storage::storage_config($cfg, $storeid);
+
+ my $volid = "$storeid:backup/$snap";
+ PVE::Storage::check_volume_access($rpcenv, $user, $cfg, undef, $volid);
+
+ my $client = PVE::PBSClient->new($scfg, $storeid);
+ my $ret = $client->file_restore_list($snap, $path, $base64);
+
+ return $ret;
+}});
+
+__PACKAGE__->register_method ({
+name => 'download',
+path => 'download',
+method => 'GET',
+proxyto => 'node',
+permissions => {
+ description => "You need read access for the volume.",
+ user => 'all',
+},
+description => "Extract a file or directory (as zip archive) from a PBS
backup.",
+parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ storage => get_standard_option('pve-storage-id'),
+ snapshot => {
+ description => "Backup snapshot identifier.",
+ type => 'string',
+ },
+ filepath => {
+ description => 'base64-path to the directory or file to
download.',
[pve-devel] [PATCH v2 00/13] Single-file-restore GUI for PBS snapshots
Implements the necessary API for allowing single file restore via the PVE web GUI. This requires some adaptations of the HTTP server, to allow data streaming without buffering - otherwise bigger restores would a) use up too much memory b) get slowed down by the pvedaemon->pveproxy->client return path. Instead, we transfer the data via a temporary FIFO. Known issues/further work: * restore VMs are not shown or stopped, they always run until timeout. Ideas would be to stop them when a user closes the restore window (but what if another user/tab/... is using the VM too?), or start a worker task for them (how to detect new VMs? parse proxmox-file-restore status before and after command?). Thoughts appreciated :) * restarting the daemon or proxy marks the worker as failed and correctly stops it, but the browser says the download is OK - just with not enough data. Not the best experience, but unsure on how to fix this. Chunked encoding maybe, and terminate with an invalid chunk in case of error/abort? v2: * attach widget-toolkit patches, forgot them in v1 * drop patch "PBSClient: allow different command execution callback" * drop applied patches, rebase * nit cleanups from Fabian's preliminary review (thanks!) * includes API permission changes to keep them uniform for pve-storage * fixups for Dominic's testing (thanks!) * now supports encrypted CT and VM backups proxmox-backup: Stefan Reiter (3): file-restore: don't force PBS_FINGERPRINT env var client-tools: add crypto_parameters_keep_fd file-restore: support encrypted VM backups src/bin/proxmox-file-restore.rs | 22 +--- src/bin/proxmox_client_tools/key_source.rs | 17 +-- src/bin/proxmox_file_restore/block_driver.rs | 1 + src/bin/proxmox_file_restore/qemu_helper.rs | 12 ++- 4 files changed, 42 insertions(+), 10 deletions(-) common: Stefan Reiter (4): PBSClient: adapt error message to include full package names PBSClient: add file_restore_list command PBSClient: add file_restore_extract function PBSClient: use crypt params for file 'list' and 'extract' src/PVE/PBSClient.pm | 71 ++-- 1 file changed, 69 insertions(+), 2 deletions(-) http-server: Stefan Reiter (2): support streaming data form fh to client allow stream download from path and over pvedaemon-proxy PVE/APIServer/AnyEvent.pm | 145 +++--- 1 file changed, 137 insertions(+), 8 deletions(-) storage: Stefan Reiter (1): add FileRestore API for PBS PVE/API2/Storage/FileRestore.pm | 160 PVE/API2/Storage/Makefile | 2 +- PVE/API2/Storage/Status.pm | 6 ++ 3 files changed, 167 insertions(+), 1 deletion(-) create mode 100644 PVE/API2/Storage/FileRestore.pm proxmox-widget-toolkit: Stefan Reiter (3): Utils: add errorCallback to monStoreErrors FileBrowser: support 'virtual'/'v' file type FileBrowser: show errors in messagebox and allow expand 'all' src/Utils.js | 6 -- src/window/FileBrowser.js | 24 +--- 2 files changed, 25 insertions(+), 5 deletions(-) -- 2.20.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 common 07/13] PBSClient: use crypt params for file 'list' and 'extract'
Necessary for accessing encrypted backups.
Signed-off-by: Stefan Reiter
---
new in v2
src/PVE/PBSClient.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
index e96f20d..537fd83 100644
--- a/src/PVE/PBSClient.pm
+++ b/src/PVE/PBSClient.pm
@@ -133,6 +133,8 @@ my $USE_CRYPT_PARAMS = {
backup => 1,
restore => 1,
'upload-log' => 1,
+list => 1,
+extract => 1,
};
my sub do_raw_client_cmd {
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 proxmox-widget-toolkit 12/13] FileBrowser: support 'virtual'/'v' file type
Denotes objects like disks ".img.fidx" files, which shouldn't be
downloadable, but should still approximate a directory entry.
Signed-off-by: Stefan Reiter
---
new in v2
src/window/FileBrowser.js | 4
1 file changed, 4 insertions(+)
diff --git a/src/window/FileBrowser.js b/src/window/FileBrowser.js
index e90b717..d138d1b 100644
--- a/src/window/FileBrowser.js
+++ b/src/window/FileBrowser.js
@@ -36,6 +36,9 @@ Ext.define('proxmox-file-tree', {
case 's': // socket
icon = 'plug';
break;
+ case 'v': // virtual
+ icon = 'cube';
+ break;
default:
icon = 'file-o';
break;
@@ -217,6 +220,7 @@ Ext.define("Proxmox.window.FileBrowser", {
case 'l': return gettext('Softlink');
case 'p': return gettext('Pipe/Fifo');
case 's': return gettext('Socket');
+ case 'v': return gettext('Virtual');
default: return Proxmox.Utils.unknownText;
}
},
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 proxmox-widget-toolkit 11/13] Utils: add errorCallback to monStoreErrors
Call a function to decide if we want to mask the component. If the
callback returns true, we assume it has already handled the error (i.e.
shown a messagebox or similar) and skip masking.
Signed-off-by: Stefan Reiter
---
new in v2
src/Utils.js | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/Utils.js b/src/Utils.js
index 3fd8f91..ee30027 100644
--- a/src/Utils.js
+++ b/src/Utils.js
@@ -317,7 +317,7 @@ utilities: {
return msg.join('');
},
-monStoreErrors: function(component, store, clearMaskBeforeLoad) {
+monStoreErrors: function(component, store, clearMaskBeforeLoad,
errorCallback) {
if (clearMaskBeforeLoad) {
component.mon(store, 'beforeload', function(s, operation, eOpts) {
Proxmox.Utils.setErrorMask(component, false);
@@ -342,7 +342,9 @@ utilities: {
let error = request._operation.getError();
let msg = Proxmox.Utils.getResponseErrorMessage(error);
- Proxmox.Utils.setErrorMask(component, msg);
+ if (!errorCallback || !errorCallback(error, msg)) {
+ Proxmox.Utils.setErrorMask(component, msg);
+ }
});
},
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 manager 1/2] backupview: add file restore button
Adds it for both BackupViews, on VM view and storage view.
Signed-off-by: Stefan Reiter
---
Forgot to resend this one for v2, it didn't change though AFAIR.
www/manager6/grid/BackupView.js| 23 +++
www/manager6/storage/BackupView.js | 19 +++
2 files changed, 42 insertions(+)
diff --git a/www/manager6/grid/BackupView.js b/www/manager6/grid/BackupView.js
index b50f52ed..7beeca0e 100644
--- a/www/manager6/grid/BackupView.js
+++ b/www/manager6/grid/BackupView.js
@@ -228,6 +228,28 @@ Ext.define('PVE.grid.BackupView', {
},
});
+ let file_restore_btn = Ext.create('Proxmox.button.Button', {
+ text: gettext('File Restore'),
+ disabled: true,
+ selModel: sm,
+ enableFn: function(rec) {
+ return !!rec && isPBS;
+ },
+ handler: function(b, e, rec) {
+ var storage = storagesel.getValue();
+ Ext.create('Proxmox.window.FileBrowser', {
+ title: gettext('File Restore') + " - " + rec.data.text,
+ listURL:
`/api2/json/nodes/localhost/storage/${storage}/file-restore/list`,
+ downloadURL:
`/api2/json/nodes/localhost/storage/${storage}/file-restore/download`,
+ extraParams: {
+ snapshot: rec.data.text,
+ },
+ archive: PVE.Utils.volume_is_qemu_backup(rec.data.volid,
rec.data.format) ?
+ 'all' : undefined,
+ }).show();
+ },
+ });
+
Ext.apply(me, {
selModel: sm,
tbar: {
@@ -238,6 +260,7 @@ Ext.define('PVE.grid.BackupView', {
delete_btn,
'-',
config_btn,
+ file_restore_btn,
'-',
{
xtype: 'proxmoxButton',
diff --git a/www/manager6/storage/BackupView.js
b/www/manager6/storage/BackupView.js
index 3dd500c2..5be18409 100644
--- a/www/manager6/storage/BackupView.js
+++ b/www/manager6/storage/BackupView.js
@@ -133,6 +133,25 @@ Ext.define('PVE.storage.BackupView', {
renderer: PVE.Utils.render_backup_verification,
},
};
+
+ me.tbar.push({
+ xtype: 'proxmoxButton',
+ text: gettext('File Restore'),
+ disabled: true,
+ selModel: sm,
+ handler: function(b, e, rec) {
+ Ext.create('Proxmox.window.FileBrowser', {
+ title: gettext('File Restore') + " - " + rec.data.text,
+ listURL:
`/api2/json/nodes/localhost/storage/${me.storage}/file-restore/list`,
+ downloadURL:
`/api2/json/nodes/localhost/storage/${me.storage}/file-restore/download`,
+ extraParams: {
+ snapshot: rec.data.text,
+ },
+ archive:
PVE.Utils.volume_is_qemu_backup(rec.data.volid, rec.data.format) ?
+ 'all' : undefined,
+ }).show();
+ },
+ });
}
me.callParent();
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v2 manager 2/2] gui: add task name for 'pbs-download'
Signed-off-by: Stefan Reiter
---
new in v2
www/manager6/Utils.js | 1 +
1 file changed, 1 insertion(+)
diff --git a/www/manager6/Utils.js b/www/manager6/Utils.js
index 7fcda854..4b5dc189 100644
--- a/www/manager6/Utils.js
+++ b/www/manager6/Utils.js
@@ -1791,6 +1791,7 @@ Ext.define('PVE.Utils', {
lvmthincreate: [gettext('LVM-Thin Storage'), gettext('Create')],
migrateall: ['', gettext('Migrate all VMs and Containers')],
'move_volume': ['CT', gettext('Move Volume')],
+ 'pbs-download': ['VM/CT', gettext('File Restore Download')],
pull_file: ['CT', gettext('Pull file')],
push_file: ['CT', gettext('Push file')],
qmclone: ['VM', gettext('Clone')],
--
2.20.1
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH firewall] fix #967: source: dest: limit length
On 22.04.21 14:30, Aaron Lauterer wrote: > iptables-restore has a buffer limit of 1024 for paramters [0]. > > If users end up adding a long list of IPs in the source or dest field > they might reach this limit. The result is that the rule will not be > applied and pve-firewall will show some error in the syslog which will > be "hidden" for most users. > > Enforcing a smaller limit ourselves should help to avoid any such > situation. 512 characters should help to not run into any problems that > stem from differences in what counts as character. If people need longer > lists, using IP sets are the better approach anyway. > > [0] http://git.netfilter.org/iptables/tree/iptables/xshared.c?h=v1.8.7#n469 > > Signed-off-by: Aaron Lauterer > --- > src/PVE/Firewall.pm | 2 ++ > 1 file changed, 2 insertions(+) > > applied, thanks! Even in the worst case IP-address length, namely Ipv4-mapped IPv6 (which we do not really support anyway, so only as theoretical worst-case), for example: "::::::192.168.100.228", there we would need 45 + 1 characters per entry plus separator, so even then one could add 11 IPs in there, which is IMO more than enough for direct apply - IPsets should be preferred, like you hint in the gui patch anyway. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [pbs-devel] [PATCH v2 proxmox-backup 01/13] file-restore: don't force PBS_FINGERPRINT env var
On 22.04.21 17:34, Stefan Reiter wrote: > It is valid to not set it, in case the server has a valid certificate. > > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/bin/proxmox_file_restore/qemu_helper.rs | 3 --- > 1 file changed, 3 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [pbs-devel] [PATCH v2 proxmox-backup 02/13] client-tools: add crypto_parameters_keep_fd
On 22.04.21 17:34, Stefan Reiter wrote: > same functionality as crypto_parameters, except it keeps the file > descriptor passed as "keyfd" open (and seeks to the beginning after > reading), if one is given. > > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/bin/proxmox_client_tools/key_source.rs | 17 +++-- > 1 file changed, 15 insertions(+), 2 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [pbs-devel] [PATCH v2 proxmox-backup 03/13] file-restore: support encrypted VM backups
On 22.04.21 17:34, Stefan Reiter wrote: > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/bin/proxmox-file-restore.rs | 22 +--- > src/bin/proxmox_file_restore/block_driver.rs | 1 + > src/bin/proxmox_file_restore/qemu_helper.rs | 9 ++-- > 3 files changed, 27 insertions(+), 5 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH widget-toolkit 4/4] panel: add StatusView from PVE
On 19.04.21 13:00, Dominik Csapak wrote:
> with 2 minor fixups:
> * one lint error
> * binding of the 'updateValues' function in the event
> (we want to avoid breaking this when used in a context where
> a controller exists, in that case using a string will only look in
> the controller and not in the component itself anymore, so use
> the function directly)
>
> Signed-off-by: Dominik Csapak
> ---
> src/Makefile| 1 +
> src/panel/StatusView.js | 125
> 2 files changed, 126 insertions(+)
> create mode 100644 src/panel/StatusView.js
applied, with a some unimportant cleanups and an important fix (or better said,
workaround)
for the use in PVE, see below.
> diff --git a/src/panel/StatusView.js b/src/panel/StatusView.js
> new file mode 100644
> index 000..059508a
> --- /dev/null
> +++ b/src/panel/StatusView.js
> +updateValues: function(store, records, success) {
> + if (!success) {
> + return; // do not update if store load was not successful
> + }
> + var me = this;
> + var itemsToUpdate = me.query('pmxInfoWidget');
does not finds the pveInfoWidgets from, well, PVE, and makes for a pretty boring
view there ;-)
> +
> + itemsToUpdate.forEach(me.updateField, me);
> +
> + me.updateTitle(store);
> +},
> +
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 proxmox-widget-toolkit 12/13] FileBrowser: support 'virtual'/'v' file type
On 22.04.21 17:34, Stefan Reiter wrote: > Denotes objects like disks ".img.fidx" files, which shouldn't be > downloadable, but should still approximate a directory entry. > > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/window/FileBrowser.js | 4 > 1 file changed, 4 insertions(+) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 proxmox-widget-toolkit 13/13] FileBrowser: show errors in messagebox and allow expand 'all'
On 22.04.21 17:34, Stefan Reiter wrote: > If an error is received upon expanding a node, chances are the rest of > the tree is still valid (i.e. opening a partition fails because it > doesn't contain a supported filesystem). Only show an error box for the > user, but don't mask the component in that case. Additionally, disable > the download button. > > Also support an archive set to 'all' to expand all children, useful for > initializing a file-restore VM on initial load. > > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/window/FileBrowser.js | 20 +--- > 1 file changed, 17 insertions(+), 3 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH widget-toolkit/pve-manager] move some code to widget-toolkit
On 19.04.21 13:00, Dominik Csapak wrote: > manager needs a depends on a bumped widget-toolkit ofc > > proxmox-widget-toolkit: > > Dominik Csapak (4): > Utils: add several render functions from PVE > bring over some icons from PVE > Utils: refactor updateColumns from pve-manager > panel: add StatusView from PVE > > pve-manger: > > Dominik Csapak (4): > ui: Utils: use render functions from widget-toolkit > ui: use some icons from widget-toolkit > ui: Utils: use updateColumns from widget-toolkit > ui: panel/StatusView: use from widget-toolkit instead > applied series, besides one issue in the status panel (replied inline) it was all OK, FWICT, thanks! Process wise it would be nicer to include the pbs one here too, or at least the widget-toolkit in the pbs-series, it's a bit hard to follow the correlation else... ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 proxmox-widget-toolkit 11/13] Utils: add errorCallback to monStoreErrors
On 22.04.21 17:34, Stefan Reiter wrote: > Call a function to decide if we want to mask the component. If the > callback returns true, we assume it has already handled the error (i.e. > shown a messagebox or similar) and skip masking. > > Signed-off-by: Stefan Reiter > --- > > new in v2 > > src/Utils.js | 6 -- > 1 file changed, 4 insertions(+), 2 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH-SERIES widget-toolkit/manager] Add checkbox for removing unreferenced disks on guest removal
On 19.04.21 15:14, Fabian Ebner wrote: > by allowing generic additional items for the SafeDestroy window in > widget-toolkit and deriving a SafeDestroyGuest child class from it. > > Also switch over the other SafeDestroy users in PVE. > > > proxmox-widget-toolkit: > > Fabian Ebner (2): > safe destroy: remove purge checkbox > safe destroy: allow specifing additional items > pve-manager: > > Fabian Ebner (5): > ui: add SafeDestroyGuest window > ui: use new SafeDestroyGuest window > ui: safe destroy guest: add checkbox for removal of unreferenced disks > ui: use safe destroy window from proxmox-widget-toolkit > ui: remove SafeDestroy window applied, much thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 common 07/13] PBSClient: use crypt params for file 'list' and 'extract'
On 22.04.21 17:34, Stefan Reiter wrote:
> Necessary for accessing encrypted backups.
>
> Signed-off-by: Stefan Reiter
> ---
>
> new in v2
>
> src/PVE/PBSClient.pm | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
> index e96f20d..537fd83 100644
> --- a/src/PVE/PBSClient.pm
> +++ b/src/PVE/PBSClient.pm
> @@ -133,6 +133,8 @@ my $USE_CRYPT_PARAMS = {
> backup => 1,
> restore => 1,
> 'upload-log' => 1,
> +list => 1,
can break the proxmox-backup-client list command... that's why I wanted to have
a rather clean split between those two..
> +extract => 1,
> };
>
> my sub do_raw_client_cmd {
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] ui: firewall: rule: maxlength for source and dest
On 22.04.21 14:30, Aaron Lauterer wrote: > Limiting the length of the source and dest paramters helps to avoid > problems with iptables-restore which would not apply a rule if a > parameter is larger than the parameter buffer (1024)[0]. As the API is > already limiting this, we should also reflect that in the GUI and give > people a hint that IP sets are most likely the better approach. > > [0] http://git.netfilter.org/iptables/tree/iptables/xshared.c?h=v1.8.7#n469 > > Signed-off-by: Aaron Lauterer > --- > This one "needs" the firewall patch 'fix #967: source: dest: limit length' > > As always when it comes to messages, someone might have a better idea > how to phrase the maxLengthText. > good enough for me :) > www/manager6/grid/FirewallRules.js | 5 - > 1 file changed, 4 insertions(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] api: network: fix #3385 check for used ports before writing
On 19.04.21 20:11, Stoiko Ivanov wrote: > Currently the check for used ports for bonds and bridges happens while > rendering '/etc/network/interfaces.new' in PVE::Inotify (pve-common). > However at that stage the new/updated interface is already merged with > the old settings, making it impossible to indicate where a NIC is > currently used. > > The code is adapted from the renderer in > PVE::Inotify::__write_etc_network_interfaces. > > Tested on a virtual PVE instance. > > Signed-off-by: Stoiko Ivanov > --- > PVE/API2/Network.pm | 37 + > 1 file changed, 37 insertions(+) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied-series: [PATCH V5 pve-cluster 0/5] sdn : add subnets management
On 28.09.20 10:33, Alexandre Derumier wrote: > Following pve-network > > > Alexandre Derumier (5): > add sdn/subnets.cfg > add sdn/ipams.cfg > add priv/ipam.db > add sdn/dns.cfg > rename sdn/.version to sdn/.running-config > > data/PVE/Cluster.pm | 6 +- > data/src/status.c | 6 +- > 2 files changed, 10 insertions(+), 2 deletions(-) > applied series, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH cluster 0/3] logging improvements
On 01.10.20 10:53, Fabian Grünbichler wrote: > noticed while debugging the (as it turns out) missing lock issue that > qb_log is not thread-safe by default, this patch series switches to > threaded, non-blocking logging. > > Fabian Grünbichler (3): > pmxcfs: switch log domain to enum > pmxcfs: set log domain in more places > pmxcfs: enable QB log thread > > data/src/cfs-utils.h | 36 > data/src/dfsm.h | 6 +++--- > data/src/loop.h | 2 +- > data/src/cfs-plug-func.c | 2 ++ > data/src/cfs-plug-link.c | 2 ++ > data/src/cfs-plug-memdb.c | 2 ++ > data/src/cfs-plug.c | 2 ++ > data/src/cfs-utils.c | 16 > data/src/confdb.c | 3 ++- > data/src/database.c | 1 + > data/src/dcdb.c | 3 ++- > data/src/dfsm.c | 18 +++--- > data/src/logger.c | 2 ++ > data/src/loop.c | 11 ++- > data/src/memdb.c | 2 ++ > data/src/pmxcfs.c | 21 - > data/src/quorum.c | 3 ++- > data/src/server.c | 1 + > data/src/status.c | 3 ++- > 19 files changed, 95 insertions(+), 41 deletions(-) > would need a rebase if we still want this ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 pve-access-control] resource pools: add resource pool cli commands
On 29.01.21 08:54, Dylan Whyte wrote: > Add commands for creating, modifying, listing, and deleting resource > pools. > > Signed-off-by: Dylan Whyte > --- > v1->v2: remove command aliases > > PVE/CLI/pveum.pm | 8 +++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH cluster] pvecm: fix typo in description for 'updatecerts'
On 01.02.21 15:47, Oguz Bektas wrote: > Signed-off-by: Oguz Bektas > --- > data/PVE/CLI/pvecm.pm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH v2 storage] pbs: allow setting up a master key
On 08.02.21 14:08, Fabian Grünbichler wrote: > similar to the existing encryption key handling, but without > auto-generation since we only have the public part here. > > Signed-off-by: Fabian Grünbichler > --- > > Notes: > v2: only use master key for backup command > > PVE/API2/Storage/Config.pm | 2 +- > PVE/CLI/pvesm.pm | 14 +- > PVE/Storage/PBSPlugin.pm | 94 +- > 3 files changed, 106 insertions(+), 4 deletions(-) > > applied, thanks! The web-interface would still need support for that, we could reuse the existing fields and detect if it's a master key when a user uploads one there (we would have the full value available), and change the param name for submission. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [RFC PATCH qemu-server] fix bootdisk_size for new bootorder config scheme
On 08.03.21 14:43, Dominik Csapak wrote: > Previously, we ever only had a single boot *disk*, while possibly > having multiple cdroms/nics in the boot order > > e.g. the config: > > boot: dnc > bootdisk: scsi0 > ide0: media=cdrom,none > scsi0: xxx > net0: ... > > would return the size of scsi0 even though it would first boot > from cdrom/network. > > When editing the bootorder with such a legacy config, we > remove the 'bootdisk' property and replace the legacy notation > with an explicit order, but we only search the first disk > for the size now. > > Restore that behaviour by iterating over all disks in the boot > order property string until we get one that is not a cdrom > and has a size. > > Signed-off-by: Dominik Csapak > --- > i cannot remember if that change was deliberate, but at least one > user ran into that: > > https://forum.proxmox.com/threads/possible-bug-boot-disk-size-shows-as-0b.85454/ > @Stefan, can you, as main author behind the change in question, please give this a review - thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH] Add API for VM import
On 12.04.21 12:08, Dominic Jäger wrote: > Extend qm importdisk/importovf functionality to the API. > > Co-authored-by: Fabian Grünbichler > Signed-off-by: Dominic Jäger > --- > v8: > - Fabian moved the import functions into the existing create_vm / > update_vm_api > - Dropped the separate API endpoints & import lock > for the record, we talked for postponing this to 7.0 to improve convergence/synergy (gosh, feels like marketing with those terms ;)) between the create and import wizard. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
