[pve-devel] partially-applied: [PATCH proxmox 1/6] notify: copy sendmail/forward fn's from proxmox_sys
Applied the first patch. The 2nd one I skipped and is now obsolete, since I instead immediately removed the functions since I was doing a major bump to the sys crate anyway. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH installer 00/14] fix #5536: implement post-(auto-)installation notification mechanism
On Thu, Jul 11, 2024 at 06:49:28PM GMT, Stefan Hanreich wrote: > Did a quick smoke test of this series by creating an ISO with an answer > file baked in and checking the response via `nc -l`. Review is inline. Thanks for the review & testing! >From a quick glance all the comments make sense, I'll address them in a v2. > > Consider this: > > Tested-By: Stefan Hanreich > Reviewed-By: Stefan Hanreich > > > On 7/10/24 15:27, Christoph Heiss wrote: > > This implements a mechanism for post-installation "notifications" via a > > POST request [0] when using the auto-installer. > > > > It's implemented as a separate, small utility to facilitate separation > > of concerns and make the information gathering easier by having it > > isolated in one place. > > > > Patches #1 through #10 are simply clean-ups, refactors, etc. that were > > done along the way, which do not impact functionality in any way. > > > > Most interesting here will be patch #12, which adds the actual > > implementation of the post-hook. (Bind-)mounting the installed host > > system is done using the existing `proxmox-chroot` utility, and the HTTP > > POST functionality can fortunately be re-used 1:1 from > > `proxmox-fetch-answer`. > > > > I've also included an example of how the JSON body (pretty-printed for > > readability) of such a post-installation request would look like below, > > for reference. > > > > Tested this with both PVE and PBS ISOs, PMG did not (yet) have a > > release with an auto-installation capable ISO. The only product-specific > > code is the version detection in `proxmox-post-hook`, which - since it > > works the same for PVE and PMG - be no obstacle. > > > > [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5536 > > > > { > > "debian-version": "12.5", > > "product-version": "pve-manager/8.2.2/9355359cd7afbae4", > > "kernel-version": "proxmox-kernel-6.8.4-2-pve-signed", > > "boot-type": "bios", > > "filesystem": "zfs (RAID1)", > > "fqdn": "host.domain", > > "machine-id": "f4bf9711783248b7aaffe3ccbca3e3dc", > > "bootdisk": [ > > { > > "size": 8589934592, > > "udev-properties": { > > "DEVNAME": "/dev/vda", [..] > > } > > }, > > { > > "size": 8589934592, > > "udev-properties": { > > "DEVNAME": "/dev/vdb", [..] > > } > > } > > ], > > "management-nic": { > > "mac": "de:ad:f0:0d:12:34", > > "address": "10.0.0.10/24", > > "udev-properties": { > > "INTERFACE": "enp6s18", [..] > > } > > }, > > "ssh-public-host-keys": { > > "ecdsa": "ecdsa-sha2-nistp256 [..] root@host", > > "ed25519": "ssh-ed25519 [..] root@host", > > "rsa": "ssh-rsa [..] root@host", > > } > > } > > > > Christoph Heiss (14): chroot: print full anyhow message > > tree-wide: fix some typos > > tree-wide: collect hardcoded installer runtime directory strings into > > constant > > common: simplify filesystem type serializing & Display trait impl > > common: setup: serialize `target_hd` as string explicitly > > common: split out installer setup files loading functionality > > debian: strip unused library dependencies > > fetch-answer: move http-related code to gated module in > > installer-common > > tree-wide: convert some more crates to use workspace dependencies > > auto-installer: tests: replace left/right with got/expected in output > > auto-installer: answer: add `posthook` section > > fix #5536: add post-hook utility for sending notifications after > > auto-install > > fix #5536: post-hook: add some unit tests > > unconfigured.sh: run proxmox-post-hook after successful auto-install > > > > Cargo.toml| 11 + > > Makefile | 8 +- > > debian/control| 1 + > > debian/install| 1 + > > debian/rules | 9 + > > proxmox-auto-install-assistant/Cargo.toml | 14 +- > > proxmox-auto-installer/Cargo.toml | 15 +- > > proxmox-auto-installer/src/answer.rs | 27 +- > > .../src/bin/proxmox-auto-installer.rs | 15 +- > > proxmox-auto-installer/src/sysinfo.rs | 10 +- > > proxmox-auto-installer/src/utils.rs | 15 +- > > proxmox-auto-installer/tests/parse-answer.rs | 42 +- > > proxmox-chroot/Cargo.toml | 8 +- > > proxmox-chroot/src/main.rs| 19 +- > > proxmox-fetch-answer/Cargo.toml | 17 +- > > .../src/fetch_plugins/http.rs | 100 +--- > > .../src/fetch_plugins/partition.rs| 14 +- > > proxmox-installer-common/Cargo.toml | 26 +- > > proxmox-installer-common/src/http.rs | 94 > > proxmox-installer-common/src/lib.rs | 5 + > > proxmox-installer-common/src/options.rs | 109 ++-- > > proxmox-installer-common/src/setup.rs | 108 +--- > > proxmox-in
Re: [pve-devel] [PATCH installer v3 4/4] low-level: install: check for already-existing `rpool` on install
only one small nit inline On 2024-07-11 13:57, Christoph Heiss wrote: .. much in the same manner as the detection for LVM works. zpools can only be renamed by importing them with a new name, so unfortunately the import-export dance is needed. Signed-off-by: Christoph Heiss --- Changes v2 -> v3: * skip rename prompt in auto-install mode Changes v1 -> v2: * use unique pool id for renaming instead of generating random id ourselves * moved renaming functionality to own subroutine in new Proxmox::Sys::ZFS module Proxmox/Install.pm | 33 + Proxmox/Sys/ZFS.pm | 20 ++-- 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 7342e4b..ba699fa 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -16,6 +16,7 @@ use Proxmox::Install::StorageConfig; use Proxmox::Sys::Block qw(get_cached_disks wipe_disk partition_bootable_disk); use Proxmox::Sys::Command qw(run_command syscmd); use Proxmox::Sys::File qw(file_read_firstline file_write_all); +use Proxmox::Sys::ZFS; use Proxmox::UI; # TODO: move somewhere better? @@ -169,9 +170,41 @@ sub btrfs_create { syscmd($cmd); } +sub zfs_ask_existing_zpool_rename { +my ($pool_name) = @_; + +# At this point, no pools should be imported/active +my $exported_pools = Proxmox::Sys::ZFS::get_exported_pools(); + +foreach (@$exported_pools) { + next if $_->{name} ne $pool_name || $_->{state} ne 'ONLINE'; + my $renamed_pool = "$_->{name}-OLD-$_->{id}"; + + my $response_ok = Proxmox::Install::Config::get_existing_storage_auto_rename(); maybe we want to name this differently to avoid confusion? response_ok -> do_rename or something in that regard? but that could be done in a follow up patch as well if we want to + if (!$response_ok) { + $response_ok = Proxmox::UI::prompt( + "A ZFS pool named '$_->{name}' (id $_->{id}) already exists on the system.\n\n" . + "Do you want to rename the pool to '$renamed_pool' before continuing " . + "or cancel the installation?" + ); + } + + # Import the pool using its id, as that is unique and works even if there are + # multiple zpools with the same name. + if ($response_ok) { + Proxmox::Sys::ZFS::rename_pool($_->{id}, $renamed_pool); + } else { + warn "Canceled installation as requested by user, due to already existing ZFS pool '$pool_name'\n"; + die "\n"; # causes abort without re-showing an error dialogue + } +} +} + [...] ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH installer v3 0/4] add check/rename for already-existing ZFS rpool
tested this series more thoroughly and checked with an existing root pool all 3 install options: * TUI * GUI * auto Worked well in 3 cases to rename. One oddity seems to be if we cancel the rename in the TUI installer. it jumps to 100% and stays there and I then need to manually abort. Jumping to a "setup cancelled" box that we confirm before quitting would be nice. This is how the GUI installer handles it. Otherwise, besides this and a tiny nit in patch 4, consider this: Reviewed-By: Aaron Lauterer Tested-By: Aaron Lauterer On 2024-07-11 13:57, Christoph Heiss wrote: Pretty straight forward overall, implements a check for an existing `rpool` on the system and ask the user whether they would like to rename it, much in the same way as it works for VGs already. Without this, the installer would silently create a second (and thus conflicting) `rpool` and cause a boot failure after the installation, since it does not know which pool to import exactly. v1: https://lists.proxmox.com/pipermail/pve-devel/2024-May/063874.html v2: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064619.html Notable changes v2 -> v3: * make low-level option lvm_auto_rename more generic * skip rename prompt in auto-install mode Notable changes v1 -> v2: * incorporated Aarons suggestions from v1 * rewrote tests to use a pre-defined input instead * moved pool renaming to own subroutine * documented all new subroutines * split out tests into own patch Christoph Heiss (4): test: add test cases for new zfs module low-level: install: check for already-existing `rpool` on install install: config: rename option lvm_auto_rename -> existing_storage_auto_rename low-level: install: automatically rename existing rpools on auto-installs Proxmox/Install.pm| 35 +++- Proxmox/Install/Config.pm | 6 +- Proxmox/Sys/ZFS.pm| 20 ++- proxmox-auto-installer/src/utils.rs | 2 +- .../resources/parse_answer/disk_match.json| 2 +- .../parse_answer/disk_match_all.json | 2 +- .../parse_answer/disk_match_any.json | 2 +- .../tests/resources/parse_answer/minimal.json | 2 +- .../resources/parse_answer/nic_matching.json | 2 +- .../resources/parse_answer/specific_nic.json | 2 +- .../tests/resources/parse_answer/zfs.json | 2 +- proxmox-installer-common/src/setup.rs | 2 +- proxmox-tui-installer/src/setup.rs| 2 +- test/Makefile | 7 ++- test/zfs-get-pool-list.pl | 57 +++ 15 files changed, 128 insertions(+), 17 deletions(-) create mode 100755 test/zfs-get-pool-list.pl ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-perl-rs v2 03/12] common: notify: add bindings for webhook API routes
Signed-off-by: Lukas Wagner --- common/src/notify.rs | 63 1 file changed, 63 insertions(+) diff --git a/common/src/notify.rs b/common/src/notify.rs index e1b006b..fe192d5 100644 --- a/common/src/notify.rs +++ b/common/src/notify.rs @@ -19,6 +19,9 @@ mod export { DeleteableSmtpProperty, SmtpConfig, SmtpConfigUpdater, SmtpMode, SmtpPrivateConfig, SmtpPrivateConfigUpdater, }; +use proxmox_notify::endpoints::webhook::{ +DeleteableWebhookProperty, WebhookConfig, WebhookConfigUpdater, +}; use proxmox_notify::matcher::{ CalendarMatcher, DeleteableMatcherProperty, FieldMatcher, MatchModeOperator, MatcherConfig, MatcherConfigUpdater, SeverityMatcher, @@ -393,6 +396,66 @@ mod export { api::smtp::delete_endpoint(&mut config, name) } +#[export(serialize_error)] +fn get_webhook_endpoints( +#[try_from_ref] this: &NotificationConfig, +) -> Result, HttpError> { +let config = this.config.lock().unwrap(); +api::webhook::get_endpoints(&config) +} + +#[export(serialize_error)] +fn get_webhook_endpoint( +#[try_from_ref] this: &NotificationConfig, +id: &str, +) -> Result { +let config = this.config.lock().unwrap(); +api::webhook::get_endpoint(&config, id) +} + +#[export(serialize_error)] +#[allow(clippy::too_many_arguments)] +fn add_webhook_endpoint( +#[try_from_ref] this: &NotificationConfig, +endpoint_config: WebhookConfig, +) -> Result<(), HttpError> { +let mut config = this.config.lock().unwrap(); +api::webhook::add_endpoint( +&mut config, +endpoint_config, +) +} + +#[export(serialize_error)] +#[allow(clippy::too_many_arguments)] +fn update_webhook_endpoint( +#[try_from_ref] this: &NotificationConfig, +name: &str, +config_updater: WebhookConfigUpdater, +delete: Option>, +digest: Option<&str>, +) -> Result<(), HttpError> { +let mut config = this.config.lock().unwrap(); +let digest = decode_digest(digest)?; + +api::webhook::update_endpoint( +&mut config, +name, +config_updater, +delete.as_deref(), +digest.as_deref(), +) +} + +#[export(serialize_error)] +fn delete_webhook_endpoint( +#[try_from_ref] this: &NotificationConfig, +name: &str, +) -> Result<(), HttpError> { +let mut config = this.config.lock().unwrap(); +api::webhook::delete_endpoint(&mut config, name) +} + #[export(serialize_error)] fn get_matchers( #[try_from_ref] this: &NotificationConfig, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-perl-rs v2 04/12] common: notify: add bindings for get_targets
This allows us to drop the impl of that function on the perl side. Signed-off-by: Lukas Wagner --- common/src/notify.rs | 9 + 1 file changed, 9 insertions(+) diff --git a/common/src/notify.rs b/common/src/notify.rs index fe192d5..0f8a35d 100644 --- a/common/src/notify.rs +++ b/common/src/notify.rs @@ -27,6 +27,7 @@ mod export { MatcherConfigUpdater, SeverityMatcher, }; use proxmox_notify::{api, Config, Notification, Severity}; +use proxmox_notify::api::Target; pub struct NotificationConfig { config: Mutex, @@ -112,6 +113,14 @@ mod export { api::common::send(&config, ¬ification) } +#[export(serialize_error)] +fn get_targets( +#[try_from_ref] this: &NotificationConfig, +) -> Result, HttpError> { +let config = this.config.lock().unwrap(); +api::get_targets(&config) +} + #[export(serialize_error)] fn test_target( #[try_from_ref] this: &NotificationConfig, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v2 06/12] api: notifications: use get_targets impl from proxmox-notify
The get_targets API endpoint is now implemented in Rust. Signed-off-by: Lukas Wagner --- PVE/API2/Cluster/Notifications.pm | 34 +-- 1 file changed, 1 insertion(+), 33 deletions(-) diff --git a/PVE/API2/Cluster/Notifications.pm b/PVE/API2/Cluster/Notifications.pm index 68fdda2a..10b611c9 100644 --- a/PVE/API2/Cluster/Notifications.pm +++ b/PVE/API2/Cluster/Notifications.pm @@ -170,39 +170,7 @@ __PACKAGE__->register_method ({ my $config = PVE::Notify::read_config(); my $targets = eval { - my $result = []; - - for my $target (@{$config->get_sendmail_endpoints()}) { - push @$result, { - name => $target->{name}, - comment => $target->{comment}, - type => 'sendmail', - disable => $target->{disable}, - origin => $target->{origin}, - }; - } - - for my $target (@{$config->get_gotify_endpoints()}) { - push @$result, { - name => $target->{name}, - comment => $target->{comment}, - type => 'gotify', - disable => $target->{disable}, - origin => $target->{origin}, - }; - } - - for my $target (@{$config->get_smtp_endpoints()}) { - push @$result, { - name => $target->{name}, - comment => $target->{comment}, - type => 'smtp', - disable => $target->{disable}, - origin => $target->{origin}, - }; - } - - $result + $config->get_targets(); }; raise_api_error($@) if $@; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-backup v2 10/12] ui: utils: enable webhook edit window
This allows users to add/edit new webhook targets. Signed-off-by: Lukas Wagner --- www/Utils.js | 5 + 1 file changed, 5 insertions(+) diff --git a/www/Utils.js b/www/Utils.js index 4853be36..b715972f 100644 --- a/www/Utils.js +++ b/www/Utils.js @@ -482,6 +482,11 @@ Ext.define('PBS.Utils', { ipanel: 'pmxGotifyEditPanel', iconCls: 'fa-bell-o', }, + webhook: { + name: 'Webhook', + ipanel: 'pmxWebhookEditPanel', + iconCls: 'fa-bell-o', + }, }; }, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-backup v2 09/12] api: notification: add API routes for webhook targets
Copied and adapted from the Gotify ones. Signed-off-by: Lukas Wagner --- src/api2/config/notifications/mod.rs | 2 + src/api2/config/notifications/webhook.rs | 175 +++ 2 files changed, 177 insertions(+) create mode 100644 src/api2/config/notifications/webhook.rs diff --git a/src/api2/config/notifications/mod.rs b/src/api2/config/notifications/mod.rs index dfe82ed0..81ca9800 100644 --- a/src/api2/config/notifications/mod.rs +++ b/src/api2/config/notifications/mod.rs @@ -22,6 +22,7 @@ pub mod matchers; pub mod sendmail; pub mod smtp; pub mod targets; +pub mod webhook; #[sortable] const SUBDIRS: SubdirMap = &sorted!([ @@ -41,6 +42,7 @@ const ENDPOINT_SUBDIRS: SubdirMap = &sorted!([ ("gotify", &gotify::ROUTER), ("sendmail", &sendmail::ROUTER), ("smtp", &smtp::ROUTER), +("webhook", &webhook::ROUTER), ]); const ENDPOINT_ROUTER: Router = Router::new() diff --git a/src/api2/config/notifications/webhook.rs b/src/api2/config/notifications/webhook.rs new file mode 100644 index ..4a040024 --- /dev/null +++ b/src/api2/config/notifications/webhook.rs @@ -0,0 +1,175 @@ +use anyhow::Error; +use serde_json::Value; + +use proxmox_notify::endpoints::webhook::{ +DeleteableWebhookProperty, WebhookConfig, WebhookConfigUpdater, +}; +use proxmox_notify::schema::ENTITY_NAME_SCHEMA; +use proxmox_router::{Permission, Router, RpcEnvironment}; +use proxmox_schema::api; + +use pbs_api_types::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA}; + +#[api( +protected: true, +input: { +properties: {}, +}, +returns: { +description: "List of webhook endpoints.", +type: Array, +items: { type: WebhookConfig }, +}, +access: { +permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false), +}, +)] +/// List all webhook endpoints. +pub fn list_endpoints( +_param: Value, +_rpcenv: &mut dyn RpcEnvironment, +) -> Result, Error> { +let config = pbs_config::notifications::config()?; + +let endpoints = proxmox_notify::api::webhook::get_endpoints(&config)?; + +Ok(endpoints) +} + +#[api( +protected: true, +input: { +properties: { +name: { +schema: ENTITY_NAME_SCHEMA, +} +}, +}, +returns: { type: WebhookConfig }, +access: { +permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_AUDIT, false), +}, +)] +/// Get a webhook endpoint. +pub fn get_endpoint(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result { +let config = pbs_config::notifications::config()?; +let endpoint = proxmox_notify::api::webhook::get_endpoint(&config, &name)?; + +rpcenv["digest"] = hex::encode(config.digest()).into(); + +Ok(endpoint) +} + +#[api( +protected: true, +input: { +properties: { +endpoint: { +type: WebhookConfig, +flatten: true, +}, +}, +}, +access: { +permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false), +}, +)] +/// Add a new webhook endpoint. +pub fn add_endpoint( +endpoint: WebhookConfig, +_rpcenv: &mut dyn RpcEnvironment, +) -> Result<(), Error> { +let _lock = pbs_config::notifications::lock_config()?; +let mut config = pbs_config::notifications::config()?; + +proxmox_notify::api::webhook::add_endpoint(&mut config, endpoint)?; + +pbs_config::notifications::save_config(config)?; +Ok(()) +} + +#[api( +protected: true, +input: { +properties: { +name: { +schema: ENTITY_NAME_SCHEMA, +}, +updater: { +type: WebhookConfigUpdater, +flatten: true, +}, +delete: { +description: "List of properties to delete.", +type: Array, +optional: true, +items: { +type: DeleteableWebhookProperty, +} +}, +digest: { +optional: true, +schema: PROXMOX_CONFIG_DIGEST_SCHEMA, +}, +}, +}, +access: { +permission: &Permission::Privilege(&["system", "notifications"], PRIV_SYS_MODIFY, false), +}, +)] +/// Update webhook endpoint. +pub fn update_endpoint( +name: String, +updater: WebhookConfigUpdater, +delete: Option>, +digest: Option, +_rpcenv: &mut dyn RpcEnvironment, +) -> Result<(), Error> { +let _lock = pbs_config::notifications::lock_config()?; +let mut config = pbs_config::notifications::config()?; +let digest = digest.map(hex::decode).transpose()?; + +proxmox_notify::api::webhook::update_endpoint( +&mut config, +&name, +updater, +delete.as_deref(), +digest.as_deref(), +)?; + +pbs_config::notifications::save_config
[pve-devel] [PATCH manager v2 07/12] api: add routes for webhook notification endpoints
These just call the API implementation via the perl-rs bindings. Signed-off-by: Lukas Wagner --- PVE/API2/Cluster/Notifications.pm | 263 +- 1 file changed, 262 insertions(+), 1 deletion(-) diff --git a/PVE/API2/Cluster/Notifications.pm b/PVE/API2/Cluster/Notifications.pm index 10b611c9..eae2d436 100644 --- a/PVE/API2/Cluster/Notifications.pm +++ b/PVE/API2/Cluster/Notifications.pm @@ -108,6 +108,7 @@ __PACKAGE__->register_method ({ { name => 'gotify' }, { name => 'sendmail' }, { name => 'smtp' }, + { name => 'webhook' }, ]; return $result; @@ -144,7 +145,7 @@ __PACKAGE__->register_method ({ 'type' => { description => 'Type of the target.', type => 'string', - enum => [qw(sendmail gotify smtp)], + enum => [qw(sendmail gotify smtp webhook)], }, 'comment' => { description => 'Comment', @@ -1094,6 +1095,266 @@ __PACKAGE__->register_method ({ } }); +my $webhook_properties = { +name => { + description => 'The name of the endpoint.', + type => 'string', + format => 'pve-configid', +}, +url => { + description => 'Server URL', + type => 'string', +}, +method => { + description => 'HTTP method', + type => 'string', + enum => [qw(post put get)], +}, +header => { + description => 'HTTP headers to set. These have to be formatted as' + . ' a property string in the format name=,value=', + type => 'array', + items => { + type => 'string', + }, + optional => 1, +}, +body => { + description => 'HTTP body, base64 encoded', + type => 'string', + optional => 1, +}, +secret => { + description => 'Secrets to set. These have to be formatted as' + . ' a property string in the format name=,value=', + type => 'array', + items => { + type => 'string', + }, + optional => 1, +}, +comment => { + description => 'Comment', + type => 'string', + optional => 1, +}, +disable => { + description => 'Disable this target', + type => 'boolean', + optional => 1, + default => 0, +}, +}; + +__PACKAGE__->register_method ({ +name => 'get_webhook_endpoints', +path => 'endpoints/webhook', +method => 'GET', +description => 'Returns a list of all webhook endpoints', +protected => 1, +permissions => { + check => ['perm', '/mapping/notifications', ['Mapping.Modify']], + check => ['perm', '/mapping/notifications', ['Mapping.Audit']], +}, +parameters => { + additionalProperties => 0, + properties => {}, +}, +returns => { + type => 'array', + items => { + type => 'object', + properties => { + %$webhook_properties, + 'origin' => { + description => 'Show if this entry was created by a user or was built-in', + type => 'string', + enum => [qw(user-created builtin modified-builtin)], + }, + }, + }, + links => [ { rel => 'child', href => '{name}' } ], +}, +code => sub { + my $config = PVE::Notify::read_config(); + my $rpcenv = PVE::RPCEnvironment::get(); + + my $entities = eval { + $config->get_webhook_endpoints(); + }; + raise_api_error($@) if $@; + + return $entities; +} +}); + +__PACKAGE__->register_method ({ +name => 'get_webhook_endpoint', +path => 'endpoints/webhook/{name}', +method => 'GET', +description => 'Return a specific webhook endpoint', +protected => 1, +permissions => { + check => ['or', + ['perm', '/mapping/notifications', ['Mapping.Modify']], + ['perm', '/mapping/notifications', ['Mapping.Audit']], + ], +}, +parameters => { + additionalProperties => 0, + properties => { + name => { + type => 'string', + format => 'pve-configid', + description => 'Name of the endpoint.' + }, + } +}, +returns => { + type => 'object', + properties => { + %$webhook_properties, + digest => get_standard_option('pve-config-digest'), + } +}, +code => sub { + my ($param) = @_; + my $name = extract_param($param, 'name'); + + my $config = PVE::Notify::read_config(); + my $endpoint = eval { + $config->get_webhook_endpoint($name) + }; + + raise_api_error($@) if $@; + $endpoint->{digest} = $config->digest(); + + return $endpoint; +} +}); + +__PACKAGE__->register_method ({ +name => 'create_webhook_endpoint', +path => 'endpoints/webhook', +
[pve-devel] [PATCH proxmox v2 01/12] notify: implement webhook targets
This target type allows users to perform HTTP requests to arbitrary third party (notification) services, for instance ntfy.sh/Discord/Slack. The configuration for these endpoints allows one to freely configure the URL, HTTP Method, headers and body. The URL, header values and body support handlebars templating to inject notification text, metadata and secrets. Secrets are stored in the protected configuration file (e.g. /etc/pve/priv/notification.cfg) as key value pairs, allowing users to protect sensitive tokens/passwords. Secrets are accessible in handlebar templating via the secrets.* namespace, e.g. if there is a secret named 'token', a body could contain '{{ secrets.token }}' to inject the token into the payload. A couple of handlebars helpers are also provided: - url-encoding (useful for templating in URLs) - escape (escape any control characters in strings) - json (print a property as json) In the configuration, the body, header values and secret values are stored in base64 encoding so that we can store any string we want. Signed-off-by: Lukas Wagner --- proxmox-notify/Cargo.toml | 9 +- proxmox-notify/src/config.rs| 23 ++ proxmox-notify/src/endpoints/mod.rs | 2 + proxmox-notify/src/endpoints/webhook.rs | 509 proxmox-notify/src/lib.rs | 17 + 5 files changed, 557 insertions(+), 3 deletions(-) create mode 100644 proxmox-notify/src/endpoints/webhook.rs diff --git a/proxmox-notify/Cargo.toml b/proxmox-notify/Cargo.toml index 7801814d..484aff19 100644 --- a/proxmox-notify/Cargo.toml +++ b/proxmox-notify/Cargo.toml @@ -9,13 +9,15 @@ exclude.workspace = true [dependencies] anyhow.workspace = true -base64.workspace = true +base64 = { workspace = true, optional = true } const_format.workspace = true handlebars = { workspace = true } +http = { workspace = true, optional = true } lettre = { workspace = true, optional = true } log.workspace = true mail-parser = { workspace = true, optional = true } openssl.workspace = true +percent-encoding = { workspace = true, optional = true } regex.workspace = true serde = { workspace = true, features = ["derive"] } serde_json.workspace = true @@ -31,10 +33,11 @@ proxmox-time.workspace = true proxmox-uuid = { workspace = true, features = ["serde"] } [features] -default = ["sendmail", "gotify", "smtp"] +default = ["sendmail", "gotify", "smtp", "webhook"] mail-forwarder = ["dep:mail-parser", "dep:proxmox-sys"] -sendmail = ["dep:proxmox-sys"] +sendmail = ["dep:proxmox-sys", "dep:base64"] gotify = ["dep:proxmox-http"] pve-context = ["dep:proxmox-sys"] pbs-context = ["dep:proxmox-sys"] smtp = ["dep:lettre"] +webhook = ["dep:base64", "dep:http", "dep:percent-encoding", "dep:proxmox-http"] diff --git a/proxmox-notify/src/config.rs b/proxmox-notify/src/config.rs index 789c4a7d..4d0b53f7 100644 --- a/proxmox-notify/src/config.rs +++ b/proxmox-notify/src/config.rs @@ -57,6 +57,17 @@ fn config_init() -> SectionConfig { GOTIFY_SCHEMA, )); } +#[cfg(feature = "webhook")] +{ +use crate::endpoints::webhook::{WebhookConfig, WEBHOOK_TYPENAME}; + +const WEBHOOK_SCHEMA: &ObjectSchema = WebhookConfig::API_SCHEMA.unwrap_object_schema(); +config.register_plugin(SectionConfigPlugin::new( +WEBHOOK_TYPENAME.to_string(), +Some(String::from("name")), +WEBHOOK_SCHEMA, +)); +} const MATCHER_SCHEMA: &ObjectSchema = MatcherConfig::API_SCHEMA.unwrap_object_schema(); config.register_plugin(SectionConfigPlugin::new( @@ -110,6 +121,18 @@ fn private_config_init() -> SectionConfig { )); } +#[cfg(feature = "webhook")] +{ +use crate::endpoints::webhook::{WebhookPrivateConfig, WEBHOOK_TYPENAME}; + +const WEBHOOK_SCHEMA: &ObjectSchema = +WebhookPrivateConfig::API_SCHEMA.unwrap_object_schema(); +config.register_plugin(SectionConfigPlugin::new( +WEBHOOK_TYPENAME.to_string(), +Some(String::from("name")), +WEBHOOK_SCHEMA, +)); +} config } diff --git a/proxmox-notify/src/endpoints/mod.rs b/proxmox-notify/src/endpoints/mod.rs index 97f79fcc..f20bee21 100644 --- a/proxmox-notify/src/endpoints/mod.rs +++ b/proxmox-notify/src/endpoints/mod.rs @@ -4,5 +4,7 @@ pub mod gotify; pub mod sendmail; #[cfg(feature = "smtp")] pub mod smtp; +#[cfg(feature = "webhook")] +pub mod webhook; mod common; diff --git a/proxmox-notify/src/endpoints/webhook.rs b/proxmox-notify/src/endpoints/webhook.rs new file mode 100644 index ..7e976f6b --- /dev/null +++ b/proxmox-notify/src/endpoints/webhook.rs @@ -0,0 +1,509 @@ +use handlebars::{ +Context as HandlebarsContext, Handlebars, Helper, HelperResult, Output, RenderContext, +RenderError as HandlebarsRenderError, +}; +use http::Request; +use percent_encoding::AsciiSet; +use proxmox_schema::property_string::PropertyString; +use
[pve-devel] [RFC many v2 00/12] notifications: add support for webhook endpoints
Sending as an RFC because I don't want this merged yet; that being said, the feature should be mostly finished at this point, I'd appreciate any reviews and feedback. This series adds support for webhook notification targets to PVE and PBS. A webhook is a HTTP API route provided by a third-party service that can be used to inform the third-party about an event. In our case, we can easily interact with various third-party notification/messaging systems and send PVE/PBS notifications via this service. The changes were tested against ntfy.sh, Discord and Slack. The configuration of webhook targets allows one to configure: - The URL - The HTTP method (GET/POST/PUT) - HTTP Headers - Body One can use handlebar templating to inject notification text and metadata in the url, headers and body. One challenge is the handling of sensitve tokens and other secrets. Since the endpoint is completely generic, we cannot know in advance whether the body/header/url contains sensitive values. Thus we add 'secrets' which are stored in the protected config only accessible by root (e.g. /etc/pve/priv/notifications.cfg). These secrets are accessible in URLs/headers/body via templating: Url: https://example.com/{{ secrets.token }} Secrets can only be set and updated, but never retrieved via the API. In the UI, secrets are handled like other secret tokens/passwords. Bumps for PVE: - libpve-rs-perl needs proxmox-notify bumped - pve-manager needs bumped proxmox-widget-toolkit and libpve-rs-perl bumped - proxmox-mail-forward needs proxmox-notify bumped Bumps for PBS: - proxmox-backup needs proxmox-notify bumped - proxmox-mail-forward needs proxmox-notify bumped Changes v1 -> v2: - Rebase proxmox-notify changes proxmox: Lukas Wagner (2): notify: implement webhook targets notify: add api for webhook targets proxmox-notify/Cargo.toml | 9 +- proxmox-notify/src/api/mod.rs | 20 + proxmox-notify/src/api/webhook.rs | 406 +++ proxmox-notify/src/config.rs| 23 ++ proxmox-notify/src/endpoints/mod.rs | 2 + proxmox-notify/src/endpoints/webhook.rs | 509 proxmox-notify/src/lib.rs | 17 + 7 files changed, 983 insertions(+), 3 deletions(-) create mode 100644 proxmox-notify/src/api/webhook.rs create mode 100644 proxmox-notify/src/endpoints/webhook.rs proxmox-perl-rs: Lukas Wagner (2): common: notify: add bindings for webhook API routes common: notify: add bindings for get_targets common/src/notify.rs | 72 1 file changed, 72 insertions(+) proxmox-widget-toolkit: Lukas Wagner (1): notification: add UI for adding/updating webhook targets src/Makefile | 1 + src/Schema.js | 5 + src/panel/WebhookEditPanel.js | 417 ++ 3 files changed, 423 insertions(+) create mode 100644 src/panel/WebhookEditPanel.js pve-manager: Lukas Wagner (2): api: notifications: use get_targets impl from proxmox-notify api: add routes for webhook notification endpoints PVE/API2/Cluster/Notifications.pm | 297 ++ 1 file changed, 263 insertions(+), 34 deletions(-) pve-docs: Lukas Wagner (1): notification: add documentation for webhook target endpoints. notifications.adoc | 93 ++ 1 file changed, 93 insertions(+) proxmox-backup: Lukas Wagner (3): api: notification: add API routes for webhook targets ui: utils: enable webhook edit window docs: notification: add webhook endpoint documentation docs/notifications.rst | 100 + src/api2/config/notifications/mod.rs | 2 + src/api2/config/notifications/webhook.rs | 175 +++ www/Utils.js | 5 + 4 files changed, 282 insertions(+) create mode 100644 src/api2/config/notifications/webhook.rs proxmox-mail-forward: Lukas Wagner (1): bump proxmox-notify dependency Cargo.toml | 2 +- debian/control | 8 2 files changed, 5 insertions(+), 5 deletions(-) Summary over all repositories: 19 files changed, 2121 insertions(+), 42 deletions(-) -- Generated by git-murpp 0.7.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-mail-forward v2 12/12] bump proxmox-notify dependency
Signed-off-by: Lukas Wagner --- Cargo.toml | 2 +- debian/control | 8 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index f39d118..49ca079 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -20,4 +20,4 @@ nix = "0.26" syslog = "6.0" proxmox-sys = "0.5.3" -proxmox-notify = {version = "0.4", features = ["mail-forwarder", "pve-context", "pbs-context"] } +proxmox-notify = {version = "0.5", features = ["mail-forwarder", "pve-context", "pbs-context"] } diff --git a/debian/control b/debian/control index 7329a24..0ab74a9 100644 --- a/debian/control +++ b/debian/control @@ -6,10 +6,10 @@ Build-Depends: cargo:native, librust-anyhow-1+default-dev, librust-log-0.4+default-dev (>= 0.4.17-~~), librust-nix-0.26+default-dev, - librust-proxmox-notify-0.4+default-dev, - librust-proxmox-notify-0.4+mail-forwarder-dev, - librust-proxmox-notify-0.4+pbs-context-dev, - librust-proxmox-notify-0.4+pve-context-dev, + librust-proxmox-notify-0.5+default-dev, + librust-proxmox-notify-0.5+mail-forwarder-dev, + librust-proxmox-notify-0.5+pbs-context-dev, + librust-proxmox-notify-0.5+pve-context-dev, librust-proxmox-sys-0.5+default-dev (>= 0.5.3-~~), librust-syslog-6+default-dev, libstd-rust-dev, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH widget-toolkit v2 05/12] notification: add UI for adding/updating webhook targets
The widgets for editing the headers/secrets were adapted from the 'Tag Edit' dialog from PVE's datacenter options. Apart from that, the new dialog is rather standard. I've decided to put the http method and url in a single row, mostly to save space and also to make it analogous to how an actual http request is structured (VERB URL, followed by headers, followed by the body). The secrets are a mechanism to store tokens/passwords in the protected notification config. Secrets are accessible via templating in the URL, headers and body via {{ secrets.NAME }}. Secrets can only be set/updated, but not retrieved/displayed. Signed-off-by: Lukas Wagner --- src/Makefile | 1 + src/Schema.js | 5 + src/panel/WebhookEditPanel.js | 417 ++ 3 files changed, 423 insertions(+) create mode 100644 src/panel/WebhookEditPanel.js diff --git a/src/Makefile b/src/Makefile index 0478251..cfaffd7 100644 --- a/src/Makefile +++ b/src/Makefile @@ -78,6 +78,7 @@ JSSRC=\ panel/StatusView.js \ panel/TfaView.js\ panel/NotesView.js \ + panel/WebhookEditPanel.js \ window/Edit.js \ window/PasswordEdit.js \ window/SafeDestroy.js \ diff --git a/src/Schema.js b/src/Schema.js index 42541e0..cd1c306 100644 --- a/src/Schema.js +++ b/src/Schema.js @@ -65,6 +65,11 @@ Ext.define('Proxmox.Schema', { // a singleton ipanel: 'pmxGotifyEditPanel', iconCls: 'fa-bell-o', }, + webhook: { + name: 'Webhook', + ipanel: 'pmxWebhookEditPanel', + iconCls: 'fa-bell-o', + }, }, // to add or change existing for product specific ones diff --git a/src/panel/WebhookEditPanel.js b/src/panel/WebhookEditPanel.js new file mode 100644 index 000..dfc7f3f --- /dev/null +++ b/src/panel/WebhookEditPanel.js @@ -0,0 +1,417 @@ +Ext.define('Proxmox.panel.WebhookEditPanel', { +extend: 'Proxmox.panel.InputPanel', +xtype: 'pmxWebhookEditPanel', +mixins: ['Proxmox.Mixin.CBind'], +onlineHelp: 'notification_targets_webhook', + +type: 'webhook', + +columnT: [ + +], + +column1: [ + { + xtype: 'pmxDisplayEditField', + name: 'name', + cbind: { + value: '{name}', + editable: '{isCreate}', + }, + fieldLabel: gettext('Endpoint Name'), + allowBlank: false, + }, +], + +column2: [ + { + xtype: 'proxmoxcheckbox', + name: 'enable', + fieldLabel: gettext('Enable'), + allowBlank: false, + checked: true, + }, +], + +columnB: [ + { + layout: 'hbox', + border: false, + margin: '0 0 5 0', + items: [ + { + xtype: 'displayfield', + value: gettext('Method/URL:'), + width: 125, + }, + { + xtype: 'proxmoxKVComboBox', + name: 'method', + //fieldLabel: gettext('Method'), + editable: false, + value: 'post', + comboItems: [ + ['post', 'POST'], + ['put', 'PUT'], + ['get', 'GET'], + ], + width: 80, + margin: '0 5 0 0', + }, + { + xtype: 'proxmoxtextfield', + //fieldLabel: gettext('URL'), + name: 'url', + allowBlank: false, + flex: 4, + }, + ], + }, + { + xtype: 'pmxWebhookKeyValueList', + name: 'header', + fieldLabel: gettext('Headers'), + maskValues: false, + cbind: { + isCreate: '{isCreate}', + }, + }, + { + xtype: 'textarea', + fieldLabel: gettext('Body'), + name: 'body', + allowBlank: true, + minHeight: '150', + fieldStyle: { + 'font-family': 'monospace', + }, + margin: '15 0 0 0', + }, + { + xtype: 'pmxWebhookKeyValueList', + name: 'secret', + fieldLabel: gettext('Secrets'), + maskValues: true, + cbind: { + isCreate: '{isCreate}', + }, + }, + { + xtype: 'proxmoxtextfield', + name: 'comment', + fieldLabel: gettext('Comment'), + cbind: { + deleteEmpty: '{!isCreate}', + }, + }, +], + +onSetValues: (values) => { + values.enable = !values.disable; + + if (values.body) { + values.body = atob
[pve-devel] [PATCH proxmox v2 02/12] notify: add api for webhook targets
All in all pretty similar to other endpoint APIs. One thing worth noting is how secrets are handled. We never ever return the values of previously stored secrets in get_endpoint(s) calls, but only a list of the names of all secrets. This is needed to build the UI, where we display all secrets that were set before in a table. For update calls, one is supposed to send all secrets that should be kept and updated. If the value should be updated, the name and value is expected, and if the current value should preseved, only the name is sent. If a secret's name is not present in the updater, it will be dropped. If 'secret' is present in the 'delete' array, all secrets will be dropped, apart from those which are also set/preserved in the same update call. Signed-off-by: Lukas Wagner --- proxmox-notify/src/api/mod.rs | 20 ++ proxmox-notify/src/api/webhook.rs | 406 ++ 2 files changed, 426 insertions(+) create mode 100644 proxmox-notify/src/api/webhook.rs diff --git a/proxmox-notify/src/api/mod.rs b/proxmox-notify/src/api/mod.rs index a7f6261c..7f823bc7 100644 --- a/proxmox-notify/src/api/mod.rs +++ b/proxmox-notify/src/api/mod.rs @@ -15,6 +15,8 @@ pub mod matcher; pub mod sendmail; #[cfg(feature = "smtp")] pub mod smtp; +#[cfg(feature = "webhook")] +pub mod webhook; // We have our own, local versions of http_err and http_bail, because // we don't want to wrap the error in anyhow::Error. If we were to do that, @@ -54,6 +56,9 @@ pub enum EndpointType { /// Gotify endpoint #[cfg(feature = "gotify")] Gotify, +/// Webhook endpoint +#[cfg(feature = "webhook")] +Webhook, } #[api] @@ -113,6 +118,17 @@ pub fn get_targets(config: &Config) -> Result, HttpError> { }) } +#[cfg(feature = "webhook")] +for endpoint in webhook::get_endpoints(config)? { +targets.push(Target { +name: endpoint.name, +origin: endpoint.origin.unwrap_or(Origin::UserCreated), +endpoint_type: EndpointType::Webhook, +disable: endpoint.disable, +comment: endpoint.comment, +}) +} + Ok(targets) } @@ -145,6 +161,10 @@ fn ensure_endpoint_exists(#[allow(unused)] config: &Config, name: &str) -> Resul { exists = exists || smtp::get_endpoint(config, name).is_ok(); } +#[cfg(feature = "webhook")] +{ +exists = exists || webhook::get_endpoint(config, name).is_ok(); +} if !exists { http_bail!(NOT_FOUND, "endpoint '{name}' does not exist") diff --git a/proxmox-notify/src/api/webhook.rs b/proxmox-notify/src/api/webhook.rs new file mode 100644 index ..b7f17c55 --- /dev/null +++ b/proxmox-notify/src/api/webhook.rs @@ -0,0 +1,406 @@ +use proxmox_http_error::HttpError; +use proxmox_schema::property_string::PropertyString; + +use crate::api::http_err; +use crate::endpoints::webhook::{ +DeleteableWebhookProperty, KeyAndBase64Val, WebhookConfig, WebhookConfigUpdater, +WebhookPrivateConfig, WEBHOOK_TYPENAME, +}; +use crate::{http_bail, Config}; + +use super::remove_private_config_entry; +use super::set_private_config_entry; + +/// Get a list of all webhook endpoints. +/// +/// The caller is responsible for any needed permission checks. +/// Returns a list of all webhook endpoints or a `HttpError` if the config is +/// erroneous (`500 Internal server error`). +pub fn get_endpoints(config: &Config) -> Result, HttpError> { +let mut endpoints: Vec = config +.config +.convert_to_typed_array(WEBHOOK_TYPENAME) +.map_err(|e| http_err!(NOT_FOUND, "Could not fetch endpoints: {e}"))?; + +for endpoint in &mut endpoints { +let priv_config: WebhookPrivateConfig = config +.private_config +.lookup(WEBHOOK_TYPENAME, &endpoint.name) +.unwrap_or_default(); + +let mut secret_names = Vec::new(); +for secret in priv_config.secret { +secret_names.push( +KeyAndBase64Val { +name: secret.name.clone(), +value: None, +} +.into(), +) +} + +endpoint.secret = secret_names; +} + +Ok(endpoints) +} + +/// Get webhook endpoint with given `name` +/// +/// The caller is responsible for any needed permission checks. +/// Returns the endpoint or a `HttpError` if the endpoint was not found (`404 Not found`). +pub fn get_endpoint(config: &Config, name: &str) -> Result { +let mut endpoint: WebhookConfig = config +.config +.lookup(WEBHOOK_TYPENAME, name) +.map_err(|_| http_err!(NOT_FOUND, "endpoint '{name}' not found"))?; + +let priv_config: Option = config +.private_config +.lookup(WEBHOOK_TYPENAME, &endpoint.name) +.ok(); + +let mut secret_names = Vec::new(); +if let Some(priv_config) = priv_config { +for secret in &priv_config.secret { +secret_names.p
[pve-devel] applied-series: [PATCH kernel 1/2] add fix for CIFS client memory leak
On 10/07/2024 13:37, Fiona Ebner wrote: > As reported in the community forum [0], there currently is a memory > leak in the CIFS client code. Reproduced by running a backup with CIFS > target storage: > >> while true; do vzdump 101 --storage cifs --prune-backups keep-last=1; echo 3 >> > /proc/sys/vm/drop_caches; done > > A fix was found on the kernel mailing list tagged for stable v6.6+ > and it does solve the issue, but is not yet included in any (stable) > kernels. > > [0]: https://forum.proxmox.com/threads/147603/post-682388 > > Signed-off-by: Fiona Ebner > --- > ...ix-pagecache-leak-when-do-writepages.patch | 108 ++ > 1 file changed, 108 insertions(+) > create mode 100644 > patches/kernel/0018-cifs-fix-pagecache-leak-when-do-writepages.patch > > applied both patches, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel