Hi! Am 01.04.25 um 18:02 schrieb Andreas Rogge: > sorry for digging up that ancient mail, but I feel that's the best > starting point for me.
For more current discussion it might be best to check out the recently posted v7 of this series, if nothing bigger comes up it should be very close to what gets applied for an initial version – i.e., one that will be supported for a long time, which does not necessarily mean that it cannot evolve. https://lore.proxmox.com/pve-devel/20250401173435.221892-1-f.eb...@proxmox.com/T/#t > Bareos investigates how to better integrate with Proxmox VE, so users > can take backups of virtual machines and restore them using Bareos. Great to hear! > Am 14.11.24 um 16:07 schrieb Fiona Ebner: >> The new_backup_provider() method can be used by storage plugins for >> external backup providers. If the method returns a provider, Proxmox >> VE will use callbacks to that provider for backups and restore instead >> of using its usual backup/restore mechanisms. > So we as a backup provider need to provide plugin code that will run as > part of PVE. If I understand correctly this must be implemented as a > Perl module. Also, PVE wants to trigger the backup itself. The interface is in Perl, the actual backup code can be whatever runs on the Proxmox VE base system, as that is based on Debian this means that there is a lot of flexibility. A colleague wrote a plugin in rust as proof of concept, we probably release that too as example. We can also decouple this more once the dust settles, there are some ideas floating around already, but they can be implemented later so ensuring that the actual base functionality works out is more important, as whatever the interface is, they will work similar but just using a different transport (e.g. say varlink over an FD instead of perl wrapper module). > This introduces a few hurdles for us: > First of all, we don't usually program in Perl, so that's going to be a > problem for us. We can probably implement something, but we probably > don't have any developer that can write Perl code that meets any kind of > professional standard. The Perl code required for a production quality interface is not that much, and certainly does not require in-depth perl experience. Perl is a mature language, for better or worse, that allowed and still allows all kind of users to create tools that can work fine if one keeps them simple, and as the core design is rather fixed here, it should not be that much work, and we certainly can help on questions or problems; just reach out on the developer list and point to your public plugin code. > To do its job[1], Bareos must be able to schedule backup and restore > jobs all by itself. If I understood correctly, backups utilizing one of > the plugins can be run by calling `vzdump`, so that this is at least > possible. Exactly, we want a native PVE implementation so that such plug-ins are actually nicely integrated and feel almost native (in the long run, missing a few bits for that), so users get a first-class experience on "both sides" of the backup and can also see what's going on at the source side of things, if they want. > However, what we've seen other vendors do is that they provide an API > where you just tell the system what VM you want to back up and then you > are provided with access to metadata (i.e. VM configuration, disk > layouts, nvram content, etc) and disk contents. You can send an API request that basically results in exactly that, just that the plugin is providing you the info then. A big advantage is that the user can also trigger backups from the PVE side and thus get a much better integration. > For restore it is mostly the same: you provide the metadata, get a set > of raw disks that you can write your data to and finally you just tell > the system that the VM restore is done. It's different but ensures that we can provide backup providers and their user a quasi native integration into PVE, ideally as close as possible to how PBS is integrated, this can IMO be also benefit for providers, and nothing that is possible with the methods you describe should become impossible with the methods we provide. >> 2. Restore API >> >> 2.1. Restore Mechanisms >> >> VM mechanism 'qemu-img': >> >> The backup provider gives a path to the disk image that will be >> restored. The path needs to be something 'qemu-img' can deal with, >> e.g. can also be an NBD URI or similar. > Um... that has nothing to do with what you provided when we take the > backup. Is there a reason PVE cannot provide a writeable block device to > restore to? > For Bareos this requirement would imply that we need an unpleasantly > large staging area on the PVE node to facilitate a restore: As we can > only stream the data we'd have to create a temporary disk image that PVE > can then read. This sounds pretty inefficient - especially when > comparing with qmrestore's ability to just read read from stdin. Bareos could provide a NBD that is streamed directly from the backup repository, this should be quite efficient w.r.t. space usage. But as this was v4 and a few things changed, and I'm less involved as the actual author it might make more sense for her to answer this. > [1] As a backup software it is Bareos' job to make sure the production > system is as strictly separated from the backup data as possible. While > we understand how nice tight integration with PVE would be, we see this > primarily as unnecessary attack surface. Nothing in the proposed methods should technically hinder such a separation, it's just the interface, no protection mechanism on the backup systems side need to be adapted or changed, and one could construct the plugin such that it can only work with jobs that got started from the backup system, albeit that would be IMO not really reasonable or at least not really a nice experience especially as the attack surfaces stays rather the same either way, as data needs effectively to be sent in both directions > For a nicely integrated solution that can bring back last week's VM > snapshot with a few clicks, there's Proxmox Backup Server, which works > great. Do you not want a nicely integrated solution? I think users want both, a nice integration into PVE and additional features and capabilities of backup solution like yours. best regards, Thomas _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
