Thanks so much for the suggestions, Rob. I'll shoot for the simplest possible
recovery plan.
My NGINX config was utterly blown away by the failed element.io install and I'm
still not sure how to browse individual files on my offsite Bacula backup, even
with Bacularis, although I've had some discussions with the Bacularis developer
on the topic.
I'll check if the bconsole is still working and, if so, will find a way to
remotely restore the content from / into /restore/ and the content from /boot
into /restore/boot. Then, I'll have to find a way to switch in the new fileset
and switch out the old fileset (keeping it around for a while just in case,
since I haven't maxed out my drive storage yet and have headroom). Kind of like
changing the tires of a car while still driving.
I'll post back once I have a working system again.
In hindsight, I should have done more dress rehearsals of restore, I'm very
slowly climbing out of the well here and learning lots on the way.
Best,
<Myles>
On 2024-06-25 10:48 a.m., Rob Gerber wrote:
> Good morning, Myles.
>
> I have a few thoughts that might contribute to recovering from this situation.
>
> 1. Sounds like the element.io installer hosed up your nginx web server files
> (which bacularis relies upon). Please note that because bacularis isn't
> integral to the function of bacula itself, the bacula installation could be
> just fine. If this is the extent of the damage, and the cloud server is
> otherwise stable, why not get a shell on it and do a restore from the command
> line bacula bconsole utility? Could be much less invasive than a full system
> restore using a (comparatively) untested restore plan. I would REALLY try
> this first. I would restore the files to a different location than the
> destination, to give yourself the ability to compare with diff or something
> prior to restoration. You'll probably want to bring the relevant system
> services down prior to restoration of files. so bring down nginx, bacularis,
> etc.
>
> 1a. If bacularis is the only thing that is broken, maybe get a temporary
> bacularis instance running to assist with this process? Probably more
> complicated and troublesome that just doing a restore using bconsole, though.
>
> 2. If the above isn't a viable option for some reason, I would suggest a
> minor alternative to a custom recovery ISO. I don't know much about crafting
> custom recovery ISO images so I'll suggest:
> 2a. Export your bacula configuration files (including any relevant certs)
> from your existing system. At minimum you'll want the contents of
> /opt/bacula/etc. I suggest copying the entire /opt/bacula directory and all
> subdirectories. Probably best to use tar to export it since that way you're
> guaranteed to be able to save user:group permissions and ownership. Chatgpt
> can help you with this.
> 2b. Export your latest catalog backup. This would require that you have
> functional bconsole access (assuming bacularis is unusable). If you have the
> ability to do this, I don't know why you wouldn't do a full recovery of your
> system from bconsole instead. Perhaps you have a reason. Either way, restore
> your most recent full catalog backup. If you don't have a recent catalog
> backup, take one from within bconsole, then export it. Will be a bacula.sql
> file that contains all the database commands to fully drop a previous bacula
> database instance and then restore your entire bacula database.
> 2c. Bacula depends on Fully Qualified Domain Names (hostname.domain) in the
> bacula configuration files. If you want this ISO to (temporarily) fully take
> the place of your damaged bacula system you'll probably need to change your
> booted ISO system's hostname to match the hostname of your damaged system.
> Please be very careful when doing this (esp during testing) because you don't
> want a hostname conflict. ALTERNATIVELY, consider just a different name for
> your recovery system and edit the bacula configuration files to match this
> name. The inclusion of your custom cert could complicate this if it's tied in
> some way to the FQDN of the damaged server.
> 2d. Instead of trying to build a fully functional bacula installation into a
> custom recovery ISO, maybe script first time installation and configuration
> of bacula. Will need to set up your repo file, install bacula, drop your old
> configuration files over top of the default bacula configuration, edit
> configuration files to match any changed details like different system
> hostname, restore bacula database backup, install bacularis, (maybe install
> bacularis sooner in the process?), then look into restoring your needed files.
>
> Issues with the above, possible alternatives:
> If you can't get a database recovery, but you can get bacula set up on
> another system and give it access to your catalog backup volumes, you could
> use the bscan utility to scan the relevant catalog backup volumes and rebuild
> bacula's awareness of what files are available. could then do a restore of
> the catalog backup, and export backup to your bacula recovery system. The
> possible need to bscan a volume to enable catalog recovery is one reason why
> I always make sure that my catalog backups go into their own dedicated
> volumes - faster / easier bscan in the event such is needed, no need to also
> scan through tons of data that include various system files.
> You might consider setting up an alternative bacula system in a VM,
> configure, get full access to your bacula backups, then restore the necessary
> backup to storage that could be accessible to the troubled system once that
> system is booted into a recovery environment. This way you wouldn't need to
> set up bacula in the recovery environment, just copy the files over into the
> system.
> If the existing bacula instance is accessible via bconsole, I'd just restore
> the backup from there. Be sure to export a copy of your bacula catalog from
> your backups too. Also, be certain that the catalog backup you exported is up
> to date and contains the latest backup.
> If you cannot export the latest catalog backup, or the latest catalog backup
> doesn't include the most recent full system backup, AND bconsole isn't
> accessible / functional, then I guess you could set up bacula in a test
> system, then bscan the volumes related to the most recent full system backup.
>
> Robert Gerber
> 402-237-8692
> r...@craeon.net
>
> On Tue, Jun 25, 2024, 8:41 AM MylesDearBusiness via Bacula-users
> <bacula-users@lists.sourceforge.net> wrote:
>
>> I`d be pleased to provide more detail, I was trying to keep my answer terse,
>> obviously I undershot !?
>>
>> I commissioned our cloud server, installing our business services and then
>> securing with community edition Bacula system backup to an offsite Koofr
>> storage backend, via rclone. I also installed and configured Bacularis with
>> appropriate Nginx reverse-proxying and production SSL cert install.
>>
>> I then attempted to install element.io (as a potential Slack replacement for
>> our company) and all he** broke loose, the microk8s installer completely
>> decimated my NGINX configuration and all services stopped working. The
>> element.io "support" team declined to help me recover my system from an
>> installer that crashed halfway through, leaving the system in an unstable
>> state.
>>
>> Thus, I wish to restore the entire bare metal cloud server from a complete
>> Bacula backup I took just prior to doing this test install.
>>
>> After trying and failing to use more low-level manual means, I`m now
>> pivoting and trying to adapt an existing Live Ubuntu ISO and install
>> appropriate Bacula packages and storage backend linkages using CUBIC. I plan
>> to test the ISO on local VirtualBox and ultimately bring it up in a virtual
>> CDROM on my cloud server`s ASMB9-iKV. I then want the ISO to come up with
>> all Bacula services primed and ready for a full system restore.
>>
>> I`m not sure what the history was that led up to the removal of the Live ISO
>> from the community Bacula builds, but it`s taking a LOT of my time to figure
>> out how to do restores on an unstable system. Is there a reason a basic
>> universal Bacula recovery ISO isn't being built with each community release ?
>>
>> Thanks,
>>
>> <Myles>
>>
>> On 2024-06-25 2:55 a.m., Davide F. wrote:
>>
>>> Hi Myles,
>>>
>>> Could you give the context ?
>>>
>>> I do t understand what’s the problem you are trying to solve, why do you
>>> want to build an ISO ?
>>>
>>> Best,
>>>
>>> Davide
>>>
>>> On Tue, Jun 25, 2024 at 03:25 MylesDearBusiness via Bacula-users
>>> <bacula-users@lists.sourceforge.net> wrote:
>>>
>>>> I spent most of today trying to create a custom ISO manually, with no
>>>> success. I'm trying to boot locally in my VirtualBox before the main event
>>>> in which I'll remotely mount it into my cloud server's IPMI ASMB9-iKVM and
>>>> boot it using a virtual CDROM drive.
>>>>
>>>> I'm aiming to try CUBIC next to customize a stock live boot ISO.
>>>>
>>>> For me, with my level of knowledge, and only ChatGPT as my support
>>>> resource, this may take many days or even weeks to achieve.
>>>>
>>>> Does anybody have an ISO that I could use as a starting point?
>>>>
>>>> I was hoping to have Bacula community edition 13.0.3 or 13.0.4 on this
>>>> recovery ISO.
>>>>
>>>> Thanks,
>>>>
>>>> <Myles>
>>>>
>>>> _______________________________________________
>>>> Bacula-users mailing list
>>>> Bacula-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>> --
>>
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
