On 2/20/25 7:19 PM, Rob Gerber wrote: > Hello, [...snip...]
Some parts are failing to upload. I verified with ls that the parts in question still remain on disk (I have set cloud cache to truncate on upload, so only part1 and parts that haven't uploaded should remain).In bconsole I ran 'cloud allpools upload' and that seemed to work, albeit slowly. It's still in process now with the few remaining part files that didn't upload. Upload speeds seem much slower than when the job was uploading part files. Maybe some sort of bandwidth limitation is in effect if the upload command was ran from within bconsole? During the actual copy job, uploads observed at my pfsense router were in the 400-800mbps range. Now that the copy jobs have finished and only the > bconsole 'cloud allpools upload' command is running, sustained upload is in the 5-10mbps range, and I still have a couple 5
> gb part files left. I know this system and my internet connection can do bette r, so why is the upload ran from the console > going so slowly? Hello Rob,I will leave this performance/bandwidth topic (mostly) alone, and just say there should not be any difference whether a part is uploaded during a job, or manually.
Maybe one of my colleagues (Eric? :) can chime in here to confirm or deny. If this is repeatable, then maybe we can dig a little into your environment.
What can I do to improve reliability here?
A lot of things can affect reliability, and who knows, maybe there are intermittent issues during backup jobs and cloud volume part uploads which are out of your control causing some parts to fail to be uploaded during the backup job itself, but those issues do not exist when you run the cloud upload command manually. Hard to tell.
> I was considering setting up an admin job to periodically run 'cloud allpools upload'. This is exactly what is recommended. I like to refer to it as a "cloud volume part sweeper job", and would run it once/day after your backups are complete.
How does one set up an admin job? I've searched the manual and read the examples, but I'm fuzzy on how it would actually work. I envisioned just setting the admin job's payload to be a bconsole command, but the examples in the manual didn't seem to be exactly that. Maybe I'm just confused.
Admin jobs are quite simple. First, they do not move any data. They are strictly meant to call scripts, or run console commands*. The configuration parser requires them to have all settings that a normal job requires (Name, Type, Client, Pool, Fileset, Storage, etc) even though some (most?) of these things are not used.
Take a look at the Job Resource section in the main documentation, and look for the RunScript section: https://docs.baculasystems.com/BETechnicalReference/Director/DirectorResourceTypes/JobResource/index.html#director-job-resourceAll you need to do is add something like the following to a new Admin Job having a `Type = Admin`
----8<----
Runscript {
RunsWhen = before
RunsOnClient = no # Default yes, theere is no client in an Admin job & Admin
Job RunScripts *only* run on the Director :)
Command = "/path/to/a/small/script.sh" # This can be `Console` if you wish to
send console commands, but don't use this*
}
----8<----
*NOTE: I don't recommend using the `Console =` option in RunsScript{} stanzas
for a few reasons.
- If you use `Console =`, any output from them appears mixed into your bacula.log file, identified as `JobID: 0` so you never
know what job actually caused the outputs
- Using `Command =` you are able to fully control everything in a script.- Any stdout in your script is printed in the Admin joblog that called it. So you can add any output you like so your Admin job log is quite verbose and you know everything that happened, and when. (bconsole: ll joblog jobid=xxxx)
- If you use `Console =` you cannot get the outputs from the above bconsole
command
Example script to call with the `Command =` setting in your RunScript:
----8<----
#!/bin/bash
bcbin="/opt/bacula/bin/bconsole"
bccfg="/opt/bacula/etc/bconsole.conf"
echo "Whatever you want to appear in Admin joblog"
echo -e "cloud allpools upload\nquit\n" | ${bcbin} -c ${bccfg} # All output
will be in your Admin Job's joblog
echo "Whatever else you want to appear in Admin joblog"
----8<----
Doing it this way, you can periodically check for failed uploads.
Manually, In bconsole:
----8<----
* ll joblog jobid=<idOfAdminJoblog>
----8<----
Could even be automated and scripted sed/grep/awk from the job log output as above, or from the bacula.log files, and send
you an email or similar...
The bigger problem seems to be that I don't know how to monitor whether a part subsequently uploaded successfully.
> I am working on deploying Bill A's job report script, and that's likely going to be my monitoring solution. See above... But, this gets me thinking about a new feature for my reporting script. :)Currently it does nothing with regards to cloud part uploads, so I may take a look at adding an option to warn when parts fail to upload. Hmmm, I really like this idea. :)
Maybe a bad exit status for an admin job tasked with uploading any leftover part files could send up a flag that something is wrong and a part file hasn't been uploaded?
Yeah, this is really not possible from what I can see.If a cloud backup job backs up everything, the Job will terminate "Backup OK" - even if there are cloud part upload errors. I am pretty sure it does not even terminate "Backup OK -- with warnings" to give you any indication. I will need to double-check, or you already can since you have some of these jobs where this happened. :)
The thought here is along the lines of "the backup succeeded, and your data is backed up and is safely under Bacula's control", so there is no consideration that the parts may no
t have made it to the cloud.I think I will definitely add some sort of cloud upload problem warning to my script for sure now that I think more about this. Stay tuned... :)
I'm using the s3 driver instead of the amazon driver. I read that the new amazon driver is much better than the s3 driver, but in my testing the amazon driver didn't seem to want to connect to a non-aws server.
Your best bet is to use the `Driver = Amazon` for your S3 type cloud resources. The `Driver = S3` is being deprecated and the awscli is, and will be much better. If you have issues using it, let's debug those. It works with Backblaze and all other vendors providing S3 end points that I tested and certified Bacula Enterprise with this past Summer (VAST, StorJ, SpectraLogic, and a few others)
Hope this helps, Bill -- Bill Arlofski w...@protonmail.com
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
