I had a similar issue in v9.x. I added "cloud storage=cloud-sd allpools allfrompool upload"
as a run after Console command in the cloud job resource. This seems to do the trick as I've not seen any orphaned volumes in the cache since. Chris On Wed, 6 Mar 2024, 18:53 K. M. Peterson, <kmp.lists+bacula-us...@gmail.com> wrote: > Hi all, > > I've been using the S3 driver for some time now, and working around > various idiosyncrasies - the most lasting one being where occasionally due > to something being shut down there are backup parts in the cache that > haven't been uploaded for some reason. > > I started using a job that ran at boot to run cloud upload > storage=$STORAGE pool=$POOL allfrompool after an interval just as a > catchall (POOL and STORAGE are set to the Cloud pool and Storage > respectively. A couple of months ago (approximately) I realized that the > Bacula Director was inoperative - hung - and I tracked this down to the > Storage Director hanging when this command is executed. Restarting the SD > clears the problem (though it also kills the upload). Otherwise, the > Cloud-based storage continues to work fine. > > I don't see any error messages anywhere, and activity in the Director and > File Daemon seem to return to normal when the Storage Daemon is > killed/restarted. I did run the SD with debug level 100, but I suspect > it's the absence of something I'm looking for. Here's the debug output > (somewhat anonymized): > > Mar 05 16:10:46 four bacula-sd[2263159]: List plugins. Hook count=0 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bsock.c:861-0 socket=4 > who=client host=[2001:0DB8::4] port=9103 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bnet_server.c:235-0 > Accept socket=2001:0DB8::4.9103:2001:0DB8::4.37168 s=0x5636929cedf8 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: dircmd.c:196-0 Got a DIR > connection at 05-Mar-2024 16:10:46 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:365-0 TLSPSK Remote need 100 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:335-0 TLSPSK Local need 100 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:563-0 TLSPSK Start PSK > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bnet.c:96-0 TLS server > negotiation established. > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:68-0 send: > auth cram-md5 challenge <104636656.1709673046@four-sd> ssl=0 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:132-0 > cram-get received: auth cram-md5 <111733638.1709673046@four-dir> ssl=0 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:156-0 sending > resp to challenge: CR+Psy+qh9NHh1+BA++VSA > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: dircmd.c:227-0 Message > channel init completed. > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: status.c:1153-0 > cmd=devices > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bsock.c:861-0 socket=4 > who=client host=[2001:0DB8::4] port=9103 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bnet_server.c:235-0 > Accept socket=2001:0DB8::4.9103:2001:0DB8::4.37182 s=0x5636929f14b8 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: dircmd.c:196-0 Got a DIR > connection at 05-Mar-2024 16:10:46 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:365-0 TLSPSK Remote need 100 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:335-0 TLSPSK Local need 100 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:563-0 TLSPSK Start PSK > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: bnet.c:96-0 TLS server > negotiation established. > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:68-0 send: > auth cram-md5 challenge <934142435.1709673046@four-sd> ssl=0 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:132-0 > cram-get received: auth cram-md5 <2101906992.1709673046@four-dir> ssl=0 > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: cram-md5.c:156-0 sending > resp to challenge: b6/yPWVPOzU/cm/0C9/+/B > Mar 05 16:10:46 four bacula-sd[2263159]: four-sd: dircmd.c:227-0 Message > channel init completed. > Mar 05 16:10:47 four bacula-sd[2263159]: List plugins. Hook count=0 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: bsock.c:861-0 socket=4 > who=client host=[2001:0DB8::4] port=9103 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: bnet_server.c:235-0 > Accept socket=2001:0DB8::4.9103:2001:0DB8::4.37186 s=0x563692a08dd8 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: dircmd.c:196-0 Got a DIR > connection at 05-Mar-2024 16:10:47 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:365-0 TLSPSK Remote need 100 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:335-0 TLSPSK Local need 100 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:563-0 TLSPSK Start PSK > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: bnet.c:96-0 TLS server > negotiation established. > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: cram-md5.c:68-0 send: > auth cram-md5 challenge <1669111798.1709673047@four-sd> ssl=0 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: cram-md5.c:132-0 > cram-get received: auth cram-md5 <1290027090.1709673047@four-dir> ssl=0 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: cram-md5.c:156-0 sending > resp to challenge: F/oa1Ssdqw1iEwcd6j/CKA > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: dircmd.c:227-0 Message > channel init completed. > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: dircmd.c:1210-0 Found > device AWS_S3_Cloud1 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: dircmd.c:1254-0 Found > device AWS_S3_Cloud1 > Mar 05 16:10:47 four bacula-sd[2263159]: four-sd: acquire.c:671-0 Attach > 0x9401bbf8 to dev "AWS_S3_Cloud1" (/data/Backup/bstor_aws_cache) > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: bsock.c:861-0 socket=6 > who=client host=[2001:0DB8::4] port=9103 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: bnet_server.c:235-0 > Accept socket=2001:0DB8::4.9103:2001:0DB8::4.37198 s=0x5636929cedf8 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: dircmd.c:196-0 Got a DIR > connection at 05-Mar-2024 16:10:48 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:365-0 TLSPSK Remote need 100 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:335-0 TLSPSK Local need 100 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:563-0 TLSPSK Start PSK > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: bnet.c:96-0 TLS server > negotiation established. > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: cram-md5.c:68-0 send: > auth cram-md5 challenge <76485448.1709673048@four-sd> ssl=0 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: cram-md5.c:132-0 > cram-get received: auth cram-md5 <76485448.1709673048@four-dir> ssl=0 > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: cram-md5.c:156-0 sending > resp to challenge: HC+BF8+G9mVUIlAV99+5LC > Mar 05 16:10:48 four bacula-sd[2263159]: four-sd: dircmd.c:227-0 Message > channel init completed. > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: bsock.c:861-0 socket=7 > who=client host=[2001:0DB8::4] port=9103 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: bnet_server.c:235-0 > Accept socket=2001:0DB8::4.9103:2001:0DB8::4.55784 s=0x5636929f14b8 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: dircmd.c:196-0 Got a DIR > connection at 05-Mar-2024 16:14:40 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:365-0 TLSPSK Remote need 100 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:335-0 TLSPSK Local need 100 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: > authenticatebase.cc:563-0 TLSPSK Start PSK > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: bnet.c:96-0 TLS server > negotiation established. > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: cram-md5.c:68-0 send: > auth cram-md5 challenge <1104246352.1709673280@four-sd> ssl=0 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: cram-md5.c:132-0 > cram-get received: auth cram-md5 <78526969.1709673280@four-dir> ssl=0 > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: cram-md5.c:156-0 sending > resp to challenge: 49UP5RldJ5ZVEg/gS++skD > Mar 05 16:14:40 four bacula-sd[2263159]: four-sd: dircmd.c:227-0 Message > channel init completed. > Mar 05 16:32:16 four systemd[1]: bacula-sd.service: Main process exited, > code=killed, status=9/KILL > Mar 05 16:32:16 four systemd[1]: bacula-sd.service: Failed with result > 'signal'. > Mar 05 16:32:51 four systemd[1]: Started Bacula Storage Daemon service. > > In the case above, I killed the daemon after 20+ minutes, but it went for > days a couple of times before I realized that it was toasted. No output > from the upload command. The SD does connect to AWS, but times out. Also, > the bconsole process can't be killed by ^C. > > I'm not sure this is going to help anyone with debugging - I'm posting to > ask whether anyone has any ideas on either what might be going wrong here, > or where I could develop more information before trying to file a bug. > > BTW, this is RHEL 8, AMD, and Bacula 13.0.2, 18Feb23. Thanks for any > ideas! > _______________________________________________ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users >
_______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
