Hello,
I have some problems with backuping Kubernetes PVCs with Bacula Kubernetes
Plugin. (I have asked it on bacula-users mailing list but got no answer.)
I am using the latest 13.0.1 Bacula from the community builds on Debian
Bullseye hosts.
Backuping only the Kubernetes objects except Persistent Volume Claims (PVC)
works like a charm. I've installed the Kubernetes plugin and the latest
Bacula File Daemon on the master node (control plane) of our Kubernetes
cluster. Bacula can access the Kubernetes cluster and backup every single
object as YAML files.
The interesting part comes with trying to backup a PVC...
First of all I could build my own Bacula Backup Proxy Pod Image from the
source and it's deployed into our local Docker image repository (repo). The
Bacula File Daemon is configured properly I guess. Backup process started
and the following things happened.
1. Bacula File Daemon deployed Bacula Backup Proxy Pod Image into the
Kubernetes cluster, so Bacula-backup container pod started.
2. I got into the pod and I could see the Baculatar application started and
running.
3. The k8s_backend application started on the Bacula File Daemon host
(kubernetes.server) in 2 instances.
4. From the Bacula-backup pod I could check that Baculatar could connect to
the k8s_backend at the default 9104 port (kubernetes.server:9104).
5. I checked the console messages of the job with Bat that Bacula File
Daemon started to process the configured PVC, started to write a pvc.tar
but nothing happened.
6. After default 600 sec, after timeout the job was cancelled.
7. It may be important that Bacula File Daemon could not delete the
Bacula-backup pod. (It could create it but could not delete it.)
Could you please tell me what's wrong?
Here are some log parts. (I've changed some sensitive data.)
Bacula File Daemon configuration:
FileSet {
Name = "Kubernetes Set"
Include {
Options {
signature = SHA512
compression = GZIP
Verify = pins3
}
Plugin = "kubernetes: \
debug=1 \
baculaimage=repo/bacula-backup:04jan23 \
namespace=namespace \
pvcdata \
pluginhost=kubernetes.server \
timeout=120 \
verify_ssl=0 \
fdcertfile=/etc/bacula/certs/bacula-backup.cert \
fdkeyfile=/etc/bacula/certs/bacula-backup.key"
}
}
Bacula File Daemon debug log (parts):
DEBUG:[baculak8s/jobs/estimation_job.py:134 in processing_loop] processing
get_annotated_namespaced_pods_data:namespace:nrfound:0
DEBUG:[baculak8s/plugins/kubernetes_plugin.py:319 in
list_pvcdata_for_namespace] list pvcdata for namespace:namespace
pvcfilter=True estimate=False
DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:108 in
pvcdata_list_namespaced] pvcfilter: True
DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:112 in
pvcdata_list_namespaced] found:some-claim
DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:127 in
pvcdata_list_namespaced] add pvc: {'name': 'some-claim', 'node_name': None,
'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
<baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
DEBUG:[baculak8s/jobs/estimation_job.py:165 in processing_loop] processing
list_pvcdata_for_namespace:namespace:nrfound:1
DEBUG:[baculak8s/jobs/estimation_job.py:172 in processing_loop]
PVCDATA:some-claim:{'name': 'some-claim', 'node_name': 'node1',
'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
<baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
I000041
Start backup volume claim: some-claim
DEBUG:[baculak8s/jobs/job_pod_bacula.py:298 in prepare_bacula_pod]
prepare_bacula_pod:token=xx88M5oggQJ....4YDbSwBRxTOhT namespace=namespace
DEBUG:[baculak8s/jobs/job_pod_bacula.py:136 in prepare_pod_yaml] pvcdata:
{'name': 'some-claim', 'node_name': 'node1', 'storage_class_name':
'nfs-client', 'capacity': '2Gi', 'fi':
<baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
DEBUG:[baculak8s/plugins/k8sbackend/baculabackup.py:102 in
prepare_backup_pod_yaml] host:kubernetes.server port:9104
namespace:namespace image:repo/bacula-backup:04jan23
job:KubernetesBackup.2023-01-04_21.05.03_10:410706
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
I000149
Prepare Bacula Pod on: node1 with: repo/bacula-backup:04jan23
<IfNotPresent> kubernetes.server:9104
DEBUG:[baculak8s/jobs/job_pod_bacula.py:198 in prepare_connection_server]
prepare_connection_server:New ConnectionServer: 0.0.0.0:9104
DEBUG:[baculak8s/util/sslserver.py:180 in listen]
ConnectionServer:Listening...
DEBUG:[baculak8s/jobs/job_pod_bacula.py:307 in prepare_bacula_pod]
prepare_bacula_pod:start pod
INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in backup_pod_isready]
backup_pod_status:isReady: False / 0
INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in backup_pod_isready]
backup_pod_status:isReady: True / 1
DEBUG:[baculak8s/jobs/estimation_job.py:183 in _estimate_file] {'name':
'some-claim', 'node_name': 'node1', 'storage_class_name': 'nfs-client',
'capacity': '2Gi', 'fi': <baculak8s.entities.file_info.FileInfo object at
0x7ffaa55bfcc0>}
DEBUG:[baculak8s/jobs/estimation_job.py:190 in _estimate_file] file_info:
{FileInfo
name:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
namespace:None type:F objtype:pvcdata cached:False}
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
C000079
FNAME:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
C000040
TSTAMP:1672861077 1672861077 1672861077
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
C000031
STAT:F 2147483648 0 0 100640 1
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
F000000
(EOD PACKET)
DEBUG:[baculak8s/jobs/backup_job.py:77 in __backup_pvcdata]
backup_pvcdata:data recv
DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
C000005
DATA
DEBUG:[baculak8s/util/sslserver.py:193 in handle_connection]
ConnectionServer:Connection from: ('192.168.XX.YY', 10541)
DEBUG:[baculak8s/util/sslserver.py:145 in gethello] ['Hello',
'KubernetesBackup.2023-01-04_21.05.03_10', '410706']
DEBUG:[baculak8s/util/token.py:57 in check_auth_data] AUTH_DATA:Token:
xx88M5oggQJuGsPbtD........ohQjeU7PkA4YDbSwBRxTOhT
DEBUG:[baculak8s/util/token.py:59 in check_auth_data]
RECV_TOKEN_DATA:Token: xx88M5oggQJuGsPbtD....ohQjeU7PkA4YDbSwBRxTOhT
DEBUG:[baculak8s/util/sslserver.py:105 in authenticate]
ConnectionServer:Authenticated
.... after timeout
DEBUG:[baculak8s/jobs/job_pod_bacula.py:121 in handle_pod_data_recv]
handle_pod_data_recv:EOT
DEBUG:[baculak8s/util/sslserver.py:201 in handle_connection]
ConnectionServer:Finish - disconnect.
DEBUG:[baculak8s/jobs/backup_job.py:85 in __backup_pvcdata]
backup_pvcdata:logs recv
Job messages:
bacula-dir No prior or suitable Full backup found in catalog for the
current FileSet. Doing FULL backup.
The FileSet "Kubernetes Set" was modified on 2023-01-04 20:20:41, this is
after the last successful backup on 2023-01-04 19:19:49.
bacula-sd Ready to append to end of Volume "Full-XXX" size=3,838,161,002
bacula-fd Connected to Storage at bacula.server:9103 with TLS
bacula-sd Volume "Full-XXXX" previously written, moving to end of data.
bacula-dir Connected to Client "bacula-fd" at kubernetes.server:9102 with
TLS
Using Device "FileStorageEeyoreFull" to write.
Connected to Storage "InternalStorageFull" at bacula.server:9103 with TLS
Start Backup JobId 410830, Job=KubernetesBackup.2023-01-04_21.05.03_10
bacula-fd kubernetes: Prepare Bacula Pod on: node with:
repo/bacula-backup:04jan23 kubernetes.server:9104
kubernetes: Processing namespace: namespace
kubernetes: Start backup volume claim: some-claim
kubernetes: Connected to Kubernetes 1.25 - v1.25.4.
bacula-dir
Error: Bacula Enterprise bacula-dir 13.0.1 (05Aug22):
Build OS: x86_64-pc-linux-gnu-bacula-enterprise debian 11.2
JobId: 410830
Job: KubernetesBackup.2023-01-04_21.05.03_10
Backup Level: Full (upgraded from Differential)
Client: "bacula-fd" 13.0.1 (05Aug22)
x86_64-pc-linux-gnu-bacula-enterprise,debian,10.11
FileSet: "Kubernetes Set" 2023-01-04 20:20:41
Pool: "Full-Pool" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "InternalStorageFull" (From Pool resource)
Scheduled time: 04-Jan-2023 21:05:03
Start time: 04-Jan-2023 21:27:04
End time: 04-Jan-2023 21:29:06
Elapsed time: 2 mins 2 secs
Priority: 10
FD Files Written: 23
SD Files Written: 0
FD Bytes Written: 52,784 (52.78 KB)
SD Bytes Written: 0 (0 B)
Rate: 0.4 KB/s
Software Compression: 100.0% 1.0:1
Comm Line Compression: 5.6% 1.1:1
Snapshot/VSS: no
Encryption: yes
Accurate: yes
Volume name(s): Full-XXXX
Volume Session Id: 43
Volume Session Time: 1672853724
Last Volume Bytes: 3,838,244,105 (3.838 GB)
Non-fatal FD errors: 3
SD Errors: 0
FD termination status: OK
SD termination status: SD despooling Attributes
Termination: *** Backup Error ***
Fatal error: catreq.c:680 Restore object create error.
bacula-fd
Error: kubernetes: PTCOMM cannot get packet header from backend.
bacula-dir Fatal error: sql_create.c:1273 Create db Object record INSERT
INTO RestoreObject
(ObjectName,PluginName,RestoreObject,ObjectLength,ObjectFullLength,ObjectIndex,ObjectType,ObjectCompression,FileIndex,JobId)
VALUES ('RestoreOptions','kubernetes: \n debug=1 \n
baculaimage=repo/bacula-backup:04jan23 \n namespace=namespace
\n pvcdata \n
pluginhost=kubernetes.server \n timeout=120 \n
verify_ssl=0 \n fdcertfile=/etc/bacula/certs/bacula-backup.cert
\n
fdkeyfile=/etc/bacula/certs/bacula-backup.key','# Plugin configuration
file\n# Version 1\nOptPrompt=\"K8S config
file\"\nOptDefault=\"*None*\"\nconfig=@STR@\n\n
OptPrompt=\"K8S API server
URL/Host\"\nOptDefault=\"*None*\"\nhost=@STR@\n\nOptPrompt=\"K8S
Bearertoken\"\nOptDefault=\"*None*\"\ntoken=@STR@\n\nOptPrompt=\"K8S API
server cert verification\"\n
OptDefault=\"True\"\nverify_ssl=@BOOL@\n\nOptPrompt=\"Custom CA Certs file
to use\"\nOptDefault=\"*None*\"\nssl_ca_cert=@STR@\n\nOptPrompt=\"Output
format when saving to file (JSON, YAML)\"\n
OptDefault=\"RAW\"\noutputformat=@STR@\n\nOptPrompt=\"The address for
listen to incoming backup pod
data\"\nOptDefault=\"*FDAddress*\"\nfdaddress=@STR@\n\n
OptPrompt=\"The port for opening socket for
listen\"\nOptDefault=\"9104\"\nfdport=@INT32@\n\nOptPrompt=\"The endpoint
address for backup pod to connect\"\n
OptDefault=\"*FDAddress*\"\npluginhost=@STR@\n\nOptPrompt=\"The endpoint
port to connect\"\nOptDefault=\"9104\"\n
pluginport=@INT32@\n\n',859,859,0,27,0,1,410830) failed. ERR=Data too long
for column 'PluginName' at row 1
bacula-sd Sending spooled attrs to the Director. Despooling 8,214 bytes ...
bacula-fd
Error: kubernetes: Error closing backend. Err=Child exited with code 1
Fatal error: kubernetes: Wrong backend response to JobEnd command.
bacula-sd Elapsed time=00:02:02, Transfer rate=659 Bytes/second
bacula-fd
Error: kubernetes: PTCOMM cannot get packet header from backend.
Error: kubernetes: Cannot successfully start bacula-backup pod in expected
time!
Error: kubernetes: Job already running in 'namespace' namespace. Check logs
or delete bacula-backup Pod manually.
Best regards,
Zsolt
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
