Hello Zsolt,
Indeed, the NetworkPolicy you've provided doesn't seem to be involved in
the issue your facing.
Let's keep trying to figure out what's going on with your setup
Could you run these commands below
kubectl get networkpolicy
and if you get some result, run
kubectl describe networkpolicy <networkpolicy-name>
In the meantime, I'll setup a "test" environement and see if I'm facing the
problem.
I'll keep you updated.
Best regards
Davide
On Wed, Jan 11, 2023 at 5:54 PM Zsolt Kozak <koza...@gmail.com> wrote:
> Hi!
>
> Yes, but only one tiny:
>
> kind: NetworkPolicy
> apiVersion: networking.k8s.io/v1
> metadata:
> name: allow-apiserver
> namespace: calico-apiserver
> ownerReferences:
> - apiVersion: operator.tigera.io/v1
> kind: APIServer
> name: default
> controller: true
> blockOwnerDeletion: true
> managedFields:
> - manager: operator
> operation: Update
> apiVersion: networking.k8s.io/v1
> spec:
> podSelector:
> matchLabels:
> apiserver: 'true'
> ingress:
> - ports:
> - protocol: TCP
> port: 5443
> policyTypes:
> - Ingress
> status: {}
>
> But I guess it's an allow, not a block policy. (I'm somewhat new to
> Kubernetes and not too familiar with network policies...)
>
> Best regards,
> Zsolt
>
> On Wed, Jan 11, 2023 at 5:47 PM Davide F. <bacula-...@dflc.ch> wrote:
>
>> Hi,
>>
>> Are you using some kind of network policy within your cluster ?
>>
>> Best,
>>
>> Davide
>>
>> On Wed, 11 Jan 2023 at 10:53 Zsolt Kozak <koza...@gmail.com> wrote:
>>
>>> Hello Davide!
>>>
>>> I am running the File Daemon on the master node, on the control plane.
>>> It's Kubernetes vanilla, version 1.25.4.
>>> No, the master node is running on the same subnet as the workers.
>>>
>>> It should be some network issue, I think.
>>>
>>> Best regards,
>>> Zsolt
>>>
>>> On Wed, Jan 11, 2023 at 8:45 AM Davide F. <bacula-...@dflc.ch> wrote:
>>>
>>>> Hello Kozak,
>>>>
>>>> I haven’t tried k8s plugin but let me try to understand what could be
>>>> the root cause of your problem.
>>>>
>>>> Could you explain further point 1 please ?
>>>> On which node are you running the file daemon ?
>>>>
>>>> Which version / flavor of Kubernetes are you using ?
>>>>
>>>> Is it Kubernetes vanilla ? OpenShift ? Tansu ?
>>>>
>>>> Depending on your feedback from the first question, does master nodes
>>>> runs in a different subnet than worker’s ?
>>>>
>>>> Thanks for your feedback
>>>>
>>>> Best,
>>>>
>>>> Davide
>>>>
>>>> On Tue, 10 Jan 2023 at 21:12 Zsolt Kozak <koza...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have some problems with backuping Kubernetes PVCs with Bacula
>>>>> Kubernetes Plugin. (I have asked it on bacula-users mailing list but got
>>>>> no
>>>>> answer.)
>>>>>
>>>>> I am using the latest 13.0.1 Bacula from the community builds on
>>>>> Debian Bullseye hosts.
>>>>>
>>>>> Backuping only the Kubernetes objects except Persistent Volume Claims
>>>>> (PVC) works like a charm. I've installed the Kubernetes plugin and the
>>>>> latest Bacula File Daemon on the master node (control plane) of our
>>>>> Kubernetes cluster. Bacula can access the Kubernetes cluster and backup
>>>>> every single object as YAML files.
>>>>>
>>>>> The interesting part comes with trying to backup a PVC...
>>>>>
>>>>> First of all I could build my own Bacula Backup Proxy Pod Image from
>>>>> the source and it's deployed into our local Docker image repository
>>>>> (repo).
>>>>> The Bacula File Daemon is configured properly I guess. Backup process
>>>>> started and the following things happened.
>>>>>
>>>>> 1. Bacula File Daemon deployed Bacula Backup Proxy Pod Image into the
>>>>> Kubernetes cluster, so Bacula-backup container pod started.
>>>>> 2. I got into the pod and I could see the Baculatar application
>>>>> started and running.
>>>>> 3. The k8s_backend application started on the Bacula File Daemon host
>>>>> (kubernetes.server) in 2 instances.
>>>>> 4. From the Bacula-backup pod I could check that Baculatar could
>>>>> connect to the k8s_backend at the default 9104 port
>>>>> (kubernetes.server:9104).
>>>>> 5. I checked the console messages of the job with Bat that Bacula File
>>>>> Daemon started to process the configured PVC, started to write a pvc.tar
>>>>> but nothing happened.
>>>>> 6. After default 600 sec, after timeout the job was cancelled.
>>>>> 7. It may be important that Bacula File Daemon could not delete the
>>>>> Bacula-backup pod. (It could create it but could not delete it.)
>>>>>
>>>>>
>>>>> Could you please tell me what's wrong?
>>>>>
>>>>>
>>>>> Here are some log parts. (I've changed some sensitive data.)
>>>>>
>>>>>
>>>>> Bacula File Daemon configuration:
>>>>>
>>>>> FileSet {
>>>>> Name = "Kubernetes Set"
>>>>> Include {
>>>>> Options {
>>>>> signature = SHA512
>>>>> compression = GZIP
>>>>> Verify = pins3
>>>>> }
>>>>> Plugin = "kubernetes: \
>>>>> debug=1 \
>>>>> baculaimage=repo/bacula-backup:04jan23 \
>>>>> namespace=namespace \
>>>>> pvcdata \
>>>>> pluginhost=kubernetes.server \
>>>>> timeout=120 \
>>>>> verify_ssl=0 \
>>>>> fdcertfile=/etc/bacula/certs/bacula-backup.cert \
>>>>> fdkeyfile=/etc/bacula/certs/bacula-backup.key"
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Bacula File Daemon debug log (parts):
>>>>>
>>>>>
>>>>> DEBUG:[baculak8s/jobs/estimation_job.py:134 in processing_loop]
>>>>> processing get_annotated_namespaced_pods_data:namespace:nrfound:0
>>>>> DEBUG:[baculak8s/plugins/kubernetes_plugin.py:319 in
>>>>> list_pvcdata_for_namespace] list pvcdata for namespace:namespace
>>>>> pvcfilter=True estimate=False
>>>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:108 in
>>>>> pvcdata_list_namespaced] pvcfilter: True
>>>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:112 in
>>>>> pvcdata_list_namespaced] found:some-claim
>>>>> DEBUG:[baculak8s/plugins/k8sbackend/pvcdata.py:127 in
>>>>> pvcdata_list_namespaced] add pvc: {'name': 'some-claim', 'node_name':
>>>>> None,
>>>>> 'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
>>>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>>>> DEBUG:[baculak8s/jobs/estimation_job.py:165 in processing_loop]
>>>>> processing list_pvcdata_for_namespace:namespace:nrfound:1
>>>>> DEBUG:[baculak8s/jobs/estimation_job.py:172 in processing_loop]
>>>>> PVCDATA:some-claim:{'name': 'some-claim', 'node_name': 'node1',
>>>>> 'storage_class_name': 'nfs-client', 'capacity': '2Gi', 'fi':
>>>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> I000041
>>>>> Start backup volume claim: some-claim
>>>>>
>>>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:298 in prepare_bacula_pod]
>>>>> prepare_bacula_pod:token=xx88M5oggQJ....4YDbSwBRxTOhT namespace=namespace
>>>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:136 in prepare_pod_yaml]
>>>>> pvcdata: {'name': 'some-claim', 'node_name': 'node1',
>>>>> 'storage_class_name':
>>>>> 'nfs-client', 'capacity': '2Gi', 'fi':
>>>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>>>> DEBUG:[baculak8s/plugins/k8sbackend/baculabackup.py:102 in
>>>>> prepare_backup_pod_yaml] host:kubernetes.server port:9104
>>>>> namespace:namespace image:repo/bacula-backup:04jan23
>>>>> job:KubernetesBackup.2023-01-04_21.05.03_10:410706
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> I000149
>>>>> Prepare Bacula Pod on: node1 with: repo/bacula-backup:04jan23
>>>>> <IfNotPresent> kubernetes.server:9104
>>>>>
>>>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:198 in
>>>>> prepare_connection_server] prepare_connection_server:New ConnectionServer:
>>>>> 0.0.0.0:9104
>>>>> DEBUG:[baculak8s/util/sslserver.py:180 in listen]
>>>>> ConnectionServer:Listening...
>>>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:307 in prepare_bacula_pod]
>>>>> prepare_bacula_pod:start pod
>>>>> INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in
>>>>> backup_pod_isready] backup_pod_status:isReady: False / 0
>>>>> INFO:[baculak8s/plugins/kubernetes_plugin.py:771 in
>>>>> backup_pod_isready] backup_pod_status:isReady: True / 1
>>>>> DEBUG:[baculak8s/jobs/estimation_job.py:183 in _estimate_file]
>>>>> {'name': 'some-claim', 'node_name': 'node1', 'storage_class_name':
>>>>> 'nfs-client', 'capacity': '2Gi', 'fi':
>>>>> <baculak8s.entities.file_info.FileInfo object at 0x7ffaa55bfcc0>}
>>>>> DEBUG:[baculak8s/jobs/estimation_job.py:190 in _estimate_file]
>>>>> file_info: {FileInfo
>>>>> name:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
>>>>> namespace:None type:F objtype:pvcdata cached:False}
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> C000079
>>>>>
>>>>> FNAME:/@kubernetes/namespaces/namespace/persistentvolumeclaims/some-claim.tar
>>>>>
>>>>>
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> C000040
>>>>> TSTAMP:1672861077 1672861077 1672861077
>>>>>
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> C000031
>>>>> STAT:F 2147483648 0 0 100640 1
>>>>>
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> F000000
>>>>> (EOD PACKET)
>>>>>
>>>>> DEBUG:[baculak8s/jobs/backup_job.py:77 in __backup_pvcdata]
>>>>> backup_pvcdata:data recv
>>>>> DEBUG:[baculak8s/io/log.py:110 in save_sent_packet] Sent Packet
>>>>> C000005
>>>>> DATA
>>>>>
>>>>> DEBUG:[baculak8s/util/sslserver.py:193 in handle_connection]
>>>>> ConnectionServer:Connection from: ('192.168.XX.YY', 10541)
>>>>> DEBUG:[baculak8s/util/sslserver.py:145 in gethello] ['Hello',
>>>>> 'KubernetesBackup.2023-01-04_21.05.03_10', '410706']
>>>>> DEBUG:[baculak8s/util/token.py:57 in check_auth_data] AUTH_DATA:Token:
>>>>> xx88M5oggQJuGsPbtD........ohQjeU7PkA4YDbSwBRxTOhT
>>>>> DEBUG:[baculak8s/util/token.py:59 in check_auth_data]
>>>>> RECV_TOKEN_DATA:Token: xx88M5oggQJuGsPbtD....ohQjeU7PkA4YDbSwBRxTOhT
>>>>> DEBUG:[baculak8s/util/sslserver.py:105 in authenticate]
>>>>> ConnectionServer:Authenticated
>>>>>
>>>>> .... after timeout
>>>>>
>>>>> DEBUG:[baculak8s/jobs/job_pod_bacula.py:121 in handle_pod_data_recv]
>>>>> handle_pod_data_recv:EOT
>>>>> DEBUG:[baculak8s/util/sslserver.py:201 in handle_connection]
>>>>> ConnectionServer:Finish - disconnect.
>>>>> DEBUG:[baculak8s/jobs/backup_job.py:85 in __backup_pvcdata]
>>>>> backup_pvcdata:logs recv
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Job messages:
>>>>>
>>>>> bacula-dir No prior or suitable Full backup found in catalog for the
>>>>> current FileSet. Doing FULL backup.
>>>>> The FileSet "Kubernetes Set" was modified on 2023-01-04 20:20:41,
>>>>> this is after the last successful backup on 2023-01-04 19:19:49.
>>>>> bacula-sd Ready to append to end of Volume "Full-XXX"
>>>>> size=3,838,161,002
>>>>> bacula-fd Connected to Storage at bacula.server:9103 with TLS
>>>>> bacula-sd Volume "Full-XXXX" previously written, moving to end of data.
>>>>> bacula-dir Connected to Client "bacula-fd" at kubernetes.server:9102
>>>>> with TLS
>>>>> Using Device "FileStorageEeyoreFull" to write.
>>>>> Connected to Storage "InternalStorageFull" at bacula.server:9103 with
>>>>> TLS
>>>>> Start Backup JobId 410830, Job=KubernetesBackup.2023-01-04_21.05.03_10
>>>>> bacula-fd kubernetes: Prepare Bacula Pod on: node with:
>>>>> repo/bacula-backup:04jan23 kubernetes.server:9104
>>>>> kubernetes: Processing namespace: namespace
>>>>> kubernetes: Start backup volume claim: some-claim
>>>>> kubernetes: Connected to Kubernetes 1.25 - v1.25.4.
>>>>> bacula-dir
>>>>> Error: Bacula Enterprise bacula-dir 13.0.1 (05Aug22):
>>>>> Build OS: x86_64-pc-linux-gnu-bacula-enterprise debian
>>>>> 11.2
>>>>> JobId: 410830
>>>>> Job: KubernetesBackup.2023-01-04_21.05.03_10
>>>>> Backup Level: Full (upgraded from Differential)
>>>>> Client: "bacula-fd" 13.0.1 (05Aug22)
>>>>> x86_64-pc-linux-gnu-bacula-enterprise,debian,10.11
>>>>> FileSet: "Kubernetes Set" 2023-01-04 20:20:41
>>>>> Pool: "Full-Pool" (From Job FullPool override)
>>>>> Catalog: "MyCatalog" (From Client resource)
>>>>> Storage: "InternalStorageFull" (From Pool resource)
>>>>> Scheduled time: 04-Jan-2023 21:05:03
>>>>> Start time: 04-Jan-2023 21:27:04
>>>>> End time: 04-Jan-2023 21:29:06
>>>>> Elapsed time: 2 mins 2 secs
>>>>> Priority: 10
>>>>> FD Files Written: 23
>>>>> SD Files Written: 0
>>>>> FD Bytes Written: 52,784 (52.78 KB)
>>>>> SD Bytes Written: 0 (0 B)
>>>>> Rate: 0.4 KB/s
>>>>> Software Compression: 100.0% 1.0:1
>>>>> Comm Line Compression: 5.6% 1.1:1
>>>>> Snapshot/VSS: no
>>>>> Encryption: yes
>>>>> Accurate: yes
>>>>> Volume name(s): Full-XXXX
>>>>> Volume Session Id: 43
>>>>> Volume Session Time: 1672853724
>>>>> Last Volume Bytes: 3,838,244,105 (3.838 GB)
>>>>> Non-fatal FD errors: 3
>>>>> SD Errors: 0
>>>>> FD termination status: OK
>>>>> SD termination status: SD despooling Attributes
>>>>> Termination: *** Backup Error ***
>>>>> Fatal error: catreq.c:680 Restore object create error.
>>>>> bacula-fd
>>>>> Error: kubernetes: PTCOMM cannot get packet header from backend.
>>>>> bacula-dir Fatal error: sql_create.c:1273 Create db Object record
>>>>> INSERT INTO RestoreObject
>>>>> (ObjectName,PluginName,RestoreObject,ObjectLength,ObjectFullLength,ObjectIndex,ObjectType,ObjectCompression,FileIndex,JobId)
>>>>> VALUES ('RestoreOptions','kubernetes: \n debug=1 \n
>>>>> baculaimage=repo/bacula-backup:04jan23 \n
>>>>> namespace=namespace
>>>>> \n pvcdata \n
>>>>> pluginhost=kubernetes.server \n timeout=120 \n
>>>>> verify_ssl=0 \n
>>>>> fdcertfile=/etc/bacula/certs/bacula-backup.cert
>>>>> \n
>>>>> fdkeyfile=/etc/bacula/certs/bacula-backup.key','# Plugin configuration
>>>>> file\n# Version 1\nOptPrompt=\"K8S config
>>>>> file\"\nOptDefault=\"*None*\"\nconfig=@STR@\n\n
>>>>> OptPrompt=\"K8S API server
>>>>> URL/Host\"\nOptDefault=\"*None*\"\nhost=@STR@\n\nOptPrompt=\"K8S
>>>>> Bearertoken\"\nOptDefault=\"*None*\"\ntoken=@STR@\n\nOptPrompt=\"K8S
>>>>> API server cert verification\"\n
>>>>> OptDefault=\"True\"\nverify_ssl=@BOOL@\n\nOptPrompt=\"Custom CA Certs
>>>>> file to
>>>>> use\"\nOptDefault=\"*None*\"\nssl_ca_cert=@STR@\n\nOptPrompt=\"Output
>>>>> format when saving to file (JSON, YAML)\"\n
>>>>> OptDefault=\"RAW\"\noutputformat=@STR@\n\nOptPrompt=\"The address for
>>>>> listen to incoming backup pod
>>>>> data\"\nOptDefault=\"*FDAddress*\"\nfdaddress=@STR@\n\n
>>>>> OptPrompt=\"The port for opening socket for
>>>>> listen\"\nOptDefault=\"9104\"\nfdport=@INT32@\n\nOptPrompt=\"The
>>>>> endpoint address for backup pod to connect\"\n
>>>>> OptDefault=\"*FDAddress*\"\npluginhost=@STR@\n\nOptPrompt=\"The
>>>>> endpoint port to connect\"\nOptDefault=\"9104\"\n
>>>>> pluginport=@INT32@\n\n',859,859,0,27,0,1,410830) failed. ERR=Data too
>>>>> long for column 'PluginName' at row 1
>>>>>
>>>>> bacula-sd Sending spooled attrs to the Director. Despooling 8,214
>>>>> bytes ...
>>>>> bacula-fd
>>>>> Error: kubernetes: Error closing backend. Err=Child exited with code 1
>>>>> Fatal error: kubernetes: Wrong backend response to JobEnd command.
>>>>> bacula-sd Elapsed time=00:02:02, Transfer rate=659 Bytes/second
>>>>> bacula-fd
>>>>> Error: kubernetes: PTCOMM cannot get packet header from backend.
>>>>>
>>>>> Error: kubernetes: Cannot successfully start bacula-backup pod in
>>>>> expected time!
>>>>>
>>>>> Error: kubernetes: Job already running in 'namespace' namespace. Check
>>>>> logs or delete bacula-backup Pod manually.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Zsolt
>>>>>
>>>>> _______________________________________________
>>>>> Bacula-devel mailing list
>>>>> Bacula-devel@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/bacula-devel
>>>>>
>>>>
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
