devinbost opened a new issue #82: URL: https://github.com/apache/pulsar-helm-chart/issues/82
**Describe the bug** After enabling TLS and Authentication, the helm chart fails to install. (The pods hang in an invalid state.) The issue preventing the pods from starting appears to be this: > MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition It is not clear why the secret cache is timing out. **To Reproduce** Here are the exact steps to reproduce this issue: ``` $ git clone https://github.com/apache/pulsar-helm-chart $ cd pulsar-helm-chart $ cat > ./examples/values-minikube.yaml ``` ``` volumes: persistence: false affinity: anti_affinity: false components: autorecovery: false zookeeper: replicaCount: 1 bookkeeper: replicaCount: 1 broker: replicaCount: 1 configData: autoSkipNonRecoverableData: "true" managedLedgerDefaultEnsembleSize: "1" managedLedgerDefaultWriteQuorum: "1" managedLedgerDefaultAckQuorum: "1" proxy: replicaCount: 1 tls: enabled: true bookie: enabled: true autorecovery: enabled: true toolset: enabled: true proxy: enabled: true broker: enabled: true zookeeper: enabled: true auth: authentication: enabled: false provider: "jwt" jwt: usingSecretKey: false authorization: enabled: true superUsers: broker: "broker-admin" proxy: "proxy-admin" client: "client-admin" ``` ctrl + c ``` $ minikube start --memory=8192 --cpus=4 $ ./scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-mini -c --pulsar-superusers superadmin,proxy-admin,broker-admin,client-admin $ ./scripts/pulsar/upload_tls.sh -k pulsar-mini -d ./.ci/tls $ helm install --values examples/values-minikube.yaml pulsar-mini apache/pulsar ``` `$ kubectl get pods -n pulsar` shows them hanging in incomplete state `$ kubectl describe pods -n pulsar` shows this issue: > Warning FailedMount 7m11s kubelet MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition Here is some additional context provided when running describe on the zookeeper pod: > Name: pulsar-mini-zookeeper-0 > Namespace: pulsar > Priority: 0 > Node: minikube/192.168.49.2 > Start Time: Mon, 09 Nov 2020 23:45:21 -0700 > Labels: app=pulsar > cluster=pulsar-mini > component=zookeeper > controller-revision-hash=pulsar-mini-zookeeper-59c4465569 > release=pulsar-mini > statefulset.kubernetes.io/pod-name=pulsar-mini-zookeeper-0 > Annotations: prometheus.io/port: 8000 > prometheus.io/scrape: true > Status: Pending > IP: > IPs: <none> > Controlled By: StatefulSet/pulsar-mini-zookeeper > Containers: > pulsar-mini-zookeeper: > Container ID: > Image: apachepulsar/pulsar-all:2.6.0 > Image ID: > Ports: 2181/TCP, 2888/TCP, 3888/TCP, 2281/TCP > Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP > Command: > sh > -c > Args: > bin/apply-config-from-env.py conf/zookeeper.conf; > /pulsar/keytool/keytool.sh zookeeper ${HOSTNAME}.pulsar-mini-zookeeper.pulsar.svc.cluster.local false; bin/generate-zookeeper-config.sh conf/zookeeper.conf; bin/pulsar zookeeper; > > State: Waiting > Reason: ContainerCreating > Ready: False > Restart Count: 0 > Requests: > cpu: 100m > memory: 256Mi > Liveness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=1s period=30s #success=1 #failure=10 > Readiness: exec [bin/pulsar-zookeeper-ruok.sh] delay=10s timeout=1s period=30s #success=1 #failure=10 > Environment Variables from: > pulsar-mini-zookeeper ConfigMap Optional: false > Environment: > ZOOKEEPER_SERVERS: pulsar-mini-zookeeper-0 > Mounts: > /pulsar/certs/ca from ca (ro) > /pulsar/certs/zookeeper from zookeeper-certs (ro) > /pulsar/data from pulsar-mini-zookeeper-data (rw) > /pulsar/keytool/keytool.sh from keytool (rw,path="keytool.sh") > /var/run/secrets/kubernetes.io/serviceaccount from default-token-vtl5l (ro) > Conditions: > Type Status > Initialized True > Ready False > ContainersReady False > PodScheduled True > Volumes: > pulsar-mini-zookeeper-data: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: <unset> > zookeeper-certs: > Type: Secret (a volume populated by a Secret) > SecretName: pulsar-mini-tls-zookeeper > Optional: false > ca: > Type: Secret (a volume populated by a Secret) > SecretName: pulsar-mini-ca-tls > Optional: false > keytool: > Type: ConfigMap (a volume populated by a ConfigMap) > Name: pulsar-mini-keytool-configmap > Optional: false > default-token-vtl5l: > Type: Secret (a volume populated by a Secret) > SecretName: default-token-vtl5l > Optional: false > QoS Class: Burstable > Node-Selectors: <none> > Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s > node.kubernetes.io/unreachable:NoExecute op=Exists for 300s > Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > Normal Scheduled 7m12s default-scheduler Successfully assigned pulsar/pulsar-mini-zookeeper-0 to minikube > Warning FailedMount 7m11s kubelet MountVolume.SetUp failed for volume "zookeeper-certs" : failed to sync secret cache: timed out waiting for the condition > Normal Pulling 7m9s kubelet Pulling image "apachepulsar/pulsar-all:2.6.0" **Expected behavior** Installing the helm chart with the provided values should start the Pulsar cluster in minikube with TLS and authentication enabled. **Environment:** 😄 minikube v1.14.2 on Darwin 10.15.7 ✨ Using the docker driver based on existing profile 🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.8 ... 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 kubectl is configured to use "minikube" by default ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
