dwn-cadams opened a new issue #241: URL: https://github.com/apache/pulsar-helm-chart/issues/241
**Describe the bug** Recently we increased zookeeper replicas from 1 to 3 when moving a cluster to prod. When we did this, we discovered the init jobs (https://github.com/apache/pulsar-helm-chart/blob/master/charts%2Fpulsar%2Ftemplates%2Fbookkeeper-cluster-initialize.yaml and https://github.com/apache/pulsar-helm-chart/blob/master/charts%2Fpulsar%2Ftemplates%2Fpulsar-cluster-initialize.yaml) cannot be upgraded if you change the zookeeper replica count. Chart upgrade fails with a `field is immutable` error. This appears to be because the `args` field in a job is immutable, but it's being changed because it's referenced in the init jobs: * https://github.com/apache/pulsar-helm-chart/blob/master/charts%2Fpulsar%2Ftemplates%2Fbookkeeper-cluster-initialize.yaml#L52 * https://github.com/apache/pulsar-helm-chart/blob/master/charts%2Fpulsar%2Ftemplates%2Fpulsar-cluster-initialize.yaml#L62 **Workarounds attempted** * We deleted the init jobs to let them be recreated. * However this led to disaster as they're not idempotent and re-initialized everything, and we had to rebuild the cluster. * Setting `initialize: false` in helm values * We cannot do this since our use-case is installing Pulsar as part of on-prem installation **To Reproduce** 1. Install the chart 2. Change `values.zookeeper.replicaCount` (eg. from 3 to 5) 3. Upgrade the chart 4. You will get a "field is immutable" error: ``` Error: UPGRADE FAILED: cannot patch "pulsar-bookie-init" with kind Job: Job.batch "pulsar-bookie-init" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"070ad83b-8c2b-4e77-bc15-fdb7f6c9915f", "job-name":"pulsar-bookie-init", "kots.io/app-slug":"REDACTED", "kots.io/backup":"velero"}, Annotations:map[string]string{"kots.io/app-slug":"REDACTED"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"wait-zookeeper-ready", Image:"streamnative/pulsar-all:2.7.2.8", Command:[]string{ "sh", "-c"}, Args:[]string{"until nslookup pulsar-zookeeper-2.pulsar-zookeeper.default; do\n sleep 3;\ndone;"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"pulsar-bookie-init", Image:"streamnative/pulsar-all:2.7.2.8", Command:[]string{"sh", "-c"}, Args:[]string{"bin/apply-config-from-env.py conf/bookkeeper.conf;\nif bin/bookkeeper shell whatisinstanceid; then\n e cho \"bookkeeper cluster already initialized\";\nelse\n bin/bookkeeper shell initnewcluster;\nfi\n"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc00dfdf3c8), SecretRef:(*core.SecretEnvSource)(nil)}}, Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc007772f88), ActiveDeadlineSecon ds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc007372480), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable && cannot patch "pulsar-pulsar-init" with kind Job: Job.batch "pulsar-pulsar-init" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{ Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"33def730-048f-4169-acdf-cac1ffed1637", "job-name":"pulsar-pulsar-init", "kots.io/app-slug":"REDACTED", "kots.io/backup":"velero"}, Annotations:map[string]string{"kots.io/app-slug":"REDACTED"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"wait-zookeeper-ready", Image:"streamnative/pulsar-all:2.7.1.3", Command:[]string{"sh", "-c"}, Args:[]string{"until nslookup pulsar-zookeeper-2.pulsar-zookeeper.default; do\n sleep 3;\ndone;"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil ), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}, core.Container{Name:"pulsar-bookkeeper-verify-clusterid", Image:"streamnative/pulsar-all:2.7.1.3", Command:[]string{"sh", "-c"}, Args:[]string{"bin/apply-config-from-env.py conf/bookkeeper.conf;\nuntil bin/bookkeeper shell whatisinstanceid; do\n sleep 3;\ndone;\n"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc0102fc888), SecretRef:(*core.SecretEn vSource)(nil)}}, Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"pulsar-pulsar-init", Image:"streamnative/pulsar-all:2.7.1.3", Command:[]string{"sh", "-c"}, Args:[]string{"\nbin/pulsar initialize-cluster-metadata \\\n --cluster pulsar \\\n --zookeeper pulsar-zookeeper:2181 \\\n --configuration-store pulsar-zookeeper:2181 \\\n --web-service-url http://pulsar-broker.default.svc.cluster.local:8080/ \\\n --web-service-url-tls https://pulsar-broker.def ault.svc.cluster.local:8443/ \\\n --broker-service-url pulsar://pulsar-broker.default.svc.cluster.local:6650/ \\\n --broker-service-url-tls pulsar+ssl://pulsar-broker.default.svc.cluster.local:6651/ || true;\n"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc00da13390), ActiveDeadlineSeconds:(*int64) (nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00618c800), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable ``` **Expected behavior** Expected the upgrade to succeed **Screenshots** n/a **Desktop (please complete the following information):** (happens on all versions) **Additional context** n/a -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
