Hi Lucas! The Flink kubernetes integration itself is responsible for mounting the configmap and overwriting the entrypoint not the operator. Therefore this is not something we can easily change from the operator side. However I think we are looking at the problem from the wrong side and there may be a solution already :)
Ideally what you want is ENV replacement in Flink configuration. This is not something that the Flink community has added yet unfortunately but we have it on our radar for the operator at least ( https://issues.apache.org/jira/browse/FLINK-27491). It will probably be added in the next 1.4.0 version. This will be possible from Flink 1.16 which introduced a small feature that allows us to inject parameters to the kubernetes entrypoints: https://issues.apache.org/jira/browse/FLINK-29123 https://github.com/apache/flink/commit/c37643031dca2e6d4c299c0d704081a8bffece1d While it's not implemented in the operator yet, you could try setting the following config in Flink 1.16.0: kubernetes.jobmanager.entrypoint.args: -D datadog.secret.conf=$MY_SECRET_ENV kubernetes.taskmanager.entrypoint.args: -D datadog.secret.conf=$MY_SECRET_ENV If you use this configuration together with the default native mode in the operator, it should work I believe. Please try and let me know! Gyula On Wed, Nov 30, 2022 at 10:36 PM Lucas Caparelli < lucas.capare...@gympass.com> wrote: > Hello folks, > > Not sure if this is the best list for this, sorry if it isn't. I'd > appreciate some pointers :-) > > When using flink-kubernetes-operator [1], docker-entrypoint.sh [2] goes > through several failures to write into $FLINK_HOME/conf/. We believe this > is due to this volume being mounted from a ConfigMap, which means it's > read-only. > > This has been reported in the past in GCP's operator, but I was unable to > find any kind of resolution for it: > https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/issues/213 > > In our use case, we want to set an API key as part of the flink-conf.yaml > file, but we don't want it to be persisted in Kubernetes or in our version > control, since it's sensitive data. This API Key is used by Flink to report > metrics to Datadog [3]. > > We have automation in place which allows us to accomplish this by setting > environment variables pointing to a path in our secret manager, which only > gets injected during runtime. That part is working fine. > > However, we're trying to inject this secret using the FLINK_PROPERTIES > variable, which is appended [4] to the flink-conf.yaml file in the > docker-entrypoint script, which fails due to the filesystem where the file > is being read-only. > > We attempted working around this in 2 different ways: > > - providing our own .spec.containers[0].command, where we copied over > /opt/flink to /tmp/flink and set FLINK_HOME=/tmp/flink. This did not work > because the operator overwrote it and replaced it with its original > command/args; > - providing an initContainer sharing the volumes so it could make the > copy without being overridden by the operator's command/args. This did not > work because the initContainer present in the spec never makes it to the > resulting Deployment, it seems the operator ignores it. > > We have some questions: > > 1. Is this overriding of the pod template present in FlinkDeployment > intentional? That is, should our custom command/args and initContainers > have been overwritten? If so, I find it a bit confusing that these fields > are present and available for use at all. > 2. Since the ConfigMap volume will always be mounted as read-only, it > seems to me there's some adjustments to be made in order for this script to > work correctly. Do you think it would make sense for the script to copy > over contents from the ConfigMap volume to a writable directory during > initialization, and then use this copy for any subsequent operation? > Perhaps copying over to $FLINK_HOME, which the user could set themselves, > maybe even with a sane default which wouldn't fail on writes (eg > /tmp/flink). > > Thanks in advance for your attention and hard work on the project! > > [1]: https://github.com/apache/flink-kubernetes-operator > [2]: > https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/docker-entrypoint.sh > [3]: https://docs.datadoghq.com/integrations/flink/ > [4]: > https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/docker-entrypoint.sh#L86-L88 >
