Hi Ingo,
Thanks for your response.
1. Not distinguishing JM/TM is reasonable, but what about the client side.
For Yarn/K8s deployment,
the local flink-conf.yaml will be shipped to JM/TM. So I am just confused
about where should the environment
variables be replaced? IIUC, it is not an issue for Ververica Platform
since it is always done in the JM/TM side.
2. I believe we should support not do the substitution for specific key. A
typical use case is "env.java.opts". If the
value contains environment variables, they are expected to be replaced
exactly when the java command is executed,
not after the java process is started. Maybe escaping with single quote is
enough.
3. The substitution only takes effects on the value makes sense to me.
Best,
Yang
Steven Wu <stevenz...@gmail.com> 于2021年1月19日周二 上午12:36写道：
> Variable substitution (proposed here) is definitely useful.
>
> For us, hierarchical override is more useful. E.g., we may have the
> default value of "state.checkpoints.dir=path1" defined in flink-conf.yaml.
> But maybe we want to override it to "state.checkpoints.dir=path2" via
> environment variable in some scenarios. Otherwise, we have to define a
> corresponding shell variable (like STATE_CHECKPOINTS_DIR) for the Flink
> config, which is annoying.
>
> As Ingo pointed, it is also annoying to handle Java property key naming
> convention (dots separated), as dots aren't allowed in shell env var naming
> (All caps, separated with underscore). Shell will complain. We have to
> bundle all env var overrides (k-v pairs) in a single property value (JSON
> and base64 encode) to avoid it.
>
> On Mon, Jan 18, 2021 at 8:15 AM Ingo Bürk <i...@ververica.com> wrote:
>
> > Hi Yang,
> >
> > thanks for your questions! I'm glad to see this feature is being received
> > positively.
> >
> > ad 1) We don't distinguish JM/TM, and I can't think of a good reason why
> a
> > user would want to do so. I'm not very experienced with Flink, however,
> so
> > please excuse me if I'm overlooking some obvious reason here. :-)
> > ad 2) Admittedly I don't have a good overview on all the configuration
> > options that exist, but from those that I do know I can't imagine someone
> > wanting to pass a value like "${MY_VAR}" verbatim. In Ververica Platform
> as
> > of now we ignore this problem. If, however, this needs to be addressed, a
> > possible solution could be to allow escaping syntax such as "\${MY_VAR}".
> >
> > Another point to consider here is when exactly the substitution takes
> > place: on the "raw" file, or on the parsed key / value separately, and if
> > so, should it support both key and value? My current thinking is that
> > substituting only the value of the parsed entry should be sufficient.
> >
> >
> > Regards
> > Ingo
> >
> > On Mon, Jan 18, 2021 at 3:48 PM Yang Wang <danrtsey...@gmail.com> wrote:
> >
> > > Thanks for kicking off the discussion.
> > >
> > > I think supporting environment variables rendering in the Flink
> > > configuration yaml file is a good idea. Especially for
> > > the Kubernetes environment since we are using the secret resource to
> > store
> > > the authentication information.
> > >
> > > But I have some questions for how to do it?
> > > 1. The environments in Flink configuration yaml will be replaced in
> > client,
> > > JobManager, TaskManager or all of them?
> > > 2. If users do not want some config options to be replaced, how to
> > > achieve that?
> > >
> > > Best,
> > > Yang
> > >
> > > Khachatryan Roman <khachatryan.ro...@gmail.com> 于2021年1月18日周一
> 下午8:55写道：
> > >
> > > > Hi Ingo,
> > > >
> > > > Thanks a lot for this proposal!
> > > >
> > > > We had a related discussion recently in the context of FLINK-19520
> > > > (randomizing tests configuration) [1].
> > > > I believe other scenarios will benefit as well.
> > > >
> > > > For the end users, I think substitution in configuration files is
> > > > preferable over parsing env vars in Flink code.
> > > > And for cases without such a file, we could have a default one on the
> > > > classpath with all substitutions defined (and then merge everything
> > from
> > > > the user-supplied file).
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-19520
> > > >
> > > > Regards,
> > > > Roman
> > > >
> > > >
> > > > On Mon, Jan 18, 2021 at 11:11 AM Ingo Bürk <i...@ververica.com>
> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > in Ververica Platform we offer a feature to use environment
> variables
> > > in
> > > > > the Flink configuration¹, e.g.
> > > > >
> > > > > ```
> > > > > s3.access-key: ${S3_ACCESS_KEY}
> > > > > ```
> > > > >
> > > > > We've been discussing internally whether contributing such a
> feature
> > to
> > > > > Flink directly would make sense and wanted to start a discussion on
> > > this
> > > > > topic.
> > > > >
> > > > > An alternative way to do so from the above would be parsing those
> > > > directly
> > > > > based on their name, so instead of having it defined in the Flink
> > > > > configuration as above, it would get automatically set if something
> > > like
> > > > > $FLINK_CONFIG_S3_ACCESS_KEY was set in the environment. This is
> > > somewhat
> > > > > similar to what e.g. Spring does, and faces similar challenges
> > (dealing
> > > > > with "."s etc.)
> > > > >
> > > > > Although I view both of these approaches as mostly orthogonal,
> > > supporting
> > > > > both very likely wouldn't make sense, of course. So I was wondering
> > > what
> > > > > your opinion is in terms of whether the project would benefit from
> > > > > environment variable support for the Flink configuration, and
> whether
> > > > there
> > > > > are tendencies as to which approach to go with.
> > > > >
> > > > > ¹
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.ververica.com/user_guide/application_operations/deployments/configure_flink.html#environment-variables
> > > > >
> > > > > Best regards
> > > > > Ingo
> > > > >
> > > >
> > >
> >
>
