Ingo,
regarding "state.checkpoints.dir: ${CHECKPOINTS_DIR:-path1}", it definitely
can work. but now users need to know that we can use "CHECKPOINTS_DIR" env
var to override "state.checkpoints.dir". That is the inconvenience that I
am trying to avoid. "state.checkpoints.dir" is well documented in the Flink
website. Now, we need to document "CHECKPOINTS_DIR" separately.
Ideally, we want to just let the user define a env var like
"state.checkpoints.dir=some/overriden/path". But it suffers the problem
that dot chars are invalid characters for shell env var names. That is the
dilemma.
That is why we bundled all the non-conformning env var overrides (where
variable names containing non-conforming chars) into a single base64
encoded string (like "NON_CONFORMING_OVERRIDES_BASE64=<base64 encoded json
string>" in our deployment infrastructure and unpack them in the container
startup. It is hacky. I am hoping that there is a more elegant solution.
If the configuration key is conforming to shell standard (like
S3_ACCESS_KEY), then we don't have a problem. but it will be deviating from
the Flink config naming convention (Java property style, dot separated).
Thanks,
Steven
On Tue, Jan 19, 2021 at 6:56 AM Till Rohrmann <trohrm...@apache.org> wrote:
> I think a short FLIP would be awesome.
>
> I guess this feature hasn't been implemented yet because it has not been
> implemented yet ;-) I agree that this feature will improve configuration
> ergonomics big time :-)
>
> Cheers,
> Till
>
> On Tue, Jan 19, 2021 at 12:28 PM Ufuk Celebi <u...@apache.org> wrote:
>
> > Hey all,
> >
> > I think that approach 2 is more idiomatic for container deployments where
> > it can be cumbersome to manually map flink-conf.yaml contents to env vars
> > [1]. The precedence order outlined by Till would also cover Steven's
> > hierarchical overwrite requirement.
> >
> > I'm really excited about this feature as it will make Flink deployments a
> > lot more ergonomic. The implementation seems to be not too complicated
> > (which makes we wonder why we didn't tackle this earlier or whether I'm
> > missing something).
> >
> > I'd also be happy to shepherd this contribution if there is consensus on
> > the need for it and the approach. Does it make sense to formalize this
> > decision a bit with a short FLIP?
> >
> > – Ufuk
> >
> > [1] In Ververica Platform, we support approach 1, because the Flink
> > configuration is part of the specification for a single Deployment and
> it's
> > minimally more convenient to have something like
> >
> > flinkConfiguration:
> > foo: ${BAR}
> >
> > for us. I don't think this approach would feel natural when manually
> > deploying Flink. There would be a clear migration path for our customers,
> > so I'm not concerned about this too much.
> >
> > On Tue, Jan 19, 2021, at 10:01 AM, Till Rohrmann wrote:
> > > Hi everyone,
> > >
> > > Thanks for starting this discussion Ingo. I think being able to use env
> > > variables to change Flink's configuration will be a very useful
> feature.
> > >
> > > Concerning the two approaches I would be in favour of the second
> approach
> > > ($FLINK_CONFIG_S3_ACCESS_KEY) because it does not require the user to
> > > prepare a special flink-conf.yaml where he inserts env variables for
> > every
> > > config value he wants to configure. Since this is not required with the
> > > second approach, I think it is more general and easier to use. Also,
> the
> > > user does not have to remember a second set of names (env names) which
> he
> > > has to/can set.
> > >
> > > For how to substitute the values, I think it should happen when we load
> > the
> > > Flink configuration. First we read the file and then overwrite values
> > > specified via an env variable or dynamic properties in some defined
> > order.
> > > For env.java.opts and other options which are used for starting the JVM
> > we
> > > might need special handling in the bash scripts.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Jan 19, 2021 at 9:46 AM Ingo Bürk <i...@ververica.com> wrote:
> > >
> > > > Hi Yang,
> > > >
> > > > 1. As you said I think this doesn't affect Ververica Platform,
> really,
> > so
> > > > I'm more than happy to hear and follow the thoughts of people more
> > > > experienced with Flink than me.
> > > > 2. I wasn't aware of env.java.opts, but that's definitely a candidate
> > where
> > > > a user may want to "escape" it so it doesn't get substituted
> > immediately, I
> > > > agree.
> > > >
> > > >
> > > > Regards
> > > > Ingo
> > > >
> > > > On Tue, Jan 19, 2021 at 4:47 AM Yang Wang <danrtsey...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Ingo,
> > > > >
> > > > > Thanks for your response.
> > > > >
> > > > > 1. Not distinguishing JM/TM is reasonable, but what about the
> client
> > > > side.
> > > > > For Yarn/K8s deployment,
> > > > > the local flink-conf.yaml will be shipped to JM/TM. So I am just
> > confused
> > > > > about where should the environment
> > > > > variables be replaced? IIUC, it is not an issue for Ververica
> > Platform
> > > > > since it is always done in the JM/TM side.
> > > > >
> > > > > 2. I believe we should support not do the substitution for specific
> > key.
> > > > A
> > > > > typical use case is "env.java.opts". If the
> > > > > value contains environment variables, they are expected to be
> > replaced
> > > > > exactly when the java command is executed,
> > > > > not after the java process is started. Maybe escaping with single
> > quote
> > > > is
> > > > > enough.
> > > > >
> > > > > 3. The substitution only takes effects on the value makes sense to
> > me.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Steven Wu <stevenz...@gmail.com> 于2021年1月19日周二 上午12:36写道：
> > > > >
> > > > > > Variable substitution (proposed here) is definitely useful.
> > > > > >
> > > > > > For us, hierarchical override is more useful. E.g., we may have
> > the
> > > > > > default value of "state.checkpoints.dir=path1" defined in
> > > > > flink-conf.yaml.
> > > > > > But maybe we want to override it to "state.checkpoints.dir=path2"
> > via
> > > > > > environment variable in some scenarios. Otherwise, we have to
> > define a
> > > > > > corresponding shell variable (like STATE_CHECKPOINTS_DIR) for the
> > Flink
> > > > > > config, which is annoying.
> > > > > >
> > > > > > As Ingo pointed, it is also annoying to handle Java property key
> > naming
> > > > > > convention (dots separated), as dots aren't allowed in shell env
> > var
> > > > > naming
> > > > > > (All caps, separated with underscore). Shell will complain. We
> > have to
> > > > > > bundle all env var overrides (k-v pairs) in a single property
> value
> > > > (JSON
> > > > > > and base64 encode) to avoid it.
> > > > > >
> > > > > > On Mon, Jan 18, 2021 at 8:15 AM Ingo Bürk <i...@ververica.com>
> > wrote:
> > > > > >
> > > > > > > Hi Yang,
> > > > > > >
> > > > > > > thanks for your questions! I'm glad to see this feature is
> being
> > > > > received
> > > > > > > positively.
> > > > > > >
> > > > > > > ad 1) We don't distinguish JM/TM, and I can't think of a good
> > reason
> > > > > why
> > > > > > a
> > > > > > > user would want to do so. I'm not very experienced with Flink,
> > > > however,
> > > > > > so
> > > > > > > please excuse me if I'm overlooking some obvious reason here.
> :-)
> > > > > > > ad 2) Admittedly I don't have a good overview on all the
> > > > configuration
> > > > > > > options that exist, but from those that I do know I can't
> imagine
> > > > > someone
> > > > > > > wanting to pass a value like "${MY_VAR}" verbatim. In Ververica
> > > > > Platform
> > > > > > as
> > > > > > > of now we ignore this problem. If, however, this needs to be
> > > > > addressed, a
> > > > > > > possible solution could be to allow escaping syntax such as
> > > > > "\${MY_VAR}".
> > > > > > >
> > > > > > > Another point to consider here is when exactly the substitution
> > takes
> > > > > > > place: on the "raw" file, or on the parsed key / value
> > separately,
> > > > and
> > > > > if
> > > > > > > so, should it support both key and value? My current thinking
> is
> > that
> > > > > > > substituting only the value of the parsed entry should be
> > sufficient.
> > > > > > >
> > > > > > >
> > > > > > > Regards
> > > > > > > Ingo
> > > > > > >
> > > > > > > On Mon, Jan 18, 2021 at 3:48 PM Yang Wang <
> danrtsey...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for kicking off the discussion.
> > > > > > > >
> > > > > > > > I think supporting environment variables rendering in the
> Flink
> > > > > > > > configuration yaml file is a good idea. Especially for
> > > > > > > > the Kubernetes environment since we are using the secret
> > resource
> > > > to
> > > > > > > store
> > > > > > > > the authentication information.
> > > > > > > >
> > > > > > > > But I have some questions for how to do it?
> > > > > > > > 1. The environments in Flink configuration yaml will be
> > replaced in
> > > > > > > client,
> > > > > > > > JobManager, TaskManager or all of them?
> > > > > > > > 2. If users do not want some config options to be replaced,
> > how to
> > > > > > > > achieve that?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yang
> > > > > > > >
> > > > > > > > Khachatryan Roman <khachatryan.ro...@gmail.com>
> 于2021年1月18日周一
> > > > > > 下午8:55写道：
> > > > > > > >
> > > > > > > > > Hi Ingo,
> > > > > > > > >
> > > > > > > > > Thanks a lot for this proposal!
> > > > > > > > >
> > > > > > > > > We had a related discussion recently in the context of
> > > > FLINK-19520
> > > > > > > > > (randomizing tests configuration) [1].
> > > > > > > > > I believe other scenarios will benefit as well.
> > > > > > > > >
> > > > > > > > > For the end users, I think substitution in configuration
> > files is
> > > > > > > > > preferable over parsing env vars in Flink code.
> > > > > > > > > And for cases without such a file, we could have a default
> > one on
> > > > > the
> > > > > > > > > classpath with all substitutions defined (and then merge
> > > > everything
> > > > > > > from
> > > > > > > > > the user-supplied file).
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-19520
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Roman
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 18, 2021 at 11:11 AM Ingo Bürk <
> > i...@ververica.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > in Ververica Platform we offer a feature to use
> environment
> > > > > > variables
> > > > > > > > in
> > > > > > > > > > the Flink configuration¹, e.g.
> > > > > > > > > >
> > > > > > > > > > ```
> > > > > > > > > > s3.access-key: ${S3_ACCESS_KEY}
> > > > > > > > > > ```
> > > > > > > > > >
> > > > > > > > > > We've been discussing internally whether contributing
> such
> > a
> > > > > > feature
> > > > > > > to
> > > > > > > > > > Flink directly would make sense and wanted to start a
> > > > discussion
> > > > > on
> > > > > > > > this
> > > > > > > > > > topic.
> > > > > > > > > >
> > > > > > > > > > An alternative way to do so from the above would be
> parsing
> > > > those
> > > > > > > > > directly
> > > > > > > > > > based on their name, so instead of having it defined in
> the
> > > > Flink
> > > > > > > > > > configuration as above, it would get automatically set if
> > > > > something
> > > > > > > > like
> > > > > > > > > > $FLINK_CONFIG_S3_ACCESS_KEY was set in the environment.
> > This is
> > > > > > > > somewhat
> > > > > > > > > > similar to what e.g. Spring does, and faces similar
> > challenges
> > > > > > > (dealing
> > > > > > > > > > with "."s etc.)
> > > > > > > > > >
> > > > > > > > > > Although I view both of these approaches as mostly
> > orthogonal,
> > > > > > > > supporting
> > > > > > > > > > both very likely wouldn't make sense, of course. So I was
> > > > > wondering
> > > > > > > > what
> > > > > > > > > > your opinion is in terms of whether the project would
> > benefit
> > > > > from
> > > > > > > > > > environment variable support for the Flink configuration,
> > and
> > > > > > whether
> > > > > > > > > there
> > > > > > > > > > are tendencies as to which approach to go with.
> > > > > > > > > >
> > > > > > > > > > ¹
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.ververica.com/user_guide/application_operations/deployments/configure_flink.html#environment-variables
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Ingo
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
