Hi,
sorry, I almost forgot, so just to update this thread: I now started a new
thread for the actual FLIP:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-161-Configuration-through-environment-variables-td48094.html
Ingo
On Wed, Jan 20, 2021 at 11:10 AM Ingo Bürk w
Hi Steven,
understood, thanks for expanding on your point. I will prepare a FLIP now
going for the solution of S3_ACCESS_KEY style environment variables since
overall this fits Flink use-cases better. I think your point that users
have to know how to convert s3.access-key to S3_ACCESS_KEY is valid
Ingo,
regarding "state.checkpoints.dir: ${CHECKPOINTS_DIR:-path1}", it definitely
can work. but now users need to know that we can use "CHECKPOINTS_DIR" env
var to override "state.checkpoints.dir". That is the inconvenience that I
am trying to avoid. "state.checkpoints.dir" is well documented in t
I think a short FLIP would be awesome.
I guess this feature hasn't been implemented yet because it has not been
implemented yet ;-) I agree that this feature will improve configuration
ergonomics big time :-)
Cheers,
Till
On Tue, Jan 19, 2021 at 12:28 PM Ufuk Celebi wrote:
> Hey all,
>
> I thi
Hey all,
I think that approach 2 is more idiomatic for container deployments where it
can be cumbersome to manually map flink-conf.yaml contents to env vars [1]. The
precedence order outlined by Till would also cover Steven's hierarchical
overwrite requirement.
I'm really excited about this fe
Hi everyone,
Thanks for starting this discussion Ingo. I think being able to use env
variables to change Flink's configuration will be a very useful feature.
Concerning the two approaches I would be in favour of the second approach
($FLINK_CONFIG_S3_ACCESS_KEY) because it does not require the use
Hi Yang,
1. As you said I think this doesn't affect Ververica Platform, really, so
I'm more than happy to hear and follow the thoughts of people more
experienced with Flink than me.
2. I wasn't aware of env.java.opts, but that's definitely a candidate where
a user may want to "escape" it so it doe
Hi Steven,
regarding the hierarchical override, we could even expand the substitution
solution to support shell syntax with default values like
state.checkpoints.dir: ${CHECKPOINTS_DIR:-path1}
such that if the environment variable doesn't exist, path1 will be used.
Regards
Ingo
On Mon, Jan 1
Hi Ingo,
Thanks for your response.
1. Not distinguishing JM/TM is reasonable, but what about the client side.
For Yarn/K8s deployment,
the local flink-conf.yaml will be shipped to JM/TM. So I am just confused
about where should the environment
variables be replaced? IIUC, it is not an issue for V
Variable substitution (proposed here) is definitely useful.
For us, hierarchical override is more useful. E.g., we may have the
default value of "state.checkpoints.dir=path1" defined in flink-conf.yaml.
But maybe we want to override it to "state.checkpoints.dir=path2" via
environment variable in
Hi Yang,
thanks for your questions! I'm glad to see this feature is being received
positively.
ad 1) We don't distinguish JM/TM, and I can't think of a good reason why a
user would want to do so. I'm not very experienced with Flink, however, so
please excuse me if I'm overlooking some obvious rea
Thanks for kicking off the discussion.
I think supporting environment variables rendering in the Flink
configuration yaml file is a good idea. Especially for
the Kubernetes environment since we are using the secret resource to store
the authentication information.
But I have some questions for ho
Hi Ingo,
Thanks a lot for this proposal!
We had a related discussion recently in the context of FLINK-19520
(randomizing tests configuration) [1].
I believe other scenarios will benefit as well.
For the end users, I think substitution in configuration files is
preferable over parsing env vars in
13 matches
Mail list logo
