Thanks a lot for writing this summary Ufuk!
> Do you agree that approach 1 has been rejected?
I don't think so. To me, using conversion is prone to errors to the same
degree as escaping in option 2 (plus inconvenience in editing).
> Is there any objection to approach 2?
Not from my side.
> Did w
Let me try to summarize the current state of the discussion. I think we now
have a few competing approaches.
# Approach 1: dynamic env var mapping
* This is the approach currently outlined in the FLIP
* The assumption that we only have dots and hyphens in config option keys seem
to be wrong, e.
Hi Ingo,
I missed that part of the discussion, sorry about that.
Would you mind putting it to the FLIP page?
I guess you are referring to this message:
> I don't think we can assume that we know all config option keys. For
instance, I might write a custom high availability service or a custom
File
Hi Yang,
yeah, this is essentially the solution we also use in Ververica Platform
(except that we use ${…}). I can't really add anything other than the
previously stated reasons for why we decided to reject this, but of course
I'm also open to this solution still. I don't think Flink necessarily n
Hi Roman,
unfortunately the configuration keys are not currently known upfront, and
enforcing this would impose strict limitations in the future. Without
knowledge of which keys exist upfront, we are forced to map an environment
variable to the config key in some specified manner without actually
On Tue, Jan 26, 2021 at 1:36 AM Till Rohrmann wrote:
> I am somehow assuming that it is easier for (human) users to specify a
> single env variable for a config option instead of defining a block of
> Flink configuration values. I wouldn't know how to properly separate the
> entries of the config
Hi all, sorry to butt in.
I am little curious about why do we have to do the overwritten via
environment variables after
loading the configuration from file. Could we support to do the
substitution while loading the
"flink-conf.yaml" file?
For example, I have the following config options in my fl
In the end DYNAMIC_PROPERTIES behaves exactly like env.java.opts;
meaning that the existing rules set by the JVM apply.
Here's an example: export DYNAMIC_PROPERTIES='-Drest.port=1234
-Dother.option="iContainAn=Sign"'
This would then be appended as is to the /java/ command.
(
Conceptually
> Here's an example: My option key is custom.my_backend_option. With the
> current design, the corresponding env variable would be
> CUSTOM_MY_BACKEND_OPTION, which would be converted into
> custom.my.backend.option .
I think we don't have to translate CUSTOM_MY_BACKEND_OPTION back. Instead,
we sh
Thinking a bit more about the DYNAMIC_PROPERTIES, I have to admit that I
like the fact that it works around the problem of encoding the key names
and that it is more powerful wrt to bulk changes. Also the fact that one
can copy past configuration snippets is quite useful. Given these aspects
and th
Hi everyone,
thanks for the livid discussion, it's great to see so many opinions and
ideas!
The point about underscores is a very valid point where the current FLIP,
if we were to stick with it, would have to be improved. I was going to say
that we should exclude that from the discussion about th
FYI: the StreamConfig actually queries the Configuration where some keys
with underscores. This isn't quite relevant for environment variables
themselves, but where exactly in the API the lookup for environment
variables is supposed to done.
In a similar vein, the metric system does not direct
Mind you that we could of course solve these character issues by first
nailing down which characters we allow in keys (presumably: [a-z0-9-.]).
On 1/26/2021 3:45 PM, Chesnay Schepler wrote:
Here's an example: My option key is custom.my_backend_option. With the
current design, the corresponding
Here's an example: My option key is custom.my_backend_option. With the
current design, the corresponding env variable would be
CUSTOM_MY_BACKEND_OPTION, which would be converted into
custom.my.backend.option .
It is true that users could still parse the original system property as
a fall-back
@Chesnay
could you explain how underscores in user-defined properties would be
affected with transformation like STATE_BACKEND -> state.backend?
IIUC, this transformation happens in Flink and doesn't alter ENV vars, so
the user app can still parse the original configuration.
OTH, I'm a bit concerne
I think we have to assume that some user has a custom config option that
uses underscores.
That said, we can probably assume that no one uses other special
characters like question marks and such, which are indeed allowed by the
YAML spec.
These kind of questions are precisely why I prefer t
@Ufuk,
I also don't find any existing options with underscores in their keys.
However, I do not find any explicit rules forbid using them either. I'm not
saying this should block the FLIP. Just it would be nice to beware of this
issue, and maybe ensure the assumption with test cases if we finally d
@Xingtong: The assumption for the mapping was that we only have dots and
hyphens in the keys. Do you have an example for a key which include
underscores? If underscores are common for keys (I couldn't find any existing
options that use it), it would certainly be a blocker for the discussed
appr
I might have joined the discussion a bit late.
My two cents:
- I'm good with EVs overwriting the flink-conf.yaml. However, I'm a bit
unsure about EVs overwriting the dynamic properties.
- Currently, dynamic properties are expected to be the final
effective configuration in many
I'm including Steven and Yang into this discussion (cc'd) who raised some good
points in the initial DISCUSS thread. Do you have any opinion on the current
discussion here?
On Tue, Jan 26, 2021, at 10:59 AM, Ufuk Celebi wrote:
> Chesnay, thanks for bringing this up. I think it's an alternative t
Chesnay, thanks for bringing this up. I think it's an alternative that's
worthwhile to discuss, although I do think that we should stick to the current
proposal in the FLIP.
My main concerns about the FLINK_PROPERTIES approach boil down to the
following:
* A single env var per config option is
It is true that we haven't properly discussed whether to support bulk
config changes via a single environment variable or not, Thomas. It is also
true that one could move the logic to the Flink processes (more
specifically to where the config is loaded).
I am somehow assuming that it is easier for
It appears that the discussion in FLIP-161 is primarily focussed on setting
individual configuration entries?
On the other hand, $FLINK_PROPERTIES (and envsubst) were introduced to set
a fragment or the complete flink-conf.yaml. It's used by at least 2 of the
Flink k8s operators.
In these cases t
I agree with Till about $FLINK_PROPERTIES being only supported by a Flink
buildfile.
Besides that,
1. having different ways of configuring different applications doesn't seem
convenient to me. For example, Kafka and ZK configured via usual properties
and Flink via concatenated one.
2. It could also
The problem I see with $FLINK_PROPERTIES is that this is only supported by
Flink's Docker entrypoint.sh. In fact this variable was introduced as a
temporary bridge to allow Docker users to change Flink's configuration
dynamically. If we had had a proper way of configuring Flink processes via
env va
Hi Chesnay,
thanks for your input! After some thought I think your proposal makes a lot
of sense, i.e. if we have one single $FLINK_DYNAMIC_PROPERTIES environment
variable, in a Kubernetes environment we could do something like
```
env:
- name: S3_KEY
valueFrom: # get from secret
- name:
The FLIP seems to disregard the existence of dynamic properties, and I'm
wondering why that is the case.
Don't they fulfill similar roles, in that they allow config options to
be set on the command-line?
What use-case do they currently not support?
I assume it's something along the lines of set
The updated design LGTM as well. Nice work Ingo!
Cheers,
Till
On Fri, Jan 22, 2021 at 3:33 PM Ingo Bürk wrote:
> Thanks, Ufuk. I think that makes sense, so I moved it from a footnote to an
> addition to prevent that in the future as well.
>
> Ingo
>
> On Fri, Jan 22, 2021 at 3:10 PM Ufuk Celebi
Thanks, Ufuk. I think that makes sense, so I moved it from a footnote to an
addition to prevent that in the future as well.
Ingo
On Fri, Jan 22, 2021 at 3:10 PM Ufuk Celebi wrote:
> LGTM. Let's see what the others think...
>
> On Thu, Jan 21, 2021, at 11:37 AM, Ingo Bürk wrote:
> > Regarding en
LGTM. Let's see what the others think...
On Thu, Jan 21, 2021, at 11:37 AM, Ingo Bürk wrote:
> Regarding env.java.opts, what special handling is needed there? AFAICT only
> the rejected alternative of substituting values would've had an effect on
> this.
Makes sense 👍
>From the FLIP:
> This mapp
Hi everyone,
I've updated the FLIP (https://cwiki.apache.org/confluence/x/ngtRCg)
according to these discussions.
Regards
Ingo
On Thu, Jan 21, 2021 at 11:37 AM Ingo Bürk wrote:
> Hi Ufuk, Till,
>
> I definitely agree that having the Configuration be (or at least feel)
> immutable and complete
Hi Ufuk, Till,
I definitely agree that having the Configuration be (or at least feel)
immutable and complete seems like a better choice, and it is probably worth
the trade-off in EV naming flexibility. Let me reshape the FLIP to propose
something along the lines of solution (3).
Regarding env.jav
Thanks for starting the discussion, Ingo!
Regarding approach 1:
I like the idea of having a mapping scheme from ConfigOption to env var(s), but
I'm concerned about the implications of lazy eval. I think it would be
preferable to keep the Configuration object as the source of truth, requiring
u
Thanks for starting this discussion Ingo!
I like the flexibility in terms of supported EV formats the lazy evaluation
gives us. However, I am a bit concerned by the fact that it is quite hard
to compute the configurational ground truth w/o knowing all requested
config keys. Moreover, it makes the
34 matches
Mail list logo
