Re: [ANNOUNCE] New Apache Flink PMC Member - Jingsong Lee

Congrats, Jingsong! On 13.06.22 08:58, Becket Qin wrote: Hi all, I'm very happy to announce that Jingsong Lee has joined the Flink PMC! Jingsong became a Flink committer in Feb 2020 and has been continuously contributing to the project since then, mainly in Flink SQL. He has been quite active

Re: [DISCUSS] FLIP-240: Introduce "ANALYZE TABLE" Syntax

imize such cases, we can implement a rule to push aggregate into table source. Currently, there is a similar rule: SupportsAggregatePushDown, which supports only pushing local aggregate into source now. Best, Godfrey Ingo Bürk 于2022年6月10日周五 17:15写道： Hi Godfrey, compared to the solution propo

Re: [DISCUSS] FLIP-240: Introduce "ANALYZE TABLE" Syntax

Hi Godfrey, compared to the solution proposed in the FLIP (using a SELECT statement), I wonder if you have considered adding APIs to catalogs / connectors to perform this task as an alternative? I could imagine that for many connectors, statistics could be implemented in a less expensive way b

Re: [DISCUSS] Deprecate SourceFunction APIs

plementing a simple Source easier before deprecating SourceFunction. Best, Jark On Sun, 5 Jun 2022 at 07:29, Jingsong Lee < lzljs3620...@apache.org wrote: +1 to David and Ingo. Before deprecate and remove SourceFunction, we should have some easier APIs to wrap new Sour

Re: [DISCUSS] Deprecate SourceFunction APIs

I +1 everything David said. The new Source API raised the complexity significantly. It's great to have such a rich, powerful API that can do everything, but in the process we lost the ability to onboard people to the APIs. Best Ingo On 04.06.22 21:21, David Anderson wrote: I'm in favor of t

Re: [DISCUSS] Flink's supported APIs and Hive query syntax

Hi, thanks Martijn for bringing this up and raising very valid concerns. I agree with the notion that Flink supporting Hive should come with a proper commitment, and otherwise we should consider not supporting it at all (in Flink itself, that is). Given that Hive is an Apache project, my fir

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

Congrats, David! On 04.03.22 12:34, Robert Metzger wrote: Hi everyone, On behalf of the PMC, I'm very happy to announce David Morávek as a new Flink committer. His first contributions to Flink date back to 2019. He has been increasingly active with reviews and driving major initiatives in the

Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser

Congrats, Martijn! On 03.03.22 16:49, Robert Metzger wrote: Hi everyone, On behalf of the PMC, I'm very happy to announce Martijn Visser as a new Flink committer. Martijn is a very active Flink community member, driving a lot of efforts on the dev@flink mailing list. He also pushes projects su

Re: [VOTE] Remove Twitter connector

+1 (binding) Best Ingo On 31.01.22 11:46, Martijn Visser wrote: Hi everyone, I would like to open up a vote to remove the Twitter connector in Flink 1.15. This was brought up previously for a discussion [1]. The vote will last for at least 72 hours, and will be accepted by a consensus of act

Re: [DISCUSS] Host StatefulFunction JavaScript SDK on npm

Hi Till, speaking as someone who regularly works with JavaScript/TypeScript, definitely a +1 from me to providing this on the npm registry to make it an easy installation for developers. Best Ingo On 24.01.22 15:37, Till Rohrmann wrote: Hi everyone, I would like to start a discussion abou

Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs

+1 (binding) Thanks for driving this much needed feature! On 14.12.21 17:45, Timo Walther wrote: Hi everyone, I'd like to start a vote on FLIP-190: Support Version Upgrades for Table API & SQL Programs [1] which has been discussed in this thread [2]. The vote will be open for at least 72 ho

Re: Some doubts about the FLINK-22848

Hi, Calcite does indeed support the (unquoted) SET statement, but the arguments are parsed as identifiers, and Flink configuration options (and values) are not valid identifiers. If we could utilize Calcite's SET implementation to correctly parse all valid configurations, it'd be of course b

Re: [VOTE] FLIP-197: API stability graduation process

+1 (binding) On 10.12.21 18:09, Till Rohrmann wrote: Hi everyone, I'd like to start a vote on FLIP-197: API stability graduation process [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://c

Re: [VOTE] FLIP-196: Source API stability guarantees

Thanks for driving this, Till. I'm very happy to see the stability guarantees being formalized more, thus +1 (binding) Ingo On 10.12.21 18:07, Till Rohrmann wrote: Hi everyone, I'd like to start a vote on FLIP-196: Source API stability guarantees [1] which has been discussed in this thread

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

Hi, I agree with Martijn on this. The lack of parity across major APIs is a frequent cause of user questions and friction, forcing users to switch between APIs etc. I would therefore also suggest expanding the scope to cover Table API + SQL as well. In general we should probably split FLIPs o

Re: [VOTE] Deprecate Java 8 support

Before more people let me know, let me update my vote to +1 (binding) (In all seriousness, thanks for the reminders!) On Mon, Dec 6, 2021, 16:54 Ingo Bürk wrote: > +1 (non-binding) > > > Ingo > > On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler > wrote: > >

Re: [VOTE] Deprecate Java 8 support

+1 (non-binding) Ingo On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler wrote: > Hello, > > after recent discussions on the dev > and > user > mailing list to dep

Re: [DISCUSS] FLIP-196: Source API stability guarantees

r uses bar(), then he opts-in to only get @Experimental > stability guarantees. > > I will add this example to the FLIP for illustrative purposes. > > Cheers, > Till > > On Fri, Dec 3, 2021 at 6:52 PM Ingo Bürk wrote: > > > > Would it be enough to say that for e

Re: [DISCUSS] FLIP-197: API stability graduation process

it be longer? > > Cheers, > Till > > On Fri, Dec 3, 2021 at 4:29 PM Ingo Bürk wrote: > > > Hi Till, > > > > Overall I whole-heartedly agree with the proposals in this FLIP. Thank > you > > for starting this discussion as well! This seems like something that

Re: [DISCUSS] FLIP-196: Source API stability guarantees

to do something, then it is not > compatible and needs to be handled differently (e.g. by offering a new > experimental interface that one can use). If we don't enforce this, then I > don't see how we can provide source stability guarantees. > > Cheers, > Till > > On Fr

Re: [DISCUSS] FLIP-196: Source API stability guarantees

al (or even internal) in an otherwise public > (evolving) API. > > If we have cases that violate the guideline, then I think we either have to > remove these methods or adjust their stability guarantees (time of > reckoning ;-). Otherwise, we won't be able to give stability guarant

Re: [DISCUSS] FLIP-197: API stability graduation process

Hi Till, Overall I whole-heartedly agree with the proposals in this FLIP. Thank you for starting this discussion as well! This seems like something that could be tested quite nicely with ArchUnit as well; I'll be happy to help should the FLIP be accepted. > I would propose per default a single re

Re: Re:Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

> > From: Yuepeng Pan > > Sent: Friday, December 3, 2021 14:14 > > To: dev@flink.apache.org > > Cc: Ingo Bürk > > Subject: Re:Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk > > > > > > > > > > Congratulat

Re: [DISCUSS] FLIP-196: Source API stability guarantees

Hi Till, Thanks for starting this discussion, and as you noticed, I qutie care for it as well. :-) When implementing the ArchUnit rules, I noticed that project-wide, different teams / modules use and understand the API annotations in different ways. Therefore, I think it is important to both get

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

Congrats, Matthias! On Thu, Dec 2, 2021 at 4:28 PM Till Rohrmann wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Matthias Pohl as a new > Flink committer. > > Matthias has worked on Flink since August last year. He helped review a ton > of PRs. He worked on a variety o

Re: [DISCUSS] FLIP-189: SQL Client Usability Improvements

I of course meant DQL, not DDL, but same thing either way. It is akin to e.g. SHOW TABLES. On Mon, Nov 29, 2021 at 9:24 AM Ingo Bürk wrote: > To me this doesn't seem too related to the FLIP – this feature would > require new DDL, and new DDL should live in Flink itself, not in the S

Re: [DISCUSS] FLIP-189: SQL Client Usability Improvements

and makes the > > SQL > > > >> >> Client an even more useful tool. > > > >> >> > > > >> >> What I miss a bit in the FLIP is the implementation details. > > > >> >> > > > >> >> For example, who is

Re: [DISCUSS] Definition of Done for Apache Flink

current flinkbot > has > > > tools to let committer to run command like "@flinkbot approve > > description". > > > However, I think many committers did not leverage this, which makes the > > bot > > > useless at most of the time. I think this disc

Re: [DISCUSS] Automated architectural tests

rent rules show already quite a few violations, some of which I will be going through in the next weeks. They range from trivial fixes to things that need to be thought about. [1] https://github.com/apache/flink/blob/master/flink-architecture-tests/README.md Ingo On Wed, Sep 1, 2021 at 11:03 AM

Re: [DISCUSS] Deprecate Java 8 support

> > We should dig deeper into the current Java version of Flink users. At > > > least make sure Java 8 is not a mainstream version. > > > > > > Receiving this signal, the user may be unhappy because his application > > > may be all on Java 8.

Re: [DISCUSS] Deprecate Java 8 support

Hi, also a +1 from me because of everything Chesnay already said. Ingo On Mon, Nov 22, 2021 at 4:41 PM Martijn Visser wrote: > Hi Chesnay, > > Thanks for bringing this up for discussion. Big +1 for dropping Java 8 and > deprecating it in 1.15, given that Java 8 support will end. We already se

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Hi Timo (& team), Thanks for driving this unquestionably difficult but important topic. This will be a major step to drive the adoption of SQL and remove a big pain point for users. Definitely a +1 on this topic. I have a couple questions on a first read: 1. Will the JSON plan's schema be conside

Re: [DISCUSS] Shall casting functions return null or throw exceptions for invalid input

Hi, first of all, thanks for the summary of both sides, and for bringing up the discussion on this. I think it is obvious that this is not something we can just "break", so the config option seems mandatory to me. Overall I agree with Martijn and Till that throwing errors is the more expected beh

Re: [DISCUSS] Definition of Done for Apache Flink

at 09:08, Francesco Guardiani < > france...@ververica.com> > > wrote: > > > > > +1 with Ingo proposal, the goal of the template should be to help > developer > > > to do a self check of his/her PR quality, not to define whether > something > > >

Re: [DISCUSS] Definition of Done for Apache Flink

Hi Joe, thank you for starting this discussion. Having a common agreement on what to expect from a PR for it to be merged is very much a worthwhile goal. I'm slightly worried about the addition to the PR template. We shouldn't make opening PRs even more difficult (unless it adds sufficient benefi

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

Congratulations, Fabian! On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink > committer. > > Fabian Paul has been actively improving the connector ecosystem by > migrating Kafka and ElasticSearch to th

Re: [DISCUSS] Conventions on assertions to use in tests

I agree on AssertJ being so much more expressive that I think it would be worth switching. However, we should be careful not to solve the issue of too many assertion frameworks by adding another one on top of everything (but also realizing that we cannot easily migrate all tests in one go). On Fri

Re: [VOTE] FLIP-189: SQL Client Usability Improvements

+1 (non-binding) I think it has all been said before. :-) On Fri, Nov 5, 2021, 17:45 Konstantin Knauf wrote: > +1 (binding) > > On Fri, Nov 5, 2021 at 4:03 PM Martijn Visser > wrote: > > > +1 (non-binding). Looking forward! > > > > Best regards, > > > > Martijn > > > > On Fri, 5 Nov 2021 at 15

Re: [DISCUSS] FLIP-189: SQL Client Usability Improvements

Hi Sergey, I think those improvements look absolutely amazing. Thanks for the little video! Best Ingo On Mon, Nov 1, 2021, 17:15 Sergey Nuyanzin wrote: > Thanks for the feedback Till. > > Martijn, I have created a short demo showing some of the features mentioned > in FLIP. > It is available

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

27;t list a 'connector' option for its tables. > > Actually, It can be divided into two steps: create and save: > - When creating a table, the table seen by HiveCatalog must have > "connector = hive", which is the hive table (Hive managed table). You > can

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

1. Is it possible to read historical data from the file store first and > > then fetch new changes from the log store? something like a hybrid > source, > > but I think we need a mechanism to get exactly-once semantic. > > 2. How the built-in table would be persisted in Catalog? &

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

Hi Jingsong, thank you for writing up the proposal. The benefits such a mechanism will bring will be very valuable! I haven't yet looked into this in detail, but one question came to my mind immediately: The DDL for these tables seems to rely on there not being a 'connector' option. However, cata

Re: [DISCUSS] Creating an external connector repository

Hi Arvid, In general I think breaking up the big repo would be a good move with many benefits (which you have outlined already). One concern would be how to proceed with our docs / examples if we were to really separate out all connectors. 1. More real-life examples would essentially now depend o

Re: The Apache Flink should pay more attention to ensuring API compatibility.

Hi, > [...] but also the new Source/Sink APIs as public I'm not really involved in the new Source/Sink APIs and will happily listen to the developers working with them here, but since they are new, we should also be careful not to mark them as stable too quickly. We've only begun updating the exi

Re: The Apache Flink should pay more attention to ensuring API compatibility.

Hi everyone, I think it would be best to support this process with tooling as much as possible, because humans are bound to make mistakes. FLINK-24138[1] should be a first step in this direction, but it wouldn't catch the cases discussed here. Maybe we should consider "capturing" the public API in

Re: Create a public open GitHub org for Flink ecosystem projects.

Hi, thank you (and the PMC) for the initiative on such a community effort. Are there already projects expected/known to move to such an organization? I think it would make sense to have at least a couple projects lined up so the org doesn't start out empty. Best Ingo On Thu, Sep 23, 2021 at 8:4

Re: [DISCUSS] Automated architectural tests

ts yet. > > Best regards, > JING ZHANG > > > Ingo Bürk 于2021年9月9日周四 下午6:49写道： > > > Great! I'll work on getting the PR into an actual, proper shape now, > > including looking at found violations more carefully and eventually > > freezing current violations

Re: [DISCUSS] Automated architectural tests

including connectors > >> & formats), having that mess of dependencies in flink-tests may > >> interfere with existing / future tests. > >> > >> As such I would prefer a separate module, as annoying as that may be. > >> > >> On 06/09/2021 15:26

Re: [DISCUSS] Automated architectural tests

noying as that may be. > > On 06/09/2021 15:26, Ingo Bürk wrote: > > I just quickly chatted with the author/maintainer of ArchUnit, and a > module > > which depends on every module that should be tested seems to be the best > > solution. How do you feel about usi

Re: [DISCUSS] Automated architectural tests

of (such as that each test should >> > extend the TestLogger as Till has already mentioned). >> > >> > I went trough your examples and ArchUnit looks really powerful and >> > expressive while still being easy to read. >> > >> > Best, >> > D. >>

Re: [DISCUSS] Automated architectural tests

rt. This could automate lot of "written rules" that are > > easy to forget about / not to be aware of (such as that each test should > > extend the TestLogger as Till has already mentioned). > > > > I went trough your examples and ArchUnit looks really powerful a

Re: [DISCUSS] Automated architectural tests

blem imo that tests need to run in order to catch stuff; > so long as it is noticed on CI. > > On 03/09/2021 08:48, Ingo Bürk wrote: > > Hi Timo, Till, > > > > thanks for your input already. I'm glad to hear that the idea resonates, > > also thanks for the add

Re: [DISCUSS] Automated architectural tests

s. The more checks the better. We > > definitely also need more guidelines (e.g. how to develop a Flink > > connector) but automation is safer then long checklists that might be > > out of date quickly. > > > > +1 to the proposal. I don't have an opinion on the tool though.

[DISCUSS] Automated architectural tests

Hello everyone, I would like to start a discussion on introducing automated tests for more architectural rather than stilistic topics. For example, here are a few things that seem worth checking to me (this is Table-API-focused since it is the subsystem I'm involved in): (a) All classes in o.a.f.

Re: Projection pushdown for metadata columns

er. Best Ingo On Mon, Aug 23, 2021 at 11:43 AM Till Rohrmann wrote: > Does this also affect Flink 1.14.0? If yes, do we want to fix this issue > for the upcoming release? If yes, then please make this issue a blocker or > at least critical. > > Cheers, > Till > > On Mon, A

Re: Projection pushdown for metadata columns

the > Spec support. > > Regards, > Timo > > > On 23.08.21 08:24, Ingo Bürk wrote: > > Hi Jingsong, > > > > thanks for your answer. Even if the source implements > > SupportsProjectionPushDown, #applyProjections will never be called with > > proje

Re: Projection pushdown for metadata columns

dMetadataNames, > newProducedType)); > } > > If there is no meta column left, we should apply again, We should tell > the source that there is no meta column left after projection. > > Best, > Jingsong > > On Fri, Aug 20, 2021 at 7:56 PM Ingo Bürk wrote: > > &g

Projection pushdown for metadata columns

Hi everyone, according to the SupportsReadableMetadata interface, the planner is supposed to project required metadata columns prior to applying them: > The planner will select required metadata columns (i.e. perform projection push down) and will call applyReadableMetadata(List, DataType) with a

Re: Configuring flink-python in IntelliJ

support configuring both Python & Java SDK for it. However, if > one works > with flink-python occasionally, using only IntelliJ IDEA is a good approach > and should be enough. > > Regards, > Dian > > On Wed, Aug 18, 2021 at 7:33 PM Ingo Bürk wrote: > > > Hi Dian, &

Re: [DISCUSS] Merge FLINK-23757 after feature freeze

> > >>> @Dian Fu could you assess how involved this > >>> change is? If the change is not very involved and the risk is limited, > >> then > >>> I'd be in favour of merging it because feature parity of APIs is quite > >>> important for our users. &

Re: Common type required when creating TableSchema

Hi Dominik, can you maybe share how you are trying to create the Schema and what you are doing with it afterwards? Best Ingo On Wed, Aug 18, 2021 at 4:08 PM Dominik Wosiński wrote: > Hey, > I've stumbled across a weird behavior and was wondering whether this is > intentional for some reason o

[DISCUSS] Merge FLINK-23757 after feature freeze

Hello dev, I was wondering whether we could also consider merging FLINK-23757[1][2] after the freeze. This is about exposing two built-in functions which we added to Table API & SQL prior to the freeze also for PyFlink. Meaning that the feature itself isn't new, we only expose it on the Python API

Re: Configuring flink-python in IntelliJ

at 5:38 PM Till Rohrmann > wrote: > > > Thanks for starting this discussion Ingo. I guess that Dian can probably > > answer this question. I am big +1 for updating our documentation for how > to > > set up the IDE since this problem will probably be encountered a couple

Configuring flink-python in IntelliJ

Hello @dev, like probably most of you, I am using IntelliJ to work on Flink. Lately I needed to also work with flink-python, and thus was wondering about how to properly set it up to work in IntelliJ. So far, I have done the following: 1. Install the Python plugin 2. Set up a custom Python SDK us

[RESULT][VOTE] FLIP-129 Register sources/sinks in Table API

Hi everyone, The vote for FLIP-129 has now been closed. We have five approving votes, four of which are binding: * Timo Walther (binding) * Jark Wu (binding) * Jingsong Li (binding) * Jing Zhang (binding) * Leonard Xu There are no disapproving votes. Thanks everyone for voting! Regards Ingo

[VOTE] FLIP-129 Register sources/sinks in Table API

Hi everyone, thanks for all the feedback so far. Based on the discussion[1] we seem to have consensus, so I would like to start a vote on FLIP-129 for which the FLIP has now also been updated[2]. The vote will last for at least 72 hours (Thu, Jun 24th 12:00 GMT) unless there is an objection or in

Re: [DISCUSS] FLIP-129 (Update): Registering sources/sinks on Table API without SQL

waiting for a long time. Let's finish it in this > > release! > > Your proposed changes look good to me. > > > > I also cc'd people who voted in previous FLIP-129. > > > > Best, > > Jark > > > > On Thu, 10 Jun 2021 at 19:46, Ingo Bürk

[DISCUSS] FLIP-129 (Update): Registering sources/sinks on Table API without SQL

Hello everyone, we would like to pick up work on FLIP-129 which aims to improve the Table API by supporting the creation of sources / sinks without having to go through SQL/DDL. This FLIP was approved a while ago, and some things have changed since then. We'd like to propose a few changes, see [1]

Re: [VOTE] FLIP-129 (Update): Registering sources/sinks on Table API without SQL

dn't start with a voting thread right away but collect more feedback > from the community in a [DISCUSS] thread before. > > Also, voting threads should be performed on the updated wiki page and > include the voting deadline. > > Regards, > Timo > > > On 10.06.21 12:02, I

[VOTE] FLIP-129 (Update): Registering sources/sinks on Table API without SQL

Hello everyone, we would like to pick up work on FLIP-129 which aims to improve the Table API by supporting the creation of sources / sinks without having to go through SQL/DDL. This FLIP was approved a while ago, and some things have changed since then. We'd like to propose a few changes, see [1]

Re: [DISCUSS]FLIP-163: SQL Client Improvements

Hi all, I have a couple questions about the SHOW CREATE TABLE statement. 1) Contrary to the example in the FLIP I think the returned DDL should always have the table identifier fully-qualified. Otherwise the DDL depends on the current context (catalog/database), which could be surprising, especia

Re: [DISCUSS]FLIP-163: SQL Client Improvements

Hi, regarding the (un-)quoted question, compatibility is of course an important argument, but in terms of consistency I'd find it a bit surprising that WITH handles it differently than SET, and I wonder if that could cause friction for developers when writing their SQL. Regards Ingo On Thu, Feb

Re: [DISCUSS] FLIP-161: Configuration through environment variables

; > > > > On Tue, Jan 26, 2021 at 6:41 PM Till Rohrmann > > wrote: > > > > > >> Thinking a bit more about the DYNAMIC_PROPERTIES, I have to admit > that I > > >> like the fact that it works around the problem of encoding the key > names > > >

Re: [DISCUSS] FLIP-161: Configuration through environment variables

uration snippets is quite useful. Given these aspects > > and that we wouldn't exclude any mentioned configuration scenarios, I > would > > also be ok following this approach given that we support it for all Flink > > processes. > > > > Cheers, > > Till >

Re: [DISCUSS] FLIP-161: Configuration through environment variables

cores in user-defined properties would be > >> affected with transformation like STATE_BACKEND -> state.backend? > >> IIUC, this transformation happens in Flink and doesn't alter ENV > >> vars, so > >> the user app can still parse the original configuratio

Re: [DISCUSS] FLIP-161: Configuration through environment variables

names? > > On 1/22/2021 3:53 PM, Till Rohrmann wrote: > > The updated design LGTM as well. Nice work Ingo! > > > > Cheers, > > Till > > > > On Fri, Jan 22, 2021 at 3:33 PM Ingo Bürk wrote: > > > >> Thanks, Ufuk. I think that makes sense, so I m

Re: [DISCUSS] FLIP-161: Configuration through environment variables

Thanks, Ufuk. I think that makes sense, so I moved it from a footnote to an addition to prevent that in the future as well. Ingo On Fri, Jan 22, 2021 at 3:10 PM Ufuk Celebi wrote: > LGTM. Let's see what the others think... > > On Thu, Jan 21, 2021, at 11:37 AM, Ingo Bürk wrote:

Re: [DISCUSS] FLIP-161: Configuration through environment variables

Hi everyone, I've updated the FLIP (https://cwiki.apache.org/confluence/x/ngtRCg) according to these discussions. Regards Ingo On Thu, Jan 21, 2021 at 11:37 AM Ingo Bürk wrote: > Hi Ufuk, Till, > > I definitely agree that having the Configuration be (or at least feel) &

Re: [DISCUSS] Flink configuration from environment variables

Hi, sorry, I almost forgot, so just to update this thread: I now started a new thread for the actual FLIP: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-161-Configuration-through-environment-variables-td48094.html Ingo On Wed, Jan 20, 2021 at 11:10 AM Ingo Bürk

Re: [DISCUSS] FLIP-161: Configuration through environment variables

should mention special cases such as > env.java.opts that are evaluated in the bash scripts and not in the Java > code. > > Cheers, > > Ufuk > > On Thu, Jan 21, 2021, at 8:57 AM, Ingo Bürk wrote: > > Hi everyone, > > > > I've now started a FLIP and

[DISCUSS] FLIP-161: Configuration through environment variables

Hi everyone, I've now started a FLIP and am opening this discussion thread. Very much looking forward to your feedback! FLIP: https://cwiki.apache.org/confluence/x/ngtRCg The first big point I'd like to discuss is about the mechanism of "when" the EVs (environment variables) are looked up. I'll

Re: [DISCUSS] Flink configuration from environment variables

does not require the user to > > > > prepare a special flink-conf.yaml where he inserts env variables for > > > every > > > > config value he wants to configure. Since this is not required with > the > > > > second approach, I think it is more gener

Re: [DISCUSS] Flink configuration from environment variables

var > naming > > (All caps, separated with underscore). Shell will complain. We have to > > bundle all env var overrides (k-v pairs) in a single property value (JSON > > and base64 encode) to avoid it. > > > > On Mon, Jan 18, 2021 at 8:15 AM Ingo Bürk wrote: > &g

Re: [DISCUSS] Flink configuration from environment variables

g to handle Java property key naming > convention (dots separated), as dots aren't allowed in shell env var naming > (All caps, separated with underscore). Shell will complain. We have to > bundle all env var overrides (k-v pairs) in a single property value (JSON > and base64 encode) to

Re: [DISCUSS] Flink configuration from environment variables

is > > preferable over parsing env vars in Flink code. > > And for cases without such a file, we could have a default one on the > > classpath with all substitutions defined (and then merge everything from > > the user-supplied file). > > > > [1] https://issue

[DISCUSS] Flink configuration from environment variables

Hi everyone, in Ververica Platform we offer a feature to use environment variables in the Flink configuration¹, e.g. ``` s3.access-key: ${S3_ACCESS_KEY} ``` We've been discussing internally whether contributing such a feature to Flink directly would make sense and wanted to start a discussion on