Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

2019-07-22 Thread Timo Walther
+1 sounds good to inform people about instabilities or other issues Regards, Timo Am 22.07.19 um 09:58 schrieb Haibo Sun: +1. Sounds good.Letting the PR creators know the build results of the master branch can help to determine more quickly whether Travis failures of their PR are an exact fa

Re: [DISCUSS] Support temporary tables in SQL API

2019-07-22 Thread Timo Walther
Thanks for summarizing our offline discussion Dawid! Even though I would prefer solution 1 instead of releasing half-baked features, I also understand that the Table API should not further block the next release. Therefore, I would be fine with solution 3 but introduce the new user-facing `crea

Re: [DISCUSS] Support computed column for Flink SQL

2019-07-29 Thread Timo Walther
Hi Danny, thanks for working on this issue and writing down the concept suggestion. We are currently still in the progress of finalizing the 1.9 release. Having proper streaming DDL support will definitely be part of Flink 1.10. I will take a look at the whole DDL efforts very soon once the 1

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

2019-08-02 Thread Timo Walther
Hi everyone, I would vote for using Optional only as method return type for non-performance critical code. Nothing more. No fields, no method parameters. Method parameters can be overloaded and internally a class can work with nulls and @Nullable. Optional is meant for API method return types

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-08 Thread Timo Walther
Hi Kurt, I posted my opinion around this particular example in FLINK-13225. Regarding the definition of "feature freeze": I think it is good to write down more of the implicit processes that we had in the past. The bylaws, coding guidelines, and a better FLIP process are very good steps towar

Re: [DISCUSS] Merging new features post-feature-freeze

2019-08-08 Thread Timo Walther
to improve, I'm all for it. Thanks, Xuefu On Thu, Aug 8, 2019 at 2:59 AM Timo Walther wrote: Hi Kurt, I posted my opinion around this particular example in FLINK-13225. Regarding the definition of "feature freeze": I think it is good to write down more of the implicit pro

Re: [VOTE] Flink Project Bylaws

2019-08-12 Thread Timo Walther
+1 Thanks for all the efforts you put into this for documenting how the project operates. Regards, Timo Am 12.08.19 um 10:44 schrieb Aljoscha Krettek: +1 On 11. Aug 2019, at 10:07, Becket Qin wrote: Hi all, I would like to start a voting thread on the project bylaws of Flink. It aims to

Re: [DISCUSS] FLIP-51: Rework of the Expression Design

2019-08-14 Thread Timo Walther
Hi Jingsong, thanks for writing down this FLIP. Big +1 from my side to finally get rid of PlannerExpressions and have consistent and well-defined behavior for Table API and SQL updated to FLIP-37. We might need to discuss some of the behavior of particular functions but this should not affec

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Timo Walther
Hi Kurt, I agree that this is a serious bug. However, I would not block the release because of this. As you said, there is a workaround and the `execute()` works in the most common case of a single execution. We can fix this in a minor release shortly after. What do others think? Regards, T

Re: [DISCUSS] FLIP-51: Rework of the Expression Design

2019-08-15 Thread Timo Walther
sts should pass still. Regards, Timo Am 15.08.19 um 10:43 schrieb JingsongLee: Hi @Timo Walther @Dawid Wysakowicz: Now, flink-planner have some legacy DataTypes: like: legacy decimal, legacy basic array type info... And If the new type inference infer a Decimal/VarChar with precision, there s

Re: [VOTE] FLIP-51: Rework of the Expression Design

2019-08-15 Thread Timo Walther
+1 for this. Thanks, Timo Am 15.08.19 um 15:57 schrieb JingsongLee: Hi Flink devs, I would like to start the voting for FLIP-51 Rework of the Expression Design. FLIP wiki: https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design Discussion thread: http:/

[DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-16 Thread Timo Walther
Hi everyone, Dawid and I are working on making parts of ExecutionConfig and TableConfig configurable via config options. This is necessary to make all properties also available in SQL. Additionally, with the new SQL DDL based on properties as well as more connectors and formats coming up, uni

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-19 Thread Timo Walther
ooks interesting. Thanks for preparing this FLIP! Client API enhancement benefit from this evolution which hopefully provides a better view of configuration of Flink. In client API enhancement, we likely make the deployment of cluster and submission of job totally defined by configuration. Will take a

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Timo Walther
ote: Hi Kurt, With the same argument as before, given that it is mentioned in the release announcement that it is a preview feature, I would not block this release because of it. Nevertheless, it would be important to mention this explicitly in the release notes [1]. Regards, Gord

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

2019-08-21 Thread Timo Walther
Thanks for summarizing the discussion Andrey, +1 to this style. Regards, Timo Am 21.08.19 um 11:57 schrieb Andrey Zagrebin: Hi All, It looks like we have reached a consensus regarding the last left question. I suggest the following final summary: - Use @Nullable annotation where you do

Re: [VOTE] FLIP-52: Remove legacy Program interface.

2019-08-21 Thread Timo Walther
+1 Am 21.08.19 um 13:21 schrieb Stephan Ewen: +1 On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas wrote: Hi all, Following the FLIP process, this is a voting thread dedicated to the FLIP-52. As shown from the corresponding discussion thread [1], we seem to agree that the Program interface can

[DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-08-21 Thread Timo Walther
Hi everyone, some of you might remember the discussion I started end of March [1] about introducing a new Java DSL for Table API that is not embedded in a string. In particular, it solves the following issues: - No possibility of deprecating functions - Missing documentation for users - Mi

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-22 Thread Timo Walther
t prepares for maybe validating the entire global configuration before submitting a job in the future. Please take another look if you find time. I hope we can proceed with the voting process if there are no objections. Regards, Timo Am 19.08.19 um 12:54 schrieb Timo Walther: Hi Stephan,

Re: [ANNOUNCE] Apache Flink 1.9.0 released

2019-08-22 Thread Timo Walther
Thanks to everyone who contributed to this release. Great team work! Regards, Timo Am 22.08.19 um 14:16 schrieb JingsongLee: Congratulations~~~ Thanks gordon and everyone~ Best, Jingsong Lee -- From:Oytun Tez Send Time:2019年8月2

[VOTE] FLIP-54: Evolve ConfigOption and Configuration

2019-08-27 Thread Timo Walther
Hi everyone, thanks for the great feedback we have received for the draft of FLIP-54. The discussion seems to have reached an agreement. Of course this doesn't mean that we can't propose further improvements on ConfigOption's and Flink configuration in general in the future. It is just one st

Re: [CODE-STYLE] Builder pattern

2019-08-27 Thread Timo Walther
Hi all, great to put this code style discussion on the mailing list because I also have found this style inconsistent in the past. Regarding Gyula's suggestions: 1. a static method `builder()` I think IDEs are also hightlight methods with this name 2. I would vote for a more declarative `prop

Re: [VOTE] FLIP-54: Evolve ConfigOption and Configuration

2019-08-27 Thread Timo Walther
" objects immutable. Best, Dawid On 27/08/2019 13:28, Timo Walther wrote: Hi everyone, thanks for the great feedback we have received for the draft of FLIP-54. The discussion seems to have reached an agreement. Of course this doesn't mean that we can't propose further improve

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-27 Thread Timo Walther
Hi everyone, I updated the FLIP proposal one more time as mentioned in the voting thread. If there are no objections, I will start a new voting thread tomorrow at 9am Berlin time. Thanks, Timo On 22.08.19 14:19, Timo Walther wrote: Hi everyone, thanks for all the feedback we have

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-08-27 Thread Timo Walther
oachable than the current Java DSL. In a training context it will be easy to teach, but I wonder if we can find a way to make it look less alien at first glance. David On Wed, Aug 21, 2019 at 1:33 PM Timo Walther wrote: Hi everyone, some of you might remember the discussion I started end of March

[VOTE] FLIP-54: Evolve ConfigOption and Configuration

2019-08-28 Thread Timo Walther
Hi everyone, after some last minute changes yesterday, I would like to start a new vote on FLIP-54. The discussion seems to have reached an agreement. Of course this doesn't mean that we can't propose further improvements on ConfigOption's and Flink configuration in general in the future. It i

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-08-28 Thread Timo Walther
;) could be interpreted to mean the value of the "foo" column, or a literal string. David On Tue, Aug 27, 2019 at 5:45 PM Timo Walther wrote: Hi David, thanks for your feedback. With the current design, the DSL would be free of any ambiguity but it is definitely more verbose esp. ar

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-29 Thread Timo Walther
On Tue, Aug 27, 2019 at 11:34 PM Timo Walther wrote: Hi everyone, I updated the FLIP proposal one more time as mentioned in the voting thread. If there are no objections, I will start a new voting thread tomorrow at 9am Berlin time. Thanks, Timo On 22.08.19 14:19, Timo Walther wrote: Hi

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-08-29 Thread Timo Walther
e: I would prefer ‘lit()’ over ‘val()’ since val is a keyword in Scala. Assuming the intention is to make the dsl ergonomic for Scala developers. Seth On Aug 28, 2019, at 7:58 AM, Timo Walther wrote: Hi David, thanks for your feedback. I was also skeptical about 1 char method names, I res

Re: [VOTE] FLIP-54: Evolve ConfigOption and Configuration

2019-08-29 Thread Timo Walther
! Best, tison. Jark Wu 于2019年8月28日周三 下午6:08写道: Hi Timo, The new changes looks good to me. +1 to the FLIP. Cheers, Jark On Wed, 28 Aug 2019 at 16:02, Timo Walther wrote: Hi everyone, after some last minute changes yesterday, I would like to start a new vote on FLIP-54. The discussion seems

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-30 Thread Timo Walther
ALLELISM and the description should clearly state all the usage of this ConfigOption. 3. I see, in that case, how about we name it something like extractConfiguration()? I am just trying to see if we can make it clear this is not something like fromBytes() and toBytes(). Thanks, Jiangjie (Becket)

[DISCUSS] FLIP-60: Restructure the Table API & SQL documentation

2019-08-30 Thread Timo Walther
Hi everyone, the Table API & SQL documentation was already in a very good shape in Flink 1.8. However, in the past it was mostly presented as an addition to DataStream API. As the Table and SQL world is growing quickly, stabilizes in its concepts, and is considered as another top-level API an

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-08-30 Thread Timo Walther
h similar to toBytes/fromBytes, but it puts itself to a Configuration. Also just wanted to make sure we adjusted this part slightly and now the ConfigOption takes ConfigurableFactory. Best, Dawid On 30/08/2019 09:39, Timo Walther wrote: Hi Becket, thanks for the discussion. 1. ConfigOptio

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-02 Thread Timo Walther
Timo said it is in the end sth similar to toBytes/fromBytes, but it puts itself to a Configuration. Also just wanted to make sure we adjusted this part slightly and now the ConfigOption takes ConfigurableFactory. Best, Dawid On 30/08/2019 09:39, Timo Walther wrote: Hi Becket, thanks for the discussi

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-09-02 Thread Timo Walther
Hi all, the FLIP looks awesome. However, I would like to discuss the changes to the user-facing parts again. Some feedback: 1. DataViews: With the current non-annotation design for DataViews, we cannot perform eager state declaration, right? At which point during execution do we know which s

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-09-02 Thread Timo Walther
Am Do., 29. Aug. 2019 um 12:15 Uhr schrieb Timo Walther : I'm fine with `lit()`. Regarding `col()`, I initially suggested `ref()` but I think Fabian and Dawid liked single char methods for the most commonly used expressions. Btw, what is your opinion on the names of commonly used methods such

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-02 Thread Timo Walther
n this is required for shipping the Configuration to TaskManager, so that we do not have to use java serializability. Best, Dawid On 02/09/2019 10:05, Timo Walther wrote: Hi Becket, Re 1 & 3: "values in configurations should actually be immutable" I would also prefer immutabilit

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Timo Walther
ses. 2. If writeToConfiguration is only for internal use cases, maybe we can avoid adding it to the configurable interface. We can add another interface such as ExtractableConfigurable for internal usage. What do you think? Thanks, Jiangjie (Becket) Qin On Mon, Sep 2, 2019 at 11:59 PM Timo Walther

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
Hi Bowen, thanks for your proposal. Here are some thoughts: 1) We should not have the restriction "hive built-in functions can only be used when current catalog is hive catalog". Switching a catalog should only have implications on the cat.db.object resolution but not functions. It would be q

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-09-03 Thread Timo Walther
our example in FLIP-58. You could you extend the example to show how to specify these attributes in the FLIP? Regards, Timo [1] https://flink.apache.org/contributing/code-style-and-quality-java.html On 02.09.19 15:35, jincheng sun wrote: Hi Timo, Great thanks for your feedback. I would li

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
This sounds exactly as the module approach I mentioned, no? Regards, Timo On 03.09.19 13:42, Danny Chan wrote: Thanks Bowen for bring up this topic, I think it’s a useful refactoring to make our function usage more user friendly. For the topic of how to organize the builtin operators and oper

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

2019-09-03 Thread Timo Walther
Hi Danny, yes, this FLIP covers all the building blocks we need also for unification of the DDL properties. Regards, Timo On 03.09.19 13:45, Danny Chan wrote: with the new SQL DDL based on properties as well as more connectors and formats coming up, unified configuration becomes more impor

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-03 Thread Timo Walther
are basically the same as what Calcite does, I think we are in the same line. Best, Danny Chan 在 2019年9月3日 +0800 PM7:57,Timo Walther ,写道: This sounds exactly as the module approach I mentioned, no? Regards, Timo On 03.09.19 13:42, Danny Chan wrote: Thanks Bowen for bring up this topic, I think

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-09-04 Thread Timo Walther
rs will not set this method and for Python functions, it will be set in the code-generated Java function by the framework. So, I think we should declare the getLanguage() in FunctionDefinition for now. (I'm not pretty sure what do you mean by saying that getKind() is final in UserDefinedFunction?) B

Re: FLIP-63: Rework table partition support

2019-09-04 Thread Timo Walther
Hi Jingsong, thanks for your proposal. Could you repost this email with the subject: "[DISCUSS] FLIP-63: Rework table partition support" Some people have filters for [DISCUSS] threads and it also makes important emails more prominent visually. Thanks, Timo On 04.09.19 09:11, JingsongLee wro

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-04 Thread Timo Walther
doc. I've modified those sections, and they are up to date now. In short, now built-in function of external systems are defined as a special kind of catalog function in Flink, and handled by Flink as following: - An external built-in function must be associated with a catalog for t

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

2019-09-04 Thread Timo Walther
O, It is always easier to expand the APIs later than reducing them. Cheers, Rong On Mon, Sep 2, 2019 at 2:37 AM Timo Walther wrote: Hi all, I see a majority votes for `lit(12)` so let's adopt that in the FLIP. The `$("field")` would consider Fabian's concerns so I would vote

Re: FLIP-24 / SQL GW

2019-09-16 Thread Timo Walther
Hi Hanan, the community is currently reworking parts of the architecture of Flink SQL first for making it a good foundation for further tools around it (see also FLIP-32 and following SQL-related FLIPs). In Flink 1.10 the SQL Client will not receive major updates but it seems likely that Flink

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-16 Thread Timo Walther
Hi Bowen, I understand the potential benefit of overriding certain built-in functions. I'm open to such a feature if many people agree. However, it would be great to still support overriding catalog functions with temporary functions in order to prototype a query even though a catalog/databas

Re: [DISCUSS] FLIP-60: Restructure the Table API & SQL documentation

2019-09-16 Thread Timo Walther
estion to consider is how about moving the User-Defined-Extensions subpages to corresponding broader topics? Sources & Sinks >> Connect to external systems Catalogs >> Connect to external systems and then have a Functions sections with subsections: functions |- built in fu

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

2019-09-16 Thread Timo Walther
Hi Dawid, thanks for the design document. It fixes big concept gaps due to historical reasons with proper support for serializability and catalog support in mind. I would not mind a registerTemporarySource/Sink, but the problem that I see is that many people think that this is the recommende

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

2019-09-17 Thread Timo Walther
nc ... Overrides built-in function with temporary function The built-in/system namespace would not be writable for permanent objects. WDYT? This way I think we can have benefits of both solutions. Best, Dawid On Tue, 17 Sep 2019, 07:24 Timo Walther, wrote: Hi Bowen, I understand the poten

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-25 Thread Timo Walther
Hi Feng, thanks for proposing this FLIP. It makes a lot of sense to finally support querying tables at a specific point in time or hopefully also ranges soon. Following time-versioned tables. Here is some feedback from my side: 1. Syntax Can you elaborate a bit on the Calcite restrictions?

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-25 Thread Timo Walther
les. Regards, Timo On 25.05.23 12:29, Timo Walther wrote: Hi Feng, thanks for proposing this FLIP. It makes a lot of sense to finally support querying tables at a specific point in time or hopefully also ranges soon. Following time-versioned tables. Here is some feedback from my side: 1. Syntax

Re: [VOTE] FLIP-346: Deprecate ManagedTable related APIs

2023-07-24 Thread Timo Walther
+1 Regards, Timo On 24.07.23 04:00, liu ron wrote: +1 Best, Ron Lincoln Lee 于2023年7月21日周五 16:09写道: +1 Best, Lincoln Lee Leonard Xu 于2023年7月21日周五 16:07写道: +1 Best, Leonard On Jul 21, 2023, at 4:02 PM, yuxia wrote: +1(binging) Best regards, Yuxia - 原始邮件 - 发件人: "Jane C

Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

2023-07-24 Thread Timo Walther
+1 But instead we should add a OpenContext there to keep the signature stable but still be able to add parameters. Regards, Timo On 21.07.23 12:24, Jing Ge wrote: +1 On Fri, Jul 21, 2023 at 10:22 AM Yuxin Tan wrote: +1 Best, Yuxin Xintong Song 于2023年7月21日周五 12:04写道: +1 Best, Xint

[DISCUSS] FLIP-348: Support System Columns in SQL and Table API

2023-07-25 Thread Timo Walther
Hi everyone, I would like to start a discussion about introducing the concept of "System Columns" in SQL and Table API. The subject sounds bigger than it actually is. Luckily, Flink SQL already exposes the concept of metadata columns. And this proposal is just a slight adjustment for how met

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

2023-08-07 Thread Timo Walther
ed any prefix with their pseudo column? 2. Some platform providers will use ${variable_name} to define their own configurations and allow them to be embedded into SQL scripts. Will there be any conflict with option 3? Best regards, Jing On Tue, Jul 25, 2023 at 7:00 PM Konstantin Knauf wro

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

2023-08-15 Thread Timo Walther
concepts. Looking forward to any kind of feedback. Thanks, Timo On 07.08.23 12:07, Timo Walther wrote: Hi everyone, thanks for the various feedback and lively discussion. Sorry, for the late reply as I was on vacation. Let me answer to some of the topics: 1) Systems columns should not be shown

Re: Plans for Schema Evolution in Table API

2023-08-15 Thread Timo Walther
Hi Ashish, sorry for the late reply. There are currently no concrete plans to support schema evolution in Table API. Until recently, Flink version evolution was the biggest topic. In the near future we can rediscuss query and state evolution in more detail. Personally, I think we will need e

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

2023-08-17 Thread Timo Walther
+1 for this proposal. Not every data team would like to enable hints. Also because they are an extension to the SQL standard. It might also be the case that custom rules would be overwritten otherwise. Setting hints could also be the exclusive task of a DevOp team. Regards, Timo On 17.08.2

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

2023-08-18 Thread Timo Walther
t not introducing any concept of system/pseudo columns for now. Best, Jark On Tue, 15 Aug 2023 at 23:25, Timo Walther wrote: Hi everyone, I would like to bump this thread up again. Esp. I would like to hear opinions on my latest suggestions to simply use METADATA VIRTUAL as system columns and

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

2023-08-18 Thread Timo Walther
27;m afraid we have to introduce a bunch of configuration, because lots of the streaming SQL syntax are extensions of SQL standard. Best, Jark On Thu, 17 Aug 2023 at 15:43, Timo Walther wrote: +1 for this proposal. Not every data team would like to enable hints. Also because they are an extensio

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

2023-08-29 Thread Timo Walther
2023 at 4:21 AM Benchao Li wrote: It sounds good to me too, that we avoid introducing the concept of "system columns" for now. Timo Walther 于2023年8月18日周五 22:38写道: Great, I also like my last suggestion as it is even more elegant. I will update the FLIP until Monday. Regards,

[VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

2023-08-31 Thread Timo Walther
Hi everyone, I'd like to start a vote on FLIP-348: Make expanding behavior of virtual metadata columns configurable [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/co

[RESULT][VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

2023-09-05 Thread Timo Walther
: +1 (binding) Best, Jark 2023年8月31日 18:54,Jing Ge 写道: +1(binding) On Thu, Aug 31, 2023 at 11:22 AM Sergey Nuyanzin wrote: +1 (binding) On Thu, Aug 31, 2023 at 9:28 AM Benchao Li wrote: +1 (binding) Martijn Visser 于2023年8月31日周四 15:24写道: +1 (binding) On Thu, Aug 31, 2023 at 9:09 AM

[DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-26 Thread Timo Walther
Hi everyone, I would like to start a discussion on FLIP-376: Add DISTRIBUTED BY clause [1]. Many SQL vendors expose the concepts of Partitioning, Bucketing, and Clustering. This FLIP continues the work of previous FLIPs and would like to introduce the concept of "Bucketing" to Flink. This

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-30 Thread Timo Walther
nterface. Best, Jingsong On Thu, Oct 26, 2023 at 5:00 PM Timo Walther wrote: Hi everyone, I would like to start a discussion on FLIP-376: Add DISTRIBUTED BY clause [1]. Many SQL vendors expose the concepts of Partitioning, Bucketing, and Clustering. This FLIP continues the work of previous FLIPs

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-30 Thread Timo Walther
iscussion. Big +1 for this. The design looks good to me! We can add some documentation for connector developers. For example: for sink, If there needs some keyby, please finish the keyby by the connector itself. SupportsBucketing is just a marker interface. Best, Jingsong On Thu, Oct 26, 2

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-30 Thread Timo Walther
r `spark-filesystem` connector. Regards, Timo On 30.10.23 10:44, Timo Walther wrote: Hi Jark, my intention was to avoid too complex syntax in the first version. In the past years, we could enable use cases also without this clause, so we should be careful with overloading it with too function

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-30 Thread Timo Walther
(uid) INTO 6 BUCKETS - For advanced users, the algorithm can be defined explicitly. - Currently, either HASH() or RANGE(). " Did you mean users can use their own algorithm? How to do it? Best regards, Jing On Mon, Oct 30, 2023 at 11:13 AM Timo Walther wrote: Let me reply to the feedback

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-10-31 Thread Timo Walther
have one more question. What does Flink check and throw exceptions for the bucketing? For example, do we check interfaces when executing create/alter DDL and when used as a source? Best, Jark On Tue, 31 Oct 2023 at 00:25, Timo Walther wrote: Hi Jing, > Have you considered using BUCKET

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-11-02 Thread Timo Walther
INTO 6". Not really used in SQL, but afaiu Spark uses the concept[1]. [1] https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrameWriter.bucketBy.html Best regards, Jing On Mon, Oct 30, 2023 at 5:25 PM Timo Walther wrote: Hi Jing, > Have you considered using B

Re: [DISCUSS] FLIP-376: Add DISTRIBUTED BY clause

2023-11-03 Thread Timo Walther
Best regards, Martijn On Thu, Nov 2, 2023 at 11:00 AM Timo Walther wrote: Hi Jing, I agree this is confusing. THe Spark API calls it bucketBy in the programmatic API. But anyway, we should discuss the SQL semantics here. It's like a "WHERE" is called "filter" in the

[VOTE] FLIP-376: Add DISTRIBUTED BY clause

2023-11-06 Thread Timo Walther
Hi everyone, I'd like to start a vote on FLIP-376: Add DISTRIBUTED BY clause[1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+D

Re: FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

2023-11-07 Thread Timo Walther
Thanks for the FLIP, Piotr. In order to follow the FLIP process[1], please prefix the email subject with "[DISCUSS]". Also, some people might have added filters to their email clients to highlight those discussions. Thanks, Timo [1] https://cwiki.apache.org/confluence/display/Flink/Flink+

[RESULT][VOTE] FLIP-376: Add DISTRIBUTED BY clause

2023-11-09 Thread Timo Walther
hread/cft2d9jc2lr8gv6dyyz7b62188mf07sj Cheers, Timo On 08.11.23 10:14, Leonard Xu wrote: +1(binding) Best, Leonard 2023年11月8日 下午1:05,Sergey Nuyanzin 写道: +1 (binding) On Wed, Nov 8, 2023 at 6:02 AM Zhanghao Chen wrote: +1 (non-binding) Best, Zhanghao Chen From:

Re: [VOTE] FLIP-393: Make QueryOperations SQL serializable

2023-11-21 Thread Timo Walther
+1 (binding) Thanks for working on this FLIP. It will be a nice continuation of the previous work. Cheers, Timo On 21.11.23 13:19, Gyula Fóra wrote: +1 (binding) Gyula On Tue, 21 Nov 2023 at 13:11, xiangyu feng wrote: +1 (non-binding) Thanks for driving this. Best, Xiangyu Feng Fer

Re: SQL return type change from 1.17 to 1.18

2023-12-07 Thread Timo Walther
Hi Peter, thanks for reaching out to the Flink community. This is indeed a serious issue. As the author of the Flink type system, DataType and many related utilities I strongly vote for reverting FLINK-33523: - It changes the Flink type system without a FLIP. - It breaks backwards compatibili

Re: [DISCUSS] Release Flink 1.18.1

2023-12-08 Thread Timo Walther
Thanks for taking care of this Jing. +1 to release 1.18.1 for this. Cheers, Timo On 08.12.23 10:00, Benchao Li wrote: I've merged FLINK-33313 to release-1.18 branch. Péter Váry 于2023年12月8日周五 16:56写道: Hi Jing, Thanks for taking care of this! +1 (non-binding) Peter Sergey Nuyanzin ezt írt

Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-12 Thread Timo Walther
Hi Xuyang, thanks for proposing this FLIP. In my opinion the FLIP touches too many topics at the same time and should be split into multiple FLIPs. We should stay focused on what is needed for Flink 2.0. Let me summarizing the topics: 1) SESSION Window TVF Aggregation This has been agreed i

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-14 Thread Timo Walther
Hi Alan, thanks for proposing this FLIP. It's a great addition to Flink and has been requested multiple times. It will be in particular interesting for accessing REST endpoints and other remote services. Great that we can generalize and reuse parts of the Python planner rules and code for th

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-14 Thread Timo Walther
The configuration in Flink is complicated and I fear we won't have enough capacity to substantially fix it. The introduction of ReadableConfig, WritableConfig, and typed ConfigOptions was a right step into making the code more maintainable. From the Flink side, every read access should go throu

Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-14 Thread Timo Walther
that. Anyway I'm happy to help here with the review as well and will reserve some time for it in coming days. On Tue, Dec 12, 2023 at 12:18 PM Timo Walther wrote: Hi Xuyang, thanks for proposing this FLIP. In my opinion the FLIP touches too many topics at the same time and should be spl

Re: [DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-14 Thread Timo Walther
Hi Feng, thank you for proposing this FLIP. This nicely completes FLIP-65 which is great for usability. I read the FLIP and have some feedback: 1) ArgumentNames annotation > Deprecate the ArgumentNames annotation as it is not user-friendly for specifying argument names with optional config

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-15 Thread Timo Walther
ot; 写道: Hi Alan, Nice FLIP, I also explore leveraging the async table function[1] to improve the throughput before. About the configs, what do you think using hints as mentioned in [1]. [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTabl

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-18 Thread Timo Walther
the more hints we introduce, the harder SQL queries get when maintaining them. @Timo: That seems like a reasonable approach to me. -Alan [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/metric_reporters/ On Fri, Dec 15, 2023 at 2:56 AM Timo Walther wrote: 1. Ove

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-19 Thread Timo Walther
to start here and we can consider UNORDERED as a followup. Then maybe we consider only INSERT mode graphs and ones where we can solve the watermark constraints. Thanks, Alan On Mon, Dec 18, 2023 at 2:36 AM Timo Walther wrote: Hi Xuyang and Alan, thanks for this productive discussion. >

Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-19 Thread Timo Walther
ed in 1.19, we can consider officially deprecating the old group window agg syntax in the release note. WDYT? -- Best! Xuyang At 2023-12-14 17:51:01, "Timo Walther" wrote: Hi Xuyang, I'm not spliting this flip is that all of these subtasks like session wind

Re: [VOTE] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-28 Thread Timo Walther
+1 (binding) Cheers, Timo > Am 28.12.2023 um 03:13 schrieb Yuepeng Pan : > > +1 (non-binding). > > Best, > Yuepeng Pan. > > > > > At 2023-12-28 09:19:37, "Lincoln Lee" wrote: >> +1 (binding) >> >> Best, >> Lincoln Lee >> >> >> Martijn Visser 于2023年12月27日周三 23:16写道: >> >>> +1 (binding

Re: [DISCUSS][FLINK-31830] Align the Nullability Handling of ROW between SQL and TableAPI

2024-01-02 Thread Timo Walther
Hi Jane, thanks for the heavy investigation and extensive summaries. I'm sorry that I ignored this discussion for too long but would like to help in shaping a sustainable long-term solution. I fear that changing: - RowType#copy() - RowType's constructor - FieldsDataType#nullable() will not so

Re: [VOTE] FLIP-387: Support named parameters for functions and call procedures

2024-01-04 Thread Timo Walther
Thanks, for starting the VOTE thread and thanks for considering my feedback. One last comment before I'm also happy to give my +1 to this: Why is ArgumentHint's default isOptinal=true? Shouldn't it be false by default? Many function implementers will forget to set this to false and suddenly ge

Re: [VOTE] FLIP-387: Support named parameters for functions and call procedures

2024-01-05 Thread Timo Walther
default and developers should explicitly state if a parameter is optional instead of using our default value. Regarding this part, I have already made modifications in the document. Best, Feng On Fri, Jan 5, 2024 at 3:38 PM Timo Walther wrote: Thanks, for starting the VOTE thread and thanks for

Re: [VOTE] Improve TableFactory to add Context

2020-02-06 Thread Timo Walther
+1 On 06.02.20 05:54, Bowen Li wrote: +1, LGTM On Tue, Feb 4, 2020 at 11:28 PM Jark Wu wrote: +1 form my side. Thanks for driving this. Btw, could you also attach a JIRA issue with the changes described in it, so that users can find the issue through the mailing list in the future. Best, J

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

2020-02-07 Thread Timo Walther
Hi Kurt, I agree with Aljoscha. We don't need to introduce a big process or do voting but we should ensure that all stakeholders are notified and have a chance to raise doubts. Regards, Timo On 07.02.20 09:51, Aljoscha Krettek wrote: I would say a ML discussion or even a Jira issue is enou

Re: [DISCUSS] Update UpsertStreamTableSink and RetractStreamTableSink to new type system

2020-02-07 Thread Timo Walther
Hi Zhenghua, Jark is right. The reason why we haven't updated those interfaces yet is because we are actually would like to introduce new interfaces. We should target new interfaces in this release. Even a short-term fix as you proposed with `getRecordDataType` does actually not help as Jingso

Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-07 Thread Timo Walther
Hi Kurt, Dawid is currently working on making a tableEnv.fromElements/values() kind of source possible in the future. We can use this to replace some of the tests. Otherwise I guess we should come up with a better test infrastructure to make defining source not necessary anymore. Regards, Ti

Re: [VOTE] FLIP-55: Introduction of a Table API Java Expression DSL

2020-02-10 Thread Timo Walther
+1 for this. It will also help in making a TableEnvironment.fromElements() possible and reduces technical debt. One entry point of TypeInformation less in the API. Regards, Timo On 10.02.20 08:31, Dawid Wysakowicz wrote: Hi all, I wanted to resurrect the thread about introducing a Java Ex

Re: [ANNOUNCE] Apache Flink 1.10.0 released

2020-02-12 Thread Timo Walther
Congratualations everyone! Great stuff :-) Regards, Timo On 12.02.20 16:05, Leonard Xu wrote: Great news! Thanks everyone involved ! Thanks Gary and Yu for being the release manager ! Best, Leonard Xu 在 2020年2月12日,23:02,Stephan Ewen 写道: Congrats to us all. A big piece of work, nicely don

Re: [DISCUSS][TABLE] Issue with package structure in the Table API

2020-02-13 Thread Timo Walther
Hi everyone, thanks for bringing our offline discussion to the mailing list, Dawid. This is a very bad mistake that has been made in the past. In general, we should discourage putting the terms "java" and "scala" in package names as this has side effects on Scala imports. I really don't like

Re: [DISCUSS] Remove registration of TableSource/TableSink in Table Env and ConnectTableDescriptor

2020-02-17 Thread Timo Walther
On Fri, Feb 7, 2020 at 10:56 PM Timo Walther wrote: Hi Kurt, Dawid is currently working on making a tableEnv.fromElements/values() kind of source possible in the future. We can use this to replace some of the tests. Otherwise I guess we should come up with a better test infrastructure to make

  1   2   3   4   5   6   7   8   9   10   >