Re: [VOTE] Release 2.0 must-have work items

2023-07-13 Thread Xintong Song
Thanks for the support, Yu. We will have the guideline before removing DataSet. We are currently prioritizing works that need to be done before the 1.18 feature freeze, and will soon get back to working on the guidelines. We expect to get the guideline ready before or soon after the 1.18 release,

Re: [VOTE] Release 2.0 must-have work items

2023-07-13 Thread Yu Li
It's great to see the discussion about what we need to improve on (completely) switching from DataSet API to DataStream API from the user perspective. I feel that these improvements would happen faster (only) when we seriously prepare to remove the DataSet APIs with a target release, just like what

Re: Flink 2.0 connectors, formats, and even user code should also be pluggable.

2023-07-13 Thread Leonard Xu
Hello Zhiqiang, Although it is strange to receive such an email on the dev mailing list, I read your email carefully. The Pluggable mechanism for Connectors, Formats is indeed necessary, and I believe that developers in the community also have this consensus. Regarding the relationship between

[jira] [Created] (FLINK-32590) Fail to read flink parquet filesystem table stored in hive metastore service.

2023-07-13 Thread Guozhen Yang (Jira)
Guozhen Yang created FLINK-32590: Summary: Fail to read flink parquet filesystem table stored in hive metastore service. Key: FLINK-32590 URL: https://issues.apache.org/jira/browse/FLINK-32590 Project

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

2023-07-13 Thread Xintong Song
Hi Matthias, I'm not aware of any decision that has already been made by the community regarding after which 1.x minor release will we ship the 2.0 major release. I personally don't think it's feasible to avoid 1.19, and even 1.20 depending on the progress. I also don't think we should push all t

[jira] [Created] (FLINK-32589) Carry over parallelism overrides to prevent users from clearing them on updates

2023-07-13 Thread Maximilian Michels (Jira)
Maximilian Michels created FLINK-32589: -- Summary: Carry over parallelism overrides to prevent users from clearing them on updates Key: FLINK-32589 URL: https://issues.apache.org/jira/browse/FLINK-32589

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

2023-07-13 Thread Jing Ge
Hi, Thanks Matthias for starting this discussion and thanks Chesnay for the clarification. I don't want to hijack this discussion but I would suggest postponing the 1.18 feature freeze over postponing the deprecations to 1.19. If there are some contributors who think it makes sense, I will raise

[DISCUSS][2.0] FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint

2023-07-13 Thread Chesnay Schepler
Hello, The job cancellation REST endpoint has a terminationMode query parameter, which in the past could be set to either CANCEL or STOP, but nowadays the job stop endpoint has subsumed the STOP functionality. Since then the cancel endpoint rejected requests that specified STOP. I propose to

[DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

2023-07-13 Thread Chesnay Schepler
Hello, The request body for the jar run/plan REST endpoints accepts program arguments as a string (programArgs) or a list of strings (programArgsList). The latter was introduced as kept running into issues with splitting the string into individual arguments./ / We ideally force users to use

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

2023-07-13 Thread Chesnay Schepler
On 13/07/2023 08:07, Mohan, Deepthi wrote: However, even these developers when explicitly asked in our conversations often comment that the website could do with a redesign Can you go into more detail as to their specific concerns? Are there functional problems with the page, or is this just

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

2023-07-13 Thread Jing Ge
yeah, saw that thread. Thanks for starting the discussion! Best regards, Jing On Thu, Jul 13, 2023 at 2:15 PM Matthias Pohl wrote: > Thanks Jing. About your question regarding FLIPs: The FLIP is necessary > when these classes touch user-facing API (which should be the case if we're > preparing

[DISCUSS][2.0] FLIP-336: Remove "now" timestamp field from REST responses

2023-07-13 Thread Chesnay Schepler
Hello, Several REST responses contain a timestamp field of the current time  There is no known use-case for said timestamp, it makes caching of responses technically sketchy (since the response differs on each call) and  it complicates testing since the timestamp field can't be easily predict

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

2023-07-13 Thread Chesnay Schepler
The issue isn't avoiding 1.19. The issue is that if things aren't deprecated in 1.18 then for every breaking change we have to start discussing exemptions from the API deprecation process, which stipulates that all Public APIs must be deprecated for at least 2 minor releases before they can be

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

2023-07-13 Thread Matthias Pohl
Thanks Jing. About your question regarding FLIPs: The FLIP is necessary when these classes touch user-facing API (which should be the case if we're preparing it for 2.0). I created a dedicated discussion thread [1] to discuss this topic because it's more general and independent of FLIP-335. Best,

[DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

2023-07-13 Thread Matthias Pohl
Jing brought up a question in the FLIP-335 discussion thread [1] which I want to move into a dedicated discussion thread as it's a bit more general: How do we handle the deprecation process of Public APIs for Flink 2.0? > > I just have a related question: Do we need to create a FLIP each time > whe

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

2023-07-13 Thread Ahmed Hamdy
+1 the design looks great. Best Regards Ahmed Hamdy On Thu, 13 Jul 2023 at 07:08, Mohan, Deepthi wrote: > Thank you all for the feedback so far. > > @Xintong, as you may know, AWS provides a managed Apache Flink service > called Kinesis Data Analytics. We regularly speak to our customers (these

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

2023-07-13 Thread Jing Ge
Hi Matthias, Thanks for raising this up! I saw your conversation with Chesnay in the PR and after reading the historical discussions, there should be no doubt to deprecate Time classes. +1 I just have a related question: Do we need to create a FLIP each time when we want to deprecate any classes?

[DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

2023-07-13 Thread Matthias Pohl
The 2.0 feature list includes the removal of Flink's Time classes in favor of the JDKs java.time.Duration class. There was already a discussion about it in [1] and FLINK-14068 [2] was created as a consequence of this discussion. I started working on marking the APIs as deprecated in FLINK-32570 [3

[jira] [Created] (FLINK-32588) Flink ML unittest BoundedPerRoundStreamIterationITCase failed

2023-07-13 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-32588: - Summary: Flink ML unittest BoundedPerRoundStreamIterationITCase failed Key: FLINK-32588 URL: https://issues.apache.org/jira/browse/FLINK-32588 Project: Flink Issu

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

2023-07-13 Thread Matt Wang
Hi Yunfeng, Thanks for the proposal. The POC showed a performance improvement of 20%, which is very exciting. But I have some questions: 1. Is the performance improvement here mainly due to the reduction of serialization, or is it due to the judgment consumption caused by tags? 2. Watermark is n

[jira] [Created] (FLINK-32587) The results returned from the CDC sql query are null or the value was changed unexpectly

2023-07-13 Thread jasonliangyc (Jira)
jasonliangyc created FLINK-32587: Summary: The results returned from the CDC sql query are null or the value was changed unexpectly Key: FLINK-32587 URL: https://issues.apache.org/jira/browse/FLINK-32587

[jira] [Created] (FLINK-32586) Enable input locality in SimpleExecutionSlotAllocator

2023-07-13 Thread xingbe (Jira)
xingbe created FLINK-32586: -- Summary: Enable input locality in SimpleExecutionSlotAllocator Key: FLINK-32586 URL: https://issues.apache.org/jira/browse/FLINK-32586 Project: Flink Issue Type: Improve

Flink 2.0 connectors, formats, and even user code should also be pluggable.

2023-07-13 Thread zhiqiang li
Flink CDC Maintainer xbjt...@gmail.com Factory Interface Timo Walther Plugin Interface Aleksey Pak Connector Base Stephan Ewen Flink File System Stephan Ewen I have observed that in [1], connectors and formats are pluggable, allowing user code to be easily integrated. The advantages of ha

Re: [DISCUSS] Update Flink Roadmap

2023-07-13 Thread Jiabao Sun
Thanks Jark and Martijn for driving this. There are two suggestions about the Table API: - Add the JSON type to adapt to the no sql database type. - Remove changelog normalize operator for upsert stream. Best, Jiabao > 2023年7月13日 下午3:49,Jark Wu 写道: > > Hi all, > > Sorry for taking so long

Re: [DISCUSS] Update Flink Roadmap

2023-07-13 Thread Jark Wu
Hi all, Sorry for taking so long back here. Martijn and I have drafted the first version of the updated roadmap, including the updated feature radar reflecting the current state of different components. https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit Feel fre