[jira] [Created] (FLINK-27424) StreamExecutionEnvironment.readFile() api monitor file directory errors

zhangqw152 created FLINK-27424: -- Summary: StreamExecutionEnvironment.readFile() api monitor file directory errors Key: FLINK-27424 URL: https://issues.apache.org/jira/browse/FLINK-27424 Project: Flink

[DISCUSS] FLIP-227: Support overdraft buffer

Hi everyone, Unaligned Checkpoint (FLIP-76 [1]) is a major feature of Flink. It effectively solves the problem of checkpoint timeout or slow checkpoint when backpressure is severe. We found that UC(Unaligned Checkpoint) does not work well when the back pressure is severe and multiple output buffe

[jira] [Created] (FLINK-27423) Upgrade Hive 3.1 connector from 3.1.2 to 3.1.3

Jeff Yang created FLINK-27423: - Summary: Upgrade Hive 3.1 connector from 3.1.2 to 3.1.3 Key: FLINK-27423 URL: https://issues.apache.org/jira/browse/FLINK-27423 Project: Flink Issue Type: Technica

[jira] [Created] (FLINK-27422) Do not create temporary pod template files for JobManager and TaskManager if not configured explicitly

Yang Wang created FLINK-27422: - Summary: Do not create temporary pod template files for JobManager and TaskManager if not configured explicitly Key: FLINK-27422 URL: https://issues.apache.org/jira/browse/FLINK-27422

Re: [DISCUSS] FLIP-168: Speculative execution for Batch Job

Hi everyone, More and more users are running their batch jobs on Flink nowadays. One major problem they encounter is slow tasks running on hot/bad nodes, resulting in very long and uncontrollable execution time of batch jobs. This problem is a pain or even unacceptable in production. Many users ha

[DISCUSS] FLIP-224: Blacklist Mechanism

Hi all, Flink job failures may happen due to cluster node issues (insufficient disk space, bad hardware, network abnormalities). Flink will take care of the failures and redeploy the tasks. However, due to data locality and limited resources, the new tasks are very likely to be redeployed to the s

[jira] [Created] (FLINK-27421) Bundle test utility classes into the PyFlink package to make users write test cases easily

Dian Fu created FLINK-27421: --- Summary: Bundle test utility classes into the PyFlink package to make users write test cases easily Key: FLINK-27421 URL: https://issues.apache.org/jira/browse/FLINK-27421 Proj

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

Hi Gyula, Thanks for the release timeline information. I would like to learn the gathered knowledge and volunteer as well. Will there be sync up meeting/call for this collaboration ? From: Gyula Fóra Date: Monday, April 25, 2022 at 11:22 AM To: dev Subject: [DISCUSS] Next Flink Kubernetes Ope

[jira] [Created] (FLINK-27420) Suspended SlotManagers fail to reregister metrics when started again

Ben Augarten created FLINK-27420: Summary: Suspended SlotManagers fail to reregister metrics when started again Key: FLINK-27420 URL: https://issues.apache.org/jira/browse/FLINK-27420 Project: Flink

Re: [DISCUSS] Docker images for SNAPSHOT

Hi Alex, After checking with ASF Infra, the only thing we need to do is create a Jira ticket with them. It's enabled per repository, so all needed is to supply the repository name for which we want to have this enabled. We could then build a workflow to push those snapshots on a (scheduled) nightl

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

Just wanted to point out that when we release 1.0.0 we inevitably also have to think about what compatibility guarantees we want to give and how we intend to enforce them. Additionally it would be good to think about the general approach of releases; how often are minor/patch releases made, ho

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

Thanks for starting the discussion. It is exciting to learn about the plan of 1.0.0 version! The timeline is fine to me. As for the SQL support, as Yang said, I have got some basic ideas and try to make a PoC for verification. It may be first implemented in upstream flink project and then utiliz

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

Thanks Gyula for starting this discussion. The release time looks good to me. The main code for the session job is complete, the doc and other side issues are on the way. I will ping you guys in the ticket after the work are completed from my side to help review together whether there is functiona

[jira] [Created] (FLINK-27419) Endpoint to cancel savepoints

Ingo Bürk created FLINK-27419: - Summary: Endpoint to cancel savepoints Key: FLINK-27419 URL: https://issues.apache.org/jira/browse/FLINK-27419 Project: Flink Issue Type: New Feature Com

[jira] [Created] (FLINK-27418) Flink SQL TopN result is wrong

zhangbin created FLINK-27418: Summary: Flink SQL TopN result is wrong Key: FLINK-27418 URL: https://issues.apache.org/jira/browse/FLINK-27418 Project: Flink Issue Type: Bug Components:

[jira] [Created] (FLINK-27417) Flink JDBC SQL Connector：SELECT * FROM table WHERE co > 100; mysql will execute SELECT * FROM table to scan the whole table

haojiawei created FLINK-27417: - Summary: Flink JDBC SQL Connector：SELECT * FROM table WHERE co > 100; mysql will execute SELECT * FROM table to scan the whole table Key: FLINK-27417 URL: https://issues.apache.org/jir

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Thanks for the reply Sebastian and Dawid, I think Sebastion has a good summary. This is a really helpful discussion. Thinking a bit more, I feel that it might still be better to add the supportsXXX() method in the Source rather than SourceReader. Generally speaking, what we are trying to do here

Re: Re: Re: Discussion about enhancing the partitioned table syntax

Hi, If this is already proposed and accepted in the FLIP, then we indeed don't need a new FLIP. Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser On Fri, 15 Apr 2022 at 11:25, 刘大龙 wrote: > Hi，Martijn > > Thanks for the Jingsong remind. This iss

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

Hi all, I'm not too familiar with Hive and HiveServer2, but I do have a couple of questions/concerns: 1. What is the relationship between this FLIP and FLIP-91? My assumption would be that this FLIP (and therefore the HiveServer2) implementation would need to be integrated in the REST Gateway, is

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi Shengkai, Thanks for opening this discussion. I did a first brief pass over the FLIP and I have a couple of questions/remarks: * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway which would be used by Flink's SQL Client (or other applications)? * From a user perspective,

[jira] [Created] (FLINK-27416) FLIP-225: Implement standalone mode support in the kubernetes operator

Usamah Jassat created FLINK-27416: - Summary: FLIP-225: Implement standalone mode support in the kubernetes operator Key: FLINK-27416 URL: https://issues.apache.org/jira/browse/FLINK-27416 Project: F

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for creating the FLIP and opening the discussion. I did get a bit confused about the title, being "query lifecycle statements in SQL client". This sounds like you want to adopt the SQL client, but you want to expand the SQL syntax with lifecycle statements, which could be used from

Re: [DISCUSS] Docker images for SNAPSHOT

Hi Alex, Thanks for bringing this up for discussion. I think it's indeed important that we make it possible that externalized connectors can be tested, both against released Flink versions but also against SNAPSHOT versions. I did a quick check at the ASF Jira service and noticed there is https:/

Re: [DISCUSS] FLIP-223: Implement standalone mode support in the kubernetes operator

+1 for the proposal :) I think this will fit nicely with the current API. If we use deployments we could simply extend the current status interfaces with taskManagerDeploymentStatus (we already have jobManagerDeploymentStatus) for the standalone mode. We would have to check how we could use the

Re: Re: [VOTE] FLIP-214: Support Advanced Function DDL

Hi all, The voting time for FLIP-214 has passed. I'm closing the vote now. There were 5 votes, 4 of which are binding: - Mang Zhang - Jark Wu (binding) - Jing Zhang (binding) - Godfrey He (binding) - Leonard Xu (binding) There were no disapproving votes. Thus, FLIP-214 has been accepted. Thank

Re: [DISCUSS] Planning Flink 1.16

Hi everyone, Thanks for starting this discussion. I would also volunteer to help out as a release manager for the 1.16 release. Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser On Tue, 26 Apr 2022 at 13:19, godfrey he wrote: > Hi Konstantin &

Re: [DISCUSS] FLIP-223: Implement standalone mode support in the kubernetes operator

Thanks for the feedback. # The TaskManager replicas Yeah I think this makes sense and explicitly stating TM replicas makes more sense in the standalone mode. I will update the FLIP and clarify this. # How the JobManager and TaskManager pods are managed? I think deployments for both TaskManager

Re: [DISCUSS] Planning Flink 1.16

Hi Konstantin & Chesnay, Thanks for driving this discussion, I am willing to volunteer as the release manager for 1.16. Best, Godfrey Konstantin Knauf 于2022年4月26日周二 18:23写道： > > Hi everyone, > > With Flink 1.15 about to be released, the community has started planning & > developing features fo

Re: [VOTE] FLIP-214: Support Advanced Function DDL

Hi, Ron Thanks for driving this FLIP, the design looks good to me. +1 (binding) Best, Leonard > 2022年4月22日 下午2:00，godfrey he 写道： > > Hi, Ron > > Thanks for the explanation, > +1 (binding) from my side > > Best, > Godfrey > > 刘大龙 于2022年4月22日周五 13:45写道： >> >> >> Hi, godfrey >> >> The a

[jira] [Created] (FLINK-27415) Read empty csv file throws exception in FileSystem table connector

Jingsong Lee created FLINK-27415: Summary: Read empty csv file throws exception in FileSystem table connector Key: FLINK-27415 URL: https://issues.apache.org/jira/browse/FLINK-27415 Project: Flink

[DISCUSS] Planning Flink 1.16

Hi everyone, With Flink 1.15 about to be released, the community has started planning & developing features for the next release, Flink 1.16. As such, I would like to start a discussion around managing this release. Specifically, Chesnay & myself would like to volunteer as release managers. Our f

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Thanks @Sebastian for the nice summary. I think most of your points aligned with the suggestions I made to the FLIP, while you were writing your reply (I believe we hit enter nearly at the same time ;) ) Two points after we synced offline 1. I changed also the supportsWatermarksSplitAlignmen

[jira] [Created] (FLINK-27414) Support operator state in PyFlink DataStream API

Juntao Hu created FLINK-27414: - Summary: Support operator state in PyFlink DataStream API Key: FLINK-27414 URL: https://issues.apache.org/jira/browse/FLINK-27414 Project: Flink Issue Type: New Fe

[jira] [Created] (FLINK-27413) Drop scala 2.11 support from the flink-docker

Yun Gao created FLINK-27413: --- Summary: Drop scala 2.11 support from the flink-docker Key: FLINK-27413 URL: https://issues.apache.org/jira/browse/FLINK-27413 Project: Flink Issue Type: Bug

Re: [DISCUSS] More public interfaces for internal data structures

Thank Jingsong for starting this discussion. I think it's reasonable to add them to public APIs which can help build connectors easier. Looking forward to a FLIP to finalize the APIs. Best, Jark On Tue, 26 Apr 2022 at 14:03, Jingsong Li wrote: > Hi everyone, > > The source sink for the Table/S

Re: [DISCUSS] FLIP-223: Implement standalone mode support in the kubernetes operator

Thanks for creating the FLIP-223 and starting the discussion. I have some quick questions. # The TaskManager replicas The TaskManager replicas need to be configured both for standalone session and application. Because it could not be calculated if the parallelism is set via java codes. # How

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Thanks for the response, Arvid! I have few comments on your message. > but could also live with an easier solution as the first step: I think that these 2 ways are mutually exclusive (originally proposed by Qingsheng and mine), because conceptually they follow the same goal, but implementation d

[jira] [Created] (FLINK-27412) Allow flinkVersion v1_13 in flink-kubernetes-operator

Yang Wang created FLINK-27412: - Summary: Allow flinkVersion v1_13 in flink-kubernetes-operator Key: FLINK-27412 URL: https://issues.apache.org/jira/browse/FLINK-27412 Project: Flink Issue Type: I

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

Thanks Gyula for starting this discussion. Some users from different companies are also very interested in flink-kubernetes-operator project and asked me in private when it will be production ready. Now I would say the release 1.0.0 aims to this mission. Given that the SQL support in flink-kubern

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

Hi folks, Sorry for being a bit silent. Many thanks for all the input and suggestions. As I'm a bit new, I needed some time to catch up and structure (for myself) the discussion and I wanted to find a way to structure the conclusions. (Also because I had the feeling that some concerns got lost in

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

@Arvid: While I also like Becket's capability approach, I fear that it doesn't work for this particular use case: Sources can always be aligned cross-task and this is just about intra-task alignment. So it's plausible to put sources into an alignment group even though they do not use