[jira] [Created] (FLINK-27751) Scala suffix check failed

Huang Xingbo created FLINK-27751: Summary: Scala suffix check failed Key: FLINK-27751 URL: https://issues.apache.org/jira/browse/FLINK-27751 Project: Flink Issue Type: Bug Component

[DISCUSS] Releasing Flink 1.14.5

Hi all, I would like to start discussing releasing Flink 1.14.5. It has already been more than two months since we released 1.14.4. There are currently 62 tickets[1] already resolved for 1.14.5, some of them quite severe. Currently, there are no issues marked as critical or blocker for 1.14.5. P

All flink-ci fail

Hi masters, All compile_ci of flink-ci[1] failed today, seemingly for the same reason: 05:03:16,982 ERROR org.apache.flink.tools.ci.suffixcheck.ScalaSuffixChecker [] - Violations found: Scala-free module 'flink-hadoop-compatibility' is referenced with scala suffix in 'flink-formats/flink-sequ

Re: StreamGraph vs JobGraph vs ExecutionGraph

Great, thanks for the details. On Tue, May 24, 2022 at 8:07 AM Sriram Ganesh wrote: > Hi Prabhu, > > For detail understanding check it out this link: > > https://www.alibabacloud.com/blog/in-depth-analysis-of-flink-job-execution-flink-advanced-tutorials_596633 > > Thanks, > Sriram G > > On Tue,

[jira] [Created] (FLINK-27750) The configuration of

dong created FLINK-27750: Summary: The configuration of Key: FLINK-27750 URL: https://issues.apache.org/jira/browse/FLINK-27750 Project: Flink Issue Type: Bug Reporter: dong -- This

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

Hi Gen Luo, You're right, the total cost time include the failover-restart time. So when the failover time exceeds the retry timeout set by the user, in fact, all the data to be retry after recovery will have no additional retry opportunities, which is equivalent to normal data. In such circumstan

Re: Application mode -yarn dependancy error

Hi. I think you should send the mail to the user mail list or stack overflow, which is about the usage and help. The dev mail list focus on the design of the Flink itself. Could you share more details for your problems, including - which version you use. - how you use the Flink, including you cod

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

Hi, all. Considering we start to vote for FLIP-91 for a while, I think we can restart the discussion about the FLIP-223. I am glad that you can give some feedback about FLIP-223. Best, Shengkai Martijn Visser 于2022年5月6日周五 19:10写道： > Hi Shengkai, > > Thanks for clarifying. > > Best regards, >

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

Hi Lincoln, Thanks for the explanation. I understand your thought, but I'm a little confused by the additional detail. Is the startTime when the record is processed for the first time? And the cost time is counted based on it even after a job recovers from a failover or is restarted? For the failo

Re: [DISCUSS] FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems

Hi all, Considering the new common table option 'lookup.max-retries' proposed in FLIP-221[1] which is commonly used for exception handling in connector implementation, we should clearly distinguish ASYNC_LOOKUP_RETRY from it to avoid confusing users. To do so, the name ASYNC_LOOKUP_RETRY can chan

Re: Json Deserialize in DataStream API with array length not fixed

Hi. In the SQL, you can just specify the `array_coordinates` type ARRAY[1]. For example, ``` CREATE TABLE source( `array_coordinates` ARRAY> ) WITH ( 'format' = 'json' ) ``` [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/ Zain Haider Nemati

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Qingsheng, Sorry for jumping into the discussion so late. It's a good idea that we can have a common table option. I have a minor comments on 'lookup.async' that not make it a common option: The table layer abstracts both sync and async lookup capabilities, connectors implementers can choose

Re: StreamGraph vs JobGraph vs ExecutionGraph

Hi Prabhu, For detail understanding check it out this link: https://www.alibabacloud.com/blog/in-depth-analysis-of-flink-job-execution-flink-advanced-tutorials_596633 Thanks, Sriram G On Tue, May 24, 2022, 07:53 Shengkai Fang wrote: > I think the question also confused me for a long time. > >

Re: StreamGraph vs JobGraph vs ExecutionGraph

I think the question also confused me for a long time. I think: - The StreamGraph is just a logical representation of the job, which contains the basic information of the Job. - The JobGraph is serializable, which is used between the client and job master. Job Master is able to assign the resourc

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

Thanks Gen Luo! Agree with you that prefer the simpler design. I’d like to share my thoughts on this choice: whether store the retry state or not only affect the recovery logic, not the per-record processing, so I just compare the two: 1. w/ retry state: simple recovery but lost precision 2. w/o

1.13.5????sql????64k????bug

 ??1.13.5sql64k

[jira] [Created] (FLINK-27748) AdaptiveScheduler should support operator fixed parallelism

john created FLINK-27748: Summary: AdaptiveScheduler should support operator fixed parallelism Key: FLINK-27748 URL: https://issues.apache.org/jira/browse/FLINK-27748 Project: Flink Issue Type: Impr

[jira] [Created] (FLINK-27747) Flink kubernetes operator helm chart release the Chart.yaml file doesn't have an apache license header

Yang Wang created FLINK-27747: - Summary: Flink kubernetes operator helm chart release the Chart.yaml file doesn't have an apache license header Key: FLINK-27747 URL: https://issues.apache.org/jira/browse/FLINK-27747

[jira] [Created] (FLINK-27746) Flink kubernetes operator docker image could not build with source release

Yang Wang created FLINK-27746: - Summary: Flink kubernetes operator docker image could not build with source release Key: FLINK-27746 URL: https://issues.apache.org/jira/browse/FLINK-27746 Project: Flink

[GitHub] [flink-connector-rabbitmq] pscls commented on pull request #1: [FLINK-20628] RabbitMQ Connector using FLIP-27 Source API

pscls commented on PR #1: URL: https://github.com/apache/flink-connector-rabbitmq/pull/1#issuecomment-1135015425 @MartijnVisser We are not exactly sure what has to be part of the root-pom. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [flink-connector-rabbitmq] pscls commented on pull request #1: [FLINK-20628] RabbitMQ Connector using FLIP-27 Source API

pscls commented on PR #1: URL: https://github.com/apache/flink-connector-rabbitmq/pull/1#issuecomment-1135011871 This is a copy from the original PR (https://github.com/apache/flink/pull/15140) against the Flink repository. -- This is an automated message from the Apache Git Service. To

[GitHub] [flink-connector-rabbitmq] pscls opened a new pull request, #1: [FLINK-20628] RabbitMQ Connector using FLIP-27 Source API

pscls opened a new pull request, #1: URL: https://github.com/apache/flink-connector-rabbitmq/pull/1 ## What is the purpose of the change This pull request ports the RabbitMQ connector implementation to the new Connector’s API described in [FLIP-27](https://cwiki.apache.org/co

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #1

Hi Yang! Thank you for preparing the RC. I have successfully verified the following: - Signatures, Hashes - No binaries in source release - Helm Repo works, Helm install works, docker image matches release commit tag - Build from source - Submit example job without errors Some problems that I ha

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

Hi Chesnay, Thanks for feedback. 1. Regarding the TM/Node id. Do you mean special characters may appear in the rest URL? Actually, I don't think so. The task manager id in REST API should be the *ResourceID* of taskmanager in Flink, there should be no special characters, and some existing REST A

Access to confluence

Hi Jark and other PMC members! Сould you give me access to the editing of the confluence page with FLIP-221 [1]? I would like to make small changes that were mentioned in discussion of this FLIP. My login is 'smiralexan' [2]. Thanks in advance! Best regards, Alexander [1] https://cwiki.apache.o

Re: Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi Jark, > "Table#createTableAs(tablePath)" seems a >little strange to me. `Table#createTableAs` is a bit misleading, I learn to Table#saveAs(tablePath). Best, Godfrey Jark Wu 于2022年5月18日周三 23:09写道： > > Hi Godfrey, > > Regarding Table API for CTAS, "Table#createTableAs(tablePath)" seems a > li

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for the update. >'SHOW QUERIES' lists all jobs in the cluster, no limit on APIs (DataStream or >SQL) or clients (SQL client or CLI). Is DataStream job a QUERY? I think not. For a QUERY, the most important concept is the statement. But the result does not contain this info. If we

[VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 1.0.0 of Apache Flink Kubernetes Operator, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) **Release Overview** As an overview, the release consists of th

[jira] [Created] (FLINK-27745) ClientUtilsTest.uploadAndSetUserArtifacts failed with NoClassDefFoundError

Huang Xingbo created FLINK-27745: Summary: ClientUtilsTest.uploadAndSetUserArtifacts failed with NoClassDefFoundError Key: FLINK-27745 URL: https://issues.apache.org/jira/browse/FLINK-27745 Project: F

[jira] [Created] (FLINK-27744) CheckPubSubEmulatorTest.testPull failed with AssertionError

Huang Xingbo created FLINK-27744: Summary: CheckPubSubEmulatorTest.testPull failed with AssertionError Key: FLINK-27744 URL: https://issues.apache.org/jira/browse/FLINK-27744 Project: Flink

Re: About Native Deployment's Autoscaling implementation

Hi Talat! One other approach that we are investigating currently is combining the Flink Kubernetes Operator with the K8S scaling capabilities (Horizontal Pod autoscaler) In this approach the HPA monitors the Taskmanager pods directly and can m

Re: StreamGraph vs JobGraph vs ExecutionGraph

They are all different representations of a job. StreamGraph is the representation made by DataStream API, which gets converted into the JobGraph (which is API-agnositic), that is submitted to Flink. From the JobGraph we then create an ExecutionGraph, which is the runtime representation of a j

StreamGraph vs JobGraph vs ExecutionGraph

Hi, Could someone give an idea of what these graphs - StreamGraph, JobGraph and ExecutionGraph are and their differences. Thanks, Prabhu Joseph

Re: Request for Review: FLINK-27507 and FLINK-27509

I've taken care of this. David On Sun, May 22, 2022 at 4:12 AM Shubham Bansal wrote: > Hi Everyone, > > I am not sure who to reach out for the reviews of these changesets, so I > am putting this on the mailing list here. > > I have raised the review for > FLINK-27507 - https://github.com/apache

[jira] [Created] (FLINK-27743) Support function params when create UDTF in flink sql

konwu created FLINK-27743: - Summary: Support function params when create UDTF in flink sql Key: FLINK-27743 URL: https://issues.apache.org/jira/browse/FLINK-27743 Project: Flink Issue Type: New Featu

TolerableCheckpointFailureNumber not always applying

Hello everyone, We're having an issue on our Flink job: it restarted because it failed a checkpoint, even though it shouldn't have. We've set the tolerableCheckpointFailureNumber to 1 million to never have the job restart because of this. However, the job did restart following a checkpoint failure

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

Hi, Jark Thanks for the feedback. > 1) All the ability interfaces begin with "Supports" instead of "Support". +1 > The "connect" word should be "collect"? Yes, it's a typo. > CatalogStatistics Yes, we should use TableStats. I forgot that TableStats and ColumnStats have been ported to the API mo

Re: [VOTE] FLIP-91: Support SQL Gateway

+1 (binding) Thanks, Timo On 23.05.22 03:53, Jingsong Li wrote: +1 Best, Jingsong On Sat, May 21, 2022 at 12:23 AM Shqiprim Bunjaku wrote: +1 (non-binding) Best, Shqiprim Bunjaku On Fri, May 20, 2022 at 5:56 PM Yufei Zhang wrote: +1 (non-binding) Best, Yufei Zhang On Fri, May 20, 20

[jira] [Created] (FLINK-27742) Fix Compatibility Issues Between Flink ML Operators.

Yunfeng Zhou created FLINK-27742: Summary: Fix Compatibility Issues Between Flink ML Operators. Key: FLINK-27742 URL: https://issues.apache.org/jira/browse/FLINK-27742 Project: Flink Issue Ty

Re: Flink UI in Application Mode

Hi Zain, you can find a link to web-ui either in the CLI output after the job submission or in the YARN ResourceManager web ui [1]. With YARN Flink needs to choose the application master port at random (could be somehow controlled by setting _yarn.application-master.port_) as there might be multip

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

Hi Lincoln, Thanks for the quick reply. 1. I understand when restarting a job with a savepoint, the retry state can ensure the total retry attempts and delay is expected. However, when failover happens while a job is running, the remaining attempts recorded in the state are actually redid, and

[jira] [Created] (FLINK-27741) Fix NPE when use dense_rank() and rank() in over aggregation

chenzihao created FLINK-27741: - Summary: Fix NPE when use dense_rank() and rank() in over aggregation Key: FLINK-27741 URL: https://issues.apache.org/jira/browse/FLINK-27741 Project: Flink Issue

[jira] [Created] (FLINK-27740) [JUnit5 Migration] Module: flink-test-utils-junit

Sergey Nuyanzin created FLINK-27740: --- Summary: [JUnit5 Migration] Module: flink-test-utils-junit Key: FLINK-27740 URL: https://issues.apache.org/jira/browse/FLINK-27740 Project: Flink Issue

Re: About Native Deployment's Autoscaling implementation

Hi Talat, This is definitely an interesting and rather complex topic. Few unstructured thoughts / notes / questions: - The main struggle has always been that it's hard to come up with a generic one-size-fits-it-all metrics for autoscaling. - Flink doesn't have knowledge of the external environ

[jira] [Created] (FLINK-27739) Add benchmark module for table store

Caizhi Weng created FLINK-27739: --- Summary: Add benchmark module for table store Key: FLINK-27739 URL: https://issues.apache.org/jira/browse/FLINK-27739 Project: Flink Issue Type: Improvement