Re: (Possible) bug in flink-kafka-connector (metrics rewriting)

Hi, I suggest you creating a ticket for it on https://issues.apache.org/jira/projects/FLINK/summary. Flink - ASF JIRA Welcome to the Apache Flink project. Apache Flink is an open source platform for scalable batch and stream data processing.

Re: Can flink dynamically detect high load and increase the job parallelism automatically?

Hi Erez, Unfortunately, autoscaling for streaming jobs is only available with reactive mode, which as you've already pointed out, is an MVP feature yet and only supports Standalone mode. Some vendors (e.g. Ververica) have already shipped their own private implementations of Flink autoscaling th

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

Hi, Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)? Best, Zhanghao Chen From: hjw <1010445...@qq.com> Sent: Tuesday, September 13, 2022 1:58 To: user Subject: A classloading question about submitting

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

@h.yuan...@gmail.com<mailto:h.yuan...@gmail.com> Any thoughts on this? Best, Zhanghao Chen From: hjw <1010445...@qq.com> Sent: Tuesday, September 13, 2022 11:24 To: zhanghao.chen ; user Subject: Re: A classloading question about submitting Flink

Re: ExecutionMode in ExecutionConfig

https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/datastream/execution_mode/ gives a comprehensive description on it Execution Mode (Batch/Streaming) | Apache Flink Execution Mode (Ba

Re: Can flink dynamically detect high load and increase the job parallelism automatically?

Flink will not try to help you do autoscaling and the parallelism is fixed unless you enable reactive mode/adaptive scheduler. Max parallelism just means the maximum parallelism with which you can rescale your job without losing states. The max parallelism limit is related to the Flink key group

Re: ExecutionMode in ExecutionConfig

Could you try setting ”execution.batch-shuffle-mode‘=‘ALL_EXCHANGES_PIPELINED’? Looks like the ExecutionMode in ExecutionConfig does not work for DataStream APIs. The default shuffling behavior for a DataStream API in batch mode is 'ALL_EXCHANGES_BLOCKING' where upstream and downstream tasks ru

Re: ExecutionMode in ExecutionConfig

It's added in Flink 1.14: https://nightlies.apache.org/flink/flink-docs-master/zh/release-notes/flink-1.14/#expose-a-consistent-globaldataexchangemode. Not sure if there's a way to change this in 1.13 Best, Zhanghao Chen From: Hailu, Andreas Sent: Wednesday, Sep

Re: Flink Operator in Golang?

Hi Mark, 1. Directly quoting from https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator: Main reasons for choosing Java over Go * Direct access to Flink Client libraries for submitting, managing jobs and handling errors * Most Flink devel

Re: Kubernetes operator and jobs with last-state upgrades

Hi, When you use last state upgrade, you simply delete the Flink deployment but keeping HA metadata where checkpoint info is stored, and the upgraded job would recover the state from the latest checkpoint. Best, Zhanghao Chen From: Alexis Sarda-Espinosa Sent: T