Re: Flink 1.14.4 HybridSource consumes lots of CPU resources

Thank you for reporting the issue. Mason has already identified the root cause and the JIRA is now assigned to him: https://issues.apache.org/jira/browse/FLINK-27479 Thomas On Tue, May 3, 2022 at 4:02 AM Martijn Visser wrote: > I'm looping in @Thomas Weise since he has expertise on > the Hybri

Re: Migrating Flink apps across cloud with state

Hey Hemanga, That's quite annoying of MirrorMaker to change the offsets on you. One solution would be to use the State Processor API[1] to read the savepoint and update the offsets to the new ones — does MirrorMaker give you these ahead of time? There might also be more specific tricks people coul

Flink serialization errors at a batch job

Hi, We're running a large Flink batch job and sometimes it throws serialization errors in the middle of the job. It is always the same operator but the error can be different. Then the following attempts work. Or sometimes attempts get exhausted, then retrying the job. The job is basically readin

Re: Migrating Flink apps across cloud with state

Any ideas, guys? On Mon, May 2, 2022 at 6:11 PM Hemanga Borah wrote: > Hello, > We are attempting to port our Flink applications from one cloud provider > to another. > > These Flink applications consume data from Kafka topics and output to > various destinations (Kafka or databases). The appl

Setting boundedness for legacy Hadoop sequence file sources

Hi all, I’m converting several batch Flink workflows to streaming, with bounded sources. Some of our sources are reading Hadoop sequence files via StreamExecutionEnvironment.createInput(HadoopInputFormat). The problem is that StreamGraphGenerator.existsUnboundedSource is returning true, becaus

Flink checkpointing with slow pipelines

Hi Flink users, Greetings. I have a question on how Flink invokes checkpoints with a slow pipeline. I have a Beam streaming pipeline with one Map() call. It is a Python program running on Flink with PortableRunner. I’ve experimented with varying amounts of sleep inside this call to simulate slown

Issue with HybridSource recovering from Savepoint

Hi, We're encountering an error using a HybridSource that is composed of a FileSource + KafkaSource, only when recovering from a savepoint [0]. This HybridSource is used to read from a Kafka topic's archives hosted on GCS via a bounded FileSource, and then automatically switch over to the data str

Re: Using the official flink operator and kubernetes secrets

Flink could not support environment replacement in the args. I think you could access the env via "*System.getenv()*" in the user main method. It should work since the user main method is executed in the JobManager side. Best, Yang Őrhidi Mátyás 于2022年4月28日周四 19:27写道： > Also, > > just declaring

Re: Prometheus metrics does not work in 1.15.0 taskmanager

> I noticed that my config of the PrometheusReporter is different here. I have: `metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter`. I will investigate if this is a problem. That's not a problem. > Which trace logs are interesting? The logging config I provi

Re: Prometheus metrics does not work in 1.15.0 taskmanager

Hi Chesnay, Thanks for the code snipped. Which trace logs are interesting? Of " org.apache.flink.metrics.prometheus.PrometheusReporter"? I could also add this logger settings in the environment where the problem is present. Other than that, I am not sure how to reproduce this issue in a local set

REMINDER - Travel Assistance available for ApacheCon NA New Orleans 2022

Hi All Contributors and Committers, This is a first reminder email that travel assistance applications for ApacheCon NA 2022 are now open! We will be supporting ApacheCon North America in New Orleans, Louisiana, on October 3rd through 6th, 2022. TAC exists to help those that would like to attend

Re: Flink 1.14.4 HybridSource consumes lots of CPU resources

I'm looping in @Thomas Weise since he has expertise on the HybridSource. On Tue, 3 May 2022 at 12:04, Arthur Li wrote: > Hi Mason, > > I upload the code and resource files to AwesomeArthurLi/quickstart: > quickstart (github.com) , > may it will he

Re: Prometheus metrics does not work in 1.15.0 taskmanager

You'd help me out greatly if you could provide me with a sample job that runs into the issue. So far I wasn't able to reproduce the issue, but it should be clear that there is some given 3 separate reports, although it is strange that so far it was only reported for Prometheus. If one of you is

Re: Flink 1.14.4 HybridSource consumes lots of CPU resources

Hi Mason, I upload the code and resource files to AwesomeArthurLi/quickstart: quickstart (github.com) , may it will help you reproduce the issue. BR. Arthur Li > 2022年5月3日 15:48，Mason Chen 写道： > > Hi Arthur, > > Coincidentally, I also encount

Re: Prometheus metrics does not work in 1.15.0 taskmanager

On Tue, May 03, 2022 at 10:32:03AM +0200, Peter Schrott wrote: > Hi! > > I also discovered problems with the PrometheusReporter on Flink 1.15.0, > coming from 1.14.4. I already consulted the mailing list: > https://lists.apache.org/thread/m8ohrfkrq1tqgq7lowr9p226z3yc0fgc > I have not found the und

Re: Prometheus metrics does not work in 1.15.0 taskmanager

On Tue, May 03, 2022 at 10:28:18AM +0200, Chesnay Schepler wrote: > Is there any warning in the logs containing "Error while handling metric"? No, we don't find any "Error while handling metric" -- ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org http://czchen.info/ Key fingerprint = BA04 346D C

Re: Prometheus metrics does not work in 1.15.0 taskmanager

Hi! I also discovered problems with the PrometheusReporter on Flink 1.15.0, coming from 1.14.4. I already consulted the mailing list: https://lists.apache.org/thread/m8ohrfkrq1tqgq7lowr9p226z3yc0fgc I have not found the underlying problem or a solution to it. Actually, after re-checking, I see th

Re: Prometheus metrics does not work in 1.15.0 taskmanager

Is there any warning in the logs containing "Error while handling metric"? On 03/05/2022 10:18, ChangZhuo Chen (陳昌倬) wrote: On Tue, May 03, 2022 at 01:00:42AM -0700, Mason Chen wrote: Hi ChangZhou, The warning log indicates that the metric was previously defined and so the runtime is handling

Re: Prometheus metrics does not work in 1.15.0 taskmanager

On Tue, May 03, 2022 at 01:00:42AM -0700, Mason Chen wrote: > Hi ChangZhou, > > The warning log indicates that the metric was previously defined and so the > runtime is handling the "duplicate" metric by ignoring it. This is > typically a benign message unless you rely on this metric. Is it possib

Re: About job execution

Hi Jack, I'm supposing you are actually referring to JobMaster by the term JobManager in your context (the JobManager referred in the doc is usually a collection of components that comprises the Dispatcher, ResourceManager, and a set of one-per-job JobMasters). The jobmaster's lifecycle is bou

Re: Prometheus metrics does not work in 1.15.0 taskmanager

Hi ChangZhou, The warning log indicates that the metric was previously defined and so the runtime is handling the "duplicate" metric by ignoring it. This is typically a benign message unless you rely on this metric. Is it possible that you are using the same task name for different tasks? It would

Re: How should I call external HTTP services with PyFlink?

Hey Francis! Thanks for the insights! I am thinking of using Java / Scala for this scenario given your input. Introducing a new language to the team, however, is going to be a big ask :-D Another option that you mentioned is pushing enrichment data instead of pulling. That would be excellent, I w