Flink autoscaler is throwing exception and shutting down the entire job

Hi, We are running our Flink streaming pipeline with following configs: (It runs on Flink 1.19.1 on AWS EMR (Yarn) taskmanager.numberOfTaskSlots: 4 job.autoscaler.enabled: 'true' jobmanager.scheduler: adaptive jobmanager.adaptive-scheduler.combined-restart.enabled: true jobmanager.adaptive-schedul

Re:Re: Re: Re: Re: Can a Table Created in Java Code be Queried Using SQL Client?

What about using "./bin/start-cluster.sh -Dtable.catalog-store.kind=file -Dtable.catalog-store.file.path=/Users/vagarwal/tmp/flink/store/" to start sql client? -- Best！ Xuyang 在 2025-01-07 00:30:19，"Vinay Agarwal" 写道： Still doesn't show. ```SQL Flink SQL> set 'table.catalog-s

For the FileSystem-based Changelog storage, for how long these changelogs are retained

Hi, I have enabled state.backend.changelog And state.backend.changelog.storage is set to filesystem I wanted to know for how long the changelog files are retained as I was storing the data to HDFS and I was quickly running out of space. What should be the ideal size of disk storage I should alloc

RE: [External] Re: Is there a way to get consolidated metrics at task level

Hi Sachin again, In case of Prometheus it’s done by means of the query against it. * See aggregation ops [1] * Each metric has properties (like dimensions) and you can aggregate over them * See examples here [2] [1] https://prometheus.io/docs/prometheus/latest/querying/operators

Re: [External] Is there a way to get consolidated metrics at task level

So how can we aggregate same metric across all the sub tasks ? Say: Task_0 is outputting a value for numRecordsInPerSecond Task_1 is outputting a value for numRecordsInPerSecond I want to consolidate numRecordsInPerSecond for the entire task which would be the sum of these values for all the numb

RE: [External] Is there a way to get consolidated metrics at task level

Hi Sachin, Happy new year … 😊 On Yarn we setup prometheus push gateway on one machine and the have al task managers export their monitoring to this push gateway. Then Prometheus would collect all metrics from push gateway. On yarn, the main problem is missing isolation of workloads and thus you

Generic log-based incremental checkpoint seems to be not working

Hello, I am running a job on apache Flink 1.19, on AWS EMR (EC2) cluster as a YARN application. I have implemented a generic log-based incremental checkpointing for faster checkpoint. It is more described in here: https://flink.apache.org/2022/05/30/improving-speed-and-stability-of-checkpointing

Re:Re: Re: Re: Can a Table Created in Java Code be Queried Using SQL Client?

Can you try to use `SET` command[1] in sql client to set the configs related catalog store? [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#sql-client-configuration [2] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/catalogs/#configure

Job failed to restart on time

I'm using Flink 1.15.4. One time, I found the job failed to restart on time, here are some of my jobManager's log: ``` 2024-12-30 18:50:32,089 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNum

Is there a way to get consolidated metrics at task level

Hello, I am running a Flink streaming pipeline with autscaler on AWS EMR (as a YARN application). My tasks are scaling independently based on load. My question is how can I get the metric, say numRecordsInPerSecond, at the task level. Basically it should sum all the values for this metric at sub t

Re: Java 17 support in Flink

On Java 17 support, what does that actually mean? We are using the official Flink 1.20 image, which is based on Java JRE 17 and in the config file we must specify a ton of those module exports, like “--add-exports=java.base/sun.net.util=ALL-UNNAMED”. That tells me that Flink codebase does not re