Flink task manager pod auto scaler

Hello, Is it possible to have dependency of auto scaler module only for having scaling metrices and do task manager pod autoscaling based on those by using HPA or KEDA rather than using complete Flink Kubernetes operator? Asking this if any way to introduce Flink Kubernetes operator in phased m

RE: Flink metrices for pending records

ttps://aka.ms/LearnAboutSenderIdentification> Hi Kamal, Each Source implementation may provide it's own set of metrics. Kafka, for instance, has a "pendingRecords" metrics. See https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/metrics/ On Wed, 21 May 2025 at 06:39,

Flink metrices for pending records

Hello, Can you please help to know if any metrices for "pending records" at source level is exposed by flink - 1.20? At below link there is nothing like that. Metrics | Apache Flink Rgds, Kamal

Re: Flink job and task manager pods auto scaling

Plz give input for below. Get Outlook for Android<https://aka.ms/AAb9ysg> From: Kamal Mittal via user Sent: Tuesday, May 20, 2025 8:31:20 AM To: User Subject: Flink job and task manager pods auto scaling Hello, Couple of questions below for Flin

Flink job and task manager pods auto scaling

Hello, Couple of questions below for Flink 1.20, please give input. 1. While using "Adaptive scheduler" for streaming job auto scaling, will the complete job re-start due to new parallelism? Any special situation? 1. While using "Adaptive scheduler" for streaming job auto scaling as wel

Flink job autoscaling metrices

Hello, Does flink operator/task metrices reset if job is auto scaled? Rgds, Kamal

Flink task manager pod auto scaling

Hello, Does flink supports vertical task manager pod auto scaling? Rgds, Kamal

RE: Flink task manager PODs autoscaling - K8s installation

task manager PODs autoscaling - K8s installation Please give input for below. From: Kamal Mittal via user mailto:user@flink.apache.org>> Sent: 16 May 2025 06:42 To: Zhanghao Chen mailto:zhanghao.c...@outlook.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: RE:

RE: Flink task manager PODs autoscaling - K8s installation

Please give input for below. From: Kamal Mittal via user Sent: 16 May 2025 06:42 To: Zhanghao Chen ; user@flink.apache.org Subject: RE: Flink task manager PODs autoscaling - K8s installation Thanks for describing. Just to know that why K8s HPA doesn't work well with Flink? Any limita

RE: Flink task manager PODs autoscaling - K8s installation

ain/docs/custom-resource/autoscaler/ [3] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#externalized-declarative-resource-management Best, Zhanghao Chen ____ From: Kamal Mittal via user mailto:user@flink.apache.org>> Sent: Mo

RE: Flink task manager PODs autoscaling - K8s installation

Please give input for below. From: Kamal Mittal via user Sent: 12 May 2025 11:05 To: user@flink.apache.org Subject: Flink task manager PODs autoscaling - K8s installation Hello, For Flink Task Manager K8s PODs autoscaling, it is mandatory to use Flink Kubernetes Operator/HPA or only flink

Flink task manager PODs autoscaling - K8s installation

Hello, For Flink Task Manager K8s PODs autoscaling, it is mandatory to use Flink Kubernetes Operator/HPA or only flink provided scheduler is sufficient as below? jobmanager.scheduler: adaptive Also what is the additional work done by below configuration? job.autoscaler.enabled: "true" Please

Flink kafka source with OAuth authentication token refresh

Hello, I am having a scenario where in flink kafka source (org.apache.flink.connector.kafka.source.KafkaSource) after fetching kafka record, there is need for OAuth authentication for one of the 3rd party REST API. This API needs authentication token and has some expiry time associated with it

Flink breaking changes in 2.0

Hello, As per link below for Flink 2.0, there are many breaking changes which will impact users. Please give inputs for below. 1. What are the alternatives for the APIs/interfaces/methods/constants which are supposed to be removed? Any documentation related to this, please share? 1. I

java.lang.Object serialization by Kryo

Hello, java.lang.Object is serialized by Kryo as it is taken as generic data type by Flink. Is there any way to switch it to some other serializer for performance improvement or not? Also can you please give some documentation for reference around it? I profiled my application and then found

RE: Flink performance

Hello Community, Please share info. for below query. Rgds, Kamal From: Kamal Mittal via user Sent: Monday, March 11, 2024 1:18 PM To: user@flink.apache.org Subject: Flink performance Hello, Can you please point me to documentation if any such available where flink talks about or documented

Flink performance

Hello, Can you please point me to documentation if any such available where flink talks about or documented performance numbers w.r.t certain use cases? Rgds, Kamal

RE: Flink custom parallel data source

ources and do processing in the subsequent operators which you can scale independently from the source parallelism. Can you describe what you are trying to achieve? Best, Alexander Fedulov On Tue, 31 Oct 2023 at 07:22, Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hel

RE: Flink custom parallel data source

in the subsequent operators which you can scale independently from the source parallelism. Can you describe what you are trying to achieve? Best, Alexander Fedulov On Tue, 31 Oct 2023 at 07:22, Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello Community, I need to have

Flink custom parallel data source

Hello Community, I need to have a custom parallel data source (Flink ParallelSourceFunction) for fetching data based on some custom logic. In this source function, opening multiple threads via java thread pool to distribute work further. These threads share Flink provided 'SourceContext' and co

RE: About Flink parquet format

Hello, Can you please share that why Flink is not able to handle exception and keeps on creating files continuously without closing? Rgds, Kamal From: Kamal Mittal via user Sent: 21 September 2023 07:58 AM To: Feng Jin Cc: user@flink.apache.org Subject: RE: About Flink parquet format Yes

RE: About Flink parquet format

nting. Best, Feng On Thu, Sep 14, 2023 at 7:27 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Tried parquet file creation with file sink bulk writer. If configured parquet page size as low as 1 byte (allowed configuration) then flink keeps on creating multiple ‘in-

RE: About Flink parquet format

le when checkpointing. Best, Feng On Thu, Sep 14, 2023 at 7:27 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Tried parquet file creation with file sink bulk writer. If configured parquet page size as low as 1 byte (allowed configuration) then flink keeps on cr

RE: About Flink parquet format

Mittal Cc: user@flink.apache.org Subject: Re: About Flink parquet format Hi Kamal Check if the checkpoint of the task is enabled and triggered correctly. By default, write parquet files will roll a new file when checkpointing. Best, Feng On Thu, Sep 14, 2023 at 7:27 PM Kamal Mittal via user

About Flink parquet format

Hello, Tried parquet file creation with file sink bulk writer. If configured parquet page size as low as 1 byte (allowed configuration) then flink keeps on creating multiple 'in-progress' state files and with content only as 'PAR1' and never closed the file. I want to know what is the reason o

RE: Re-start strategy without checkpointing enabled

and writes to mysql. If the job failover without checkpointing, the tasks will consume Kafka from the earliest offset again. I think it is best to enable checkpointing to restart job from the position where the job stopped reading. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org&

Re-start strategy without checkpointing enabled

Hello, If checkpointing is NOT enabled and re-start strategy is configured then flink retries the whole job execution i.e. enabling checkpointing is must for re-try or not? Rgds, Kamal

RE: Flink AVRO to Parquet writer - Row group size/Page size

Hello Community, Please help me to find out inputs for below query. Rgds, Kamal From: Kamal Mittal via user Sent: 18 August 2023 08:04 AM To: user@flink.apache.org Subject: RE: Flink AVRO to Parquet writer - Row group size/Page size Hello Community, Please share views for below. Rgds, Kamal

RE: Flink AVRO to Parquet writer - Row group size/Page size

Mittal via user mailto:user@flink.apache.org>> Sent: 16 August 2023 04:35 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Flink AVRO to Parquet writer - Row group size/Page size Hello, For Parquet, default row group size is 128 MB and Page size is 1MB but Flink

RE: Flink AVRO to Parquet writer - Row group size/Page size

Hello Community, Please share views for below. Rgds, Kamal From: Kamal Mittal via user Sent: 16 August 2023 04:35 PM To: user@flink.apache.org Subject: Flink AVRO to Parquet writer - Row group size/Page size Hello, For Parquet, default row group size is 128 MB and Page size is 1MB but Flink

Flink AVRO to Parquet writer - Row group size/Page size

Hello, For Parquet, default row group size is 128 MB and Page size is 1MB but Flink Bulk writer using file sink create the files based on checkpointing interval only. So is there any significance of configured row group size and page size for Flink parquet bulk writer? How Flink uses these two

Global/Shared objects

Hello, Is it possible to create global/shared objects like static which are shared among slots in a task manager? Is it ok to create such objects in flink? Rgds, Kamal

RE: Flink operator task opens threads internally

. Best, Ron Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月5日周六 10:50写道： Hello, Can you please share views for below? Rgds, Kamal From: Kamal Mittal mailto:kamal.mit...@ericsson.com>> Sent: 04 August 2023 11:19 AM To: Kamal Mittal mailto:kamal.mit...@ericsson.com&

RE: Flink operator task opens threads internally

connection which can be 1000 in nos. Flink server socket application is running over K8s environment. Do let me know please that it is ok to open multiple threads for accepting client socket connections inside flink application? Rgds, Kamal From: Kamal Mittal via user mailto:user@flink.apache.org

Flink task manager failure scenario

Hello, How flink behaves in case one of task manager POD fails out of a set of task managers PODs over say K8s environment? In my case, job remains in failed state even after giving re-start strategy with fixed delay (5 sec) and no. of attempts (5) with error as "Could not acquire minimum reso

RE: Flink operator task opens threads internally

application? Rgds, Kamal From: Kamal Mittal via user Sent: 03 August 2023 09:27 AM To: Paul Lam Cc: user@flink.apache.org Subject: RE: Flink operator task opens threads internally Hello, We have a client sending TCP traffic towards Flink application and to support that there is server socket

RE: Flink netty connector for TCP source

d events to the source subtasks. Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY

RE: Flink netty connector for TCP source

Hello Shammon, Please have a look for below and share views. Rgds, Kamal From: Kamal Mittal via user Sent: 02 August 2023 08:02 AM To: Shammon FY ; user@flink.apache.org Subject: RE: Flink netty connector for TCP source Thanks Shammon. Purpose of opening server socket in Split Enumerator was

RE: Flink operator task opens threads internally

Flink jobs, as the socket server address is dynamic and requires service discovery. Would you like to share more about the background? Best, Paul Lam 2023年8月3日 10:26，Kamal Mittal via user mailto:user@flink.apache.org>> 写道： Hello Community, Please share views for the below mail. Rgds, Kamal

RE: Flink operator task opens threads internally

Hello Community, Please share views for the below mail. Rgds, Kamal From: Kamal Mittal via user Sent: 02 August 2023 08:19 AM To: user@flink.apache.org Subject: Flink operator task opens threads internally Hello Community, I have a operator pipeline like as below, is it ok if "source&

Flink operator task opens threads internally

Hello Community, I have a operator pipeline like as below, is it ok if "source" task opens threads by using java thread pool and parallelize the work? This is needed for accepting multiple client socket connections in "single custom source server socket function". Single Custom source server s

RE: Flink netty connector for TCP source

f you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits. Best, Shammon FY On Tue, Aug 1, 2023 at 12:02 PM Kamal Mitta

No. of task managers vs No. of task slots

Hell Community, Need info. for below - 1. How many task managers a job manager can handle? Is there any upper limit also? 1. How to decide no. of task managers, is there any way? 1. What is the difference between high no. of task managers vs high no. of task slots (with low no.

RE: Flink netty connector for TCP source

SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user mailto:user@flink.apache.org>> 于2023年8月1日周二 10:43写道： Thanks. I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute

RE: Flink netty connector for TCP source

Stream` and you can refer to [1] for more details. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program Best, Shammon FY On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector

RE: Custom TCP server socket source

Hello, Can you please share views about below mail? Rgds, Kamal From: Kamal Mittal via user Sent: 28 July 2023 07:59 AM To: Martijn Visser Cc: user@flink.apache.org Subject: RE: Custom TCP server socket source Hello Martijn, I followed the same link and created Enumerator but that

RE: Custom TCP server socket source

know what else you've tried already and what issues you've encountered. Best regards, Martijn On Thu, Jul 27, 2023 at 6:54 AM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, I need to write “Custom server socket source” which accepts client connections over a

Custom TCP server socket source

Hello, I need to write "Custom server socket source" which accepts client connections over a port. 1. How to scale it across task managers with parallelism <= no. of task managers and with same single port 2. This is needed w.r.t Kubernetes POD deployment model where each POD is running

RE: Flink netty connector for TCP source

at 4:26 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal

Flink netty connector for TCP source

Hello, Does flink provides netty connector for custom TCP source? Any documentation details please share? Rgds, Kamal

RE: TCP server socket with Kubernetes Cluster

below mail. Rgds, Kamal From: Kamal Mittal via user mailto:user@flink.apache.org>> Sent: 21 July 2023 02:02 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: TCP server socket with Kubernetes Cluster Hello, Created a TCP server socket single source function and it is

RE: TCP server socket with Kubernetes Cluster

Hello Community, Please share views for below mail. Rgds, Kamal From: Kamal Mittal via user Sent: 21 July 2023 02:02 PM To: user@flink.apache.org Subject: TCP server socket with Kubernetes Cluster Hello, Created a TCP server socket single source function and it is opened on a single POD

TCP server socket with Kubernetes Cluster

Hello, Created a TCP server socket single source function and it is opened on a single POD (taskmanager) of Kubernetes cluster out of a set of PODs (taskmanager) by Flink. Is there any way to know on which POD (taskmanager) it is opened? Does Flink gives any such information? This is needed

RE: About cluster.evenly-spread-out-slots

spread-out-slots` is set to true. Best, Shammon FY On Thu, Jul 20, 2023 at 3:47 PM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, If property “cluster.evenly-spread-out-slots” is set to TRUE then Flink guarantees that same operator multiple tasks are never executed/

About cluster.evenly-spread-out-slots

Hello, If property "cluster.evenly-spread-out-slots" is set to TRUE then Flink guarantees that same operator multiple tasks are never executed/scheduled on same task manager? Definitely this will depend upon parallelism value used for an operator and no. of task slots available. Like in below

RE: TCP Socket stream scalability

Hello Community, Please share views for below mail. Rgds, Kamal From: Kamal Mittal via user Sent: 14 July 2023 12:55 PM To: user@flink.apache.org Subject: TCP Socket stream scalability Hello, TCP Socket stream can be scaled across task managers similarly to file enumerator and source reader

TCP Socket stream scalability

Hello, TCP Socket stream can be scaled across task managers similarly to file enumerator and source reader below? Job is submitted with TCP socket source function and a socket will bind on a port once and by a task manager. Is it possible to open socket at job manager and then scale / divide t

Checkpointed data size is zero

Hello Community, I have a requirement to read data coming over TCP socket stream and for the same written one custom source function reading data by TCP socket. Job is running successfully but in flink dashboard checkpoint overview, checkpointed data size is 0. Can you please help if there is

Flink TCP socket custom source - savepoint cannot be taken

Hello Community, I have a requirement to read data coming over TCP socket stream and for the same written one custom source function reading data by TCP socket. Job is running successfully but while trying to take a savepoint, error comes that savepoint cannot be taken. Is there any limitation

RE: Flink TCP custom source - secured server socket

would be good to stress this out. Best, Jan On 6/29/23 12:53, Kamal Mittal via user wrote: Hello Community, I have created TCP stream custom source and reading data from TCP stream source. But this TCP connection needs to be secured i.e. SSL based, query is how to configure/provide certificates

Flink TCP custom source - secured server socket

Hello Community, I have created TCP stream custom source and reading data from TCP stream source. But this TCP connection needs to be secured i.e. SSL based, query is how to configure/provide certificates via Flink for Client-Server secured TCP connection? Rgds, Kamal

RE: Flink bulk and record file source format metrices

counter) in a FilterFunction immediately following the source. — Ken On Jun 16, 2023, at 2:11 AM, Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Any way-forward, please suggest. Rgds, Kamal From: Kamal Mittal via user mailto:user@flink.apache.org>> Sent: 15

RE: Flink bulk and record file source format metrices

Hello, Any way-forward, please suggest. Rgds, Kamal From: Kamal Mittal via user Sent: 15 June 2023 10:39 AM To: Shammon FY Cc: user@flink.apache.org Subject: RE: Flink bulk and record file source format metrices Hello, I need one counter matric for no. of corrupt records while decoding

RE: Flink bulk and record file source format metrices

Mittal via user mailto:user@flink.apache.org>> wrote: Hello, Using Flink record stream format file source API as below for parquet records reading. FileSource.FileSourceBuilder source = FileSource.forRecordStreamFormat(streamformat, path); source.monitorContinuously(Duration.ofMillis(1));

Flink File source with Parquet reader API - 1.16.x

Hello, Flink File source API as below used for reading AVRO - Parquet records. If any record comes "null or is made null in between of a split in AVROParquetReader > read() method" due to "bad record" situation then whole file split is stopped/discarded. Can you please guide about this that ho

Flink bulk and record file source format metrices

Hello, Using Flink record stream format file source API as below for parquet records reading. FileSource.FileSourceBuilder source = FileSource.forRecordStreamFormat(streamformat, path); source.monitorContinuously(Duration.ofMillis(1)); Want to log/generate metrices for corrupt records and

RE: Parquet decoding exception - Flink 1.16.x

: Kamal Mittal via user Sent: 07 June 2023 05:48 PM To: Martijn Visser Cc: Kamal Mittal via user Subject: RE: Parquet decoding exception - Flink 1.16.x Hello, Metrices link given in below mail doesn’t give any way to create metrices for source function right? I am using below Flink API to read

RE: Parquet decoding exception - Flink 1.16.x

via user Subject: Re: Raise alarm for corrupt records Hi Kamal, Documentation on the metrics can be found at https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ Best regards, Martijn On Wed, Jun 7, 2023 at 10:13 AM Kamal Mittal via user mailto:user@flink.apache.org

RE: Raise alarm for corrupt records

alerting solution. Best regards, Martijn On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello Community, Is there any way Flink provides out of box to raise alarm for corrupt records (e.g. due to decoding failure) in between of running data pi

Raise alarm for corrupt records

Hello Community, Is there any way Flink provides out of box to raise alarm for corrupt records (e.g. due to decoding failure) in between of running data pipeline and send this alarm to outside of task manager process? Rgds, Kamal