Re: flink add multiple sink in sequence

o sinks, you may need to > create a new combined sink and do it yourself. > > On Thu, Jan 5, 2023 at 11:48 PM Great Info wrote: > >> >> I have a stream from Kafka, after reading it and doing some >> transformations/enrichment I need to store the final data stream in th

flink add multiple sink in sequence

I have a stream from Kafka, after reading it and doing some transformations/enrichment I need to store the final data stream in the database and publish it to Kafka so I am planning to add two sinks like below *finalInputStream.addSink(dataBaseSink); // Sink1finalInputStream.addSink( flinkKafkaPr

Utilizing Kafka headers in Flink Kafka connector

I have some flink applications that read streams from Kafka, now the producer side code has introduced some additional information in Kafka headers while producing records. Now I need to change my consumer-side logic to process the records if the header contains a specific value, if the header valu

Flink FaultTolerant at operator level

I have flink job and the current flow looks like below Source_Kafka->*operator-1*(key by partition)->*operator-2*(enrich the record)-*Sink1-Operator* & *Sink2-Operator * With this flow the current problem is at operator-2, the core logic runs here is to lookup some reference status data from redi

Re: Flink running same task on different Task Manager

to set different parallelism for each slotSharingGourp On Thu, Jul 14, 2022 at 10:12 PM Great Info wrote: > -> If so, I think you can set Task1 and Task2 to the same parallelism and > set them in the same slot sharing group. In this way, Task1 and Task2 will > be deployed into the sa

Re: Flink running same task on different Task Manager

arted at > the same time (to ensure the data consistency). > You can get more details about failover strategy in [1] > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/task_failure_recovery/#failover-strategies > > Best, > Lijie > >

Re: Flink running same task on different Task Manager

22年6月15日周三 13:16写道： > >> I don't really understand how task2 reads static data from task1, >> but I think you can integrate the logic of getting static data from http >> in >> task1 into task2 and keep only one kind of task. >> >> Best, >> Weihua >

Flink running same task on different Task Manager

Hi, I have one flink job which has two tasks Task1- Source some static data over https and keep it in memory, this keeps refreshing it every 1 hour Task2- Process some real-time events from Kafka and uses static data to validate something and transform, then forward to other Kafka topic. so far, e

flink Job is throwing depdnecy issue when submitted to clusrer

I have one flink job which reads files from s3 and processes them. Currently, it is running on flink 1.9.0, I need to upgrade my cluster to 1.13.5, so I have done the changes in my job pom and brought up the flink cluster using 1.13.5 dist. when I submit my application I am getting the below error

Re: how to initialize few things at task managers

t;> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/config/#yarn-ship-files >> >> 2022年4月16日 下午11:44，Great Info 写道： >> >> I need to download Keystore and use it while creating the source >> connector, currently, I am over

how to initialize few things at task managers

I need to download Keystore and use it while creating the source connector, currently, I am overriding the open method

Flink does not cleanup some disk memory after submitting jar over rest

I have deployed my own flink setup in AWS ECS. One Service for JobManager and one Service for task Managers. I am running one ECS task for a job manager and 3 ecs tasks for TASK managers. I have a kind of batch job which I upload using flink rest every-day with changing new arguments, when I submi