Reducing Checkpoint Count for Chain Operator

Hi, We have a job that is reading from kafka and writing some endpoints. The job does not have any shuffling steps. I implement it with multiple steps. Flink chained those operators in one operator in submission time. However I see all operators are doing checkpointing. Is there any way to crea

Re: Reducing Checkpoint Count for Chain Operator

o checkpoint > independently for each task, this is not supported. > > > Best regards, > > Weijie > > > Talat Uyarer via user 于2023年2月1日周三 15:34写道： > >> Hi, >> >> We have a job that is reading from kafka and writing some endpoints. The >> job does no

Re: Reducing Checkpoint Count for Chain Operator

; operator creates their own checkpoint at checkpointing time . Rather than > creating a checkpoint per operator in checkpointing time. Can I have one > checkpoint per OperatorChain? This is my question. > > > > Thanks > > > > On Wed, Feb 1, 2023 at 1:02 AM weijie guo &g

Re: Reducing Checkpoint Count for Chain Operator

not operator > chain. So what is the significance of checkpoint based on the granularity > of operator chain？ > > > > Best regards, > > Weijie > > > > > > Talat Uyarer 于2023年2月2日周四 02:20写道： > > Hi Weijie, > > > > Thanks for replying back. > &

Task Failure Strategy for Adaptive Scheduler

Hi All, We use Flink 1.13 with reactive mode for our streaming jobs. When we have an issue/exception on our pipeline. Flink rescheduled all tasks. Is there any way to reschedule only task that had exceptions ? Thanks

Re: Task Failure Strategy for Adaptive Scheduler

, > D. > > On Tue, Apr 11, 2023 at 4:35 AM Weihua Hu wrote: > >> Hi, >> >> AFAIK, the reactive mode always restarts the whole pipeline now. >> >> Best, >> Weihua >> >> >> On Tue, Apr 11, 2023 at 8:38 AM Talat Uyarer via user < >&

Re: Task Failure Strategy for Adaptive Scheduler

WpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=io1uqfos6mZy0_MjhX-EMn6Kv1O-J51WEdXoLFL2-JdSmUURlkVSK9Jo06K7PXbt&s=duMv5JJ1nSCwJNlA1eG3uaopaPpgwqrBfBnbegFEb7s&e=> >>> >>> Best, >>> D. >>> >>> On

Flink Kubernetes Operator Scale Issue

Hi All, We are using Flink Kubernetes Operator on our production. We have 3k+ jobs in standalone mode. But after 2.5k jobs operator getting slow. Now when we submit a job it takes 10+ minutes to the job runs. Does anyone use similar scale or more job ? Now we run as a single pod. Does operator su

Re: Evolving Flink SQL statement set and restoring from savepoints

Hi Keith, When you add a new insert statement to your EXECUTE STATEMENT you change your job graph with independent two graphs.Unfortunately, Flink doesn't currently provide a way to directly force specific UIDs for operators through configuration or SQL hints. This is primarily due to how Flink's

Re: Exception in Flink 1.18 (Time should be non negative)

Hi Lasse, If there's a significant difference in the system time between Flink TaskManagers, it can lead to negative time calculations when comparing timestamps from different sources. On Mon, May 6, 2024 at 5:40 AM Lasse Nedergaard < lassenedergaardfl...@gmail.com> wrote: > Hi. > > In Flink job

Re: Issue in PrefetchCount

Hi ajay, When you have 3 parallelisms you will have 3 independent clients. If you want to keep prefetch count 3 you need to set setRequestedChannelMax as 1 and setParallelism 3. So All 3 clients can have one connection. Talat On Tue, May 7, 2024 at 5:52 AM ajay pandey wrote: > Hi Flink Team, >

Apache Beam Summit 2024 is September 4 2024!

Hey Flink Users, Heads up! The Apache Beam Summit 2024 is happening next month( September 04, 2024) at the Google Campus in Sunnyvale, CA. It's two days packed with learning and networking with the Beam community. Key highlights: - Improving Stability for Running Python SDK with Flink Runner: im