Understanding pipelined regions

Hi, I'm quite new to the world of stream and batch processing. I've been reading about pipelined regions in Flink and am quite confused by what it means. My specific problem involves a streaming job that looks like the following: 1. There is a Kafka source that takes in an input data that sets of

Re: Understanding pipelined regions

;> >> >> >> Sincere greetings from the supposed close-by Zurich 😊 >> >> >> >> Thias >> >> >> >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpo

Re: Understanding pipelined regions

any pipelined regions are there? > Since all edges are forward edges, A, B, C and D must have the same > parallelism, and there will be the same pipelined regions as the > parallelism. > > On Wed, Dec 21, 2022 at 5:14 PM Raihan Sunny > wrote: > >> Hello Gen, >> &g

Flink HDFS Integration

Hi, I'm trying to integrate HDFS in Flink for checkpoint and savepoint storage. I have an HDFS cluster running on Docker. I have made the following changes to the flink configuration: state.backend: filesystem state.savepoints.dir: hdfs://namenode:9000/user/root/savepoints state.checkpoints.dir:

Re: Flink HDFS Integration

doop-dependencies > > Best regards, > > Martijn > > On Mon, Jan 9, 2023 at 10:40 AM Raihan Sunny > wrote: > >> Hi, >> >> I'm trying to integrate HDFS in Flink for checkpoint and savepoint >> storage. I have an HDFS cluster running on D

Flink Kubernetes Operator Application mode multiple jobs

Hi everyone, I need some help with deploying multiple jobs from a single main function in Application mode using Flink Kubernetes Operator. As per the documentation [1] it should be possible to use multiple "executeAsync()" to deploy multiple jobs from the same file. This indeed is the case whe

Re: Flink Kubernetes Operator Application mode multiple jobs

rce, deployConfig); } After this modification, everything works perfectly fine as we expected in our environment. On 2024/05/07 05:48:02 Raihan Sunny wrote: > Hi everyone, > > I need some help with deploying multiple jobs from a single main function in > Application mode using Flink Ku

Flink Kubernetes Operator autoscaling GPU-based workload

Hi, I have a workload that depends on the GPU. I have only 1 GPU card. As per the documentation I have added the necessary configurations and can run the GPU workload in standalone REACTIVE mode with as many taskmanager instances as required. I have set the number of task slots to 1 so that a rai

Re: Flink Kubernetes Operator autoscaling GPU-based workload

nt. > After that you can check the autoscaler logs for more info. > > Gyula > > On Tue, Aug 1, 2023 at 10:33 AM Raihan Sunny via user < > user@flink.apache.org> wrote: > >> Hi, >> >> I have a workload that depends on the GPU. I have only 1 GPU card. As per

Flink HDFS with Flink Kubernetes Operator

Hi, I've been using HDFS with Flink for checkpoint and savepoint storage which works perfectly fine. Now I have another use case where I want to read and write to HDFS from the application code as well. For this, I'm using the "pyarrow" library which is already installed with PyFlink as a dependen

Re: Flink HDFS with Flink Kubernetes Operator

decorators. It's true by default, but by setting it to false > Flink won't add this mount. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/#kubernetes-decorator-hadoop-conf-mount-enabled > > Regards, > Mate > > Raihan

Flink metrics to Prometheus on Kubernetes

Hi, I have a few Flink jobs running on Kubernetes using the Flink Kubernetes Operator. By following the documentation [1] I was able to set up monitoring for the Operator itself. As for the jobs themselves, I'm a bit confused about how to properly set it up. Here's my FlinkDeployment configuration

Backpressure causing operators to stop ingestion completely

Hello, I have an aggregator job that experiences backpressure after running for a while and completely stops processing. It doesn't take any further input from the source. Here's a bit of context: - There are 3 producer jobs, all of which write data to a common Kafka topic - The aggregator job r

Re: Re:Backpressure causing operators to stop ingestion completely

imes while other times there's a more or less contant flow of data, is there any specific network buffer settings that I might try to tweak and observe the changes? From: Jake.zhang Sent: Tuesday, October 22, 2024 2:40 PM To: Raihan Sunny ; user Subject: