Fwd: Backpressure handling in FileSource APIs - Flink 1.16

Added Flink community DL as well. -- Forwarded message - From: Kamal Mittal Date: Tue, May 23, 2023 at 7:57 AM Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 To: Shammon FY Hello, Yes, want to take some custom actions and also if there is any default behavi

Re: Data Stream Processing applications testing

Seems interesting, thanks for sharing On Tue, May 23, 2023 at 2:04 AM Alexandre Strapacao Guedes Vianna < a...@cin.ufpe.br> wrote: > Hey everyone, > > I wanted to share my latest paper, "A Grey Literature Review on Data > Stream Processing Applications Testing," in the Journal of Systems and > So

Re: IRSA with Flink S3a connector

Hello, Please provide some pointers on this issue. Thanks !! Regards Anuj On Fri, May 19, 2023 at 1:34 PM Anuj Jain wrote: > Hi Community, > Looking forward to some advice on the problem. > > I also found this similar Jira, but not sure if a fix has been done for > the Hadoop connector - can s

Re: Question about Flink exception handling

Thanks for your response. For simplicity, I want to capture exceptions in a centralized manner and log them for further analysis, without interrupting the job's execution or causing it to restart. On Tue, May 23, 2023 at 6:31 AM Shammon FY wrote: > Hi Sharif, > > I would like to know what do yo

Re: Backpressure handling in FileSource APIs - Flink 1.16

Hi Kamal, If I understand correctly, do you want the source to do some custom actions, such as current limiting, when there is backpressure in the job? Best, Shammon FY On Mon, May 22, 2023 at 2:12 PM Kamal Mittal wrote: > Hello Community, > > Can you please share views about the query asked

Re: Question about Flink exception handling

Hi Sharif, I would like to know what do you want to do with the exception after catching it? There are different ways for different requirements, for example, Flink has already reported these exceptions. Best, Shammon FY On Mon, May 22, 2023 at 4:45 PM Sharif Khan via user wrote: > Hi, commun

Data Stream Processing applications testing

Hey everyone, I wanted to share my latest paper, "A Grey Literature Review on Data Stream Processing Applications Testing," in the Journal of Systems and Software (JSS), Elsevier. This paper provides unique industry insights, addresses the challenges faced in Data Stream Processing (DSP) applicat

Re: Flink Kubernetes Operator lifecycle state count metrics question

Also! I do have 2 FlinkDeployments deployed with this operator, but they are in different namespaces, and each of the per namespace metrics reports that it has 2 Deployments in them, even though there is only one according to kubectl. Actually...we just tried to deploy a change (enabling some che

Re: Flink Kubernetes Operator lifecycle state count metrics question

Oh, FWIW, I do have operator HA enabled with 2 replicas running, but in my examples there, I am curl-ing the leader flink operator pod. On Mon, May 22, 2023 at 2:47 PM Andrew Otto wrote: > Hello! > > I'm doing some grafana+prometheus dashboarding for > flink-kubernetes-operator. Reading metri

Flink Kubernetes Operator lifecycle state count metrics question

Hello! I'm doing some grafana+prometheus dashboarding for flink-kubernetes-operator. Reading metrics docs , I see that I have nice per k8s namespace lifecycle current count gauge metrics in Prometheus. Using kubectl, I can see that I have one FlinkDeployment

Re: Maven plugin to detect issues early on

cc user ML to get more attention, since the plugin will be used by Flink application developers. Best regards, Jing On Mon, May 22, 2023 at 3:32 PM Jing Ge wrote: > Hi Emre, > > Thanks for clarifying it. Afaiac, it is a quite interesting proposal, > especially for Flink job developers who are h

Question about Flink exception handling

Hi, community. Can anyone please let me know? 1. What is the best practice in terms of handling exceptions in Flink jobs? 2. Is there any way to catch exceptions globally in Flink jobs? Basically, I want to catch exceptions from any operators in one place (globally). my expectation is let's say