Thanks Shammon for clarification. On Wed, May 24, 2023 at 11:01 AM Shammon FY <zjur...@gmail.com> wrote:
> Hi Kamal, > > The source will slow down when there is backpressure in the flink job, you > can refer to docs [1] and [2] to get more detailed information about > backpressure mechanism. > > Currently there's no API or Callback in source for users to do some > customized operations for backpressure, but users can collect the metrics > of the job and analysis, for example, the metrics in [1] and [3]. I hope > this can help you. > > [1] > https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed > . > [2] > https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632 > [3] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/ > > On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <kspmm...@gmail.com> wrote: > >> Hello Community, >> >> Can you please share views about the query asked above w.r.t back >> pressure for FileSource APIs for Bulk and Record stream formats. >> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >> conversion. >> >> Rgds, >> Kamal >> >> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <kspmm...@gmail.com> wrote: >> >>> Added Flink community DL as well. >>> >>> ---------- Forwarded message --------- >>> From: Kamal Mittal <kspmm...@gmail.com> >>> Date: Tue, May 23, 2023 at 7:57 AM >>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 >>> To: Shammon FY <zjur...@gmail.com> >>> >>> >>> Hello, >>> >>> Yes, want to take some custom actions and also if there is any default >>> behavior of slowing down sending data in pipeline further or reading data >>> from source somehow? >>> >>> Rgds, >>> Kamal >>> >>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zjur...@gmail.com> wrote: >>> >>>> Hi Kamal, >>>> >>>> If I understand correctly, do you want the source to do some custom >>>> actions, such as current limiting, when there is backpressure in the job? >>>> >>>> Best, >>>> Shammon FY >>>> >>>> >>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <kspmm...@gmail.com> >>>> wrote: >>>> >>>>> Hello Community, >>>>> >>>>> Can you please share views about the query asked above w.r.t back >>>>> pressure for FileSource APIs for Bulk and Record stream formats. >>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >>>>> conversion. >>>>> >>>>> Rgds, >>>>> Kamal >>>>> >>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <kspmm...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello Community, >>>>>> >>>>>> Does FileSource APIs for Bulk and Record stream formats handle back >>>>>> pressure by any way like slowing down sending data in piepline further or >>>>>> reading data from source somehow? >>>>>> Or does it give any callback/handle so that any action can be taken? >>>>>> Can you please share details if any? >>>>>> >>>>>> Rgds, >>>>>> Kamal >>>>>> >>>>>
