Re: MSI Auth to Azure Storage Account with Flink Apache Operator not working

2023-05-16 Thread Surendra Singh Lilhore
Hi DEROCCO, Flink uses shaded jars for the Hadoop Azure Storage plugin, so in order to correct the ClassNotFoundException, you need to adjust the configuration. Please configure the MSITokenProvider as shown below. fs.azure.account.oauth.provider.type: *org.apache.flink.fs.shaded.hadoop3.org.apac

Re: Query on RestartPipelinedRegionFailoverStrategy

2023-05-16 Thread Shammon FY
Hi Prabhu, Whether the tasks are in the same region depends on the DistributionPattern between upstream and downstream Operators. For example, if the DistributionPattern from A to B is ALL_TO_ALL, all subtasks for A and B will be in the same range. Otherwise, if the DistributionPattern is POINTWIS

RE: MSI Auth to Azure Storage Account with Flink Apache Operator not working

2023-05-16 Thread Ivan Webber via user
When you create your cluster you probably need to ensure the following settings are set. I briefly looked into MSI but ended up using Azure Key Vault with CSI-storage driver for initial prototype (https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/aks/csi-secrets-store-driver.md#upgr

Re: [DISCUSS] Status of Statefun Project

2023-05-16 Thread Galen Warren via user
Hi Martijn, since you opened this discussion thread, I'm curious what your thoughts are in light of the responses? Thanks. On Wed, Apr 19, 2023 at 1:21 PM Galen Warren wrote: > I use Apache Flink for stream processing, and StateFun as a hand-off point >> for the rest of the application. >> It se

Dynamin Windowing in with Pyflink

2023-05-16 Thread Nawaz Nayeem via user
Hey, I’ve been trying to emulate the behavior of a dynamic window, as Flink does not support dynamic window sizes. My operator inherits from KeyedProcessFunction, and I’m only using KeyedStates to manipulate the window_size. I’m clearing the KeyedStates when my bucket(window) is complete, to reset

(无主题)

2023-05-16 Thread 湘晗刚
Hi all  Linux system rngd.service ,entropy_avail is small cause flink task back pressure  high,how to solve other than restart rngd.service every time Thanks in advance Tian

Re: Query on RestartPipelinedRegionFailoverStrategy

2023-05-16 Thread Prabhu Joseph
Yes i expected the same. But all the tasks goes into one region and RestartPipelinedRegionFailoverStrategy restarts all of them. I see this strategy does not make any difference from RestartAllFailoverStrategy in stream execution mode. It could only help in Batch execution mode where Blocking resul

Best practices for handling exceptions in Apache Flink 1.16.0 and global exception handling

2023-05-16 Thread Sharif Khan via user
I'm currently working on a Flink job using version 1.16.0 of Apache Flink and I would like to know what are the best practices for handling exceptions in my application. I'm interested in learning about recommended approaches for handling exceptions in Flink, and how to ensure the robustness and re

[no subject]

2023-05-16 Thread Sharif Khan via user
sharif.k...@selise.ch -- SELISE Group Zürich: The Circle 37, 8058 Zürich-Airport, Switzerland Munich: Tal 44, 80331 München, Germany Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab Emirates Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh Thimphu