OffsetCommitMode.Kafka_periodic with checkpointing enabled

Hi Team, I need to get kafka-lag to prepare a graph and its dependent on kafka committed offset. Flink is updating the offsets only after checkpointing to make it consistent. Default Behaviour as per doc : If checkpoint is enabled, but consumer.setCommitOffsetsOnCheckpoints set to false, then off

Re: flink-kubernetes-operator HA k8s RoleBinding for Leases?

Hey! Sounds like a bug :) Could you please open a jira / PR (in case you fixed this already)? Thanks Gyula On Mon, 8 May 2023 at 22:20, Andrew Otto wrote: > Hi, > > I'm trying to enable HA for flink-kubernetes-operator >

Re: akka.remote.OversizedPayloadException after we upgrade to Flink 1.15

Hi Wei, >From the error message, I guess the reason for the issue is that the events sent by SplitEnumerator to the source exceeds the default size of akka. You can add the option 'akka.framesize' to set the akka packet size, or try to decrease the event size. When you use 'FlinkKafkaConsumer' to

RE: MSI Auth to Azure Storage Account with Flink Apache Operator not working

Shammon, I’m still having trouble setting the package in my cluster environment. I have these lines added to my dockerfile mkdir ./plugins/azure-fs-hadoop cp ./opt/flink-azure-fs-hadoop-1.16.0.jar ./plugins/azure-fs-hadoop/ according to the flink docs here (https://nightlies.apache.org/fli

flink-kubernetes-operator HA k8s RoleBinding for Leases?

Hi, I'm trying to enable HA for flink-kubernetes-operator with Helm. We are using namespaced RBAC via watchedNamespaces. I've followed instructions and se

akka.remote.OversizedPayloadException after we upgrade to Flink 1.15

Hi Team, We hit an issue after we upgrade our job from Flink 1.12 to 1.15, there's a consistent akka.remote.OversizedPayloadException after job restarts: Transient association error (association remains live) akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[akka.

Mounting files into native k8s Flink pods

Hi all!Running Flink on k8s (native, not using the operator) I was trying to mount Hadoop configuration files into a certain directory within the pods but I can't manage to do so.My existing deployment consists in a Job spec which launches a .sh file that will trigger the flink run-application bit

Question about RexNodeExtractor formatting UDF names

Hi Flink Team: I have a question about RexNodeExtractor in Flink 1.16.0. I am trying to push down UDFs (with function names in the format ST_XXX, including underscores, e.g. ST_Contains) into TableSourceScan, and I have implemented applyFilters and handling of pushdown functions based on Suppor

Re: Encryption of parameters in flink-conf.yaml

Hi Anuj, To my best knowledge, flink does not provide the encryption strategy support for now. If you are using flink on k8s, it is possible to achieve the encryption of parameters using the init container. You can check this SO

Re: Flink SQL Async UDF

What im curious is about the completable future in the function signature (eval) public final void eval(CompletableFuture> future, Object... keys) is is injected automatically? Best On Mon, May 8, 2023 at 10:02 AM Giannis Polyzos wrote: > Hi Biao, > yeah, this means that since it's a Table Fun

Re: Flink SQL Async UDF

Hi Biao, yeah, this means that since it's a Table Function UDF, I need to combine it with lateral correct? I have tried both approaches to be honest. You can find attached the log output with verbose=true for both queries On Mon, May 8, 2023 at 9:49 AM Biao Geng wrote: > Hi Giannis, > Would you