Re: Access to collector in the process function

Hi, Oscar The collector object is created in [1] open method, so we can't get it in ProcessFunciton open method. One possible way is you change the source code, place it in ProcessFunction#Context. [1] https://github.com/apache/flink/blob/aa8d93ea239f5be79066b7e5caad08d966c86ab2/flink-streaming-j

Re: Job graph

Hi, Nikolaos As far as I know, JobGraph is a relatively low-level concept, and currently we don't expose it directly to users, and don't provide a direct Restful API to get it from JobManager. Why do you need to get JobGraph and what is your real need? Best, Ron Nikolaos Paraskakis 于2023年8月31日周

Re: 退订

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-list

Re: flink k8s operator - problem with patching seession cluster

Just want to broth this up in case it was missed in the other messages/queries :) TL:DR How to add TM to Flink Session cluster via Java K8s client if Session Cluster has running jobs? Thanks, Krzysztof pt., 25 sie 2023 o 23:48 Krzysztof Chmielewski < krzysiek.chmielew...@gmail.com> napisał(a):

Re: Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

I agree with Yaroslav, generally speaking PVs are not necessary or even recommended for RocksDB because the state doesn't need to be shared, recovered later anyways. It's usually faster and cheaper to go with instance level SSDs. Gyula On Wed, Aug 30, 2023 at 8:37 PM Yaroslav Tkachenko wrote: >

Re: Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

It depends on your requirements. Personally, I don't use PVs and, instead, mount a volume from a host with a fast instance-level SSD. On Wed, Aug 30, 2023 at 11:26 AM Tony Chen wrote: > We used to have a Persistent Volume (PV), attached to the pod, for storing > the RocksDB data while using the

Re: Uneven TM Distribution of Flink on YARN

No. we don't use yarn.taskmanager.node-label Best Lu On Tue, Aug 29, 2023 at 12:17 AM Geng Biao wrote: > Maybe you can check if you have set yarn.taskmanager.node-label for some > flink jobs? > > Best, > Biao Geng > > 发送自 Outlook for iOS > --

Re: Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

We used to have a Persistent Volume (PV), attached to the pod, for storing the RocksDB data while using the GoogleCloudPlatform operator. For the Apache flink-kubernetes-operator, do the pods need a PV attached to it to use RocksDB? If not, do you have recommendations on memory configuration for th

Re: Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

Hi! Rocksdb is supported and every other state backend as well. You can simply set this in you config like before :) Cheers Gyula On Wed, 30 Aug 2023 at 19:22, Tony Chen wrote: > Hi Flink Community, > > Does the flink-kubernetes-operator support RocksDB as the state backend > for FlinkDeploym

Re: Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

Hey Tony, Pretty much all Flink configuration is supported, including the RocksDB state backend. On Wed, Aug 30, 2023 at 9:05 AM Tony Chen wrote: > Hi Flink Community, > > Does the flink-kubernetes-operator support RocksDB as the state backend > for FlinkDeployment? > > We have some Flink appli

Enable RocksDB in FlinkDeployment with flink-kubernetes-operator

Hi Flink Community, Does the flink-kubernetes-operator support RocksDB as the state backend for FlinkDeployment? We have some Flink applications that have large states, and we were able to deal with these large states in the past with RocksDB. If there is no support for RocksDB, are there any rec

Re: Blue green deployment with Flink Apache Operator

Hey! I don't know if anyone has implemented this or not but one way to approach this problem (and this may not be the right way, just an idea :) ) is to add a new Custom Resource type that sits on top of the FlinkDeployment / FlinkSessionJob resources and add a small controller for this. This new

Job graph

Hello folks, I am trying to get the job graph of a running flink job. I want to use flink libraries. For now, I have the RestClusterClient and the job IDs. Tell me please how to get the job graph. Thank you.

Access to collector in the process function

Hi! We would like to use hexagonal architecture in our design and treat the collector as an output port when sending events from the use case. For that, we would like to call an interface from the use case that effectively sends the event ultimately via out.collect The problem is that for instant

Blue green deployment with Flink Apache Operator

Hi, >From https://issues.apache.org/jira/browse/FLINK-29199 it seems that support for blue green deployment will not be supported or will not happen soon. I'd like to know if some of you have built a custom mechanism on top of this operator to support the blue green deployment and if you would ha

Semantics of purging with global windows

Hello, According to the javadoc of TriggerResult.PURGE, "All elements in the window are cleared and the window is discarded, without evaluating the window function or emitting any elements." However, I've noticed that using a GlobalWindow (with a custom trigger) followed by an AggregateFunction wi

RE: Rate Limit / Throttle Data to Send

Hi Patricia, What you try to implement can be achieved out-of-the-box by windowing. I assume these packets of 100 event are by key but globally. In that case use non-keyed windowing [1] with count trigger (100) [3] and maybe add a processing time trigger if it takes too long time to collect all

Flink local mini cluster is causing memory leak when triggered multiple times

Hi Team, We have a data pipeline which is built using Apache Beam SDK and we use Apache Flink Runner to execute Beam pipelines. We use the local embedded execution mode of Flink for running the pipelines. Currently,