Execution model when using AsyncIO

2023-09-25 Thread Vignesh Kumar Kathiresan via user
Hi Flink users, I was reading the asyncIO operator and couldn't understand the execution model. I started to think about the flink execution model in general and a few questions arose. 1. For a given subtask(one parallel instance of N chained operators) how do the elements get executed?. Do they

Re: [E] RE: Recommendations on using multithreading in flink map functions in java

2023-08-15 Thread Vignesh Kumar Kathiresan via user
mmendations on using multithreading in flink map > functions in java > > > > Hi, Vignesh > > > > Flink is a distributed parallel computing framework, each MapFunction is > actually a separate thread. If you want more threads to process the data, > you can increase the p

Recommendations on using multithreading in flink map functions in java

2023-08-14 Thread Vignesh Kumar Kathiresan via user
Hello All, *Problem statement * For a given element, I have to perform multiple(lets say N) operations on it. All the N operations are independent of each other. And for achieving lowest latency, I want to do them concurrently. I want to understand what's the best way to perform it in flink?. I u

Flink Kubernetes rocksdb backend volume type

2022-10-27 Thread Vignesh Kumar Kathiresan via user
Hi all, What is the recommended practice of volume mounting for the rocksdb backend in a flink kubernetes standalone setup. We currently have a "local" type. We have local persistent volumes and persistent volume claims for each node's local ssd. And use one PVC in the task manager deployment. A

Re: [E] Re: Presto S3 filesystem access issue - checkpointing - EKS

2022-10-18 Thread Vignesh Kumar Kathiresan via user
__;Iw!!Op6eflyXZCqGR5I!DcCBzrPl2tynm_thNuOqHR0rMjkoeuxgfe2zLSpAbgosfUuu1ulePuoXe-l0_sQCnOR2CvoEeYksEhgi6Ic$> > > Best, > Yanfei > > > > > Vignesh Kumar Kathiresan via user 于2022年10月18日周二 > 03:48写道: > >> Hello all, >> >> I am trying to achieve flink applicat

Presto S3 filesystem access issue - checkpointing - EKS

2022-10-17 Thread Vignesh Kumar Kathiresan via user
Hello all, I am trying to achieve flink application checkpointing to s3 using the recommended presto s3 filesystem plugin. My application is deployed in a kubernetes cluster (EKS) in flink application mode. When I start the application I am getting a forbidden 403 response ```Caused by: com.face

[Flink Kubernetes Operator] FlinkSessionJob crd spec jarURI

2022-09-06 Thread Vignesh Kumar Kathiresan via user
Hi, Have a session cluster deployed in kubernetes. Trying to submit a job following the example given in docs. When I give 1) spec.job.jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar getting Error: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not fi

Re: [E] Re: Kubernetes operator expose UI rest service as NodePort instead of default clusterIP

2022-09-02 Thread Vignesh Kumar Kathiresan via user
/github.com/apache/flink-kubernetes-operator/blob/release-1.1/helm/flink-kubernetes-operator/values.yaml*L34-L38__;Iw!!Op6eflyXZCqGR5I!DTn70pqhttQzBpwxuX_IzpnrchfomQ2-Qj8DIHnahai7tLLDx3MX9lmkcnZvRdz4f-LCTpuVvlqTdV-w$> > > On Thu, Sep 1, 2022 at 11:45 PM Vignesh Kumar Kathiresan via user &l

Re: [E] Re: Kubernetes operator expose UI rest service as NodePort instead of default clusterIP

2022-09-01 Thread Vignesh Kumar Kathiresan via user
erations/ingress/__;!!Op6eflyXZCqGR5I!FZvr8XAEWiEa176d0PfqyLJQoxTGIsDkpV-xqs5JNRCJc3Kv43nm-sa2l275jTPk50K2mjrI3COxrj0op5P5cw$> > > Best, > Yang > > Vignesh Kumar Kathiresan via user 于2022年9月2日周五 > 04:57写道: > >> Hello Flink community, >> >> Need some help with "flink k

Kubernetes operator expose UI rest service as NodePort instead of default clusterIP

2022-09-01 Thread Vignesh Kumar Kathiresan via user
Hello Flink community, Need some help with "flink kubernetes operator" based cluster setup. My flink cluster is set up using the flink-kubernetes-operator in AWS EKS. The required resources(deployments, pods, services, configmaps etc) are created as expected. But the service "*-rest" is created a