Re: [Question] How to scale application based on 'reactive' mode

Hello, Thanks for your notice. 1. In "Flink 1.18 + non-reactive", is parallelism being changed by the number of TM? 2. In the document( https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.6/docs/custom-resource/autoscaler/), it said "we are not using any container memory /

Re: Job graph

This may or may not help, but you can get the execution plan from inside the client, by doing something like this (I printed the plan to stderr): ... System.err.println(env.getExecutionPlan()); env.execute("my job"); The result is a JSON-encoded representation of the job graph, which

Re: Blue green deployment with Flink Apache Operator

Back in 2020, there was a Flink Forward talk [1] about how Lyft was doing blue green deployments. Earlier (all the way back in 2017) Drivetribe described [2] how they were doing so as well. David [1] https://www.youtube.com/watch?v=Hyt3YrtKQAM [2] https://www.ververica.com/blog/drivetribe-cqrs-ap

Re: Flink local mini cluster is causing memory leak when triggered multiple times

An update: The extra memory that is being acquired/kept after each pipeline run seems to be coming from off-heap space, especially using `Unsafe.allocateMemory`. I have added some notes here but the TL;DR; is that adding

Re: [Question] How to scale application based on 'reactive' mode

Pretty much, except that with Flink 1.18 autoscaler can scale the job in place without restarting the JM (even without reactive mode ) So actually best option is autoscaler with Flink 1.18 native mode (no reactive) Gyula On Fri, 1 Sep 2023 at 13:54, Dennis Jung wrote: > Thanks for feedback. >

Re: [Question] How to scale application based on 'reactive' mode

Thanks for feedback. Could you check whether I understand correctly? *Only using 'reactive' mode:* By manually adding TaskManager(TM) (such as using './bin/taskmanager.sh start'), parallelism will be increased. For example, when job parallelism is 1 and TM is 1, and if adding 1 new TM, JobManager

Re: [Question] How to scale application based on 'reactive' mode

I would look at reactive scaling as a way to increase / decrease parallelism. It’s not a way to automatically decide when to actually do it as you need to create new TMs . The autoscaler could use reactive mode to change the parallelism but you need the autoscaler itself to decide when new resour

Re: Send data asynchronously to a 3rd party via SinkFunction

hi, patricia I suggest using the generic asynchronous base sink. https://flink.apache.org/2022/03/16/the-generic-asynchronous-base-sink/ Best, Feng On Fri, Sep 1, 2023 at 6:07 PM patricia lee wrote: > > I'd like to ask if there is a way to send data to a vendor (SDK plugin, > which is also a

Re: [Question] How to scale application based on 'reactive' mode

For now, the thing I've found about 'reactive' mode is that it automatically adjusts 'job parallelism' when TaskManager is increased/decreased. https://www.slideshare.net/FlinkForward/autoscaling-flink-with-reactive-mode Is there some other feature that only 'reactive' mode offers for scaling? T

Send data asynchronously to a 3rd party via SinkFunction

I'd like to ask if there is a way to send data to a vendor (SDK plugin, which is also an HTTP request) asynchronously in flink 1.17? After transformation on the data, I usually collate them as a List to my custom SinkFunction. I initialized a CompleteableFuture inside the invoke() method. However

Re: [Question] How to scale application based on 'reactive' mode

Hello, Thank you for your response. I have few more questions in following: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/ *Reactive Mode configures a job so that it always uses all resources available in the cluster. Adding a TaskManager will scale up