For now, the thing I've found about 'reactive' mode is that it automatically adjusts 'job parallelism' when TaskManager is increased/decreased.
https://www.slideshare.net/FlinkForward/autoscaling-flink-with-reactive-mode Is there some other feature that only 'reactive' mode offers for scaling? Thanks. Regards. 2023년 9월 1일 (금) 오후 4:56, Dennis Jung <inylov...@gmail.com>님이 작성: > Hello, > Thank you for your response. I have few more questions in following: > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/ > > *Reactive Mode configures a job so that it always uses all resources > available in the cluster. Adding a TaskManager will scale up your job, > removing resources will scale it down. Flink will manage the parallelism of > the job, always setting it to the highest possible values.* > => Does this mean when I add/remove TaskManager in 'non-reactive' mode, > resource(CPU/Memory/Etc.) of the cluster is not being changed? > > *Reactive Mode restarts a job on a rescaling event, restoring it from the > latest completed checkpoint. This means that there is no overhead of > creating a savepoint (which is needed for manually rescaling a job). Also, > the amount of data that is reprocessed after rescaling depends on the > checkpointing interval, and the restore time depends on the state size.* > => As I know 'rescaling' also works in non-reactive mode, with restoring > checkpoint. What is the difference of using 'reactive' here? > > *The Reactive Mode allows Flink users to implement a powerful autoscaling > mechanism, by having an external service monitor certain metrics, such as > consumer lag, aggregate CPU utilization, throughput or latency. As soon as > these metrics are above or below a certain threshold, additional > TaskManagers can be added or removed from the Flink cluster.* > => Why is this only possible in 'reactive' mode? Seems this is more > related to 'autoscaler'. Are there some specific features/API which can > control TaskManager/Parallelism only in 'reactive' mode? > > Thank you. > > 2023년 9월 1일 (금) 오후 3:30, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > >> The reactive mode reacts to available resources. The autoscaler reacts to >> changing load and processing capacity and adjusts resources. >> >> Completely different concepts and applicability. >> Most people want the autoscaler , but this is a recent feature and is >> specific to the k8s operator at the moment. >> >> Gyula >> >> On Fri, 1 Sep 2023 at 04:50, Dennis Jung <inylov...@gmail.com> wrote: >> >>> Hello, >>> Thanks for your notice. >>> >>> Than what is the purpose of using 'reactive', if this doesn't do >>> anything itself? >>> What is the difference if I use auto-scaler without 'reactive' mode? >>> >>> Regards, >>> Jung >>> >>> >>> >>> 2023년 8월 18일 (금) 오후 7:51, Gyula Fóra <gyula.f...@gmail.com>님이 작성: >>> >>>> Hi! >>>> >>>> I think what you need is probably not the reactive mode but a proper >>>> autoscaler. The reactive mode as you say doesn't do anything in itself, you >>>> need to build a lot of logic around it. >>>> >>>> Check this instead: >>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ >>>> >>>> The Kubernetes Operator has a built in autoscaler that can scale jobs >>>> based on kafka data rate / processing throughput. It also doesn't rely on >>>> the reactive mode. >>>> >>>> Cheers, >>>> Gyula >>>> >>>> On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung <inylov...@gmail.com> >>>> wrote: >>>> >>>>> Hello, >>>>> Sorry for frequent questions. This is a question about 'reactive' mode. >>>>> >>>>> 1. As far as I understand, though I've setup `scheduler-mode: >>>>> reactive`, it will not change parallelism automatically by itself, by CPU >>>>> usage or Kafka consumer rate. It needs additional resource monitor >>>>> features >>>>> (such as Horizontal Pod Autoscaler, or else). Is this correct? >>>>> 2. Is it possible to create a custom resource monitor provider >>>>> application? For example, if I want to increase/decrease parallelism by >>>>> Kafka consumer rate, do I need to send specific API from outside, to order >>>>> rescaling? >>>>> 3. If 2 is correct, what is the difference when using 'reactive' mode? >>>>> Because as far as I think, calling a specific API will rescale either >>>>> using >>>>> 'reactive' mode or not...(or is the API just working based on this mode)? >>>>> >>>>> Thanks. >>>>> >>>>> Regards >>>>> >>>>>
