Hello Chen, Thanks for your reply! I have further questions as following...
1. In case of non-reactive mode in Flink 1.18, if the autoscaler adjusts parallelism, what is the difference by using 'reactive' mode? 2. In case if I use Flink 1.15~1.17 without autoscaler, is the difference of using 'reactive' mode is, changing parallelism dynamically by change of TM number (manually, or by custom scaler)? Regards, Jung 2023년 9월 5일 (화) 오후 3:59, Chen Zhanghao <zhanghao.c...@outlook.com>님이 작성: > Hi Dennis, > > > 1. In Flink 1.18 + non-reactive mode, autoscaler adjusts the job's > parallelism and the job will request for extra TMs if the current ones > cannot satisfy its need and redundant TMs will be released automatically > later for being idle. In other words, parallelism changes cause TM number > change. > 2. The core metrics used is busy time (the amount of time spent on > task processing per 1 second = 1 s - backpressured time - idle time), it is > considered to be superior as it counts I/O cost etc into account as well. > Also, the metrics is on a per-task granularity and allows us to identify > bottleneck tasks. > 3. Autoscaler feature currently only works for K8s opeartor + native > K8s mode. > > > Best, > Zhanghao Chen > ------------------------------ > *发件人:* Dennis Jung <inylov...@gmail.com> > *发送时间:* 2023年9月2日 12:58 > *收件人:* Gyula Fóra <gyula.f...@gmail.com> > *抄送:* user@flink.apache.org <user@flink.apache.org> > *主题:* Re: [Question] How to scale application based on 'reactive' mode > > Hello, > Thanks for your notice. > > 1. In "Flink 1.18 + non-reactive", is parallelism being changed by the > number of TM? > 2. In the document( > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.6/docs/custom-resource/autoscaler/), > it said "we are not using any container memory / CPU utilization metrics > directly here". Which metrics are these using internally? > 3. I'm using standalone k8s( > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/) > for deployment. Is autoscaler features only available by using the "flink > k8s operator"(sorry I don't understand this clearly yet...)? > > Regards > > > 2023년 9월 1일 (금) 오후 10:20, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > > Pretty much, except that with Flink 1.18 autoscaler can scale the job in > place without restarting the JM (even without reactive mode ) > > So actually best option is autoscaler with Flink 1.18 native mode (no > reactive) > > Gyula > > On Fri, 1 Sep 2023 at 13:54, Dennis Jung <inylov...@gmail.com> wrote: > > Thanks for feedback. > Could you check whether I understand correctly? > > *Only using 'reactive' mode:* > By manually adding TaskManager(TM) (such as using './bin/taskmanager.sh > start'), parallelism will be increased. For example, when job parallelism > is 1 and TM is 1, and if adding 1 new TM, JobManager will be restarted and > parallelism will be 2. > But the number of TM is not being controlled automatically. > > *Autoscaler + non-reactive:* > It can flexibilly control the number of TM by several metrics(CPU usage, > throughput, ...), and JobManager will be restarted when scaling. But job > parallelism is the same after the number of TM has been changed. > > *Autoscaler + 'reactive' mode*: > It can control numbers of TM by metric, and increase/decrease job > parallelism by changing TM. > > Regards, > Jung > > 2023년 9월 1일 (금) 오후 8:16, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > > I would look at reactive scaling as a way to increase / decrease > parallelism. > > It’s not a way to automatically decide when to actually do it as you need > to create new TMs . > > The autoscaler could use reactive mode to change the parallelism but you > need the autoscaler itself to decide when new resources should be added > > On Fri, 1 Sep 2023 at 13:09, Dennis Jung <inylov...@gmail.com> wrote: > > For now, the thing I've found about 'reactive' mode is that it > automatically adjusts 'job parallelism' when TaskManager is > increased/decreased. > > > https://www.slideshare.net/FlinkForward/autoscaling-flink-with-reactive-mode > > Is there some other feature that only 'reactive' mode offers for scaling? > > Thanks. > Regards. > > > > 2023년 9월 1일 (금) 오후 4:56, Dennis Jung <inylov...@gmail.com>님이 작성: > > Hello, > Thank you for your response. I have few more questions in following: > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/ > > *Reactive Mode configures a job so that it always uses all resources > available in the cluster. Adding a TaskManager will scale up your job, > removing resources will scale it down. Flink will manage the parallelism of > the job, always setting it to the highest possible values.* > => Does this mean when I add/remove TaskManager in 'non-reactive' mode, > resource(CPU/Memory/Etc.) of the cluster is not being changed? > > *Reactive Mode restarts a job on a rescaling event, restoring it from the > latest completed checkpoint. This means that there is no overhead of > creating a savepoint (which is needed for manually rescaling a job). Also, > the amount of data that is reprocessed after rescaling depends on the > checkpointing interval, and the restore time depends on the state size.* > => As I know 'rescaling' also works in non-reactive mode, with restoring > checkpoint. What is the difference of using 'reactive' here? > > *The Reactive Mode allows Flink users to implement a powerful autoscaling > mechanism, by having an external service monitor certain metrics, such as > consumer lag, aggregate CPU utilization, throughput or latency. As soon as > these metrics are above or below a certain threshold, additional > TaskManagers can be added or removed from the Flink cluster.* > => Why is this only possible in 'reactive' mode? Seems this is more > related to 'autoscaler'. Are there some specific features/API which can > control TaskManager/Parallelism only in 'reactive' mode? > > Thank you. > > 2023년 9월 1일 (금) 오후 3:30, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > > The reactive mode reacts to available resources. The autoscaler reacts to > changing load and processing capacity and adjusts resources. > > Completely different concepts and applicability. > Most people want the autoscaler , but this is a recent feature and is > specific to the k8s operator at the moment. > > Gyula > > On Fri, 1 Sep 2023 at 04:50, Dennis Jung <inylov...@gmail.com> wrote: > > Hello, > Thanks for your notice. > > Than what is the purpose of using 'reactive', if this doesn't do anything > itself? > What is the difference if I use auto-scaler without 'reactive' mode? > > Regards, > Jung > > > > 2023년 8월 18일 (금) 오후 7:51, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > > Hi! > > I think what you need is probably not the reactive mode but a proper > autoscaler. The reactive mode as you say doesn't do anything in itself, you > need to build a lot of logic around it. > > Check this instead: > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ > > The Kubernetes Operator has a built in autoscaler that can scale jobs > based on kafka data rate / processing throughput. It also doesn't rely on > the reactive mode. > > Cheers, > Gyula > > On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung <inylov...@gmail.com> wrote: > > Hello, > Sorry for frequent questions. This is a question about 'reactive' mode. > > 1. As far as I understand, though I've setup `scheduler-mode: reactive`, > it will not change parallelism automatically by itself, by CPU usage or > Kafka consumer rate. It needs additional resource monitor features (such as > Horizontal Pod Autoscaler, or else). Is this correct? > 2. Is it possible to create a custom resource monitor provider > application? For example, if I want to increase/decrease parallelism by > Kafka consumer rate, do I need to send specific API from outside, to order > rescaling? > 3. If 2 is correct, what is the difference when using 'reactive' mode? > Because as far as I think, calling a specific API will rescale either using > 'reactive' mode or not...(or is the API just working based on this mode)? > > Thanks. > > Regards > >
