Hi Konstantin, We found that Flink REST URL does not support the format ":merge" , which will be recognized as a parameter in the URL(due to start with a colon).
We will keep the previous way, i.e. POST: http://{jm_rest_address:port}/blocklist/taskmanagers and the "id" and "merge" flag are put into the request body. Best, Lijie Lijie Wang <wangdachui9...@gmail.com> 于2022年5月18日周三 09:35写道： > Hi Weihua, > thanks for feedback. > > 1. Yes, only *Manually* is supported in this FLIP, but it's the first step > towards auto-detection. > 2. We wii print the blocked nodes in logs. Maybe also put it into the > exception of insufficient resources. > 3. No. This FLIP won't change the WebUI. The blocklist information can be > obtained through REST API and metrics. > > Best, > Lijie > > Weihua Hu <huweihua....@gmail.com> 于2022年5月17日周二 21:41写道： > >> Hi, >> Thanks for creating this FLIP. >> We have implemented an automatic blocklist detection mechanism >> internally, which is indeed very effective for handling node failures. >> Due to the large number of nodes, although SREs already support automatic >> offline failure nodes, the detection is not 100% accurate and there is some >> delay. >> So the blocklist mechanism can make flink job recover from failure much >> faster. >> >> Here are some of my thoughts: >> 1. In this FLIP, it needs users to locate machine failure manually, there >> is a certain cost of use >> 2. What happens if too many nodes are blocked, resulting in insufficient >> resources? Will there be a special Exception for the user? >> 3. Will we display the blocklist information in the WebUI? The blocklist >> is for cluster level, and if multiple users share a cluster, some users may >> be a little confused when resources are not enough, or when resources are >> applied for more. >> >> Also, Looking forward to the next FLIP on auto-detection. >> >> Best, >> Weihua >> >> > 2022年5月16日 下午11:22，Lijie Wang <wangdachui9...@gmail.com> 写道： >> > >> > Hi Konstantin, >> > >> > Maybe change it to the following: >> > >> > 1. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id} >> > Merge is not allowed. If the {id} already exists, return error. >> Otherwise, >> > create a new item. >> > >> > 2. POST: http:// >> {jm_rest_address:port}/blocklist/taskmanagers/{id}:merge >> > Merge is allowed. If the {id} already exists, merge. Otherwise, create a >> > new item. >> > >> > WDYT? >> > >> > Best, >> > Lijie >> > >> > Konstantin Knauf <kna...@apache.org> 于2022年5月16日周一 20:07写道： >> > >> >> Hi Lijie, >> >> >> >> hm, maybe the following is more appropriate in that case >> >> >> >> POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge >> >> >> >> Best, >> >> >> >> Konstantin >> >> >> >> Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang < >> >> wangdachui9...@gmail.com>: >> >> >> >>> Hi Konstantin, >> >>> thanks for your feedback. >> >>> >> >>> From what I understand, PUT should be idempotent. However, we have a >> >>> *timeout* field in the request. This means that initiating the same >> >> request >> >>> at two different times will lead to different resource status >> (timestamps >> >>> of the items to be removed will be different). >> >>> >> >>> Should we use PUT in this case? WDYT? >> >>> >> >>> Best, >> >>> Lijie >> >>> >> >>> Konstantin Knauf <kna...@apache.org> 于2022年5月13日周五 17:20写道： >> >>> >> >>>> Hi Lijie, >> >>>> >> >>>> wouldn't the REST API-idiomatic way for an update/replace be a PUT on >> >> the >> >>>> resource? >> >>>> >> >>>> PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id} >> >>>> >> >>>> Best, >> >>>> >> >>>> Konstantin >> >>>> >> >>>> >> >>>> >> >>>> Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang < >> >>>> wangdachui9...@gmail.com>: >> >>>> >> >>>>> Hi everyone, >> >>>>> >> >>>>> I've had an offline discussion with Becket Qin and Zhu Zhu, and made >> >>> the >> >>>>> following changes on REST API: >> >>>>> 1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose >> >>> one. >> >>>> If >> >>>>> both are specified, will return error. >> >>>>> 2. If the specified item is already there, the *ADD* operation has >> >> two >> >>>>> behaviors: *return error*(default value) or *merge/update*, and we >> >>> add a >> >>>>> flag to the request body to control it. You can find more details >> >>> "Public >> >>>>> Interface" section. >> >>>>> >> >>>>> If there is no more feedback, we will start the vote thread next >> >> week. >> >>>>> >> >>>>> Best, >> >>>>> Lijie >> >>>>> >> >>>>> Lijie Wang <wangdachui9...@gmail.com> 于2022年5月10日周二 17:14写道： >> >>>>> >> >>>>>> Hi Becket Qin, >> >>>>>> >> >>>>>> Thanks for your suggestions. I have moved the description of >> >>>>>> configurations, metrics and REST API into "Public Interface" >> >> section, >> >>>> and >> >>>>>> made a few updates according to your suggestion. And in this FLIP, >> >>>> there >> >>>>>> no public java Interfaces or pluggables that users need to >> >> implement >> >>> by >> >>>>>> themselves. >> >>>>>> >> >>>>>> Answers for you questions: >> >>>>>> 1. Yes, there 2 block actions: MARK_BLOCKED and. >> >>>>>> MARK_BLOCKED_AND_EVACUATE_TASKS (has renamed). Currently, block >> >> items >> >>>> can >> >>>>>> only be added through the REST API, so these 2 action are mentioned >> >>> in >> >>>>> the >> >>>>>> REST API part (The REST API part has beed moved to public interface >> >>>> now). >> >>>>>> 2. I agree with you. I have changed the "Cause" field to String, >> >> and >> >>>>> allow >> >>>>>> users to specify it via REST API. >> >>>>>> 3. Yes, it is useful to allow different timeouts. As mentioned >> >> above, >> >>>> we >> >>>>>> will introduce 2 fields : *timeout* and *endTimestamp* into the ADD >> >>>> REST >> >>>>>> API to specify when to remove the blocked item. These 2 fields are >> >>>>>> optional, if neither is specified, it means that the blocked item >> >> is >> >>>>>> permanent and will not be removed. If both are specified, the >> >> minimum >> >>>> of >> >>>>>> *currentTimestamp+tiemout *and* endTimestamp* will be used as the >> >>> time >> >>>> to >> >>>>>> remove the blocked item. To keep the configurations more minimal, >> >> we >> >>>> have >> >>>>>> removed the *cluster.resource-blocklist.item.timeout* configuration >> >>>>>> option. >> >>>>>> 4. Yes, the block item will be overridden if the specified item >> >>> already >> >>>>>> exists. The ADD operation is *ADD or UPDATE*. >> >>>>>> 5. Yes. On JM/RM side, all the blocklist information is maintained >> >> in >> >>>>>> JMBlocklistHandler/RMBlocklistHandler. The blocklist handler(or >> >>>>> abstracted >> >>>>>> to other interfaces) will be propagated to different components. >> >>>>>> >> >>>>>> Best, >> >>>>>> Lijie >> >>>>>> >> >>>>>> Becket Qin <becket....@gmail.com> 于2022年5月10日周二 11:26写道： >> >>>>>> >> >>>>>>> Hi Lijie, >> >>>>>>> >> >>>>>>> Thanks for updating the FLIP. It looks like the public interface >> >>>> section >> >>>>>>> did not fully reflect all the user sensible behavior and API. Can >> >>> you >> >>>>> put >> >>>>>>> everything that users may be aware of there? That would include >> >> the >> >>>> REST >> >>>>>>> API, metrics, configurations, public java Interfaces or pluggables >> >>>> that >> >>>>>>> users may see or implement by themselves, as well as a brief >> >> summary >> >>>> of >> >>>>>>> the >> >>>>>>> behavior of the public API. >> >>>>>>> >> >>>>>>> Besides that, I have a few questions: >> >>>>>>> >> >>>>>>> 1. According to the conversation in the discussion thread, it >> >> looks >> >>>> like >> >>>>>>> the BlockAction will have "MARK_BLOCKLISTED" and >> >>>>>>> "MARK_BLOCKLISTED_AND_EVACUATE_TASKS". Is that the case? If so, >> >> can >> >>>> you >> >>>>>>> add >> >>>>>>> that to the public interface as well? >> >>>>>>> >> >>>>>>> 2. At this point, the "Cause" field in the BlockingItem is a >> >>> Throwable >> >>>>> and >> >>>>>>> is not reflected in the REST API. Should that be included in the >> >>> query >> >>>>>>> response? And should we change that field to be a String so users >> >>> may >> >>>>>>> specify the cause via the REST API when they block some nodes / >> >> TMs? >> >>>>>>> >> >>>>>>> 3. Would it be useful to allow users to have different timeouts >> >> for >> >>>>>>> different blocked items? So while there is a default timeout, >> >> users >> >>>> can >> >>>>>>> also override it via the REST API when they block an entity. >> >>>>>>> >> >>>>>>> 4. Regarding the ADD operation, if the specified item is already >> >>>> there, >> >>>>>>> will the block item be overridden? For example, if the user wants >> >> to >> >>>>>>> extend >> >>>>>>> the timeout of a blocked item, can they just issue an ADD command >> >>>>> again? >> >>>>>>> >> >>>>>>> 5. I am not quite familiar with the details of this, but is there >> >> a >> >>>>> source >> >>>>>>> of truth for the blocked list? I think it might be good to have a >> >>>> single >> >>>>>>> source of truth for the blocked list and just propagate that list >> >> to >> >>>>>>> different components to take the action of actually blocking the >> >>>>> resource. >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> Jiangjie (Becket) Qin >> >>>>>>> >> >>>>>>> On Mon, May 9, 2022 at 5:54 PM Lijie Wang < >> >> wangdachui9...@gmail.com >> >>>> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Hi everyone, >> >>>>>>>> >> >>>>>>>> Based on the discussion in the mailing list, I updated the FLIP >> >>> doc, >> >>>>> the >> >>>>>>>> changes include: >> >>>>>>>> 1. Changed the description of the motivation section to more >> >>> clearly >> >>>>>>>> describe the problem this FLIP is trying to solve. >> >>>>>>>> 2. Only *Manually* is supported. >> >>>>>>>> 3. Adopted some suggestions, such as *endTimestamp*. >> >>>>>>>> >> >>>>>>>> Best, >> >>>>>>>> Lijie >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Roman Boyko <ro.v.bo...@gmail.com> 于2022年5月7日周六 19:25写道： >> >>>>>>>> >> >>>>>>>>> Hi Lijie! >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> *a) “Probably storing inside Zookeeper/Configmap might be >> >>>>>>> helpfulhere.” >> >>>>>>>>> Can you explain it in detail? I don't fully understand that. >> >> In >> >>>>>>>> myopinion, >> >>>>>>>>> non-active and active are the same, and no special treatment >> >>>>>>> isrequired.* >> >>>>>>>>> >> >>>>>>>>> Sorry this was a misunderstanding from my side. I thought we >> >>> were >> >>>>>>> talking >> >>>>>>>>> about the HA mode (but not about Active and Standalone >> >>>>>>> ResourceManager). >> >>>>>>>>> And the original question was - how to handle the blacklisted >> >>>> nodes >> >>>>>>> list >> >>>>>>>> at >> >>>>>>>>> the moment of leader change? Should we simply forget about >> >> them >> >>> or >> >>>>>>> try to >> >>>>>>>>> pre-save that list on the remote storage? >> >>>>>>>>> >> >>>>>>>>> On Sat, 7 May 2022 at 10:51, Yang Wang <danrtsey...@gmail.com >> >>> >> >>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Thanks Lijie and ZhuZhu for the explanation. >> >>>>>>>>>> >> >>>>>>>>>> I just overlooked the "MARK_BLOCKLISTED". For tasks level, >> >> it >> >>> is >> >>>>>>> indeed >> >>>>>>>>>> some functionalities the external tools(e.g. kubectl taint) >> >>>> could >> >>>>>>> not >> >>>>>>>>>> support. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Best, >> >>>>>>>>>> Yang >> >>>>>>>>>> >> >>>>>>>>>> Lijie Wang <wangdachui9...@gmail.com> 于2022年5月6日周五 22:18写道： >> >>>>>>>>>> >> >>>>>>>>>>> Thanks for your feedback, Jiangang and Martijn. >> >>>>>>>>>>> >> >>>>>>>>>>> @Jiangang >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>>> For auto-detecting, I wonder how to make the strategy >> >> and >> >>>>> mark a >> >>>>>>>> node >> >>>>>>>>>>> blocked? >> >>>>>>>>>>> >> >>>>>>>>>>> In fact, we currently plan to not support auto-detection >> >> in >> >>>> this >> >>>>>>>> FLIP. >> >>>>>>>>>> The >> >>>>>>>>>>> part about auto-detection may be continued in a separate >> >>> FLIP >> >>>> in >> >>>>>>> the >> >>>>>>>>>>> future. Some guys have the same concerns as you, and the >> >>>>>>> correctness >> >>>>>>>>> and >> >>>>>>>>>>> necessity of auto-detection may require further discussion >> >>> in >> >>>>> the >> >>>>>>>>> future. >> >>>>>>>>>>> >> >>>>>>>>>>>> In session mode, multi jobs can fail on the same bad >> >> node >> >>>> and >> >>>>>>> the >> >>>>>>>>> node >> >>>>>>>>>>> should be marked blocked. >> >>>>>>>>>>> By design, the blocklist information will be shared among >> >>> all >> >>>>> jobs >> >>>>>>>> in a >> >>>>>>>>>>> cluster/session. The JM will sync blocklist information >> >> with >> >>>> RM. >> >>>>>>>>>>> >> >>>>>>>>>>> @Martijn >> >>>>>>>>>>> >> >>>>>>>>>>>> I agree with Yang Wang on this. >> >>>>>>>>>>> As Zhu Zhu and I mentioned above, we think the >> >>>>>>> MARK_BLOCKLISTED(Just >> >>>>>>>>>> limits >> >>>>>>>>>>> the load of the node and does not kill all the processes >> >> on >> >>>> it) >> >>>>>>> is >> >>>>>>>>> also >> >>>>>>>>>>> important, and we think that external systems (*yarn >> >> rmadmin >> >>>> or >> >>>>>>>> kubectl >> >>>>>>>>>>> taint*) cannot support it. So we think it makes sense even >> >>>> only >> >>>>>>>>>> *manually*. >> >>>>>>>>>>> >> >>>>>>>>>>>> I also agree with Chesnay that magical mechanisms are >> >>> indeed >> >>>>>>> super >> >>>>>>>>> hard >> >>>>>>>>>>> to get right. >> >>>>>>>>>>> Yes, as you see, Jiangang(and a few others) have the same >> >>>>> concern. >> >>>>>>>>>>> However, we currently plan to not support auto-detection >> >> in >> >>>> this >> >>>>>>>> FLIP, >> >>>>>>>>>> and >> >>>>>>>>>>> only *manually*. In addition, I'd like to say that the >> >> FLIP >> >>>>>>> provides >> >>>>>>>> a >> >>>>>>>>>>> mechanism to support MARK_BLOCKLISTED and >> >>>>>>>>>>> MARK_BLOCKLISTED_AND_EVACUATE_TASKS, >> >>>>>>>>>>> the auto-detection may be done by external systems. >> >>>>>>>>>>> >> >>>>>>>>>>> Best, >> >>>>>>>>>>> Lijie >> >>>>>>>>>>> >> >>>>>>>>>>> Martijn Visser <mart...@ververica.com> 于2022年5月6日周五 >> >>> 19:04写道： >> >>>>>>>>>>> >> >>>>>>>>>>>>> If we only support to block nodes manually, then I >> >> could >> >>>> not >> >>>>>>> see >> >>>>>>>>>>>> the obvious advantages compared with current SRE's >> >>>>> approach(via >> >>>>>>>> *yarn >> >>>>>>>>>>>> rmadmin or kubectl taint*). >> >>>>>>>>>>>> >> >>>>>>>>>>>> I agree with Yang Wang on this. >> >>>>>>>>>>>> >> >>>>>>>>>>>>> To me this sounds yet again like one of those magical >> >>>>>>> mechanisms >> >>>>>>>>>> that >> >>>>>>>>>>>> will rarely work just right. >> >>>>>>>>>>>> >> >>>>>>>>>>>> I also agree with Chesnay that magical mechanisms are >> >>> indeed >> >>>>>>> super >> >>>>>>>>> hard >> >>>>>>>>>>> to >> >>>>>>>>>>>> get right. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Best regards, >> >>>>>>>>>>>> >> >>>>>>>>>>>> Martijn >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Fri, 6 May 2022 at 12:03, Jiangang Liu < >> >>>>>>>> liujiangangp...@gmail.com >> >>>>>>>>>> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Thanks for the valuable design. The auto-detecting can >> >>>>> decrease >> >>>>>>>>> great >> >>>>>>>>>>> work >> >>>>>>>>>>>>> for us. We have implemented the similar feature in our >> >>>> inner >> >>>>>>> flink >> >>>>>>>>>>>>> version. >> >>>>>>>>>>>>> Below is something that I care about: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 1. For auto-detecting, I wonder how to make the >> >>> strategy >> >>>>> and >> >>>>>>>>> mark a >> >>>>>>>>>>>>> node >> >>>>>>>>>>>>> blocked? Sometimes the blocked node is hard to be >> >>>>> detected, >> >>>>>>> for >> >>>>>>>>>>>>> example, >> >>>>>>>>>>>>> the upper node or the down node will be blocked when >> >>>>> network >> >>>>>>>>>>>>> unreachable. >> >>>>>>>>>>>>> 2. I see that the strategy is made in JobMaster >> >> side. >> >>>> How >> >>>>>>> about >> >>>>>>>>>>>>> implementing the similar logic in resource manager? >> >> In >> >>>>>>> session >> >>>>>>>>>> mode, >> >>>>>>>>>>>>> multi >> >>>>>>>>>>>>> jobs can fail on the same bad node and the node >> >> should >> >>>> be >> >>>>>>>> marked >> >>>>>>>>>>>>> blocked. >> >>>>>>>>>>>>> If the job makes the strategy, the node may be not >> >>>> marked >> >>>>>>>> blocked >> >>>>>>>>>> if >> >>>>>>>>>>>>> the >> >>>>>>>>>>>>> fail times don't exceed the threshold. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Zhu Zhu <reed...@gmail.com> 于2022年5月5日周四 23:35写道： >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thank you for all your feedback! >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Besides the answers from Lijie, I'd like to share >> >> some >> >>> of >> >>>>> my >> >>>>>>>>>> thoughts: >> >>>>>>>>>>>>>> 1. Whether to enable automatical blocklist >> >>>>>>>>>>>>>> Generally speaking, it is not a goal of FLIP-224. >> >>>>>>>>>>>>>> The automatical way should be something built upon >> >> the >> >>>>>>> blocklist >> >>>>>>>>>>>>>> mechanism and well decoupled. It was designed to be a >> >>>>>>>> configurable >> >>>>>>>>>>>>>> blocklist strategy, but I think we can further >> >> decouple >> >>>> it >> >>>>> by >> >>>>>>>>>>>>>> introducing a abnormal node detector, as Becket >> >>>> suggested, >> >>>>>>> which >> >>>>>>>>>> just >> >>>>>>>>>>>>>> uses the blocklist mechanism once bad nodes are >> >>> detected. >> >>>>>>>> However, >> >>>>>>>>>> it >> >>>>>>>>>>>>>> should be a separate FLIP with further dev >> >> discussions >> >>>> and >> >>>>>>>>> feedback >> >>>>>>>>>>>>>> from users. I also agree with Becket that different >> >>> users >> >>>>>>> have >> >>>>>>>>>>> different >> >>>>>>>>>>>>>> requirements, and we should listen to them. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> 2. Is it enough to just take away abnormal nodes >> >>>> externally >> >>>>>>>>>>>>>> My answer is no. As Lijie has mentioned, we need a >> >> way >> >>> to >> >>>>>>> avoid >> >>>>>>>>>>>>>> deploying tasks to temporary hot nodes. In this case, >> >>>> users >> >>>>>>> may >> >>>>>>>>> just >> >>>>>>>>>>>>>> want to limit the load of the node and do not want to >> >>>> kill >> >>>>>>> all >> >>>>>>>> the >> >>>>>>>>>>>>>> processes on it. Another case is the speculative >> >>>>> execution[1] >> >>>>>>>>> which >> >>>>>>>>>>>>>> may also leverage this feature to avoid starting >> >> mirror >> >>>>>>> tasks on >> >>>>>>>>>> slow >> >>>>>>>>>>>>>> nodes. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>> Zhu >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Lijie Wang <wangdachui9...@gmail.com> 于2022年5月5日周四 >> >>>>> 15:56写道： >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi everyone, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks for your feedback. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> There's one detail that I'd like to re-emphasize >> >> here >> >>>>>>> because >> >>>>>>>> it >> >>>>>>>>>> can >> >>>>>>>>>>>>>> affect the value and design of the blocklist >> >> mechanism >> >>>>>>> (perhaps >> >>>>>>>> I >> >>>>>>>>>>> should >> >>>>>>>>>>>>>> highlight it in the FLIP). We propose two actions in >> >>>> FLIP: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> 1) MARK_BLOCKLISTED: Just mark the task manager or >> >>> node >> >>>>> as >> >>>>>>>>>> blocked. >> >>>>>>>>>>>>>> Future slots should not be allocated from the blocked >> >>>> task >> >>>>>>>> manager >> >>>>>>>>>> or >> >>>>>>>>>>>>> node. >> >>>>>>>>>>>>>> But slots that are already allocated will not be >> >>>> affected. >> >>>>> A >> >>>>>>>>> typical >> >>>>>>>>>>>>>> application scenario is to mitigate machine hotspots. >> >>> In >> >>>>> this >> >>>>>>>>> case, >> >>>>>>>>>> we >> >>>>>>>>>>>>> hope >> >>>>>>>>>>>>>> that subsequent resource allocations will not be on >> >> the >> >>>> hot >> >>>>>>>>> machine, >> >>>>>>>>>>> but >> >>>>>>>>>>>>>> tasks currently running on it should not be affected. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> 2) MARK_BLOCKLISTED_AND_EVACUATE_TASKS: Mark the >> >> task >> >>>>>>> manager >> >>>>>>>> or >> >>>>>>>>>>> node >> >>>>>>>>>>>>> as >> >>>>>>>>>>>>>> blocked, and evacuate all tasks on it. Evacuated >> >> tasks >> >>>> will >> >>>>>>> be >> >>>>>>>>>>>>> restarted on >> >>>>>>>>>>>>>> non-blocked task managers. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> For the above 2 actions, the former may more >> >>> highlight >> >>>>> the >> >>>>>>>>> meaning >> >>>>>>>>>>> of >> >>>>>>>>>>>>>> this FLIP, because the external system cannot do >> >> that. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Regarding *Manually* and *Automatically*, I >> >> basically >> >>>>> agree >> >>>>>>>> with >> >>>>>>>>>>>>> @Becket >> >>>>>>>>>>>>>> Qin: different users have different answers. Not all >> >>>> users’ >> >>>>>>>>>> deployment >> >>>>>>>>>>>>>> environments have a special external system that can >> >>>>> perform >> >>>>>>> the >> >>>>>>>>>>> anomaly >> >>>>>>>>>>>>>> detection. In addition, adding pluggable/optional >> >>>>>>> auto-detection >> >>>>>>>>>>> doesn't >> >>>>>>>>>>>>>> require much extra work on top of manual >> >> specification. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I will answer your other questions one by one. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> @Yangze >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> a) I think you are right, we do not need to expose >> >>> the >> >>>>>>>>>>>>>> >> >>> `cluster.resource-blocklist.item.timeout-check-interval` >> >>>> to >> >>>>>>>> users. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> b) We can abstract the `notifyException` to a >> >>> separate >> >>>>>>>> interface >> >>>>>>>>>>>>> (maybe >> >>>>>>>>>>>>>> BlocklistExceptionListener), and the >> >>>>>>>>> ResourceManagerBlocklistHandler >> >>>>>>>>>>> can >> >>>>>>>>>>>>>> implement it in the future. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> @Martijn >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> a) I also think the manual blocking should be done >> >> by >> >>>>>>> cluster >> >>>>>>>>>>>>> operators. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> b) I think manual blocking makes sense, because >> >>>> according >> >>>>>>> to >> >>>>>>>> my >> >>>>>>>>>>>>>> experience, users are often the first to perceive the >> >>>>> machine >> >>>>>>>>>> problems >> >>>>>>>>>>>>>> (because of job failover or delay), and they will >> >>> contact >> >>>>>>>> cluster >> >>>>>>>>>>>>> operators >> >>>>>>>>>>>>>> to solve it, or even tell the cluster operators which >> >>>>>>> machine is >> >>>>>>>>>>>>>> problematic. From this point of view, I think the >> >>> people >> >>>>> who >> >>>>>>>>> really >> >>>>>>>>>>> need >> >>>>>>>>>>>>>> the manual blocking are the users, and it’s just >> >>>> performed >> >>>>> by >> >>>>>>>> the >> >>>>>>>>>>>>> cluster >> >>>>>>>>>>>>>> operator, so I think the manual blocking makes sense. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> @Chesnay >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> We need to touch the logic of JM/SlotPool, because >> >>> for >> >>>>>>>>>>>>> MARK_BLOCKLISTED >> >>>>>>>>>>>>>> , we need to know whether the slot is blocklisted >> >> when >> >>>> the >> >>>>>>> task >> >>>>>>>> is >> >>>>>>>>>>>>>> FINISHED/CANCELLED/FAILED. If so, SlotPool should >> >>>> release >> >>>>>>> the >> >>>>>>>>> slot >> >>>>>>>>>>>>>> directly to avoid assigning other tasks (of this job) >> >>> on >> >>>>> it. >> >>>>>>> If >> >>>>>>>> we >> >>>>>>>>>>> only >> >>>>>>>>>>>>>> maintain the blocklist information on the RM, JM >> >> needs >> >>> to >> >>>>>>>> retrieve >> >>>>>>>>>> it >> >>>>>>>>>>> by >> >>>>>>>>>>>>>> RPC. I think the performance overhead of that is >> >>>> relatively >> >>>>>>>> large, >> >>>>>>>>>> so >> >>>>>>>>>>> I >> >>>>>>>>>>>>>> think it's worth maintaining the blocklist >> >> information >> >>> on >> >>>>>>> the JM >> >>>>>>>>>> side >> >>>>>>>>>>>>> and >> >>>>>>>>>>>>>> syncing them. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> @Роман >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> a) “Probably storing inside Zookeeper/Configmap >> >>>> might >> >>>>>>> be >> >>>>>>>>>> helpful >> >>>>>>>>>>>>>> here.” Can you explain it in detail? I don't fully >> >>>>>>> understand >> >>>>>>>>> that. >> >>>>>>>>>>> In >> >>>>>>>>>>>>> my >> >>>>>>>>>>>>>> opinion, non-active and active are the same, and no >> >>>> special >> >>>>>>>>>> treatment >> >>>>>>>>>>> is >> >>>>>>>>>>>>>> required. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> b) I agree with you, the `endTimestamp` makes >> >> sense, >> >>> I >> >>>>> will >> >>>>>>>> add >> >>>>>>>>> it >> >>>>>>>>>>> to >> >>>>>>>>>>>>>> FLIP. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> @Yang >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> As mentioned above, AFAK, the external system >> >> cannot >> >>>>>>> support >> >>>>>>>> the >> >>>>>>>>>>>>>> MARK_BLOCKLISTED action. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Looking forward to your further feedback. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Lijie >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Yang Wang <danrtsey...@gmail.com> 于2022年5月3日周二 >> >>>> 21:09写道： >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Thanks Lijie and Zhu for creating the proposal. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> I want to share some thoughts about Flink cluster >> >>>>>>> operations. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> In the production environment, the SRE(aka Site >> >>>>>>> Reliability >> >>>>>>>>>>> Engineer) >> >>>>>>>>>>>>>>>> already has many tools to detect the unstable >> >> nodes, >> >>>>> which >> >>>>>>>>> could >> >>>>>>>>>>> take >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>> system logs/metrics into consideration. >> >>>>>>>>>>>>>>>> Then they use graceful-decomission in YARN and >> >> taint >> >>>> in >> >>>>>>> K8s >> >>>>>>>> to >> >>>>>>>>>>>>> prevent >> >>>>>>>>>>>>>> new >> >>>>>>>>>>>>>>>> allocations on these unstable nodes. >> >>>>>>>>>>>>>>>> At last, they will evict all the containers and >> >> pods >> >>>>>>> running >> >>>>>>>> on >> >>>>>>>>>>> these >> >>>>>>>>>>>>>> nodes. >> >>>>>>>>>>>>>>>> This mechanism also works for planned maintenance. >> >>> So >> >>>> I >> >>>>> am >> >>>>>>>>> afraid >> >>>>>>>>>>>>> this >> >>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>> not the typical use case for FLIP-224. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> If we only support to block nodes manually, then I >> >>>> could >> >>>>>>> not >> >>>>>>>>> see >> >>>>>>>>>>>>>>>> the obvious advantages compared with current SRE's >> >>>>>>>> approach(via >> >>>>>>>>>>> *yarn >> >>>>>>>>>>>>>>>> rmadmin or kubectl taint*). >> >>>>>>>>>>>>>>>> At least, we need to have a pluggable component >> >>> which >> >>>>>>> could >> >>>>>>>>>> expose >> >>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>> potential unstable nodes automatically and block >> >>> them >> >>>> if >> >>>>>>>>> enabled >> >>>>>>>>>>>>>> explicitly. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>> Yang >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Becket Qin <becket....@gmail.com> 于2022年5月2日周一 >> >>>> 16:36写道： >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thanks for the proposal, Lijie. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> This is an interesting feature and discussion, >> >> and >> >>>>>>> somewhat >> >>>>>>>>>>> related >> >>>>>>>>>>>>>> to the >> >>>>>>>>>>>>>>>>> design principle about how people should operate >> >>>>> Flink. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> I think there are three things involved in this >> >>>> FLIP. >> >>>>>>>>>>>>>>>>> a) Detect and report the unstable node. >> >>>>>>>>>>>>>>>>> b) Collect the information of the unstable >> >>> node >> >>>>> and >> >>>>>>>>> form a >> >>>>>>>>>>>>>> blocklist. >> >>>>>>>>>>>>>>>>> c) Take the action to block nodes. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> My two cents: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> 1. It looks like people all agree that Flink >> >>> should >> >>>>> have >> >>>>>>>> c). >> >>>>>>>>> It >> >>>>>>>>>>> is >> >>>>>>>>>>>>>> not only >> >>>>>>>>>>>>>>>>> useful for cases of node failures, but also >> >> handy >> >>>> for >> >>>>>>> some >> >>>>>>>>>>> planned >> >>>>>>>>>>>>>>>>> maintenance. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> 2. People have different opinions on b), i.e. >> >> who >> >>>>>>> should be >> >>>>>>>>> the >> >>>>>>>>>>>>> brain >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> make the decision to block a node. I think this >> >>>>> largely >> >>>>>>>>> depends >> >>>>>>>>>>> on >> >>>>>>>>>>>>>> who we >> >>>>>>>>>>>>>>>>> talk to. Different users would probably give >> >>>> different >> >>>>>>>>> answers. >> >>>>>>>>>>> For >> >>>>>>>>>>>>>> people >> >>>>>>>>>>>>>>>>> who do have a centralized node health management >> >>>>>>> service, >> >>>>>>>> let >> >>>>>>>>>>> Flink >> >>>>>>>>>>>>>> do just >> >>>>>>>>>>>>>>>>> do a) and c) would be preferred. So essentially >> >>>> Flink >> >>>>>>> would >> >>>>>>>>> be >> >>>>>>>>>>> one >> >>>>>>>>>>>>> of >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> sources that may detect unstable nodes, report >> >> it >> >>> to >> >>>>>>> that >> >>>>>>>>>>> service, >> >>>>>>>>>>>>>> and then >> >>>>>>>>>>>>>>>>> take the command from that service to block the >> >>>>>>> problematic >> >>>>>>>>>>> nodes. >> >>>>>>>>>>>>> On >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> other hand, for users who do not have such a >> >>>> service, >> >>>>>>>> simply >> >>>>>>>>>>>>> letting >> >>>>>>>>>>>>>> Flink >> >>>>>>>>>>>>>>>>> be clever by itself to block the suspicious >> >> nodes >> >>>>> might >> >>>>>>> be >> >>>>>>>>>>> desired >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> ensure the jobs are running smoothly. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> So that indicates a) and b) here should be >> >>>> pluggable / >> >>>>>>>>>> optional. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> In light of this, maybe it would make sense to >> >>> have >> >>>>>>>> something >> >>>>>>>>>>>>>> pluggable >> >>>>>>>>>>>>>>>>> like a UnstableNodeReporter which exposes >> >> unstable >> >>>>> nodes >> >>>>>>>>>>> actively. >> >>>>>>>>>>>>> (A >> >>>>>>>>>>>>>> more >> >>>>>>>>>>>>>>>>> general interface should be JobInfoReporter<T> >> >>> which >> >>>>>>> can be >> >>>>>>>>>> used >> >>>>>>>>>>> to >> >>>>>>>>>>>>>> report >> >>>>>>>>>>>>>>>>> any information of type <T>. But I'll just keep >> >>> the >> >>>>>>> scope >> >>>>>>>>>>> relevant >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>> FLIP here). Personally speaking, I think it is >> >> OK >> >>> to >> >>>>>>> have a >> >>>>>>>>>>> default >> >>>>>>>>>>>>>>>>> implementation of a reporter which just tells >> >>> Flink >> >>>> to >> >>>>>>> take >> >>>>>>>>>>> action >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>> block >> >>>>>>>>>>>>>>>>> problematic nodes and also unblocks them after >> >>>>> timeout. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Mon, May 2, 2022 at 3:27 PM Роман Бойко < >> >>>>>>>>>> ro.v.bo...@gmail.com >> >>>>>>>>>>>> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Thanks for good initiative, Lijie and Zhu! >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> If it's possible I'd like to participate in >> >>>>>>> development. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I agree with 3rd point of Konstantin's reply - >> >>> we >> >>>>>>> should >> >>>>>>>>>>> consider >> >>>>>>>>>>>>>> to move >> >>>>>>>>>>>>>>>>>> somehow the information of blocklisted >> >> nodes/TMs >> >>>>> from >> >>>>>>>>> active >> >>>>>>>>>>>>>>>>>> ResourceManager to non-active ones. Probably >> >>>> storing >> >>>>>>>> inside >> >>>>>>>>>>>>>>>>>> Zookeeper/Configmap might be helpful here. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> And I agree with Martijn that a lot of >> >>>> organizations >> >>>>>>>> don't >> >>>>>>>>>> want >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>> expose >> >>>>>>>>>>>>>>>>>> such API for a cluster user group. But I think >> >>>> it's >> >>>>>>>>> necessary >> >>>>>>>>>>> to >> >>>>>>>>>>>>>> have the >> >>>>>>>>>>>>>>>>>> mechanism for unblocking the nodes/TMs anyway >> >>> for >> >>>>>>>> avoiding >> >>>>>>>>>>>>> incorrect >> >>>>>>>>>>>>>>>>>> automatic behaviour. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> And another one small suggestion - I think it >> >>>> would >> >>>>> be >> >>>>>>>>> better >> >>>>>>>>>>> to >> >>>>>>>>>>>>>> extend >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>> *BlocklistedItem* class with the >> >> *endTimestamp* >> >>>>> field >> >>>>>>> and >> >>>>>>>>>> fill >> >>>>>>>>>>> it >> >>>>>>>>>>>>>> at the >> >>>>>>>>>>>>>>>>>> item creation. This simple addition will allow >> >>> to: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> - >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Provide the ability to users to setup the >> >>> exact >> >>>>>>> time >> >>>>>>>> of >> >>>>>>>>>>>>>> blocklist end >> >>>>>>>>>>>>>>>>>> through RestAPI >> >>>>>>>>>>>>>>>>>> - >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Not being tied to a single value of >> >>>>>>>>>>>>>>>>>> *cluster.resource-blacklist.item.timeout* >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On Mon, 2 May 2022 at 14:17, Chesnay Schepler >> >> < >> >>>>>>>>>>>>> ches...@apache.org> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I do share the concern between blurring the >> >>>> lines >> >>>>> a >> >>>>>>>> bit. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> That said, I'd prefer to not have any >> >>>>> auto-detection >> >>>>>>>> and >> >>>>>>>>>> only >> >>>>>>>>>>>>>> have an >> >>>>>>>>>>>>>>>>>>> opt-in mechanism >> >>>>>>>>>>>>>>>>>>> to manually block processes/nodes. To me >> >> this >> >>>>> sounds >> >>>>>>>> yet >> >>>>>>>>>>> again >> >>>>>>>>>>>>>> like one >> >>>>>>>>>>>>>>>>>>> of those >> >>>>>>>>>>>>>>>>>>> magical mechanisms that will rarely work >> >> just >> >>>>> right. >> >>>>>>>>>>>>>>>>>>> An external system can leverage way more >> >>>>> information >> >>>>>>>>> after >> >>>>>>>>>>> all. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Moreover, I'm quite concerned about the >> >>>> complexity >> >>>>>>> of >> >>>>>>>>> this >> >>>>>>>>>>>>>> proposal. >> >>>>>>>>>>>>>>>>>>> Tracking on both the RM/JM side; syncing >> >>> between >> >>>>>>>>>> components; >> >>>>>>>>>>>>>>>>> adjustments >> >>>>>>>>>>>>>>>>>>> to the >> >>>>>>>>>>>>>>>>>>> slot and resource protocol. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> In a way it seems overly complicated. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> If we look at it purely from an active >> >>> resource >> >>>>>>>>> management >> >>>>>>>>>>>>>> perspective, >> >>>>>>>>>>>>>>>>>>> then there >> >>>>>>>>>>>>>>>>>>> isn't really a need to touch the slot >> >> protocol >> >>>> at >> >>>>>>> all >> >>>>>>>> (or >> >>>>>>>>>> in >> >>>>>>>>>>>>> fact >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>> anything in the JobMaster), >> >>>>>>>>>>>>>>>>>>> because there isn't any point in keeping >> >>> around >> >>>>>>> blocked >> >>>>>>>>> TMs >> >>>>>>>>>>> in >> >>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> first >> >>>>>>>>>>>>>>>>>>> place. >> >>>>>>>>>>>>>>>>>>> They'd just be idling, potentially shutting >> >>> down >> >>>>>>> after >> >>>>>>>> a >> >>>>>>>>>>> while >> >>>>>>>>>>>>> by >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> RM >> >>>>>>>>>>>>>>>>>>> because of >> >>>>>>>>>>>>>>>>>>> it (unless we _also_ touch that logic). >> >>>>>>>>>>>>>>>>>>> Here the blocking of a process (be it by >> >>>> blocking >> >>>>>>> the >> >>>>>>>>>> process >> >>>>>>>>>>>>> or >> >>>>>>>>>>>>>> node) >> >>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>> equivalent with shutting down the blocked >> >>>>>>> process(es). >> >>>>>>>>>>>>>>>>>>> Once the block is lifted we can just spin it >> >>>> back >> >>>>>>> up. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> And I do wonder whether we couldn't apply >> >> the >> >>>> same >> >>>>>>> line >> >>>>>>>>> of >> >>>>>>>>>>>>>> thinking to >> >>>>>>>>>>>>>>>>>>> standalone resource management. >> >>>>>>>>>>>>>>>>>>> Here being able to stop/restart a >> >> process/node >> >>>>>>> manually >> >>>>>>>>>>> should >> >>>>>>>>>>>>> be >> >>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>> core >> >>>>>>>>>>>>>>>>>>> requirement for a Flink deployment anyway. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> On 02/05/2022 08:49, Martijn Visser wrote: >> >>>>>>>>>>>>>>>>>>>> Hi everyone, >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Thanks for creating this FLIP. I can >> >>>> understand >> >>>>>>> the >> >>>>>>>>>> problem >> >>>>>>>>>>>>> and >> >>>>>>>>>>>>>> I see >> >>>>>>>>>>>>>>>>>>> value >> >>>>>>>>>>>>>>>>>>>> in the automatic detection and >> >>> blocklisting. I >> >>>>> do >> >>>>>>>> have >> >>>>>>>>>> some >> >>>>>>>>>>>>>> concerns >> >>>>>>>>>>>>>>>>>> with >> >>>>>>>>>>>>>>>>>>>> the ability to manually specify to be >> >>> blocked >> >>>>>>>>> resources. >> >>>>>>>>>> I >> >>>>>>>>>>>>> have >> >>>>>>>>>>>>>> two >> >>>>>>>>>>>>>>>>>>>> concerns; >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> * Most organizations explicitly have a >> >>>>> separation >> >>>>>>> of >> >>>>>>>>>>>>> concerns, >> >>>>>>>>>>>>>>>>> meaning >> >>>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>> there's a group who's responsible for >> >>>> managing a >> >>>>>>>>> cluster >> >>>>>>>>>>> and >> >>>>>>>>>>>>>> there's >> >>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>> user >> >>>>>>>>>>>>>>>>>>>> group who uses that cluster. With the >> >>>>>>> introduction of >> >>>>>>>>>> this >> >>>>>>>>>>>>>> mechanism, >> >>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>> latter group now can influence the >> >>>>> responsibility >> >>>>>>> of >> >>>>>>>>> the >> >>>>>>>>>>>>> first >> >>>>>>>>>>>>>> group. >> >>>>>>>>>>>>>>>>>> So >> >>>>>>>>>>>>>>>>>>> it >> >>>>>>>>>>>>>>>>>>>> can be possible that someone from the user >> >>>> group >> >>>>>>>> blocks >> >>>>>>>>>>>>>> something, >> >>>>>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>>>>>>> causes an outage (which could result in >> >>> paging >> >>>>>>>>> mechanism >> >>>>>>>>>>>>>> triggering >> >>>>>>>>>>>>>>>>>> etc) >> >>>>>>>>>>>>>>>>>>>> which impacts the first group. >> >>>>>>>>>>>>>>>>>>>> * How big is the group of people who can >> >> go >> >>>>>>> through >> >>>>>>>> the >> >>>>>>>>>>>>> process >> >>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>>>> manually >> >>>>>>>>>>>>>>>>>>>> identifying a node that isn't behaving as >> >> it >> >>>>>>> should >> >>>>>>>>> be? I >> >>>>>>>>>>> do >> >>>>>>>>>>>>>> think >> >>>>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>> group is relatively limited. Does it then >> >>> make >> >>>>>>> sense >> >>>>>>>> to >> >>>>>>>>>>>>>> introduce >> >>>>>>>>>>>>>>>>> such >> >>>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>> feature, which would only be used by a >> >>> really >> >>>>>>> small >> >>>>>>>>> user >> >>>>>>>>>>>>> group >> >>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>>> Flink? >> >>>>>>>>>>>>>>>>>>> We >> >>>>>>>>>>>>>>>>>>>> still have to maintain, test and support >> >>> such >> >>>> a >> >>>>>>>>> feature. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> I'm +1 for the autodetection features, but >> >>> I'm >> >>>>>>>> leaning >> >>>>>>>>>>>>> towards >> >>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>> exposing >> >>>>>>>>>>>>>>>>>>>> this to the user group but having this >> >>>> available >> >>>>>>>>> strictly >> >>>>>>>>>>> for >> >>>>>>>>>>>>>> cluster >> >>>>>>>>>>>>>>>>>>>> operators. They could then also set up >> >> their >> >>>>>>>>>>>>>> paging/metrics/logging >> >>>>>>>>>>>>>>>>>>> system >> >>>>>>>>>>>>>>>>>>>> to take this into account. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Best regards, >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Martijn Visser >> >>>>>>>>>>>>>>>>>>>> https://twitter.com/MartijnVisser82 >> >>>>>>>>>>>>>>>>>>>> https://github.com/MartijnVisser >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> On Fri, 29 Apr 2022 at 09:39, Yangze Guo < >> >>>>>>>>>>> karma...@gmail.com >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Thanks for driving this, Zhu and Lijie. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> +1 for the overall proposal. Just share >> >>> some >> >>>>>>> cents >> >>>>>>>>> here: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - Why do we need to expose >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>> cluster.resource-blacklist.item.timeout-check-interval >> >>>>>>>>>> to >> >>>>>>>>>>>>> the >> >>>>>>>>>>>>>> user? >> >>>>>>>>>>>>>>>>>>>>> I think the semantics of >> >>>>>>>>>>>>>> `cluster.resource-blacklist.item.timeout` >> >>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>> sufficient for the user. How to guarantee >> >>> the >> >>>>>>>> timeout >> >>>>>>>>>>>>>> mechanism is >> >>>>>>>>>>>>>>>>>>>>> Flink's internal implementation. I think >> >> it >> >>>>> will >> >>>>>>> be >> >>>>>>>>> very >> >>>>>>>>>>>>>> confusing >> >>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>> we do not need to expose it to users. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - ResourceManager can notify the >> >> exception >> >>>> of a >> >>>>>>> task >> >>>>>>>>>>>>> manager to >> >>>>>>>>>>>>>>>>>>>>> `BlacklistHandler` as well. >> >>>>>>>>>>>>>>>>>>>>> For example, the slot allocation might >> >> fail >> >>>> in >> >>>>>>> case >> >>>>>>>>> the >> >>>>>>>>>>>>> target >> >>>>>>>>>>>>>> task >> >>>>>>>>>>>>>>>>>>>>> manager is busy or has a network jitter. >> >> I >> >>>>> don't >> >>>>>>>> mean >> >>>>>>>>> we >> >>>>>>>>>>>>> need >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> cover >> >>>>>>>>>>>>>>>>>>>>> this case in this version, but we can >> >> also >> >>>>> open a >> >>>>>>>>>>>>>> `notifyException` >> >>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>>>> `ResourceManagerBlacklistHandler`. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - Before we sync the blocklist to >> >>>>>>> ResourceManager, >> >>>>>>>>> will >> >>>>>>>>>>> the >> >>>>>>>>>>>>>> slot of >> >>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>>> blocked task manager continues to be >> >>> released >> >>>>> and >> >>>>>>>>>>> allocated? >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>> Yangze Guo >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 28, 2022 at 3:11 PM Lijie >> >> Wang >> >>> < >> >>>>>>>>>>>>>>>>> wangdachui9...@gmail.com> >> >>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>> Hi Konstantin, >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback. I will >> >> response >> >>>>> your 4 >> >>>>>>>>>> remarks: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 1) Thanks for reminding me of the >> >>>>> controversy. I >> >>>>>>>>> think >> >>>>>>>>>>>>>> “BlockList” >> >>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>>> good >> >>>>>>>>>>>>>>>>>>>>>> enough, and I will change it in FLIP. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 2) Your suggestion for the REST API is a >> >>>> good >> >>>>>>> idea. >> >>>>>>>>>> Based >> >>>>>>>>>>>>> on >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> above, I >> >>>>>>>>>>>>>>>>>>>>>> would change REST API as following: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> POST/GET <host>/blocklist/nodes >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> POST/GET <host>/blocklist/taskmanagers >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> DELETE >> >> <host>/blocklist/node/<identifier> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> DELETE >> >>>>> <host>/blocklist/taskmanager/<identifier> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 3) If a node is blocking/blocklisted, it >> >>>> means >> >>>>>>> that >> >>>>>>>>> all >> >>>>>>>>>>>>> task >> >>>>>>>>>>>>>>>>> managers >> >>>>>>>>>>>>>>>>>>> on >> >>>>>>>>>>>>>>>>>>>>>> this node are blocklisted. All slots on >> >>>> these >> >>>>>>> TMs >> >>>>>>>> are >> >>>>>>>>>> not >> >>>>>>>>>>>>>>>>> available. >> >>>>>>>>>>>>>>>>>>> This >> >>>>>>>>>>>>>>>>>>>>>> is actually a bit like TM losts, but >> >> these >> >>>> TMs >> >>>>>>> are >> >>>>>>>>> not >> >>>>>>>>>>>>> really >> >>>>>>>>>>>>>> lost, >> >>>>>>>>>>>>>>>>>>> they >> >>>>>>>>>>>>>>>>>>>>>> are in an unavailable status, and they >> >> are >> >>>>> still >> >>>>>>>>>>> registered >> >>>>>>>>>>>>>> in this >> >>>>>>>>>>>>>>>>>>> flink >> >>>>>>>>>>>>>>>>>>>>>> cluster. They will be available again >> >> once >> >>>> the >> >>>>>>>>>>>>> corresponding >> >>>>>>>>>>>>>>>>>> blocklist >> >>>>>>>>>>>>>>>>>>>>> item >> >>>>>>>>>>>>>>>>>>>>>> is removed. This behavior is the same in >> >>>>>>>>>>> active/non-active >> >>>>>>>>>>>>>>>>> clusters. >> >>>>>>>>>>>>>>>>>>>>>> However in the active clusters, these >> >> TMs >> >>>> may >> >>>>> be >> >>>>>>>>>> released >> >>>>>>>>>>>>> due >> >>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> idle >> >>>>>>>>>>>>>>>>>>>>>> timeouts. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 4) For the item timeout, I prefer to >> >> keep >> >>>> it. >> >>>>>>> The >> >>>>>>>>>> reasons >> >>>>>>>>>>>>> are >> >>>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>>>>>> following: >> >>>>>>>>>>>>>>>>>>>>>> a) The timeout will not affect users >> >>> adding >> >>>> or >> >>>>>>>>> removing >> >>>>>>>>>>>>> items >> >>>>>>>>>>>>>> via >> >>>>>>>>>>>>>>>>>> REST >> >>>>>>>>>>>>>>>>>>>>> API, >> >>>>>>>>>>>>>>>>>>>>>> and users can disable it by configuring >> >> it >> >>>> to >> >>>>>>>>>>>>> Long.MAX_VALUE . >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> b) Some node problems can recover after >> >> a >> >>>>>>> period of >> >>>>>>>>>> time >> >>>>>>>>>>>>>> (such as >> >>>>>>>>>>>>>>>>>>> machine >> >>>>>>>>>>>>>>>>>>>>>> hotspots), in which case users may >> >> prefer >> >>>> that >> >>>>>>>> Flink >> >>>>>>>>>> can >> >>>>>>>>>>> do >> >>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>>>> automatically instead of requiring the >> >>> user >> >>>> to >> >>>>>>> do >> >>>>>>>> it >> >>>>>>>>>>>>> manually. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Lijie >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Konstantin Knauf <kna...@apache.org> >> >>>>>>> 于2022年4月27日周三 >> >>>>>>>>>>>>> 19:23写道： >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Hi Lijie, >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> I think, this makes sense and +1 to >> >> only >> >>>>>>> support >> >>>>>>>>>>> manually >> >>>>>>>>>>>>>> blocking >> >>>>>>>>>>>>>>>>>>>>>>> taskmanagers and nodes. Maybe the >> >>> different >> >>>>>>>>> strategies >> >>>>>>>>>>> can >> >>>>>>>>>>>>>> also be >> >>>>>>>>>>>>>>>>>>>>>>> maintained outside of Apache Flink. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> A few remarks: >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> 1) Can we use another term than >> >>>> "bla.cklist" >> >>>>>>> due >> >>>>>>>> to >> >>>>>>>>>> the >> >>>>>>>>>>>>>>>>> controversy >> >>>>>>>>>>>>>>>>>>>>> around >> >>>>>>>>>>>>>>>>>>>>>>> the term? [1] There was also a Jira >> >>> Ticket >> >>>>>>> about >> >>>>>>>>> this >> >>>>>>>>>>>>> topic a >> >>>>>>>>>>>>>>>>> while >> >>>>>>>>>>>>>>>>>>>>> back >> >>>>>>>>>>>>>>>>>>>>>>> and there was generally a consensus to >> >>>> avoid >> >>>>>>> the >> >>>>>>>>> term >> >>>>>>>>>>>>>> blacklist & >> >>>>>>>>>>>>>>>>>>>>> whitelist >> >>>>>>>>>>>>>>>>>>>>>>> [2]? We could use "blocklist" >> >> "denylist" >> >>> or >> >>>>>>>>>>> "quarantined" >> >>>>>>>>>>>>>>>>>>>>>>> 2) For the REST API, I'd prefer a >> >>> slightly >> >>>>>>>> different >> >>>>>>>>>>>>> design >> >>>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>> verbs >> >>>>>>>>>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>>>>>>>> add/remove often considered an >> >>> anti-pattern >> >>>>> for >> >>>>>>>> REST >> >>>>>>>>>>> APIs. >> >>>>>>>>>>>>>> POST >> >>>>>>>>>>>>>>>>> on a >> >>>>>>>>>>>>>>>>>>>>> list >> >>>>>>>>>>>>>>>>>>>>>>> item is generally the standard to add >> >>>> items. >> >>>>>>>> DELETE >> >>>>>>>>> on >> >>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>> individual >> >>>>>>>>>>>>>>>>>>>>>>> resource is standard to remove an item. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> POST <host>/quarantine/items >> >>>>>>>>>>>>>>>>>>>>>>> DELETE >> >>>>> <host>/quarantine/items/<itemidentifier> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> We could also consider to separate >> >>>>> taskmanagers >> >>>>>>>> and >> >>>>>>>>>>> nodes >> >>>>>>>>>>>>> in >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>> REST >> >>>>>>>>>>>>>>>>>>>>> API >> >>>>>>>>>>>>>>>>>>>>>>> (and internal data structures). Any >> >>> opinion >> >>>>> on >> >>>>>>>> this? >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> POST/GET <host>/quarantine/nodes >> >>>>>>>>>>>>>>>>>>>>>>> POST/GET <host>/quarantine/taskmanager >> >>>>>>>>>>>>>>>>>>>>>>> DELETE >> >>> <host>/quarantine/nodes/<identifier> >> >>>>>>>>>>>>>>>>>>>>>>> DELETE >> >>>>>>> <host>/quarantine/taskmanager/<identifier> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> 3) How would blocking nodes behave with >> >>>>>>> non-active >> >>>>>>>>>>>>> resource >> >>>>>>>>>>>>>>>>>> managers, >> >>>>>>>>>>>>>>>>>>>>> i.e. >> >>>>>>>>>>>>>>>>>>>>>>> standalone or reactive mode? >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> 4) To keep the implementation even more >> >>>>>>> minimal, >> >>>>>>>> do >> >>>>>>>>> we >> >>>>>>>>>>>>> need >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> timeout >> >>>>>>>>>>>>>>>>>>>>>>> behavior? If items are added/removed >> >>>> manually >> >>>>>>> we >> >>>>>>>>> could >> >>>>>>>>>>>>>> delegate >> >>>>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>>> to the >> >>>>>>>>>>>>>>>>>>>>>>> user easily. In my opinion the timeout >> >>>>> behavior >> >>>>>>>>> would >> >>>>>>>>>>>>> better >> >>>>>>>>>>>>>> fit >> >>>>>>>>>>>>>>>>>> into >> >>>>>>>>>>>>>>>>>>>>>>> specific strategies at a later point. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Looking forward to your thoughts. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Cheers and thank you, >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Konstantin >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> https://en.wikipedia.org/wiki/Blacklist_(computing)#Controversy_over_use_of_the_term >> >>>>>>>>>>>>>>>>>>>>>>> [2] >> >>>>>>>>> https://issues.apache.org/jira/browse/FLINK-18209 >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Am Mi., 27. Apr. 2022 um 04:04 Uhr >> >>> schrieb >> >>>>>>> Lijie >> >>>>>>>>> Wang >> >>>>>>>>>> < >> >>>>>>>>>>>>>>>>>>>>>>> wangdachui9...@gmail.com>: >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Hi all, >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Flink job failures may happen due to >> >>>> cluster >> >>>>>>> node >> >>>>>>>>>>> issues >> >>>>>>>>>>>>>>>>>>>>> (insufficient >> >>>>>>>>>>>>>>>>>>>>>>> disk >> >>>>>>>>>>>>>>>>>>>>>>>> space, bad hardware, network >> >>>> abnormalities). >> >>>>>>>> Flink >> >>>>>>>>>> will >> >>>>>>>>>>>>>> take care >> >>>>>>>>>>>>>>>>>> of >> >>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>>> failures and redeploy the tasks. >> >>> However, >> >>>>> due >> >>>>>>> to >> >>>>>>>>> data >> >>>>>>>>>>>>>> locality >> >>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>>>> limited >> >>>>>>>>>>>>>>>>>>>>>>>> resources, the new tasks are very >> >> likely >> >>>> to >> >>>>> be >> >>>>>>>>>>> redeployed >> >>>>>>>>>>>>>> to the >> >>>>>>>>>>>>>>>>>> same >> >>>>>>>>>>>>>>>>>>>>>>>> nodes, which will result in continuous >> >>>> task >> >>>>>>>>>>> abnormalities >> >>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>> affect >> >>>>>>>>>>>>>>>>>>>>> job >> >>>>>>>>>>>>>>>>>>>>>>>> progress. >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Currently, Flink users need to >> >> manually >> >>>>>>> identify >> >>>>>>>>> the >> >>>>>>>>>>>>>> problematic >> >>>>>>>>>>>>>>>>>>>>> node and >> >>>>>>>>>>>>>>>>>>>>>>>> take it offline to solve this problem. >> >>> But >> >>>>>>> this >> >>>>>>>>>>> approach >> >>>>>>>>>>>>> has >> >>>>>>>>>>>>>>>>>>>>> following >> >>>>>>>>>>>>>>>>>>>>>>>> disadvantages: >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> 1. Taking a node offline can be a >> >> heavy >> >>>>>>> process. >> >>>>>>>>>> Users >> >>>>>>>>>>>>> may >> >>>>>>>>>>>>>> need >> >>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>>>> contact >> >>>>>>>>>>>>>>>>>>>>>>>> cluster administors to do this. The >> >>>>> operation >> >>>>>>> can >> >>>>>>>>>> even >> >>>>>>>>>>> be >> >>>>>>>>>>>>>>>>> dangerous >> >>>>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>>>>>>> allowed during some important business >> >>>>> events. >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> 2. Identifying and solving this kind >> >> of >> >>>>>>> problems >> >>>>>>>>>>> manually >> >>>>>>>>>>>>>> would >> >>>>>>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>>>>>> slow >> >>>>>>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>>>>> a waste of human resources. >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> To solve this problem, Zhu Zhu and I >> >>>> propose >> >>>>>>> to >> >>>>>>>>>>>>> introduce a >> >>>>>>>>>>>>>>>>>> blacklist >> >>>>>>>>>>>>>>>>>>>>>>>> mechanism for Flink to filter out >> >>>>> problematic >> >>>>>>>>>>> resources. >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> You can find more details in >> >>> FLIP-224[1]. >> >>>>>>> Looking >> >>>>>>>>>>> forward >> >>>>>>>>>>>>>> to your >> >>>>>>>>>>>>>>>>>>>>>>> feedback. >> >>>>>>>>>>>>>>>>>>>>>>>> [1] >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-224%3A+Blacklist+Mechanism >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Best, >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Lijie >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> Best regards, >> >>>>>>>>> Roman Boyko >> >>>>>>>>> e.: ro.v.bo...@gmail.com >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> https://twitter.com/snntrable >> >>>> https://github.com/knaufk >> >>>> >> >>> >> >> >> >> >> >> -- >> >> https://twitter.com/snntrable >> >> https://github.com/knaufk >> >> >> >>
