Hi, thank you all for supporting this proposal! Recently, I will split this huge PR to small PRs.
Thank you again! - zhangjian > 2024年5月30日 11:57，Xiaoqiao He <hexiaoq...@apache.org> 写道： > > Great! It looks like there are no other nothing blockers. > > @zhangjian <1361320...@qq.com> If no other furthermore comments, we should > go to the next step: > a. Create a dev branch for this proposal. > b. Split this huge PR to some small JIRA and PRs. > c. Involve some folks to review PR. > > Please ping here if you need any help. Thanks again. > Good Luck! > > Best Regards, > - He Xiaoqiao > > On Wed, May 29, 2024 at 11:46 AM Ayush Saxena <ayush...@gmail.com> wrote: > >> Thanx for the details, sounds cool, good luck with the feature!!! >> >> -Ayush >> >>> On 29 May 2024, at 8:56 AM, zhangjian <1361320...@qq.com> wrote: >>> >>> Thank you for taking the time to review this proposal. >>> Your opinion does point out the key issues to designing an asynchronous >> router, but my proposal can address these issues: >>> 1. My design does not affect the functionality of existing synchronous >> routers in throwing stanby or retry exceptions and other aspects, the async >> router still inherits these implementations. >>> 2. Currently, both asynchronous router and sync router support >> backpressure on client requests when they exceed a certain limit ( >>> asynchronous router : cannot obtain semaphores through the handler, >>> sync router : block through handler synchronization, unable to obtain >> available handler >>> ) >>> and return standby exception to allow the client to retry other routers >> (RouterRpcFairnessPolicyController mechanism). >>> >>> Thank you again! >>> zhangjian >>> >>>> 2024年5月29日 07:05，Ayush Saxena <ayush...@gmail.com> 写道： >>>> Thanx folks, I had a very quick pass on the PDF and it looks good. >>>> Maybe some doubts around the fact where it was mentioned that if a >>>> Namenode returns a StandbyException or something on similar lines, the >>>> Router will retry, I think we have some logic in RouterRpcClient >>>> checking for such case, if it is StandByException it does try the >>>> other Namenode, but for all other Retryable Exceptions, we return them >>>> back to the client & let the client operate according to its Retry >>>> Policy, I think we should preserve that behaviour, if the intentions >>>> were to change it. >>>> Regarding controlling the concurrency to prevent OOM at the router, >>>> maybe we should consider rejecting the client requests beyond a >>>> certain limit/backlog & return back a relevant Retriable Exception to >>>> the client, so that it can retry on another Router rather than >>>> overloading one Router when there are other available, most of the >>>> deployments I believe would be running considerable number of Routers >>>> Rest I scratched my head for possible scenario where things can go >>>> south, but I think mostly the scenarios that came into my mind are >>>> covered >>>> Nothing blocker from my side, Good Luck!!! >>>> -Ayush >>>>> On Tue, 28 May 2024 at 21:52, Sangjin Lee <sj...@apache.org> wrote: >>>>> Sounds good. Thanks for sharing your findings. >>>>> On Sat, May 25, 2024 at 2:24 AM zhangjian <1361320...@qq.com> wrote: >>>>>> Hello everyone, I conducted a performance comparison test between >> sync and asynchronous router, and the test results showed that in single ns >> or multi ns scenarios, Asynchronous router in terms of throughput The >> utilization of CPU and thread, as well as the average processing time of >> client requests, are better than those of sync router, especially when >> downstream ns have performance bottlenecks, The performance of the async >> router is far greater than that of the sync router; And in terms of >> isolation, Asynchronous router is also better than sync router. >>>>>> Detailed testing PDF: >> https://issues.apache.org/jira/browse/HDFS-17531 Comparison of Async >> router & sync router performance.pdf >>>>>> 2024年5月24日 14:13，Yuanbo Liu <liuyuanb...@gmail.com> 写道： >>>>>> good job! >>>>>> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com> wrote: >>>>>>> Hello everyone, currently, I have tested the performance of async >> and sync router for a downstream ns: >>>>>>> 1. The throughput, CPU, and thread performance of the async router >> are better than those of the sync router, and its memory performance is >> within an acceptable range compared to the synchronous router. >>>>>>> 2. Asynchronous router can apply pressure downstream to better >> utilize the performance of downstream ns, and can almost fill the call >> queue of downstream ns. >>>>>>> Due to the large size of the test result pdf, it cannot be sent via >> email, >>>>>>> please see: https://issues.apache.org/jira/browse/HDFS-17531 >>>>>>>> 2024年5月23日 17:03，Xiaoqiao He <hexiaoq...@apache.org> 写道： >>>>>>>> Great. Thanks for your addendum information. >>>>>>>> cc @Ayush Saxena <ayush...@gmail.com> @inigo...@apache.org >>>>>>>> <inigo...@apache.org> Any more feedback for this proposal? >>>>>>>> IMO The feature of asynchronous router RPC is a helpful >> improvement. For my >>>>>>>> internal practice, it will improve the throughput of requests >> forward >>>>>>>> significantly >>>>>>>> and is very valuable to push it forward. >>>>>>>> Thanks again and good luck! >>>>>>>> Best Regards, >>>>>>>> - He Xiaoqiao >>>>>>>> On Wed, May 22, 2024 at 9:59 AM zhangjian <1361320...@qq.com> >> wrote: >>>>>>>>> Hi, Sangjin Lee, thank you for your attention. I will use my free >> time to >>>>>>>>> do a performance comparison recently. >>>>>>>>>> 2024年5月22日 03:42，Sangjin Lee <sj...@apache.org> 写道： >>>>>>>>>> Thanks for the great proposal, Zhangjian. On point #3, I suspect >> it >>>>>>>>> should >>>>>>>>>> be fairly straightforward to create a small isolated synthetic >> test to >>>>>>>>>> prove (or disprove) the benefits of this approach. By driving a >>>>>>>>> controlled >>>>>>>>>> amount of requests per second, you could see latency, memory, >> CPU, etc. >>>>>>>>>> Ideally, it should show meaningful improvements without much >> degradation >>>>>>>>> in >>>>>>>>>> other metrics. Would you be able to spend some time doing that? >>>>>>>>>> Thanks, >>>>>>>>>> Sangjin >>>>>>>>>> On Tue, May 21, 2024 at 5:13 AM zhangjian >> <1361320...@qq.com.invalid> >>>>>>>>> wrote: >>>>>>>>>>> Hi, xiaoqiao he, thank you for your reply. >>>>>>>>>>> 1.Currently, the server and client protocols within router can be >>>>>>>>>>> implemented by extends existing protocols and adding asynchronous >>>>>>>>>>> functionality, so it will not affect existing synchronization >> protocols. >>>>>>>>>>> RouterClientNamenodeProtocolServerSideTranslatorPB >>>>>>>>>>> RouterClientProtocolTranslatorPB >>>>>>>>>>> RouterGetUserMappingsProtocolServerSideTranslatorPB >>>>>>>>>>> RouterGetUserMappingsProtocolTranslatorPB >>>>>>>>>>> RouterNamenodeProtocolServerSideTranslatorPB >>>>>>>>>>> RouterNamenodeProtocolTranslatorPB >>>>>>>>>>> RouterRefreshUserMappingsProtocolServerSideTranslatorPB >>>>>>>>>>> RouterRefreshUserMappingsProtocolTranslatorPB >>>>>>>>>>> The following issues have implemented asynchronous callbacks for >>>>>>>>>>> Rpc.server, but I have not found any other modules to use related >>>>>>>>> functions >>>>>>>>>>> Server HADOOP-11552 HADOOP-17046 >>>>>>>>>>> In the implementation of asynchronous Rpc.client, this issue is >> directly >>>>>>>>>>> used >>>>>>>>>>> Client HADOOP-13226 >>>>>>>>>>> Therefore, I believe that asynchronous routers are safe for >> modifying >>>>>>>>> the >>>>>>>>>>> RPC protocol, RPC server, and client >>>>>>>>>>> 2. Forwarding requests to multiple downstream ns, the >> synchronous router >>>>>>>>>>> handler adds requests from multiple downstream ns to the thread >> pool >>>>>>>>>>> (RouterRpcClient.executorService), and then waits for responses >> from all >>>>>>>>>>> downstream ns before returning. Since threads in the thread pool >> also >>>>>>>>>>> process rpc requests synchronously, similar to a handler, the >> number of >>>>>>>>>>> threads in the thread pool directly affects the performance of >>>>>>>>>>> invoiceConcurrent, which in turn affects the performance of the >> handler. >>>>>>>>>>> In asynchronous router implementation, the handler calls >>>>>>>>> invoiceConcurrent >>>>>>>>>>> to simply convert a request into multiple requests and add them >> to the >>>>>>>>> asyn >>>>>>>>>>> handler thread pool, which can then process the next request in >> the call >>>>>>>>>>> queue; When a connection thread of a downstream ns receives a >> response, >>>>>>>>> it >>>>>>>>>>> will hand it over to the async response for processing. The async >>>>>>>>> response >>>>>>>>>>> thread will determine whether it has received all responses from >> the >>>>>>>>>>> downstream ns. If it does, it will continue to process the >> response. >>>>>>>>>>> Otherwise, the async response thread will process the next >> response. The >>>>>>>>>>> asynchronous router uses CompletableFuture.allOf() to implement >>>>>>>>>>> asynchronous invoiceConcurrent, and the handler, async handler, >> async >>>>>>>>>>> response, and connection thread still does not need to wait >>>>>>>>> synchronously. >>>>>>>>>>> In addition, synchronous routers not only have drawbacks in >> multi ns >>>>>>>>>>> environments, but also in single downstream ns situations, it is >> often >>>>>>>>>>> difficult to decide how many handlers to set for the router, >> setting it >>>>>>>>> too >>>>>>>>>>> much will waste thread resources, and setting it too small will >> not be >>>>>>>>> able >>>>>>>>>>> to give pressure to downstream ns; Asynchronous routers can push >>>>>>>>> requests >>>>>>>>>>> to downstream ns without considering how to set handlers. >> Asynchronous >>>>>>>>>>> routers can also better connect to more downstream storage >> services that >>>>>>>>>>> support the HDFS protocol, with better scalability. >>>>>>>>>>> 3.Since I have not yet deployed asynchronous routers to our own >> cluster, >>>>>>>>>>> there is no performance comparison. However, theoretically, I >> believe >>>>>>>>> that >>>>>>>>>>> asynchronous routers will occupy more memory than synchronous >> routers. >>>>>>>>>>> However, I do not believe that it will occupy a lot, especially >> since we >>>>>>>>>>> can control the maximum number of requests entering the router, >> as >>>>>>>>>>> CompletableFuture is stable and widely used; In other aspects, it >>>>>>>>> should be >>>>>>>>>>> far superior to synchronous routers, especially in downstream >> scenarios >>>>>>>>>>> with more ns.If anyone is interested, you can also help to make a >>>>>>>>>>> performance comparison >>>>>>>>>>>> 2024年5月21日 11:39，Xiaoqiao He <hexiaoq...@apache.org> 写道： >>>>>>>>>>>> Thanks for this great proposal! >>>>>>>>>>>> Some questions after reviewing the design doc (sorry didn't >> review PR >>>>>>>>>>>> carefully which is too large.) >>>>>>>>>>>> 1. This solution will involve RPC framework update, will it >> affect >>>>>>>>> other >>>>>>>>>>>> modules and how to >>>>>>>>>>>> keep other modules off these changes. >>>>>>>>>>>> 2. Some RPC requests should be forward concurrently to all >> downstream >>>>>>>>> NS, >>>>>>>>>>>> will it cover >>>>>>>>>>>> this case in this solution. >>>>>>>>>>>> 3. Considering there is one init-version implementation, did you >>>>>>>>> collect >>>>>>>>>>>> some benchmark vs >>>>>>>>>>>> the current synchronous model of DFSRouter? >>>>>>>>>>>> Thanks again. >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> - He Xiaoqiao >>>>>>>>>>>> On Tue, May 21, 2024 at 11:21 AM zhangjian >> <1361320...@qq.com.invalid> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> Thank you for your positive attitude towards this feature. You >> can >>>>>>>>> debug >>>>>>>>>>>>> the UTs provided in PR to better understand the current >> asynchronous >>>>>>>>>>>>> calling function. >>>>>>>>>>>>>> 2024年5月21日 02:04，Simbarashe Dzinamarira < >> simbadz...@apache.org> 写道： >>>>>>>>>>>>>> Excited to see this feature as well. I'll spend more time >>>>>>>>> understanding >>>>>>>>>>>>> the >>>>>>>>>>>>>> proposal and implementation. >>>>>>>>>>>>>> On Mon, May 20, 2024 at 7:55 AM zhangjian >> <1361320...@qq.com.invalid >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> Hi, Yuanbo liu, thank you for your interest in this >> feature, I >>>>>>>>> think >>>>>>>>>>>>> the >>>>>>>>>>>>>>> difficulty of an asynchronous router is not only to implement >>>>>>>>>>>>> asynchronous >>>>>>>>>>>>>>> functions, but also to consider the readability and >> reusability of >>>>>>>>> the >>>>>>>>>>>>>>> code, so as to facilitate the development of the community. >> I also >>>>>>>>>>>>> planned >>>>>>>>>>>>>>> to do the virtual thread you mentioned at the beginning, >> virtual >>>>>>>>>>> Threads >>>>>>>>>>>>>>> can achieve asynchronousization elegantly at the code level, >> but the >>>>>>>>>>>>>>> biggest problem is that it is not easy to upgrade the jdk >> version, >>>>>>>>> no >>>>>>>>>>>>>>> matter in the community or in the actual production >> environment. >>>>>>>>>>>>> Therefore, >>>>>>>>>>>>>>> I later used CompletableFuture, which is currently supported >> by jdk >>>>>>>>> 8, >>>>>>>>>>>>> to >>>>>>>>>>>>>>> achieve asynchronousization. The router is stateless, and >> the router >>>>>>>>>>> rpc >>>>>>>>>>>>>>> process is very clear. Therefore, even if CompletableFuture >> itself >>>>>>>>> is >>>>>>>>>>>>> not >>>>>>>>>>>>>>> as readable as the virtual thread, if we design it well, we >> can make >>>>>>>>>>> the >>>>>>>>>>>>>>> asynchronous process look very clear. >>>>>>>>>>>>>>>> 2024年5月20日 10:56，Yuanbo Liu <liuyuanb...@gmail.com> 写道： >>>>>>>>>>>>>>>> Nice to see this feature brought up. I tried to implement >> this >>>>>>>>>>> feature >>>>>>>>>>>>> in >>>>>>>>>>>>>>>> our internal clusters, and know that it's a very complicated >>>>>>>>> feature, >>>>>>>>>>>>> CC >>>>>>>>>>>>>>>> hdfs-dev to bring more discussion. >>>>>>>>>>>>>>>> By the way, I'm not sure whether virtual thread of higher >> jdk will >>>>>>>>>>> help >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> this case. >>>>>>>>>>>>>>>> On Mon, May 20, 2024 at 10:10 AM zhangjian >>>>>>>>> <1361320...@qq.com.invalid >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> Hello everyone, currently there are some shortcomings in >> the RPC >>>>>>>>> of >>>>>>>>>>>>> HDFS >>>>>>>>>>>>>>>>> router： >>>>>>>>>>>>>>>>> Currently the router's handler thread is synchronized, >> when the >>>>>>>>>>>>>>> *handler* thread >>>>>>>>>>>>>>>>> adds the call to connection.calls, it needs to wait until >> the >>>>>>>>>>>>>>> *connection* notifies >>>>>>>>>>>>>>>>> the call to complete, and then Only after the response is >> put into >>>>>>>>>>> the >>>>>>>>>>>>>>>>> response queue can a new call be obtained from the call >> queue and >>>>>>>>>>>>>>>>> processed. Therefore, the concurrency performance of the >> router is >>>>>>>>>>>>>>> limited >>>>>>>>>>>>>>>>> by the number of handlers; a simple example is as follows: >> If the >>>>>>>>>>>>>>> number of >>>>>>>>>>>>>>>>> handlers is 1 and the maximum number of calls in the >> connection >>>>>>>>>>> thread >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> 10, then even if the connection thread can send 10 >> requests to the >>>>>>>>>>>>>>>>> downstream ns, since the number of handlers is 1, the >> router can >>>>>>>>>>> only >>>>>>>>>>>>>>>>> process one request after another. >>>>>>>>>>>>>>>>> Since the performance of router rpc is mainly limited by >> the >>>>>>>>> number >>>>>>>>>>> of >>>>>>>>>>>>>>>>> handlers, the most effective way to improve rpc performance >>>>>>>>>>> currently >>>>>>>>>>>>>>> is to >>>>>>>>>>>>>>>>> increase the number of handlers. Letting the router create >> a large >>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>> of handler threads will also increase the number of thread >>>>>>>>> switches >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> cannot maximize the use of machine performance. >>>>>>>>>>>>>>>>> There are usually multiple ns downstream of the router. If >> the >>>>>>>>>>> handler >>>>>>>>>>>>>>>>> forwards the request to an ns with poor performance, it >> will cause >>>>>>>>>>> the >>>>>>>>>>>>>>>>> handler to wait for a long time. Due to the reduction of >> available >>>>>>>>>>>>>>>>> handlers, the router's ability to handle ns requests with >> normal >>>>>>>>>>>>>>>>> performance will be reduced. From the perspective of the >> client, >>>>>>>>> the >>>>>>>>>>>>>>>>> performance of the downstream ns of the router has >> deteriorated at >>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> time. We often find that the call queue of the downstream >> ns is >>>>>>>>> not >>>>>>>>>>>>>>> high, >>>>>>>>>>>>>>>>> but the call queue of the router is very high. >>>>>>>>>>>>>>>>> Therefore, although the main function of the router is to >> federate >>>>>>>>>>> and >>>>>>>>>>>>>>>>> handle requests from multiple NSs, the current synchronous >> RPC >>>>>>>>>>>>>>> performance >>>>>>>>>>>>>>>>> cannot satisfy the scenario where there are many NSs >> downstream of >>>>>>>>>>> the >>>>>>>>>>>>>>>>> router. Even if the concurrent performance of the router >> can be >>>>>>>>>>>>>>> improved by >>>>>>>>>>>>>>>>> increasing the number of handlers, it is still relatively >> slow. >>>>>>>>> More >>>>>>>>>>>>>>>>> threads will increase the CPU context switching time, and >> in fact >>>>>>>>>>> many >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> the handler threads are in a blocked state, which is >> undoubtedly a >>>>>>>>>>>>>>> waste of >>>>>>>>>>>>>>>>> thread resources. When a request enters the router, there >> is no >>>>>>>>>>>>>>> guarantee >>>>>>>>>>>>>>>>> that there will be a running handler at this time. >>>>>>>>>>>>>>>>> Therefore, I consider asynchronous router rpc. Please view >> the >>>>>>>>>>> issues: >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-17531 for the >>>>>>>>> complete >>>>>>>>>>>>>>>>> solution. >>>>>>>>>>>>>>>>> And you can also view this PR: >>>>>>>>>>>>>>> https://github.com/apache/hadoop/pull/6838, >>>>>>>>>>>>>>>>> which is just a demo, but it completes the core >> asynchronous RPC >>>>>>>>>>>>>>> function. >>>>>>>>>>>>>>>>> If you think asynchronous routing is feasible, we can >> consider >>>>>>>>>>>>> splitting >>>>>>>>>>>>>>>>> this PR for easy review in the future. >>>>>>>>>>>>>>>>> The PDF is attached and can also be viewed through issues. >>>>>>>>>>>>>>>>> Welcome everyone to exchange and discuss! >>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe, e-mail: >> common-dev-unsubscr...@hadoop.apache.org >>>>>>>>>>>>>>> For additional commands, e-mail: >> common-dev-h...@hadoop.apache.org >>>>>>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >>>>>>>>>>>>> For additional commands, e-mail: >> hdfs-dev-h...@hadoop.apache.org >>>>>>>>>>> >> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>>>>>>>>>> For additional commands, e-mail: >> common-dev-h...@hadoop.apache.org >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
