Great! It looks like there are no other nothing blockers. @zhangjian <1361320...@qq.com> If no other furthermore comments, we should go to the next step: a. Create a dev branch for this proposal. b. Split this huge PR to some small JIRA and PRs. c. Involve some folks to review PR.
Please ping here if you need any help. Thanks again. Good Luck! Best Regards, - He Xiaoqiao On Wed, May 29, 2024 at 11:46 AM Ayush Saxena <ayush...@gmail.com> wrote: > Thanx for the details, sounds cool, good luck with the feature!!! > > -Ayush > > > On 29 May 2024, at 8:56 AM, zhangjian <1361320...@qq.com> wrote: > > > > Thank you for taking the time to review this proposal. > > Your opinion does point out the key issues to designing an asynchronous > router, but my proposal can address these issues: > > 1. My design does not affect the functionality of existing synchronous > routers in throwing stanby or retry exceptions and other aspects, the async > router still inherits these implementations. > > 2. Currently, both asynchronous router and sync router support > backpressure on client requests when they exceed a certain limit ( > > asynchronous router : cannot obtain semaphores through the handler, > > sync router : block through handler synchronization, unable to obtain > available handler > > ) > > and return standby exception to allow the client to retry other routers > (RouterRpcFairnessPolicyController mechanism). > > > > Thank you again! > > zhangjian > > > >> 2024年5月29日 07:05，Ayush Saxena <ayush...@gmail.com> 写道： > >> Thanx folks, I had a very quick pass on the PDF and it looks good. > >> Maybe some doubts around the fact where it was mentioned that if a > >> Namenode returns a StandbyException or something on similar lines, the > >> Router will retry, I think we have some logic in RouterRpcClient > >> checking for such case, if it is StandByException it does try the > >> other Namenode, but for all other Retryable Exceptions, we return them > >> back to the client & let the client operate according to its Retry > >> Policy, I think we should preserve that behaviour, if the intentions > >> were to change it. > >> Regarding controlling the concurrency to prevent OOM at the router, > >> maybe we should consider rejecting the client requests beyond a > >> certain limit/backlog & return back a relevant Retriable Exception to > >> the client, so that it can retry on another Router rather than > >> overloading one Router when there are other available, most of the > >> deployments I believe would be running considerable number of Routers > >> Rest I scratched my head for possible scenario where things can go > >> south, but I think mostly the scenarios that came into my mind are > >> covered > >> Nothing blocker from my side, Good Luck!!! > >> -Ayush > >>> On Tue, 28 May 2024 at 21:52, Sangjin Lee <sj...@apache.org> wrote: > >>> Sounds good. Thanks for sharing your findings. > >>> On Sat, May 25, 2024 at 2:24 AM zhangjian <1361320...@qq.com> wrote: > >>>> Hello everyone, I conducted a performance comparison test between > sync and asynchronous router, and the test results showed that in single ns > or multi ns scenarios, Asynchronous router in terms of throughput The > utilization of CPU and thread, as well as the average processing time of > client requests, are better than those of sync router, especially when > downstream ns have performance bottlenecks, The performance of the async > router is far greater than that of the sync router; And in terms of > isolation, Asynchronous router is also better than sync router. > >>>> Detailed testing PDF: > https://issues.apache.org/jira/browse/HDFS-17531 Comparison of Async > router & sync router performance.pdf > >>>> 2024年5月24日 14:13，Yuanbo Liu <liuyuanb...@gmail.com> 写道： > >>>> good job! > >>>> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com> wrote: > >>>>> Hello everyone, currently, I have tested the performance of async > and sync router for a downstream ns: > >>>>> 1. The throughput, CPU, and thread performance of the async router > are better than those of the sync router, and its memory performance is > within an acceptable range compared to the synchronous router. > >>>>> 2. Asynchronous router can apply pressure downstream to better > utilize the performance of downstream ns, and can almost fill the call > queue of downstream ns. > >>>>> Due to the large size of the test result pdf, it cannot be sent via > email, > >>>>> please see: https://issues.apache.org/jira/browse/HDFS-17531 > >>>>>> 2024年5月23日 17:03，Xiaoqiao He <hexiaoq...@apache.org> 写道： > >>>>>> Great. Thanks for your addendum information. > >>>>>> cc @Ayush Saxena <ayush...@gmail.com> @inigo...@apache.org > >>>>>> <inigo...@apache.org> Any more feedback for this proposal? > >>>>>> IMO The feature of asynchronous router RPC is a helpful > improvement. For my > >>>>>> internal practice, it will improve the throughput of requests > forward > >>>>>> significantly > >>>>>> and is very valuable to push it forward. > >>>>>> Thanks again and good luck! > >>>>>> Best Regards, > >>>>>> - He Xiaoqiao > >>>>>> On Wed, May 22, 2024 at 9:59 AM zhangjian <1361320...@qq.com> > wrote: > >>>>>>> Hi, Sangjin Lee, thank you for your attention. I will use my free > time to > >>>>>>> do a performance comparison recently. > >>>>>>>> 2024年5月22日 03:42，Sangjin Lee <sj...@apache.org> 写道： > >>>>>>>> Thanks for the great proposal, Zhangjian. On point #3, I suspect > it > >>>>>>> should > >>>>>>>> be fairly straightforward to create a small isolated synthetic > test to > >>>>>>>> prove (or disprove) the benefits of this approach. By driving a > >>>>>>> controlled > >>>>>>>> amount of requests per second, you could see latency, memory, > CPU, etc. > >>>>>>>> Ideally, it should show meaningful improvements without much > degradation > >>>>>>> in > >>>>>>>> other metrics. Would you be able to spend some time doing that? > >>>>>>>> Thanks, > >>>>>>>> Sangjin > >>>>>>>> On Tue, May 21, 2024 at 5:13 AM zhangjian > <1361320...@qq.com.invalid> > >>>>>>> wrote: > >>>>>>>>> Hi, xiaoqiao he, thank you for your reply. > >>>>>>>>> 1.Currently, the server and client protocols within router can be > >>>>>>>>> implemented by extends existing protocols and adding asynchronous > >>>>>>>>> functionality, so it will not affect existing synchronization > protocols. > >>>>>>>>> RouterClientNamenodeProtocolServerSideTranslatorPB > >>>>>>>>> RouterClientProtocolTranslatorPB > >>>>>>>>> RouterGetUserMappingsProtocolServerSideTranslatorPB > >>>>>>>>> RouterGetUserMappingsProtocolTranslatorPB > >>>>>>>>> RouterNamenodeProtocolServerSideTranslatorPB > >>>>>>>>> RouterNamenodeProtocolTranslatorPB > >>>>>>>>> RouterRefreshUserMappingsProtocolServerSideTranslatorPB > >>>>>>>>> RouterRefreshUserMappingsProtocolTranslatorPB > >>>>>>>>> The following issues have implemented asynchronous callbacks for > >>>>>>>>> Rpc.server, but I have not found any other modules to use related > >>>>>>> functions > >>>>>>>>> Server HADOOP-11552 HADOOP-17046 > >>>>>>>>> In the implementation of asynchronous Rpc.client, this issue is > directly > >>>>>>>>> used > >>>>>>>>> Client HADOOP-13226 > >>>>>>>>> Therefore, I believe that asynchronous routers are safe for > modifying > >>>>>>> the > >>>>>>>>> RPC protocol, RPC server, and client > >>>>>>>>> 2. Forwarding requests to multiple downstream ns, the > synchronous router > >>>>>>>>> handler adds requests from multiple downstream ns to the thread > pool > >>>>>>>>> (RouterRpcClient.executorService), and then waits for responses > from all > >>>>>>>>> downstream ns before returning. Since threads in the thread pool > also > >>>>>>>>> process rpc requests synchronously, similar to a handler, the > number of > >>>>>>>>> threads in the thread pool directly affects the performance of > >>>>>>>>> invoiceConcurrent, which in turn affects the performance of the > handler. > >>>>>>>>> In asynchronous router implementation, the handler calls > >>>>>>> invoiceConcurrent > >>>>>>>>> to simply convert a request into multiple requests and add them > to the > >>>>>>> asyn > >>>>>>>>> handler thread pool, which can then process the next request in > the call > >>>>>>>>> queue; When a connection thread of a downstream ns receives a > response, > >>>>>>> it > >>>>>>>>> will hand it over to the async response for processing. The async > >>>>>>> response > >>>>>>>>> thread will determine whether it has received all responses from > the > >>>>>>>>> downstream ns. If it does, it will continue to process the > response. > >>>>>>>>> Otherwise, the async response thread will process the next > response. The > >>>>>>>>> asynchronous router uses CompletableFuture.allOf() to implement > >>>>>>>>> asynchronous invoiceConcurrent, and the handler, async handler, > async > >>>>>>>>> response, and connection thread still does not need to wait > >>>>>>> synchronously. > >>>>>>>>> In addition, synchronous routers not only have drawbacks in > multi ns > >>>>>>>>> environments, but also in single downstream ns situations, it is > often > >>>>>>>>> difficult to decide how many handlers to set for the router, > setting it > >>>>>>> too > >>>>>>>>> much will waste thread resources, and setting it too small will > not be > >>>>>>> able > >>>>>>>>> to give pressure to downstream ns; Asynchronous routers can push > >>>>>>> requests > >>>>>>>>> to downstream ns without considering how to set handlers. > Asynchronous > >>>>>>>>> routers can also better connect to more downstream storage > services that > >>>>>>>>> support the HDFS protocol, with better scalability. > >>>>>>>>> 3.Since I have not yet deployed asynchronous routers to our own > cluster, > >>>>>>>>> there is no performance comparison. However, theoretically, I > believe > >>>>>>> that > >>>>>>>>> asynchronous routers will occupy more memory than synchronous > routers. > >>>>>>>>> However, I do not believe that it will occupy a lot, especially > since we > >>>>>>>>> can control the maximum number of requests entering the router, > as > >>>>>>>>> CompletableFuture is stable and widely used; In other aspects, it > >>>>>>> should be > >>>>>>>>> far superior to synchronous routers, especially in downstream > scenarios > >>>>>>>>> with more ns.If anyone is interested, you can also help to make a > >>>>>>>>> performance comparison > >>>>>>>>>> 2024年5月21日 11:39，Xiaoqiao He <hexiaoq...@apache.org> 写道： > >>>>>>>>>> Thanks for this great proposal! > >>>>>>>>>> Some questions after reviewing the design doc (sorry didn't > review PR > >>>>>>>>>> carefully which is too large.) > >>>>>>>>>> 1. This solution will involve RPC framework update, will it > affect > >>>>>>> other > >>>>>>>>>> modules and how to > >>>>>>>>>> keep other modules off these changes. > >>>>>>>>>> 2. Some RPC requests should be forward concurrently to all > downstream > >>>>>>> NS, > >>>>>>>>>> will it cover > >>>>>>>>>> this case in this solution. > >>>>>>>>>> 3. Considering there is one init-version implementation, did you > >>>>>>> collect > >>>>>>>>>> some benchmark vs > >>>>>>>>>> the current synchronous model of DFSRouter? > >>>>>>>>>> Thanks again. > >>>>>>>>>> Best Regards, > >>>>>>>>>> - He Xiaoqiao > >>>>>>>>>> On Tue, May 21, 2024 at 11:21 AM zhangjian > <1361320...@qq.com.invalid> > >>>>>>>>>> wrote: > >>>>>>>>>>> Thank you for your positive attitude towards this feature. You > can > >>>>>>> debug > >>>>>>>>>>> the UTs provided in PR to better understand the current > asynchronous > >>>>>>>>>>> calling function. > >>>>>>>>>>>> 2024年5月21日 02:04，Simbarashe Dzinamarira < > simbadz...@apache.org> 写道： > >>>>>>>>>>>> Excited to see this feature as well. I'll spend more time > >>>>>>> understanding > >>>>>>>>>>> the > >>>>>>>>>>>> proposal and implementation. > >>>>>>>>>>>> On Mon, May 20, 2024 at 7:55 AM zhangjian > <1361320...@qq.com.invalid > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> Hi, Yuanbo liu, thank you for your interest in this > feature, I > >>>>>>> think > >>>>>>>>>>> the > >>>>>>>>>>>>> difficulty of an asynchronous router is not only to implement > >>>>>>>>>>> asynchronous > >>>>>>>>>>>>> functions, but also to consider the readability and > reusability of > >>>>>>> the > >>>>>>>>>>>>> code, so as to facilitate the development of the community. > I also > >>>>>>>>>>> planned > >>>>>>>>>>>>> to do the virtual thread you mentioned at the beginning, > virtual > >>>>>>>>> Threads > >>>>>>>>>>>>> can achieve asynchronousization elegantly at the code level, > but the > >>>>>>>>>>>>> biggest problem is that it is not easy to upgrade the jdk > version, > >>>>>>> no > >>>>>>>>>>>>> matter in the community or in the actual production > environment. > >>>>>>>>>>> Therefore, > >>>>>>>>>>>>> I later used CompletableFuture, which is currently supported > by jdk > >>>>>>> 8, > >>>>>>>>>>> to > >>>>>>>>>>>>> achieve asynchronousization. The router is stateless, and > the router > >>>>>>>>> rpc > >>>>>>>>>>>>> process is very clear. Therefore, even if CompletableFuture > itself > >>>>>>> is > >>>>>>>>>>> not > >>>>>>>>>>>>> as readable as the virtual thread, if we design it well, we > can make > >>>>>>>>> the > >>>>>>>>>>>>> asynchronous process look very clear. > >>>>>>>>>>>>>> 2024年5月20日 10:56，Yuanbo Liu <liuyuanb...@gmail.com> 写道： > >>>>>>>>>>>>>> Nice to see this feature brought up. I tried to implement > this > >>>>>>>>> feature > >>>>>>>>>>> in > >>>>>>>>>>>>>> our internal clusters, and know that it's a very complicated > >>>>>>> feature, > >>>>>>>>>>> CC > >>>>>>>>>>>>>> hdfs-dev to bring more discussion. > >>>>>>>>>>>>>> By the way, I'm not sure whether virtual thread of higher > jdk will > >>>>>>>>> help > >>>>>>>>>>>>> in > >>>>>>>>>>>>>> this case. > >>>>>>>>>>>>>> On Mon, May 20, 2024 at 10:10 AM zhangjian > >>>>>>> <1361320...@qq.com.invalid > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> Hello everyone, currently there are some shortcomings in > the RPC > >>>>>>> of > >>>>>>>>>>> HDFS > >>>>>>>>>>>>>>> router： > >>>>>>>>>>>>>>> Currently the router's handler thread is synchronized, > when the > >>>>>>>>>>>>> *handler* thread > >>>>>>>>>>>>>>> adds the call to connection.calls, it needs to wait until > the > >>>>>>>>>>>>> *connection* notifies > >>>>>>>>>>>>>>> the call to complete, and then Only after the response is > put into > >>>>>>>>> the > >>>>>>>>>>>>>>> response queue can a new call be obtained from the call > queue and > >>>>>>>>>>>>>>> processed. Therefore, the concurrency performance of the > router is > >>>>>>>>>>>>> limited > >>>>>>>>>>>>>>> by the number of handlers; a simple example is as follows: > If the > >>>>>>>>>>>>> number of > >>>>>>>>>>>>>>> handlers is 1 and the maximum number of calls in the > connection > >>>>>>>>> thread > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>> 10, then even if the connection thread can send 10 > requests to the > >>>>>>>>>>>>>>> downstream ns, since the number of handlers is 1, the > router can > >>>>>>>>> only > >>>>>>>>>>>>>>> process one request after another. > >>>>>>>>>>>>>>> Since the performance of router rpc is mainly limited by > the > >>>>>>> number > >>>>>>>>> of > >>>>>>>>>>>>>>> handlers, the most effective way to improve rpc performance > >>>>>>>>> currently > >>>>>>>>>>>>> is to > >>>>>>>>>>>>>>> increase the number of handlers. Letting the router create > a large > >>>>>>>>>>>>> number > >>>>>>>>>>>>>>> of handler threads will also increase the number of thread > >>>>>>> switches > >>>>>>>>>>> and > >>>>>>>>>>>>>>> cannot maximize the use of machine performance. > >>>>>>>>>>>>>>> There are usually multiple ns downstream of the router. If > the > >>>>>>>>> handler > >>>>>>>>>>>>>>> forwards the request to an ns with poor performance, it > will cause > >>>>>>>>> the > >>>>>>>>>>>>>>> handler to wait for a long time. Due to the reduction of > available > >>>>>>>>>>>>>>> handlers, the router's ability to handle ns requests with > normal > >>>>>>>>>>>>>>> performance will be reduced. From the perspective of the > client, > >>>>>>> the > >>>>>>>>>>>>>>> performance of the downstream ns of the router has > deteriorated at > >>>>>>>>>>> this > >>>>>>>>>>>>>>> time. We often find that the call queue of the downstream > ns is > >>>>>>> not > >>>>>>>>>>>>> high, > >>>>>>>>>>>>>>> but the call queue of the router is very high. > >>>>>>>>>>>>>>> Therefore, although the main function of the router is to > federate > >>>>>>>>> and > >>>>>>>>>>>>>>> handle requests from multiple NSs, the current synchronous > RPC > >>>>>>>>>>>>> performance > >>>>>>>>>>>>>>> cannot satisfy the scenario where there are many NSs > downstream of > >>>>>>>>> the > >>>>>>>>>>>>>>> router. Even if the concurrent performance of the router > can be > >>>>>>>>>>>>> improved by > >>>>>>>>>>>>>>> increasing the number of handlers, it is still relatively > slow. > >>>>>>> More > >>>>>>>>>>>>>>> threads will increase the CPU context switching time, and > in fact > >>>>>>>>> many > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>> the handler threads are in a blocked state, which is > undoubtedly a > >>>>>>>>>>>>> waste of > >>>>>>>>>>>>>>> thread resources. When a request enters the router, there > is no > >>>>>>>>>>>>> guarantee > >>>>>>>>>>>>>>> that there will be a running handler at this time. > >>>>>>>>>>>>>>> Therefore, I consider asynchronous router rpc. Please view > the > >>>>>>>>> issues: > >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-17531 for the > >>>>>>> complete > >>>>>>>>>>>>>>> solution. > >>>>>>>>>>>>>>> And you can also view this PR: > >>>>>>>>>>>>> https://github.com/apache/hadoop/pull/6838, > >>>>>>>>>>>>>>> which is just a demo, but it completes the core > asynchronous RPC > >>>>>>>>>>>>> function. > >>>>>>>>>>>>>>> If you think asynchronous routing is feasible, we can > consider > >>>>>>>>>>> splitting > >>>>>>>>>>>>>>> this PR for easy review in the future. > >>>>>>>>>>>>>>> The PDF is attached and can also be viewed through issues. > >>>>>>>>>>>>>>> Welcome everyone to exchange and discuss! > >>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>>> To unsubscribe, e-mail: > common-dev-unsubscr...@hadoop.apache.org > >>>>>>>>>>>>> For additional commands, e-mail: > common-dev-h...@hadoop.apache.org > >>>>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >>>>>>>>>>> For additional commands, e-mail: > hdfs-dev-h...@hadoop.apache.org > >>>>>>>>> > --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > >>>>>>>>> For additional commands, e-mail: > common-dev-h...@hadoop.apache.org > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > --------------------------------------------------------------------- > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
