Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/ [Feb 13, 2025, 1:04:48 PM] (github) YARN-11758. [UI2] On the Cluster Metrics page make the Resource Usage… (#7331) [Feb 13, 2025, 6:09:49 PM] (Chris Nauroth) HADOOP-19375: Organize JDK version-specific code in IDEA friendly approach [Feb 13, 2025, 8:14:40 PM] (Chris Nauroth) HADOOP-19377: Avoid initializing useless HashMap in protocolImplMapArray. -1 overall The following subsystems voted -1: blanks hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) shadedclient unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestListFilesInDFS hadoop.hdfs.crypto.TestHdfsCryptoStreams cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-compile-javac-root.txt [8.0K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-hadolint.txt [20K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/results-javadoc-javadoc-root.txt [240K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1853/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [216K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.
+1. In addition to thanking the developers, special thanks to Xiaoqiao He for pushing this feature. Best Regards, - Shilun Fan On Thu, Feb 13, 2025 at 2:29 PM Xiaoqiao He wrote: > Great. +1 from my side. Thanks. > > Best Regards, > - He Xiaoqiao > > On Thu, Feb 13, 2025 at 10:15 AM jian zhang wrote: > >> Hi, He Xiaoqiao >> >> I have rebased HDFS-17531 again and resolved the conflicts. The current >> pipeline failure is unrelated to the ARR feature and was introduced by >> slfan1989's PR: HADOOP-19415. [JDK17] Upgrade JUnit from 4 to 5 in >> hadoop-common Part 1. (#7339). slfan1989 will fix it later. >> >> Best Regards, >> - Jian Zhang >> >> Xiaoqiao He 于2025年2月11日周二 11:15写道： >> >> > Hi Jian Zhang, Thanks for your great work. Please fix the conflict >> first, >> > others make sense to me. >> > I will give my +1 once it is ready. >> > Another thing, before check in we need to launch another official vote >> > thread. Good luck. >> > >> > BTW, Happy lunar new year! >> > >> > Best Regards, >> > - He Xiaoqiao >> > >> > On Thu, Feb 6, 2025 at 5:30 PM jian zhang >> wrote: >> > >> >> Hi, all, >> >> Currently this feature has been developed and passed the pipeline. >> Please >> >> continue to help review this feature. >> >> >> >> Best Regards, >> >> Jian Zhang >> >> >> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >> >> >> >>> @Hui Fei Hi, Sir: >> >>> For the first opinion, I have create an umbrella JIRA >> >>> https://issues.apache.org/jira/browse/HDFS-17716 >> >>> and move non-core JIRA under it. >> >>> >> >>> Best Wishes >> >>> Haobo Zhang >> >>> >> >>> Replied Message >> >>> From Hui Fei >> >>> Date 01/22/2025 17:37 >> >>> To jian zhang >> >>> Cc Hdfs-dev , >> >>> , >> >>> Xiaoqiao He , >> >>> >> >>> >> >>> Subject Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk. >> >>> Got your idea. Thank you! >> >>> - How about removing unfinished tasks and placing them under a new >> task >> >>> as >> >>> subtasks, like ARR improvements? If this feature is completed but >> there >> >>> are >> >>> still some open tasks, it looks strange. >> >>> - Will it take a long time to add documentation? Discussion may last >> for >> >>> several days. If it takes a long time, I think it may block the trunk >> >>> release and All community members need to remember that there is >> >>> documentation required. It doesn't look good. That's my thought, and >> we >> >>> can >> >>> wait for others' opinions >> >>> >> >>> jian zhang 于2025年1月22日周三 16:53写道： >> >>> >> >>> Hi, Hui Fei, >> >>> - The remaining 3 sub tasks are not related to the core functions of >> >>> the asynchronous router, and these sub tasks have little impact on the >> >>> trunk branch, we can wait until HDFS-17531 is merged into the trunk, >> and >> >>> then submit the remaining PRs directly to the trunk. >> >>> - It is indeed necessary to add a documentation to >> >>> "HDFSRouterFederation.md", how about submitting a PR to do this after >> >>> merging HDFS-17531 into the trunk branch? >> >>> >> >>> Best Regards, >> >>> Jian Zhang >> >>> >> >>> Hui Fei 于2025年1月22日周三 16:24写道： >> >>> >> >>> Thanks for your great work, looking forward to this feature. >> >>> >> >>> Some comments from me. >> >>> - I checked and found that there are still 3 sub tasks under this >> >>> feature jira ticket, are they necessary to be solved? >> >>> - I didn't find the documentation for this feature. It's a key >> feature, >> >>> Is it necessary to add documentation to HDFSRouterFederation.md? >> >>> >> >>> jian zhang 于2025年1月22日周三 10:29写道： >> >>> >> >>> Hi, all, the development of the asynchronous router functionality has >> >>> been completed. The development branch is HDFS-17531, and it is ready >> to >> >>> be >> >>> merged into the trunk branch. >> >>> >> >>> JIRA: HDFS-17531 https://issues.apache.org/jira/browse/HDFS-17531 >> >>> PR: https://github.com/apache/hadoop/pull/7308 >> >>> >> >>> Here is the functionality introduction of the asynchronous router for >> >>> everyone to review: >> >>> I. Overview >> >>> >> >>> The asynchronous router aims to address the performance bottleneck >> >>> issues of the synchronous router in high - concurrency and multi - >> >>> nameservices scenarios. By introducing an asynchronous processing >> >>> mechanism, it optimizes the request handling process, improves the >> >>> system's >> >>> concurrency ability and resource utilization, and is particularly >> >>> suitable >> >>> for the federated scenarios where multiple downstream services (NS) >> need >> >>> to >> >>> be processed. >> >>> >> >>> II. Problems of the Synchronous Router >> >>> >> >>> - Performance Bottleneck: The performance of the synchronous router >> >>> is limited by the number of handler threads. Even if the connection >> >>> thread >> >>> can still forward requests to the downstream namenode, the handler >> must >> >>> wait for each request to complete before processing the next one, >> >>> resulting >> >>> in limited processing capacity. >> >>> - Thread Resou
Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.
+1. In addition to thanking the developers, special thanks to XiaoQiao He for pushing this feature. Best Regards, - Shilun Fan On Fri, Feb 14, 2025 at 3:40 PM Hui Fei wrote: > > Did some test work referring to the documentation. > >- Compiled source codes, built local cluster and the async feature >worked fine. >- it is disabled by default >- can increase or decrease the thread number by changing the related >configurations > > The feature is as described in the documentation. Great work, +1 > > Xiaoqiao He 于2025年2月13日周四 14:29写道： > >> Great. +1 from my side. Thanks. >> >> Best Regards, >> - He Xiaoqiao >> >> On Thu, Feb 13, 2025 at 10:15 AM jian zhang >> wrote: >> >>> Hi, He Xiaoqiao >>> >>> I have rebased HDFS-17531 again and resolved the conflicts. The current >>> pipeline failure is unrelated to the ARR feature and was introduced by >>> slfan1989's PR: HADOOP-19415. [JDK17] Upgrade JUnit from 4 to 5 in >>> hadoop-common Part 1. (#7339). slfan1989 will fix it later. >>> >>> Best Regards, >>> - Jian Zhang >>> >>> Xiaoqiao He 于2025年2月11日周二 11:15写道： >>> >>> > Hi Jian Zhang, Thanks for your great work. Please fix the conflict >>> first, >>> > others make sense to me. >>> > I will give my +1 once it is ready. >>> > Another thing, before check in we need to launch another official vote >>> > thread. Good luck. >>> > >>> > BTW, Happy lunar new year! >>> > >>> > Best Regards, >>> > - He Xiaoqiao >>> > >>> > On Thu, Feb 6, 2025 at 5:30 PM jian zhang >>> wrote: >>> > >>> >> Hi, all, >>> >> Currently this feature has been developed and passed the pipeline. >>> Please >>> >> continue to help review this feature. >>> >> >>> >> Best Regards, >>> >> Jian Zhang >>> >> >>> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >>> >> >>> >>> @Hui Fei Hi, Sir: >>> >>> For the first opinion, I have create an umbrella JIRA >>> >>> https://issues.apache.org/jira/browse/HDFS-17716 >>> >>> and move non-core JIRA under it. >>> >>> >>> >>> Best Wishes >>> >>> Haobo Zhang >>> >>> >>> >>> Replied Message >>> >>> From Hui Fei >>> >>> Date 01/22/2025 17:37 >>> >>> To jian zhang >>> >>> Cc Hdfs-dev , >>> >>> , >>> >>> Xiaoqiao He , >>> >>> >>> >>> >>> >>> Subject Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk. >>> >>> Got your idea. Thank you! >>> >>> - How about removing unfinished tasks and placing them under a new >>> task >>> >>> as >>> >>> subtasks, like ARR improvements? If this feature is completed but >>> there >>> >>> are >>> >>> still some open tasks, it looks strange. >>> >>> - Will it take a long time to add documentation? Discussion may last >>> for >>> >>> several days. If it takes a long time, I think it may block the trunk >>> >>> release and All community members need to remember that there is >>> >>> documentation required. It doesn't look good. That's my thought, and >>> we >>> >>> can >>> >>> wait for others' opinions >>> >>> >>> >>> jian zhang 于2025年1月22日周三 16:53写道： >>> >>> >>> >>> Hi, Hui Fei, >>> >>> - The remaining 3 sub tasks are not related to the core functions of >>> >>> the asynchronous router, and these sub tasks have little impact on >>> the >>> >>> trunk branch, we can wait until HDFS-17531 is merged into the trunk, >>> and >>> >>> then submit the remaining PRs directly to the trunk. >>> >>> - It is indeed necessary to add a documentation to >>> >>> "HDFSRouterFederation.md", how about submitting a PR to do this after >>> >>> merging HDFS-17531 into the trunk branch? >>> >>> >>> >>> Best Regards, >>> >>> Jian Zhang >>> >>> >>> >>> Hui Fei 于2025年1月22日周三 16:24写道： >>> >>> >>> >>> Thanks for your great work, looking forward to this feature. >>> >>> >>> >>> Some comments from me. >>> >>> - I checked and found that there are still 3 sub tasks under this >>> >>> feature jira ticket, are they necessary to be solved? >>> >>> - I didn't find the documentation for this feature. It's a key >>> feature, >>> >>> Is it necessary to add documentation to HDFSRouterFederation.md? >>> >>> >>> >>> jian zhang 于2025年1月22日周三 10:29写道： >>> >>> >>> >>> Hi, all, the development of the asynchronous router functionality has >>> >>> been completed. The development branch is HDFS-17531, and it is >>> ready to >>> >>> be >>> >>> merged into the trunk branch. >>> >>> >>> >>> JIRA: HDFS-17531 https://issues.apache.org/jira/browse/HDFS-17531 >>> >>> PR: https://github.com/apache/hadoop/pull/7308 >>> >>> >>> >>> Here is the functionality introduction of the asynchronous router for >>> >>> everyone to review: >>> >>> I. Overview >>> >>> >>> >>> The asynchronous router aims to address the performance bottleneck >>> >>> issues of the synchronous router in high - concurrency and multi - >>> >>> nameservices scenarios. By introducing an asynchronous processing >>> >>> mechanism, it optimizes the request handling process, improves the >>> >>> system's >>> >>> concurrency ability and resource utilization, and is particularly >>> >>> suitable >>> >>> for the federated scenarios where multiple down
Re: [VOTE] Request to merge branch HDFS-17531 into trunk.
+1 Best Regards, - Shilun Fan On Fri, Feb 14, 2025 at 3:22 PM jian zhang wrote: > Hi, all, the development of the asynchronous router functionality has been > completed. Thanks to all the contributors who participated in this feature. > The development branch is HDFS-17531, and it is ready to be merged into the > trunk branch. > > JIRA: HDFS-17531 https://issues.apache.org/jira/browse/HDFS-17531 > PR: https://github.com/apache/hadoop/pull/7308 > > DISCUSS: https://lists.apache.org/thread/02y3dtpfxt21bxjgmyl3kxnv4m1vwz44 > > Here is the functionality introduction of the asynchronous router for > everyone to review: > I. Overview > > The asynchronous router aims to address the performance bottleneck > issues of the synchronous router in high - concurrency and multi - > nameservices scenarios. By introducing an asynchronous processing > mechanism, it optimizes the request handling process, improves the system's > concurrency ability and resource utilization, and is particularly suitable > for the federated scenarios where multiple downstream services (NS) need to > be processed. > > II. Problems of the Synchronous Router > > - Performance Bottleneck: The performance of the synchronous router is > limited by the number of handler threads. Even if the connection thread can > still forward requests to the downstream namenode, the handler must wait > for each request to complete before processing the next one, resulting in > limited processing capacity. > - Thread Resource Waste: To improve performance, increasing the number > of handler threads will lead to more thread switches, which instead reduces > the system efficiency. At the same time, a large number of handler threads > are in a blocked state, wasting thread resources. > - Poor Isolation in Multi - ns: If the performance of a certain > nameservice in the downstream nameservice is poor, it will cause the > handler to wait for a long time, thus affecting the forwarding of requests > to other normal - performance ns, resulting in a decrease in the overall > performance of the downstream ns services perceived by the client. > - Ineffective Utilization of Federation Multi - ns Performance: In high > - concurrency scenarios, a large number of requests may be backlogged in > the router's request queue, while the queues of downstream services are not > fully utilized, leading to unreasonable resource allocation. > > III. Design and Improvements of the Asynchronous Router > > The asynchronous router solves the above problems by redesigning the > request handling process and introducing an asynchronous processing > mechanism. Its core improvements include: > > - Handler: Retrieves requests from the request queue for preliminary > processing. If there are exceptions in the request (such as the mount point > does not exist, etc.), it directly puts the response into the response > queue; otherwise, it sends the request to the asynchronous handler thread > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately without > blocking and waiting. > - Async Responder: Is responsible for processing the responses received > by the connection thread. If the request needs to be re - initiated (such > as the downstream service returns a standby exception), it re - adds the > request to the asynchronous handler thread pool; otherwise, it puts the > response into the response queue. > - Responder: Retrieves the response from the response queue and returns > it to the client. > > IV. Advantages of the Asynchronous Router > > - High - Concurrency Performance: Through the asynchronous processing > mechanism, the asynchronous router can handle a large number of requests > simultaneously, significantly improving the system's concurrent processing > ability. > - High Resource Utilization: It avoids thread blocking and frequent > switching, reduces thread resource waste, and improves the overall > efficiency of the system. > - Isolation: Different ns are processed by different async handler > thread pools, achieving isolation of different downstream services. Even if > the performance of a certain service is poor, it will not affect the > processing ability of other services. > > V. Summary > > The asynchronous router solves the performance bottleneck problem of > the traditional synchronous router in high - concurrency scenarios by > introducing an asynchronous processing mechanism. It not only improves the > system's concurrency ability and resource utilization but also achieves > isolation of downstream services through the queue mechanism, enhancing the > system's stability and adaptability. In the federated scenarios where > multiple downstream services need to be processed, the asynchronous router > is a more efficient and reliable solution. > VI. Performance Testing > > > > https://docs.google.com/document/d/1meHOCvhm3XRHlIMwvKFidfUSjveTJrb8yAMas
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.server.namenode.ha.TestPipelinesFailover hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.8M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1668/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branc
Re: [VOTE] Request to merge branch HDFS-17531 into trunk.
+1 Best Regards, - He Xiaoqiao On Fri, Feb 14, 2025 at 3:53 PM Ayush Saxena wrote: > +1 > > -Ayush > > > On 14 Feb 2025, at 1:20 PM, Hui Fei wrote: > > > > Did some test work referring to the documentation. > > > > - Compiled source codes, built a local cluster and the async feature > > worked fine. > > - it is disabled by default > > - can increase or decrease the thread number by changing the related > > configurations > > > > +1 > > > > jian zhang 于2025年2月14日周五 15:21写道： > > > >> Hi, all, the development of the asynchronous router functionality has > been > >> completed. Thanks to all the contributors who participated in this > feature. > >> The development branch is HDFS-17531, and it is ready to be merged into > the > >> trunk branch. > >> > >> JIRA: HDFS-17531 https://issues.apache.org/jira/browse/HDFS-17531 > >> PR: https://github.com/apache/hadoop/pull/7308 > >> > >> DISCUSS: > https://lists.apache.org/thread/02y3dtpfxt21bxjgmyl3kxnv4m1vwz44 > >> > >> Here is the functionality introduction of the asynchronous router for > >> everyone to review: > >> I. Overview > >> > >>The asynchronous router aims to address the performance bottleneck > >> issues of the synchronous router in high - concurrency and multi - > >> nameservices scenarios. By introducing an asynchronous processing > >> mechanism, it optimizes the request handling process, improves the > system's > >> concurrency ability and resource utilization, and is particularly > suitable > >> for the federated scenarios where multiple downstream services (NS) > need to > >> be processed. > >> > >> II. Problems of the Synchronous Router > >> > >>- Performance Bottleneck: The performance of the synchronous router > is > >> limited by the number of handler threads. Even if the connection thread > can > >> still forward requests to the downstream namenode, the handler must wait > >> for each request to complete before processing the next one, resulting > in > >> limited processing capacity. > >>- Thread Resource Waste: To improve performance, increasing the > number > >> of handler threads will lead to more thread switches, which instead > reduces > >> the system efficiency. At the same time, a large number of handler > threads > >> are in a blocked state, wasting thread resources. > >>- Poor Isolation in Multi - ns: If the performance of a certain > >> nameservice in the downstream nameservice is poor, it will cause the > >> handler to wait for a long time, thus affecting the forwarding of > requests > >> to other normal - performance ns, resulting in a decrease in the overall > >> performance of the downstream ns services perceived by the client. > >>- Ineffective Utilization of Federation Multi - ns Performance: In > >> high - concurrency scenarios, a large number of requests may be > backlogged > >> in the router's request queue, while the queues of downstream services > are > >> not fully utilized, leading to unreasonable resource allocation. > >> > >> III. Design and Improvements of the Asynchronous Router > >> > >>The asynchronous router solves the above problems by redesigning the > >> request handling process and introducing an asynchronous processing > >> mechanism. Its core improvements include: > >> > >>- Handler: Retrieves requests from the request queue for preliminary > >> processing. If there are exceptions in the request (such as the mount > point > >> does not exist, etc.), it directly puts the response into the response > >> queue; otherwise, it sends the request to the asynchronous handler > thread > >> pool. > >>- Async Handler: Puts the request into the call queue > >> (connection.calls) of the connection thread and returns immediately > without > >> blocking and waiting. > >>- Async Responder: Is responsible for processing the responses > >> received by the connection thread. If the request needs to be re - > >> initiated (such as the downstream service returns a standby exception), > it > >> re - adds the request to the asynchronous handler thread pool; > otherwise, > >> it puts the response into the response queue. > >>- Responder: Retrieves the response from the response queue and > >> returns it to the client. > >> > >> IV. Advantages of the Asynchronous Router > >> > >>- High - Concurrency Performance: Through the asynchronous processing > >> mechanism, the asynchronous router can handle a large number of requests > >> simultaneously, significantly improving the system's concurrent > processing > >> ability. > >>- High Resource Utilization: It avoids thread blocking and frequent > >> switching, reduces thread resource waste, and improves the overall > >> efficiency of the system. > >>- Isolation: Different ns are processed by different async handler > >> thread pools, achieving isolation of different downstream services. > Even if > >> the performance of a certain service is poor, it will not affect the > >> processing ability of other services. > >> > >> V.
Re: [VOTE] Request to merge branch HDFS-17531 into trunk.
+1 - Zhang Haobo Replied Message | From | Xiaoqiao He | | Date | 02/14/2025 19:38 | | To | Ayush Saxena | | Cc | Hui Fei , jian zhang , Hdfs-dev , , | | Subject | Re: [VOTE] Request to merge branch HDFS-17531 into trunk. | +1 Best Regards, - He Xiaoqiao On Fri, Feb 14, 2025 at 3:53 PM Ayush Saxena wrote: +1 -Ayush On 14 Feb 2025, at 1:20 PM, Hui Fei wrote: Did some test work referring to the documentation. - Compiled source codes, built a local cluster and the async feature worked fine. - it is disabled by default - can increase or decrease the thread number by changing the related configurations +1 jian zhang 于2025年2月14日周五 15:21写道： Hi, all, the development of the asynchronous router functionality has been completed. Thanks to all the contributors who participated in this feature. The development branch is HDFS-17531, and it is ready to be merged into the trunk branch. JIRA: HDFS-17531 https://issues.apache.org/jira/browse/HDFS-17531 PR: https://github.com/apache/hadoop/pull/7308 DISCUSS: https://lists.apache.org/thread/02y3dtpfxt21bxjgmyl3kxnv4m1vwz44 Here is the functionality introduction of the asynchronous router for everyone to review: I. Overview The asynchronous router aims to address the performance bottleneck issues of the synchronous router in high - concurrency and multi - nameservices scenarios. By introducing an asynchronous processing mechanism, it optimizes the request handling process, improves the system's concurrency ability and resource utilization, and is particularly suitable for the federated scenarios where multiple downstream services (NS) need to be processed. II. Problems of the Synchronous Router - Performance Bottleneck: The performance of the synchronous router is limited by the number of handler threads. Even if the connection thread can still forward requests to the downstream namenode, the handler must wait for each request to complete before processing the next one, resulting in limited processing capacity. - Thread Resource Waste: To improve performance, increasing the number of handler threads will lead to more thread switches, which instead reduces the system efficiency. At the same time, a large number of handler threads are in a blocked state, wasting thread resources. - Poor Isolation in Multi - ns: If the performance of a certain nameservice in the downstream nameservice is poor, it will cause the handler to wait for a long time, thus affecting the forwarding of requests to other normal - performance ns, resulting in a decrease in the overall performance of the downstream ns services perceived by the client. - Ineffective Utilization of Federation Multi - ns Performance: In high - concurrency scenarios, a large number of requests may be backlogged in the router's request queue, while the queues of downstream services are not fully utilized, leading to unreasonable resource allocation. III. Design and Improvements of the Asynchronous Router The asynchronous router solves the above problems by redesigning the request handling process and introducing an asynchronous processing mechanism. Its core improvements include: - Handler: Retrieves requests from the request queue for preliminary processing. If there are exceptions in the request (such as the mount point does not exist, etc.), it directly puts the response into the response queue; otherwise, it sends the request to the asynchronous handler thread pool. - Async Handler: Puts the request into the call queue (connection.calls) of the connection thread and returns immediately without blocking and waiting. - Async Responder: Is responsible for processing the responses received by the connection thread. If the request needs to be re - initiated (such as the downstream service returns a standby exception), it re - adds the request to the asynchronous handler thread pool; otherwise, it puts the response into the response queue. - Responder: Retrieves the response from the response queue and returns it to the client. IV. Advantages of the Asynchronous Router - High - Concurrency Performance: Through the asynchronous processing mechanism, the asynchronous router can handle a large number of requests simultaneously, significantly improving the system's concurrent processing ability. - High Resource Utilization: It avoids thread blocking and frequent switching, reduces thread resource waste, and improves the overall efficiency of the system. - Isolation: Different ns are processed by different async handler thread pools, achieving isolation of different downstream services. Even if the performance of a certain service is poor, it will not affect the processing ability of other services. V. Summary The asynchronous router solves the performance bottleneck problem of the traditional synchronous router in high - concurrency scenarios by introducing an asynchronous processing mechanism. It not only improves the system's concurrency ability and resource utilization but also achieves i
