Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2024-05-25 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.ipc.TestIPC 
   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.TestDFSInotifyEventInputStream 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.router.webapp.TestRouterWebServicesREST 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-compile-javac-root.txt
  [488K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-mvnsite-root.txt
  [572K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [224K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [460K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1403/artifact/out/patch-unit-hadoop-tools_ha

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-25 Thread zhangjian 
Hello everyone, I conducted a performance comparison test between sync and 
asynchronous router, and the test results showed that in single ns or multi ns 
scenarios, Asynchronous router in terms of throughput The utilization of CPU 
and thread, as well as the average processing time of client requests, are 
better than those of sync router, especially when downstream ns have 
performance bottlenecks, The performance of the async router is far greater 
than that of the sync router; And in terms of isolation, Asynchronous router is 
also better than sync router.
Detailed testing PDF: https://issues.apache.org/jira/browse/HDFS-17531  
Comparison of Async router & sync router performance.pdf

> 2024年5月24日 14:13，Yuanbo Liu  写道：
> 
> good job!
> 
> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com 
> > wrote:
>> Hello everyone, currently, I have tested the performance of async and sync 
>> router for a downstream ns:
>> 1. The throughput, CPU, and thread performance of the async router are 
>> better than those of the sync router, and its memory performance is within 
>> an acceptable range compared to the synchronous router.
>> 2. Asynchronous router can apply pressure downstream to better utilize the 
>> performance of downstream ns, and can almost fill the call queue of 
>> downstream ns.
>> 
>> Due to the large size of the test result pdf, it cannot be sent via email, 
>> please see: https://issues.apache.org/jira/browse/HDFS-17531
>> 
>> > 2024年5月23日 17:03，Xiaoqiao He > > > 写道：
>> > 
>> > Great. Thanks for your addendum information.
>> > 
>> > cc @Ayush Saxena mailto:ayush...@gmail.com>> 
>> > @inigo...@apache.org 
>> > mailto:inigo...@apache.org>> Any more feedback for 
>> > this proposal?
>> > 
>> > IMO The feature of asynchronous router RPC is a helpful improvement. For my
>> > internal practice, it will improve the throughput of requests forward
>> > significantly
>> > and is very valuable to push it forward.
>> > Thanks again and good luck!
>> > 
>> > Best Regards,
>> > - He Xiaoqiao
>> > 
>> > On Wed, May 22, 2024 at 9:59 AM zhangjian <1361320...@qq.com 
>> > > wrote:
>> > 
>> >> Hi, Sangjin Lee, thank you for your attention. I will use my free time to
>> >> do a performance comparison recently.
>> >> 
>> >>> 2024年5月22日 03:42，Sangjin Lee > >>> > 写道：
>> >>> 
>> >>> Thanks for the great proposal, Zhangjian. On point #3, I suspect it
>> >> should
>> >>> be fairly straightforward to create a small isolated synthetic test to
>> >>> prove (or disprove) the benefits of this approach. By driving a
>> >> controlled
>> >>> amount of requests per second, you could see latency, memory, CPU, etc.
>> >>> Ideally, it should show meaningful improvements without much degradation
>> >> in
>> >>> other metrics. Would you be able to spend some time doing that?
>> >>> 
>> >>> Thanks,
>> >>> Sangjin
>> >>> 
>> >>> On Tue, May 21, 2024 at 5:13 AM zhangjian <1361320...@qq.com.invalid 
>> >>> >
>> >> wrote:
>> >>> 
>>  Hi, xiaoqiao he, thank you for your reply.
>>  
>>  1.Currently, the server and client protocols within router can be
>>  implemented by extends existing protocols and adding asynchronous
>>  functionality, so it will not affect existing synchronization protocols.
>>  RouterClientNamenodeProtocolServerSideTranslatorPB
>>  RouterClientProtocolTranslatorPB
>>  RouterGetUserMappingsProtocolServerSideTranslatorPB
>>  RouterGetUserMappingsProtocolTranslatorPB
>>  RouterNamenodeProtocolServerSideTranslatorPB
>>  RouterNamenodeProtocolTranslatorPB
>>  RouterRefreshUserMappingsProtocolServerSideTranslatorPB
>>  RouterRefreshUserMappingsProtocolTranslatorPB
>>  
>>  The following issues have implemented asynchronous callbacks for
>>  Rpc.server, but I have not found any other modules to use related
>> >> functions
>>  Server HADOOP-11552 HADOOP-17046
>>  In the implementation of asynchronous Rpc.client, this issue is directly
>>  used
>>  Client HADOOP-13226
>>  Therefore, I believe that asynchronous routers are safe for modifying
>> >> the
>>  RPC protocol, RPC server, and client
>>  
>>  2. Forwarding requests to multiple downstream ns, the synchronous router
>>  handler adds requests from multiple downstream ns to the thread pool
>>  (RouterRpcClient.executorService), and then waits for responses from all
>>  downstream ns before returning. Since threads in the thread pool also
>>  process rpc requests synchronously, similar to a handler, the number of
>>  threads in the thread pool directly affects the performance of
>>  invoiceConcurrent, which in turn affects the performance of the handler.
>>  In asynchronous router implementation, the handler calls
>> >> invoiceConcurrent
>>  to simply convert a request into

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2024-05-25 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1591/

[May 23, 2024, 2:10:10 PM] (github) HADOOP-18325: ABFS: Add correlated metric 
support for ABFS operations (#6314)
[May 24, 2024, 1:41:08 AM] (github) HDFS-17529. RBF: Improve router state store 
cache entry deletion (#6833)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2024-05-25 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/681/

[May 23, 2024, 2:10:10 PM] (github) HADOOP-18325: ABFS: Add correlated metric 
support for ABFS operations (#6314)
[May 24, 2024, 1:41:08 AM] (github) HDFS-17529. RBF: Improve router state store 
cache entry deletion (#6833)
[May 24, 2024, 4:40:37 PM] (github) HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org