[jira] [Created] (HDFS-15904) Flaky test TestBalancer#testBalancerWithSortTopNodes()
Viraj Jasani created HDFS-15904: --- Summary: Flaky test TestBalancer#testBalancerWithSortTopNodes() Key: HDFS-15904 URL: https://issues.apache.org/jira/browse/HDFS-15904 Project: Hadoop HDFS Issue Type: Test Reporter: Viraj Jasani Assignee: Viraj Jasani Fix For: 3.4.0 TestBalancer#testBalancerWithSortTopNodes shows some flakes in around ~10 runs or so. It's reproducible locally also. Basically, balancing either moves 2 blocks of size 100+100 bytes or it moves 3 blocks of size 100+100+50 bytes (2nd case causes flakies). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/ [Mar 17, 2021 6:16:18 AM] (Takanobu Asanuma) HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) [Mar 17, 2021 7:34:52 AM] (Akira Ajisaka) HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.util.TestDiskCheckerWithDiskIo hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-compile-javac-root.txt [456K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-mvnsite-root.txt [396K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/diff-javadoc-javadoc-root.txt [20K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [212K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [272K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [124K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [96K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/241/artifact/out/patch
[jira] [Created] (HDFS-15905) Improve Router performance with router redirection
Aihua Xu created HDFS-15905: --- Summary: Improve Router performance with router redirection Key: HDFS-15905 URL: https://issues.apache.org/jira/browse/HDFS-15905 Project: Hadoop HDFS Issue Type: New Feature Components: rbf Affects Versions: 3.1.0 Reporter: Aihua Xu Assignee: Aihua Xu Router implementation currently takes the proxy approach to handle the client requests: the routers receive the requests from the clients and send the requests to the target clusters on behalf of the clients. This approach works well, while after moving more clusters on top of routers, we are seeing that routers are becoming the bottleneck since e.g., without RBF, the clients themselves manage the connections for themselves, while with RBF, the limited routers manage much more connections for the clients; we also keep idle connections to boost the connection performance. We have done some work to tune connection management but it doesn't help much. We are proposing to reduce the functionality on the router side and use them as actual router instead of proxy: the clients talk to routers to resolve target cluster info given a path and get router delegation token; the clients directly send the requests to target cluster. A big challenge here is the token authentication against target cluster with router token only. One approach: we can ask router to return target cluster token along with router token so the clients can authenticate against target cluster. Second approach: similar to block token mechanism, the router exchanges secret keys with target clusters through heart-beats so the clients can authenticate with target cluster with that router token. I would like to know your feedback. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/ [Mar 17, 2021 3:00:05 AM] (noreply) HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) [Mar 17, 2021 5:14:26 AM] (Takanobu Asanuma) HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. [Mar 17, 2021 6:12:36 AM] (noreply) HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) [Mar 17, 2021 12:38:20 PM] (Peter Bacsko) YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu. [Mar 17, 2021 5:57:11 PM] (noreply) HADOOP-17578. Improve UGI debug log to help troubleshooting TokenCach… (#2762) [Mar 17, 2021 6:11:37 PM] (Eric Badger) YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots hadoop.hdfs.TestHDFSFileSystemContract hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.tools.TestDistCpSystem hadoop.tools.fedbalance.procedure.TestBalanceProcedureScheduler hadoop.tools.fedbalance.TestDistCpProcedure hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-compile-cc-root.txt [116K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-compile-javac-root.txt [368K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [340K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [32K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/patch-unit-hadoop-tools_hadoop-federation-balance.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [8.0K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/450/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [24K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr.
