[INFO] branch HDFS-17531 rebase onto trunk
Hi All, Just rebase development feature branch HDFS-17531(RBF: Asynchronous router RPC) onto trunk, then keep branch HDFS-17531 closely with trunk. Thanks. Best Regards, - He Xiaoqiao
Re: [INFO] branch HDFS-17531 rebase onto trunk
Hi Xiaoqiao He Thank you for taking the time to do this work! Best Regards, - Jian Zhang Xiaoqiao He 于2024年9月12日周四 15:54写道: > Hi All, > > Just rebase development feature branch HDFS-17531(RBF: Asynchronous router > RPC) onto trunk, > then keep branch HDFS-17531 closely with trunk. Thanks. > > Best Regards, > - He Xiaoqiao >
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ipc.TestIPC hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-mvnsite-root.txt [568K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [452K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1513/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) shadedclient unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs-httpfs Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] spotbugs : module:hadoop-yarn-project/hadoop-yarn org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] spotbugs : module:hadoop-hdfs-project Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] spotbugs : module:hadoop-yarn-project org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] spotbugs : module:root Redundant nullcheck of xAttrs, which is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:is known to be non-null in org.apache.hadoop.fs.http.client.HttpFSFileSystem.getXAttr(Path, String) Redundant null check at HttpFSFileSystem.java:[line 1373] org.apache.hadoop.yarn.service.ServiceScheduler$1.load(ConfigFile) may return null, but is declared @Nonnull At ServiceScheduler.java:is declared @Nonnull At ServiceScheduler.java:[line 558] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1698/artifact/out/results-c
[jira] [Created] (HDFS-17623) RBF:The router service fails to delete a mount table with multiple subclusters mounted on it through MultipleDestinationMountTableResolver
Guo Wei created HDFS-17623: -- Summary: RBF:The router service fails to delete a mount table with multiple subclusters mounted on it through MultipleDestinationMountTableResolver Key: HDFS-17623 URL: https://issues.apache.org/jira/browse/HDFS-17623 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.4.0 Reporter: Guo Wei Please see the error message in the following example: [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p hdfs://hh-rbf-test1/guov100/data [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p hdfs://hh-rbf-test2/guov100/data [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -add /guov100/data hh-rbf-test1,hh-rbf-test2 /guov100/data -order RANDOM Successfully added mount point /guov100/data [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -ls Mount Table Entries: Source Destinations Owner Group Mode Quota/Usage /guov100/data hh-rbf-test1->/guov100/data,hh-rbf-test2->/guov100/data hdfs hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir hdfs://test-fed/guov100/data/test [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls hdfs://hh-rbf-test1/guov100/data/test/ Found 1 items -rw-r--r-- 3 hdfs hdfs 0 2024-09-13 09:56 hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls hdfs://hh-rbf-test2/guov100/data/test/ Found 1 items -rw-r--r-- 3 hdfs hdfs 0 2024-09-13 09:56 hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls hdfs://test-fed/guov100/data/test/ Found 2 items -rw-r--r-- 3 hdfs hdfs 0 2024-09-13 09:56 hdfs://test-fed/guov100/data/test/file-test1.txt -rw-r--r-- 3 hdfs hdfs 0 2024-09-13 09:56 hdfs://test-fed/guov100/data/test/file-test2.txt [hdfs@sjsy-hh202-zbxh55w root]$ [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -rm -r hdfs://test-fed/guov100/data/test/ rm: Failed to move to trash: hdfs://test-fed/guov100/data/test: Rename of /guov100/data/test to /user/hdfs/.Trash/Current/guov100/data/test is not allowed, no eligible destination in the same namespace was found. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17623) RBF:The router service fails to delete a mount table with multiple subclusters mounted on it through MultipleDestinationMountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guo Wei resolved HDFS-17623. Fix Version/s: 3.4.0 Resolution: Works for Me > RBF:The router service fails to delete a mount table with multiple > subclusters mounted on it through MultipleDestinationMountTableResolver > -- > > Key: HDFS-17623 > URL: https://issues.apache.org/jira/browse/HDFS-17623 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Guo Wei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Please see the error message in the following example: > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p > hdfs://hh-rbf-test1/guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p > hdfs://hh-rbf-test2/guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -add /guov100/data > hh-rbf-test1,hh-rbf-test2 /guov100/data -order RANDOM > Successfully added mount point /guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -ls > Mount Table Entries: > Source Destinations Owner > Group Mode Quota/Usage > /guov100/data > hh-rbf-test1->/guov100/data,hh-rbf-test2->/guov100/data hdfs > hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir > hdfs://test-fed/guov100/data/test > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch > hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch > hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://hh-rbf-test1/guov100/data/test/ > Found 1 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://hh-rbf-test2/guov100/data/test/ > Found 1 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://test-fed/guov100/data/test/ > Found 2 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://test-fed/guov100/data/test/file-test1.txt > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://test-fed/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -rm -r > hdfs://test-fed/guov100/data/test/ > {color:#FF}rm: Failed to move to trash: > hdfs://test-fed/guov100/data/test: Rename of /guov100/data/test to > /user/hdfs/.Trash/Current/guov100/data/test is not allowed, no eligible > destination in the same namespace was found.{color} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-17623) RBF:The router service fails to delete a mount table with multiple subclusters mounted on it through MultipleDestinationMountTableResolver
[ https://issues.apache.org/jira/browse/HDFS-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guo Wei reopened HDFS-17623: > RBF:The router service fails to delete a mount table with multiple > subclusters mounted on it through MultipleDestinationMountTableResolver > -- > > Key: HDFS-17623 > URL: https://issues.apache.org/jira/browse/HDFS-17623 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Guo Wei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Please see the error message in the following example: > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p > hdfs://hh-rbf-test1/guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir -p > hdfs://hh-rbf-test2/guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -add /guov100/data > hh-rbf-test1,hh-rbf-test2 /guov100/data -order RANDOM > Successfully added mount point /guov100/data > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfsrouteradmin -ls > Mount Table Entries: > Source Destinations Owner > Group Mode Quota/Usage > /guov100/data > hh-rbf-test1->/guov100/data,hh-rbf-test2->/guov100/data hdfs > hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -mkdir > hdfs://test-fed/guov100/data/test > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch > hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -touch > hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://hh-rbf-test1/guov100/data/test/ > Found 1 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://hh-rbf-test1/guov100/data/test/file-test1.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://hh-rbf-test2/guov100/data/test/ > Found 1 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://hh-rbf-test2/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -ls > hdfs://test-fed/guov100/data/test/ > Found 2 items > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://test-fed/guov100/data/test/file-test1.txt > {-}rw-r{-}{-}r{-}- 3 hdfs hdfs 0 2024-09-13 09:56 > hdfs://test-fed/guov100/data/test/file-test2.txt > [hdfs@sjsy-hh202-zbxh55w root]$ > [hdfs@sjsy-hh202-zbxh55w root]$ hdfs dfs -rm -r > hdfs://test-fed/guov100/data/test/ > {color:#FF}rm: Failed to move to trash: > hdfs://test-fed/guov100/data/test: Rename of /guov100/data/test to > /user/hdfs/.Trash/Current/guov100/data/test is not allowed, no eligible > destination in the same namespace was found.{color} > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org