[jira] [Resolved] (HDFS-16677) Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes.
[ https://issues.apache.org/jira/browse/HDFS-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie resolved HDFS-16677. - Resolution: Duplicate > Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes. > > > Key: HDFS-16677 > URL: https://issues.apache.org/jira/browse/HDFS-16677 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding, hdfs >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Copy from HDFS-15006 > In HDFS-14989, we add a new Namenode operation "swapBlockList" to replace the > set of blocks in an INode File with a new set of blocks. This JIRA will track > the effort to add an FS Edit log op to persist this operation. > We also need to increase the NamenodeLayoutVersion for this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-mvnsite-root.txt [564K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [432K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [116K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/734/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt
Re: [VOTE] Release Apache Hadoop 3.3.4
cancelling this RC so i can issue a new one with an updated reload4j https://issues.apache.org/jira/browse/HADOOP-18354 i will also do a pr to update aws sdk, which is less critical (the jackson databind classes don't seem to get used), -upgrading will stop audit tools overreacting On Thu, 21 Jul 2022 at 19:07, Steve Loughran wrote: > > I have put together a release candidate (RC0) for Hadoop 3.3.4 > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/ > > The git tag is release-3.3.4-RC0, commit c679bc76d26 > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1356/ > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md > > There's a very small number of changes, primarily critical code/packaging > issues and security fixes. > > See the release notes for details. > > Please try the release and vote. The vote will run for 5 days. > > -Steve > > >
[jira] [Created] (HDFS-16692) Add detailed scope info in NotEnoughReplicas Reason logs.
ZanderXu created HDFS-16692:
---
Summary: Add detailed scope info in NotEnoughReplicas Reason logs.
Key: HDFS-16692
URL: https://issues.apache.org/jira/browse/HDFS-16692
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: ZanderXu
Assignee: ZanderXu
When we writing some ec data from client not in hdfs cluster, there are a large
number of INFO log output, as blew:
{code:shell}
2022-07-26 15:50:40,973 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=17}
2022-07-26 15:50:40,974 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=18}
2022-07-26 15:50:40,974 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=17}
2022-07-26 15:50:40,975 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=18}
2022-07-26 15:50:40,975 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=17}
2022-07-26 15:50:40,976 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=18}
2022-07-26 15:50:40,976 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=18}
2022-07-26 15:50:40,977 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=18}
2022-07-26 15:50:40,977 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=19}
2022-07-26 15:50:40,977 INFO blockmanagement.BlockPlacementPolicy
(BlockPlacementPolicyDefault.java:chooseRandom(912)) - Not enough replicas was
chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1, TOO_MANY_NODES_ON_RACK=3}
{code}
I feel that we should add detailed scope info in this log to show the scope
that we cannot select any good nodes from.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16693) RBF: Fix NPE in Quota
Weison Wei created HDFS-16693: - Summary: RBF: Fix NPE in Quota Key: HDFS-16693 URL: https://issues.apache.org/jira/browse/HDFS-16693 Project: Hadoop HDFS Issue Type: Bug Reporter: Weison Wei Assignee: Weison Wei when the switch `dfs.federation.router.quota.enable` is false, Quota#isMountEntry will throw NPE; -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/ [Jul 25, 2022, 9:39:25 AM] (noreply) HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output (#4541). Contributed by Max Xie. [Jul 25, 2022, 5:05:45 PM] (noreply) YARN-11161. Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation (#4610) [Jul 25, 2022, 5:25:38 PM] (noreply) HDFS-16681. Do not pass GCC flags for MSVC in libhdfspp (#4615) [Jul 25, 2022, 6:38:59 PM] (noreply) MAPREDUCE-7372 MapReduce set permission too late in copyJar method (#4026). Contributed by Zhang Dongsheng. [Jul 25, 2022, 6:55:40 PM] (noreply) YARN-10883. [Router] Router Audit Log Add Client IP Address. (#4426) [Jul 25, 2022, 8:30:00 PM] (noreply) HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. (#4155) -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/933/artifact/out/results-javadoc-javadoc-root.txt [400K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
