[jira] [Created] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode and dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
Haiyang Hu created HDFS-16287: - Summary: Support to make dfs.namenode.avoid.read.slow.datanode and dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable Key: HDFS-16287 URL: https://issues.apache.org/jira/browse/HDFS-16287 Project: Hadoop HDFS Issue Type: Improvement Reporter: Haiyang Hu Assignee: Haiyang Hu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-compile-javac-root.txt [496K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-mvnsite-root.txt [584K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-javadoc-root.txt [32K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [232K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [436K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [40K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/463/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [32K] https://ci-hadoop.apache.org/j
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/670/ [Oct 26, 2021 1:16:50 AM] (noreply) HADOOP-17946. Upgrade commons-lang to 3.12.0 (#3575) [Oct 26, 2021 6:54:11 AM] (noreply) HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16288) Native Test Case #35 Fails in RHEL 8.4
Paula Logan created HDFS-16288:
--
Summary: Native Test Case #35 Fails in RHEL 8.4
Key: HDFS-16288
URL: https://issues.apache.org/jira/browse/HDFS-16288
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs, libhdfs++, native, test
Affects Versions: 3.3.1
Environment: RHEL 8.4
Reporter: Paula Logan
When running the following maven command, Native Test Case #35 Fails and the
tests are halted.
mvn test -Pnative,parallel-tests,yarn-ui -Dparallel-tests=true
-Dtests=allNative -Drequire.bzip2=true -Drequire.fuse=true -Drequire.isal=true
-Disal.prefix=/usr/local -Disal.lib=/usr/local/lib64 -Dbundle.isal=true
-Drequire.openssl=true -Dopenssl.prefix=/usr -Dopenssl.include=/usr/include
-Dopenssl.lib=/usr/lib64 -Dbundle.openssl=true -Drequire.pmdk=true
-Dpmdk.lib=/usr/lib64 -Dbundle.pmdk=true -Drequire.snappy=true
-Dsnappy.prefix=/usr -Dsnappy.include=/usr/include -Dsnappy.lib=/usr/lib64
-Dbundle.snappy=true -Drequire.valgrind=true -Dhbase.profile=2.0
-Drequire.zstd=true -Dzstd.prefix=/usr -Dzstd.include=/usr/include
-Dzstd.lib=/usr/lib64 -Dbundle.zstd=true
I added the following to assist with the debugging:
-Dnative_ctest_args="-V -VV --debug --stop-on-failure"
-Dnative_cmake_args="-D_LIBHDFS_JNI_HELPER_DEBUGGING_ON_
-DLIBHDFSPP_C_API_ENABLE_DEBUG" -Droot.log.level=DEBUG
After much work tracing through Test Case #35 functions in
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c,
I found the problem, and it deals with both the test_libhdfs_threaded.c calls
and the functions listed in
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c
file.
test_libhdfs_threaded.c uses two file systems: hdfs (C) and hdfspp (C++). It
makes a call to hdfsGetPathInfo of hdfs_shim.c passing in a C++ FS which calls
the libhdfspp_hdfsGetPathInfo, which is correct. Then test_libhdfs_threaded.c
wants to know if the file is encrypted, so it calls hdfsFileIsEncrypted of
hdfs_shim.c (knowing it is using a C++ PathInfo structure) which calls
libhdfs_hdfsFileIsEncrypted, which is not correct.
In hdfs_shim.c:
hdfsFileInfo *hdfsGetPathInfo(hdfsFS fs, const char* path) {
return (hdfsFileInfo *)libhdfspp_hdfsGetPathInfo(fs->libhdfsppRep, path);
}
int hdfsFileIsEncrypted(hdfsFileInfo *hdfsFileInfo) {
return libhdfs_hdfsFileIsEncrypted
((libhdfs_hdfsFileInfo *) hdfsFileInfo);
}
The crux of the problem deals with the extendedHdfsFileInfo which exists in the
C structures but not the C++ structures.
In RHEL 8.4, I get a 1 returned from hdfsFileIsEncrypted (uses bitwise and)
when the file is not encrypted.
I put in a quick fix to get passed Test Case #35, but then found that it
clashed with #40.
I modified the following files by adding new functions appropriate for the FS
type and Path structures returned just to get all (40) of the Native Test Cases
to pass to be able to test the remaining test cases: hdfs.h,
libhdfs_wrapper_defines.h, libhdfs_wrapper_undefs.h, libhdfspp_wrapper.h,
hdfs_shim.c, hdfs.c, test_libhdfs_threaded.c.
Your fix, I would think, would be to add the extendedHdfsFileInfo to the C++
code or change test_libhdfs_threaded.c to call hdfsFileIsEncrypted using a C
Path structure.
I've communicated with a Hadoop Native developer who said he didn't have any
problem in Ubuntu. My guess would be that the memory is initialized by default
on that system to 0. Not sure as I haven't used Ubuntu.
Notes:
I followed the CentOS 8 procedures in the BUILDING.txt.
HDFS-9359 has a note that might be relevant to this problem. "hdfs_shim.c is
created to support testing part of libhdfs++ that have been implemented.
Functions not implemented in libhdfs++ are delegated to libhdfs."
The hdfsFileIsEncrypted does care which FS is being used as the Path structures
returned are different sizes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
