Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [548K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/725/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] asflicense:
[jira] [Resolved] (HDFS-16654) Link OpenSSL lib for CMake deps check
[
https://issues.apache.org/jira/browse/HDFS-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gautham Banasandra resolved HDFS-16654.
---
Fix Version/s: 3.4.0
Resolution: Fixed
Merged PR https://github.com/apache/hadoop/pull/4538 to trunk.
> Link OpenSSL lib for CMake deps check
> -
>
> Key: HDFS-16654
> URL: https://issues.apache.org/jira/browse/HDFS-16654
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> CMake checks whether the required components of OpenSSL are available prior
> to building HDFS native client -
> https://github.com/apache/hadoop/blob/fac895828f714b5587b57900d588acac69880c1e/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt#L130
> {code}
> check_c_source_compiles("#include
> \"${OPENSSL_INCLUDE_DIR}/openssl/evp.h\"\nint main(int argc, char **argv) {
> return !EVP_aes_256_ctr; }" HAS_NEW_ENOUGH_OPENSSL)
> {code}
> This check compiles but fails while linking on Windows -
> {code}
> src.obj : error LNK2019: unresolved external symbol EVP_aes_256_ctr
> referenced in function main
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\CMakeFiles\CMakeTmp\cmTC_e391b.vcxproj]
> H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\CMakeFiles\CMakeTmp\Debug\cmTC_e391b.exe
> : fatal error LNK1120: 1 unresolved externals
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\CMakeFiles\CMakeTmp\cmTC_e391b.vcxproj]
> Source file was:
> #include
> int main(int argc, char **argv) { return !EVP_aes_256_ctr; }
> {code}
> Thus, we need to link to the OpenSSL library prior to running this check.
> Please note that this check doesn't fail on Linux since CMake is able to pick
> it up from the standard location where libs are installed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16665) Fix duplicate sources for hdfspp_test_shim_static
Gautham Banasandra created HDFS-16665:
-
Summary: Fix duplicate sources for hdfspp_test_shim_static
Key: HDFS-16665
URL: https://issues.apache.org/jira/browse/HDFS-16665
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs++
Affects Versions: 3.4.0
Environment: Windows 10
Reporter: Gautham Banasandra
Assignee: Gautham Banasandra
The library target *hdfspp_test_shim_static* is built using the following
sources, which causes duplicate symbols to be defined -
1. hdfs_shim.c
2. ${LIBHDFSPP_BINDING_C}/hdfs.cc
https://github.com/apache/hadoop/blob/8774f178686487007dcf8c418c989b785a529000/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt#L153
And fails the compilation -
{code}
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsAllowSnapshot already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsAvailable already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsBuilderConfSetStr already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\hdfspp_test_shim_static.lib(hdfs_shim.obj)
: error LNK2005: hdfsBuilderConnect already defined in
hdfspp_test_shim_static.lib(hdfs.obj)
{code}
Duplicate symbols defined by hdfs_shim.c -
https://github.com/apache/hadoop/blob/440f4c2b28515d2007b81ac00b549bbf14fa9f64/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c#L583-L585.
Adding the source *${LIBHDFSPP_BINDING_C}/hdfs.cc* is redundant here since this
file is transitively included in hdfs_shim.c through *libhdfspp_wrapper.h* -
https://github.com/apache/hadoop/blob/440f4c2b28515d2007b81ac00b549bbf14fa9f64/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c#L20.
Thus, we need to exclude *${LIBHDFSPP_BINDING_C}/hdfs.cc* to fix this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16666) Parameterize CMake args for Windows
Gautham Banasandra created HDFS-1: - Summary: Parameterize CMake args for Windows Key: HDFS-1 URL: https://issues.apache.org/jira/browse/HDFS-1 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs++ Affects Versions: 3.4.0 Environment: Windows 10 Reporter: Gautham Banasandra Assignee: Gautham Banasandra Need to pass the required HDFS CMake related argument for building on Windows. Currently, these are passed - https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml#L150. We need to pass these arguments as well for Windows - https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml#L219-L223 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/ No changes -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/924/artifact/out/results-javadoc-javadoc-root.txt [400K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/335/ [Jul 15, 2022, 8:52:12 PM] (noreply) HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) [Jul 15, 2022, 9:18:46 PM] (noreply) HADOOP-13144. Enhancing IPC client throughput via multiple connections per user (#4542) -1 overall The following subsystems voted -1: blanks pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState)) Redundant null check at ResourceLocalizationService.java:is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalRes
[jira] [Created] (HDFS-16667) Use malloc for buffer allocation in uriparser2
Gautham Banasandra created HDFS-16667:
-
Summary: Use malloc for buffer allocation in uriparser2
Key: HDFS-16667
URL: https://issues.apache.org/jira/browse/HDFS-16667
Project: Hadoop HDFS
Issue Type: Bug
Components: libhdfs++
Affects Versions: 3.4.0
Environment: Windows 10
Reporter: Gautham Banasandra
Assignee: Gautham Banasandra
Currently, a variable is used to specify the array size in *uriparser2* -
{code:cpp}
static int parse_int(const char *first, const char *after_last) {
const int size = after_last - first;
if (size) {
char buffer[size + 1];
memcpyz(buffer, first, size);
return atoi(buffer);
}
return 0;
}
{code}
https://github.com/apache/hadoop/blob/34e548cb62ed21c5bba7a82f5f1489ca6bdfb8c4/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser2.c#L71-L79
This results in the following error on Windows -
{code}
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\out\build\x64-Debug\cl :
command line warning D9025: overriding '/W4' with '/w'
uriparser2.c
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,23):
error C2057: expected constant expression
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,23):
error C2466: cannot allocate an array of constant size 0
H:\hadoop-hdfs-project\hadoop-hdfs-native-client\src\main\native\libhdfspp\third_party\uriparser2\uriparser2\uriparser2.c(74,24):
error C2133: 'buffer': unknown size
{code}
Thus, we need to use malloc to fix this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
