[jira] [Created] (HDFS-11046) Duplicate '-' in the daemon log name
Akira Ajisaka created HDFS-11046:
Summary: Duplicate '-' in the daemon log name
Key: HDFS-11046
URL: https://issues.apache.org/jira/browse/HDFS-11046
Project: Hadoop HDFS
Issue Type: Bug
Components: logging
Reporter: Akira Ajisaka
Assignee: Akira Ajisaka
When {{HADOOP_SECURE_DN_USER}} and {{HADOOP_PRIVILEGED_NFS_USER}} are not set,
there are duplicate '-' in the log name.
{noformat}
[centos@ip-172-31-21-203 logs]$ ls *.log
hadoop--centos-datanode-ip-172-31-21-203.ap-northeast-1.compute.internal.log
hadoop--centos-namenode-ip-172-31-21-203.ap-northeast-1.compute.internal.log
hadoop-centos-nodemanager-ip-172-31-21-203.ap-northeast-1.compute.internal.log
hadoop-centos-resourcemanager-ip-172-31-21-203.ap-northeast-1.compute.internal.log
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/ [Oct 23, 2016 3:25:37 PM] (xyao) Revert "HDFS-10757. KMSClientProvider combined with KeyProviderCache can [Oct 23, 2016 5:58:36 PM] (xyao) HADOOP-13749. KMSClientProvider combined with KeyProviderCache can -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-kms Exception is caught when Exception is not thrown in org.apache.hadoop.crypto.key.kms.server.KMS.createKey(Map) At KMS.java:is not thrown in org.apache.hadoop.crypto.key.kms.server.KMS.createKey(Map) At KMS.java:[line 169] Exception is caught when Exception is not thrown in org.apache.hadoop.crypto.key.kms.server.KMS.generateEncryptedKeys(String, String, int) At KMS.java:is not thrown in org.apache.hadoop.crypto.key.kms.server.KMS.generateEncryptedKeys(String, String, int) At KMS.java:[line 501] Failed junit tests : hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.client.cli.TestLogsCLI hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Timed out junit tests : org.apache.hadoop.http.TestHttpServerLifecycle org.apache.hadoop.hdfs.TestLeaseRecovery2 cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/branch-findbugs-hadoop-common-project_hadoop-kms-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [16K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/204/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Enhance FileUtil#list to throw ACE when the dir is not readable
Hi all,
This came out of discussion in HADOOP-12718. What do you think if I place
extra permission check into {{FileUtil#list}}?
{code}
public static String[] list(File dir) throws IOException {
+ if (!canRead(dir)) {
+ throw new AccessControlException("Permission denied for dir: " +
+ dir.toString());
}
String[] fileNames = dir.list();
if(fileNames == null) {
throw new IOException("Invalid directory or I/O error occurred for
dir: "
+ dir.toString());
}
return fileNames;
}
{code}
Currently {{FileUtil#list}} is only called by:
{code}
hadoop-common (1 usage found)
org.apache.hadoop.fs (1 usage found)
RawLocalFileSystem (1 usage found)
listStatus(Path) (1 usage found)
474String[] names = FileUtil.list(localf);
hadoop-hdfs (3 usages found)
org.apache.hadoop.hdfs.server.datanode (2 usages found)
BlockPoolSliceStorage (1 usage found)
cleanupDetachDir(File) (1 usage found)
518if (FileUtil.list(detachDir).length != 0) {
DataStorage (1 usage found)
cleanupDetachDir(File) (1 usage found)
910if (FileUtil.list(detachDir).length != 0 ) {
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl (1 usage
found)
FsVolumeImpl (1 usage found)
isBPDirEmpty(String) (1 usage found)
1035if (rbwDir.exists() &&
FileUtil.list(rbwDir).length != 0) {
{code}
These callers seem ok with the change. ACE is a subclass of IOE, I imagine
the impact should be minimum with the benefit of give more precise
exception class and message.
Thanks,
John Zhuge
Software Engineer, Cloudera
[jira] [Created] (HDFS-11047) Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode
Xiaobing Zhou created HDFS-11047: Summary: Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode Key: HDFS-11047 URL: https://issues.apache.org/jira/browse/HDFS-11047 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, fs Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/ [Oct 23, 2016 3:25:37 PM] (xyao) Revert "HDFS-10757. KMSClientProvider combined with KeyProviderCache can [Oct 23, 2016 5:58:36 PM] (xyao) HADOOP-13749. KMSClientProvider combined with KeyProviderCache can -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.TestMaintenanceState hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.cli.TestLogsCLI hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMRIntermediateDataEncryption compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-compile-root.txt [308K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-compile-root.txt [308K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-compile-root.txt [308K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [196K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/134/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-plu
[jira] [Created] (HDFS-11048) Audit Log should escape control characters
Eric Badger created HDFS-11048: -- Summary: Audit Log should escape control characters Key: HDFS-11048 URL: https://issues.apache.org/jira/browse/HDFS-11048 Project: Hadoop HDFS Issue Type: Bug Reporter: Eric Badger Assignee: Eric Badger Allowing control characters without escaping them allows for spoofing audit log entries at worst and accidentally breaking log parsing at best. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
