Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/ [Sep 16, 2016 7:08:47 AM] (aajisaka) HDFS-10862. Typos in 4 log messages. Contributed by Mehran Hassani. [Sep 16, 2016 9:16:25 PM] (weichiu) HADOOP-13411. Checkstyle suppression by annotation or comment. [Sep 16, 2016 9:26:42 PM] (weichiu) HADOOP-13412. Move dev-support/checkstyle/suppressions.xml to [Sep 16, 2016 9:53:30 PM] (weichiu) HADOOP-13580. If user is unauthorized, log "unauthorized" instead of [Sep 16, 2016 10:37:36 PM] (wang) HDFS-10823. Implement HttpFSFileSystem#listStatusIterator. [Sep 18, 2016 12:43:36 AM] (kai.zheng) YARN-5163. Migrate TestClientToAMTokens and TestClientRMTokens tests [Sep 18, 2016 1:03:15 AM] (kai.zheng) MAPREDUCE-6774. Add support for HDFS erasure code policy to TestDFSIO. [Sep 17, 2016 4:22:39 AM] (naganarasimha_gr) YARN-4232. TopCLI console support for HA mode. Contributed by Bibin A [Sep 17, 2016 4:49:59 AM] (naganarasimha_gr) MAPREDUCE-6777. Typos in 4 log messages. Contributed by Mehran Hassani [Sep 17, 2016 5:05:39 AM] (naganarasimha_gr) YARN-5642. Typos in 9 log messages. Contributed by Mehran Hassani -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [40K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [8.0K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/167/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Using HDFS DataNode as a part of a project, where to start
Hello Ravi, Thank you for your response. I have another question: I am trying to trace a call from org.apache.hadoop.fs.FsShell to NameNode. I am running a simple basic "ls" to understand how the mechanism works. What I want to know is which classes are used on the way when the "ls" is computed. To understand the mechanism, I print some messages on the way. Below is my prints. I get lost from line 34 to 35. I also could not find how DFSClient, DistributedFileSystem and FileSystem are being used on this way when running "ls". Any comment/help is appreciated. I basically want to know the call sequence of a basic shell command "ls". Thanks Yasin 1 - FsShell.main: [-ls, /test] 2 - FsShell.run: [-ls, /test] 3 - FsShell.init 1 4 - FsShell.init 2 5 - FsShell.registerCommands 6 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.FsCommand 7 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.AclCommands 8 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.CopyCommands 9 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Count 10 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Delete 11 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Display 12 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.find.Find 13 - CommandFactory.registerCommands org.apache.hadoop.fs.FsShellPermissions 14 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.FsUsage 15 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Ls 16 - LS.registerCommands: org.apache.hadoop.fs.shell.CommandFactory@51c8530f 17 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Mkdir 18 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.MoveCommands 19 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.SetReplication 20 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Stat 21 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Tail 22 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Test 23 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Touch 24 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.Truncate 25 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.SnapshotCommands 26 - CommandFactory.registerCommands org.apache.hadoop.fs.shell.XAttrCommands 27 - FsShell.run:cmd: -ls 28 - FsShell.run:Command Name: ls 29 - FsShell.run: Started:** org.apache.hadoop.fs.shell.Ls@3bd94634 30 - Command.run 31 - LS.processOptions: args[0]: /test 32 - Command.run. After ProcessOptions 33 - Command.processRawArguments 34 - Command.expandArguments: /test 35 - UserGroupInformation.getCurrentUser() 36 - UserGroupInformation.getLoginUser() 1 37 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.ensureInitialized() 1 38 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.initialize: 1 39 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.initialize: 2 40 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.initialize: 3 41 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.initialize: 4 42 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.ensureInitialized() 2 43 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 1 yasin (auth:null) 44 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 2 45 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 3 46 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 4 47 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.spawnAutoRenewalThreadForUserCreds() 1 48 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 7 49 - 16/09/17 14:34:05 INFO security.UserGroupInformation: UserGroupInformation.loginUserFromSubject: 8 50 - UserGroupInformation.getLoginUser() 2 51 - UserGroupInformation.getCurrentUser() 52 - DistributedFileSystem.DistributedFileSystem() 53 - DistributedFileSystem.DistributedFileSystem() 54 - FileSystem.createFileSystemhdfs://localhost:54310 55 - FileSystem.initializehdfs://localhost:54310 56 - DistributedFileSystem.initialize hdfs://localhost:54310 57 - UserGroupInformation.getCurrentUser() 58 - DFSClient.DFSClient 2: 59 - NamenodeProxies.creatProx
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/ [Sep 16, 2016 9:16:25 PM] (weichiu) HADOOP-13411. Checkstyle suppression by annotation or comment. [Sep 16, 2016 9:26:42 PM] (weichiu) HADOOP-13412. Move dev-support/checkstyle/suppressions.xml to [Sep 16, 2016 9:53:30 PM] (weichiu) HADOOP-13580. If user is unauthorized, log "unauthorized" instead of [Sep 16, 2016 10:37:36 PM] (wang) HDFS-10823. Implement HttpFSFileSystem#listStatusIterator. [Sep 18, 2016 12:43:36 AM] (kai.zheng) YARN-5163. Migrate TestClientToAMTokens and TestClientRMTokens tests [Sep 18, 2016 1:03:15 AM] (kai.zheng) MAPREDUCE-6774. Add support for HDFS erasure code policy to TestDFSIO. [Sep 17, 2016 4:22:39 AM] (naganarasimha_gr) YARN-4232. TopCLI console support for HA mode. Contributed by Bibin A [Sep 17, 2016 4:49:59 AM] (naganarasimha_gr) MAPREDUCE-6777. Typos in 4 log messages. Contributed by Mehran Hassani [Sep 17, 2016 5:05:39 AM] (naganarasimha_gr) YARN-5642. Typos in 9 log messages. Contributed by Mehran Hassani -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.TestPersistBlocks hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestLargeBlockReport org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMROpportunisticMaps compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patch-compile-root.txt [308K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patch-compile-root.txt [308K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patch-compile-root.txt [308K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [240K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/97/artifact/out/patc
