Re: About 2.6.5 and 2.7.4 release
Yes. I can help with releasing them. -Akira On 9/9/16 15:31, Tsuyoshi Ozawa wrote: Chiwan Park reported Hadoop incompatibility by the change of HADOOP-11252. Now, HADOOP-13579 is committed to branch-2.6 and branch-2.7. complement: HADOOP-13579 fixes the problem of incompatibility by HADOOP-11252. Thanks, - Tsuyoshi On Fri, Sep 9, 2016 at 2:42 PM, Tsuyoshi Ozawa wrote: Hi developers, Chiwan Park reported Hadoop incompatibility by the change of HADOOP-11252. Now, HADOOP-13579 is committed to branch-2.6 and branch-2.7. Should we release 2.6.5 and 2.7.4 soon? Thanks, - Tsuyoshi - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
> On 8 Sep 2016, at 16:08, Allen Wittenauer wrote: > > >> On Sep 8, 2016, at 2:50 AM, Steve Loughran wrote: >> >> I'm trying to do the review effort here even though I don't know detailed >> bash, as I expect I don't know any less than others, and what better way to >> learn than reviewing code written by people that do know bash? > > Just a heads up that I'm using bash variable references. While not > exactly rare, they are uncommon. [We use them in lots of places in the > shell code already, so no new ground being broken.] > >> Could you submit a PR of that HADOOP-13341 branch, so I can review it there. > > Sure. https://github.com/apache/hadoop/pull/126 has been opened. > > Thanks! LGTM: +1. Added the vote on the JIRA too - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/ [Sep 8, 2016 3:49:22 PM] (aajisaka) HDFS-10844. test_libhdfs_threaded_hdfs_static and [Sep 8, 2016 4:34:34 PM] (aajisaka) HDFS-10847. Complete the document for FileDistribution processor in [Sep 8, 2016 6:54:33 PM] (zhz) HDFS-8901. Use ByteBuffer in striping positional read. Contributed by [Sep 8, 2016 7:40:30 PM] (aw) Revert "YARN-5567. Fix script exit code checking in [Sep 8, 2016 8:29:30 PM] (cdouglas) HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. [Sep 9, 2016 12:46:12 AM] (cdouglas) HADOOP-13519. Make Path Serializable. Contributed by Steve Loughran [Sep 9, 2016 12:53:40 AM] (cdouglas) HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen [Sep 9, 2016 1:30:18 AM] (wang) HDFS-10831. Add log when URLConnectionFactory.openConnection failed. [Sep 9, 2016 2:26:56 AM] (aengineer) HDFS-10553. DiskBalancer: Rename Tools/DiskBalancer class to [Sep 9, 2016 3:00:42 AM] (aengineer) HDFS-9849. DiskBalancer: reduce lock path in shutdown code. Contributed -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestDFSShell hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/159/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10708) libhdfs++: Expose an InputStream interface for the apache ORC project
[ https://issues.apache.org/jira/browse/HDFS-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer resolved HDFS-10708. Resolution: Won't Fix Closing this. It's already covered by ORC-17 and it's better to make this an ORC task anyway. > libhdfs++: Expose an InputStream interface for the apache ORC project > -- > > Key: HDFS-10708 > URL: https://issues.apache.org/jira/browse/HDFS-10708 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > > It seems fitting to connect a pure c++ implementation of the HDFS client to a > pure c++ implementation of a parser for the ORC file format. Implementing > the orc::InputStream API is pretty straightforward. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10850) getEZForPath should NOT throw FNF
Daryn Sharp created HDFS-10850: -- Summary: getEZForPath should NOT throw FNF Key: HDFS-10850 URL: https://issues.apache.org/jira/browse/HDFS-10850 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 2.8.0 Reporter: Daryn Sharp Priority: Blocker HDFS-9433 made an incompatible change to the semantics of getEZForPath. It used to return the EZ of the closest ancestor path. It never threw FNF. A common use of getEZForPath to determining if a file can be renamed, or must be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [Release thread] 2.6.5 release activities
Hi all, I wanted to give an update on the Hadoop 2.6.5 release efforts. Here is what has been done so far: 1. I have gone through all of the potential backports and recorded the commit hashes for each of them from the branch that seems the most appropriate (i.e. if there was a backport to 2.7.x then I used the hash from the backport). 2. I verified if the cherry pick for each commit is clean. This was best effort as some of the patches are in parts of the code that I am less familiar with. This is recorded in the public spread sheet here: https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3ol WpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing I am going to need help from committers to get these backports committed. If there are any committers that have some spare cycles, especially if you were involved with the initial commit for one of these issues, please look at the spreadsheet and volunteer to backport one of the issues. As always, please let me know if you have any questions or feel that I have missed something. Thank you! Chris Trezzo On Mon, Aug 15, 2016 at 10:55 AM, Allen Wittenauer wrote: > > > On Aug 12, 2016, at 8:19 AM, Junping Du wrote: > > > > In this community, we are so aggressive to drop Java 7 support in 3.0.x > release. Here, why we are so conservative to keep releasing new bits to > support Java 6? > > I don't view a group of people putting bug fixes into a micro > release as particularly conservative. If a group within the community > wasn't interested in doing it, 2.6.5 wouldn't be happening. > > But let's put the releases into context, because I think it tells > a more interesting story. > > * hadoop 2.6.x = EOLed JREs (6,7) > * hadoop 2.7 -> hadoop 2.x = transitional (7,8) > * hadoop 3.x = JRE 8 > * hadoop 4.x = JRE 9 > > There are groups of people still using JDK6 and they want bug > fixes in a maintenance release. Boom, there's 2.6.x. > > Hadoop 3.x has been pushed off for years for "reasons". So we > still have releases coming off of branch-2. If 2.7 had been released as > 3.x, this chart would look less weird. But it wasn't thus 2.x has this > weird wart in the middle of that supports JDK7 and JDK8. Given the public > policy and roadmaps of at least one major vendor at the time of this > writing, we should expect to see JDK7 support for at least the next two > years after 3.x appears. Bang, there's 2.x, where x is some large number. > > Then there is the future. People using JRE 8 want to use newer > dependencies. A reasonable request. Some of these dependency updates won't > work with JRE 7. We can't do that in hadoop 2.x in any sort of compatible > way without breaking the universe. (Tons of JIRAs on this point.) This > means we can only do it in 3.x (re: Hadoop Compatibility Guidelines). > Kapow, there's 3.x > > The log4j community has stated that v1 won't work with JDK9. In > turn, this means we'll need to upgrade to v2 at some point. Upgrading to > v2 will break the log4j properties file (and maybe other things?). Another > incompatible change and it likely won't appear until Apache Hadoop v4 > unless someone takes the initiative to fix it before v3 hits store > shelves. This makes JDK9 the likely target for Apache Hadoop v4. > > Having major release cadences tied to JRE updates isn't > necessarily a bad thing and definitely forces the community to a) actually > stop beating around the bush on majors and b) actually makes it relatively > easy to determine what the schedule looks like to some degree. > > > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string
Daryn Sharp created HDFS-10851: -- Summary: FSDirStatAndListingOp: stop passing path as string Key: HDFS-10851 URL: https://issues.apache.org/jira/browse/HDFS-10851 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Reporter: Daryn Sharp Assignee: Daryn Sharp Path strings should be resolved once into INodesInPath. The IIP should be used extensively from that point forward. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
+1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative. I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true. On 9/7/16, 6:44 AM, "Allen Wittenauer" wrote: > > I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at > 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was > developed exclusively by me. As usual with large shell script changes, it's > been broken up into several smaller commits to make it easier to read. The > core of the functionality is almost entirely in hadoop-functions.sh with the > majority of the rest of the new additions either being documentation or test > code. In addition, large swaths of code is removed from the hadoop, hdfs, > mapred, and yarn executables. > > Here's a quick summary: > >* makes the rules around _OPTS consistent across all the projects >* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, >and yarn subcommand >* with the exception of deprecations, removes all of the custom daemon _OPTS >handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands >* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it >consistent for non-daemon subcommands >* makes the _USER blocker consistent with _OPTS as well as providing better >documentation around this feature's existence. Note that this is an >incompatible change against -alpha1. >* by consolidating all of this code, makes it possible to finally fix a good >chunk of the "directory name containing spaces blows up the bash code" >problems that's been around since the beginning of the project > > Thanks! > > >- >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-10852) Verify existence of all Tool classes mentioned in HDFS script
Manoj Govindassamy created HDFS-10852:
-
Summary: Verify existence of all Tool classes mentioned in HDFS
script
Key: HDFS-10852
URL: https://issues.apache.org/jira/browse/HDFS-10852
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.0.0-alpha2
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
Priority: Minor
HDFS script (hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs) delegates user
requests to {{org.apache.hadoop.util.Tool}} implementers based on the classname
mappings in the script. If Tools are refactored like classnames are changed,
then the HDFS script has to be updated to carry the new Tool classname.
Existing unit tests do not go via hdfs script as they construct the Tools
directly and run the operations. So, missing to update HDFS script on any
refactoring might break the CLI. Lets have a test to at least verify the
existence of all Tool classes mentioned in HDFS script.
{noformat}
:grep "HADOOP_CLASSNAME=" ./hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.server.balancer.Balancer
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.CacheAdmin
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.CryptoAdmin
HADOOP_CLASSNAME="org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.datanode.DataNode'
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.tools.DebugAdmin'
HADOOP_CLASSNAME=org.apache.hadoop.fs.FsShell
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSAdmin
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DiskBalancerCLI
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.erasurecode.ECCli
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DelegationTokenFetcher
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSck
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.GetConf
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.GetGroups
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSHAAdmin
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.qjournal.server.JournalNode'
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.JMXGet
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.snapshot.LsSnapshottableDir
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.server.mover.Mover
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.namenode.NameNode'
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.nfs.nfs3.Nfs3
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer
HADOOP_CLASSNAME=org.apache.hadoop.portmap.Portmap
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode'
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.snapshot.SnapshotDiff
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.StoragePolicyAdmin
HADOOP_CLASSNAME=org.apache.hadoop.util.VersionInfo
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.tools.DFSZKFailoverController'
HADOOP_CLASSNAME="${subcmd}"
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
> On Sep 9, 2016, at 2:15 PM, Anu Engineer wrote: > > +1, Thanks for the effort. It brings in a world of consistency to the hadoop > vars; and as usual reading your bash code was very educative. Thanks! There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established. > I had a minor suggestion though. since we have classified the _OPTS to client > and daemon opts, for new people it is hard to know which of these subcommands > are daemon vs. a client command. Maybe we can add a special char in the help > message to indicate which are daemons or just document it? Only way I know > right now is to look the appropriate script and see if > HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true. That's a great suggestion. Would it be better if the usage output was more like: ---snip--- Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] OPTIONS is none or any of: --buildpaths attempt to add class files from build tree --config dir Hadoop config directory --daemon (start|status|stop) operate on a daemon --debugturn on shell script debug mode --help usage information --hostnames list[,of,host,names] hosts to use in worker mode --hosts filename list of hosts to use in worker mode --loglevel level set the log4j level for this command --workers turn on worker mode SUBCOMMAND is one of: Clients: cacheadmin configure the HDFS cache classpathprints the class path needed to get the hadoop jar and the required libraries crypto configure HDFS encryption zones ... Daemons: balancer run a cluster balancing utility datanode run a DFS datanode namenode run the DFS name node ... ---snip--- We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10853) Fix TestWebHdfsProxySelector in branch-2.8
Eric Badger created HDFS-10853: -- Summary: Fix TestWebHdfsProxySelector in branch-2.8 Key: HDFS-10853 URL: https://issues.apache.org/jira/browse/HDFS-10853 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Eric Badger Assignee: Eric Badger Similar to HDFS-8948, we need to use GenericTestUtils to set log levels to avoid multiple binding errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
> SUBCOMMAND is one of: > > >Clients: > cacheadmin configure the HDFS cache > classpathprints the class path needed to get the hadoop jar > and the required libraries > crypto configure HDFS encryption zones > ... > >Daemons: > balancer run a cluster balancing utility > datanode run a DFS datanode > namenode run the DFS name node >... >---snip--- Absolutely, that is a great output, very clear and provides a very good user experience. Thanks Anu On 9/9/16, 3:06 PM, "Allen Wittenauer" wrote: > >> On Sep 9, 2016, at 2:15 PM, Anu Engineer wrote: >> >> +1, Thanks for the effort. It brings in a world of consistency to the hadoop >> vars; and as usual reading your bash code was very educative. > > Thanks! > > There's still a handful of HDFS and MAPRED vars that begin with HADOOP, > but those should be trivial to knock out after a pattern has been established. > >> I had a minor suggestion though. since we have classified the _OPTS to >> client and daemon opts, for new people it is hard to know which of these >> subcommands are daemon vs. a client command. Maybe we can add a special >> char in the help message to indicate which are daemons or just document it? >> Only way I know right now is to look the appropriate script and see if >> HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true. > > > That's a great suggestion. Would it be better if the usage output was > more like: > >---snip--- >Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] > > OPTIONS is none or any of: > >--buildpaths attempt to add class files from build tree >--config dir Hadoop config directory >--daemon (start|status|stop) operate on a daemon >--debugturn on shell script debug mode >--help usage information >--hostnames list[,of,host,names] hosts to use in worker mode >--hosts filename list of hosts to use in worker mode >--loglevel level set the log4j level for this command >--workers turn on worker mode > > SUBCOMMAND is one of: > > >Clients: > cacheadmin configure the HDFS cache > classpathprints the class path needed to get the hadoop jar > and the required libraries > crypto configure HDFS encryption zones > ... > >Daemons: > balancer run a cluster balancing utility > datanode run a DFS datanode > namenode run the DFS name node >... >---snip--- > > We do something similar in Apache Yetus and shouldn't be too hard to do > in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but > as long as people put their new commands in the correct section in > hadoop_usage, it should work. > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10854) Remove createStripedFile and addBlockToFile by creating real EC files
Zhe Zhang created HDFS-10854:
Summary: Remove createStripedFile and addBlockToFile by creating
real EC files
Key: HDFS-10854
URL: https://issues.apache.org/jira/browse/HDFS-10854
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding, test
Affects Versions: 3.0.0-alpha2
Reporter: Zhe Zhang
{{DFSTestUtil#createStripedFile}} and {{addBlockToFile}} were developed before
we completed EC client. They were used to test the {{NameNode}} EC logic when
the client was unable to really create/read/write EC files.
They are causing confusions in other issues about {{NameNode}}. For example, in
one of the patches under {{HDFS-10301}},
{{testProcessOverReplicatedAndMissingStripedBlock}} fails because in the test
we fake a block report from a DN, with a randomly generated storage ID. The DN
itself is never aware of that storage. This is not possible in a real
production environment.
{code}
DatanodeStorage storage = new
DatanodeStorage(UUID.randomUUID().toString());
StorageReceivedDeletedBlocks[] reports = DFSTestUtil
.makeReportForReceivedBlock(block,
ReceivedDeletedBlockInfo.BlockStatus.RECEIVING_BLOCK, storage);
for (StorageReceivedDeletedBlocks report : reports) {
ns.processIncrementalBlockReport(dn.getDatanodeId(), report);
}
{code}
Now that we have a fully functional EC client, we should remove the old testing
logic and use similar logic as non-EC tests (creating real files and emulate
blocks missing / being corrupt).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
+1 (also on JIRA) -C On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer wrote: > > I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 > at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch > was developed exclusively by me. As usual with large shell script changes, > it's been broken up into several smaller commits to make it easier to read. > The core of the functionality is almost entirely in hadoop-functions.sh with > the majority of the rest of the new additions either being documentation or > test code. In addition, large swaths of code is removed from the hadoop, > hdfs, mapred, and yarn executables. > > Here's a quick summary: > > * makes the rules around _OPTS consistent across all the projects > * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, > and yarn subcommand > * with the exception of deprecations, removes all of the custom daemon _OPTS > handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands > * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it > consistent for non-daemon subcommands > * makes the _USER blocker consistent with _OPTS as well as providing better > documentation around this feature's existence. Note that this is an > incompatible change against -alpha1. > * by consolidating all of this code, makes it possible to finally fix a good > chunk of the "directory name containing spaces blows up the bash code" > problems that's been around since the beginning of the project > > Thanks! > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
