Port 0.20-append changes onto 0.20-security-203
-----------------------------------------------
Key: HDFS-1795
URL: https://issues.apache.org/jira/browse/HDFS-1795
Project: Hadoop HDFS
Issue Type: Task
Reporter: Andrew Purtell
Attachments: security-append-patches.zip
Port 0.20-append changes onto 0.20-security-203.
I started with a Git repository cloned from
git://git.apache.org/hadoop-common.git . Branch 'branch-0.20-security-203' was
used as the starting point for the work. I then enumerated over the 0.20-append
specific patches in 'branch-0.20-append'. Each was applied if not already via
cherry pick except for as noted below. This process in effect replayed the
evolution of 0.20-append branch on top of 0.20-security-203.
The specific functional changes that HBase absolutely relies upon are specially
mentioned.
Generally I ran the full test suite after each change. There were a couple of
exceptions where pairs of adjacent change sets were strongly related, in which
case I applied them in sequence, then ran the test suite. During this process I
encountered no test failures except for one test in TestFileAppend4, a test
brought in from the append branch, and I still need to dig in to see if this is
a real problem or if the test needs to be changed to work on top of
security-203.
{noformat}
commit b9ad012eaf3915c2169a02a7130b54cbcc1d8a89
Author: Dhruba Borthakur <[email protected]>
Date: Fri Jun 4 07:20:10 2010 +0000
HDFS-200. Support append and sync for hadoop 0.20 branch.
Required for HBase
commit c968e11b5a60fc6f28e4e43fbbc8a99e7e49a659
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 9 23:09:07 2010 +0000
HDFS-101. DFSClient correctly detects second datanode failure in write
pipeline. (Nicolas Spiegelberg via dhruba)
Excluded
Already in 0.20-security-203 according to search of Git change log
commit 9f7e5ed2ff47444a1dcd12ed34796929d5b9f7d5
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 9 23:12:21 2010 +0000
HDFS-988. Fix bug where savenameSpace can corrupt edits log.
(Nicolas Spiegelberg via dhruba)
commit dfbbd6fbadaa95c54a1040b4fe8854b1b858d7a5
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 10 18:46:03 2010 +0000
HDFS-826. Allow a mechanism for an application to detect that
datanode(s) have died in the write pipeline. (dhruba)
Required for HBase
Commit be8d32503d30208a2d7772b3b4b2a270938a4004
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 10 22:25:39 2010 +0000
HDFS-142. Blocks that are being written by a client are stored in the
blocksBeingWritten directory.
(Dhruba Borthakur, Nicolas Spiegelberg, Todd Lipcon via dhruba)
commit 856efc2e95aaacc597d669c1b053634ff752dbec
Author: Dhruba Borthakur <[email protected]>
Date: Fri Jun 11 00:48:41 2010 +0000
HDFS-630. Client can exclude specific nodes in the write pipeline.
(Nicolas Spiegelberg via dhruba)
Required for HBase
commit 2da1a05fc0cc0429229e87694977bae2ba370625
Author: Dhruba Borthakur <[email protected]>
Date: Fri Jun 11 01:02:13 2010 +0000
HDFS-457. Better handling of volume failure in DataNode Storage.
(Nicolas Spiegelberg via dhruba)
Excluded
Already in 0.20-security-203 according to search of Git change log
commit bd42393cd3a3a731ea98b25ddb528ad03a1ab4af
Author: Dhruba Borthakur <[email protected]>
Date: Fri Jun 11 23:37:38 2010 +0000
HDFS-1054. remove sleep before retry for allocating a block.
(Todd Lipcon via dhruba)
commit 120441b9e571a5703ac39b47608e87182f0f4972
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 20:53:12 2010 +0000
HDFS-445. pread should refetch block locations when necessary.
(Todd Lipcon via dhruba)
Excluded
Already in 0.20-security-203 according to search of Git change log
commit 2004aa453ba6b7ee2045093ba313ef8551a7f8da
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 20:59:10 2010 +0000
HDFS-561. Fix write pipeline
commit 2a8227b0e6be8937fc4a654899be2a22c1f6efbe
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 21:13:24 2010 +0000
HDFS-927. DFSInputStream retries too many times for new block
locations. (Todd Lipcon via dhruba)
Excluded
Already in 0.20-security-203 according to search of Git change log
commit b1e49dbf50a429cf01b636caa2666ff81ed2a016
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 21:21:45 2010 +0000
HDFS-1215. Fix unti test TestNodeCount.
(Todd Lipcon via dhruba)
Excluded
TestNodeCount not failing on current working branch
commit b5078c57478deda9974ed40897045d38b100b9fc
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 21:25:38 2010 +0000
HDFS-606. Fix ConcurrentModificationException in
invalidateCorruptReplicas. (Todd Lipcon via dhruba)
commit c3d35f8cdd761cbcd0a4ef851e37ccdb2d25d7e0
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 21:42:53 2010 +0000
HDFS-1216. Upgrade to junit 4.5
(Todd Lipcon via dhruba)
Excluded
Already using Junit 4.5 on current working branch
commit c21b6834e982b0bf2edaf61dae8a0028076ccb60
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 17 01:34:56 2010 +0000
HDFS-1207. FSNamesystem.stallReplicationWork should be volatile.
(Todd Lipcon via dhruba)
commit e8a1a09fec716bb013f94f1c14f4ae8f8409ee68
Author: Dhruba Borthakur <[email protected]>
Date: Wed Jun 16 21:57:39 2010 +0000
HDFS-1141. completeFile does not check lease ownership.
(Todd Lipcon via dhruba)
commit ca458a07a6d8cb5cda8c589b1a23dabffacef7fe
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 17 01:44:32 2010 +0000
HDFS-1204. Lease expiration should recover single files,
not entire lease holder (Sam Rash via dhruba)
commit 62441fbd516ec9132619d448a1051554d29d2dba
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 17 01:52:50 2010 +0000
HDFS-1210. DFSClient should log exception when block recovery fails.
(Todd Lipcon via dhruba)
Not a required change but a minor useful one
commit 08b67cdf9ff7dada8aa86a2443a0ffc1a0a83c20
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jun 17 01:59:30 2010 +0000
HDFS-1211. Block receiver should not log "rewind" packets at INFO
level. (Todd Lipcon via dhruba)
Not a required change but a minor useful one
commit 0c9911a623891e82c40d4c1c3a523f3a85a85216
Author: Dhruba Borthakur <[email protected]>
Date: Sun Jun 20 08:31:54 2010 +0000
HDFS-1118. Fix socketleak on DFSClient.
(Zheng Shao via dhruba)
commit db9989c091b37446e4f4d468b24c4bd08b1ea337
Author: Tsz-wo Sze <[email protected]>
Date: Mon Jun 21 21:26:45 2010 +0000
MAPREDUCE-1880. Fix BigDecimal.divide(..) in the pi example.
Excluded
Not an HDFS change
commit 11fa325540526da57a9e178463017853058ba921
Author: Dhruba Borthakur <[email protected]>
Date: Tue Jun 22 05:55:48 2010 +0000
HDFS-1254. Support append/sync via the default configuration.
(dhruba)
commit 9e405ca177aba719b04cfd7d33d400d311bf8db7
Author: Dhruba Borthakur <[email protected]>
Date: Thu Jul 1 08:37:30 2010 +0000
HDFS-1057. Concurrent readers hit ChecksumExceptions if following
a writer to very end of file (Sam Rash via dhruba)
commit 6512b527cda8efa0c622a8657af96ac785625773
Author: Dhruba Borthakur <[email protected]>
Date: Tue Jul 6 06:52:20 2010 +0000
HDFS-1202. DataBlockScanner throws NPE when updated before
initialized. (Todd Lipcon via dhruba)
commit 116e71fea80c31042520f364625ba7b1647cdb09
Author: Tsz-wo Sze <[email protected]>
Date: Tue Jul 6 17:39:25 2010 +0000
HDFS-1258. Clearing namespace quota on "/" corrupts fs image.
commit ca88978b319debe146e9988d882393033b1d2d62
Author: Tsz-wo Sze <[email protected]>
Date: Wed Sep 22 20:52:25 2010 +0000
MAPREDUCE-1280. Update Eclipse plugin to the new eclipse.jdt API.
Excluded
Not an HDFS change
commit a68b163fc31c3d82c63b69ae99e9954fef4df4af
Author: Hairong Kuang <[email protected]>
Date: Mon Sep 27 18:14:20 2010 +0000
HDFS-1346. DFSClient receives out of order packet ack.
commit 7083b46208f56776be9ada2d0e60d95cb4fdb67e
Author: Hairong Kuang <[email protected]>
Date: Mon Nov 1 06:17:40 2010 +0000
HDFS-724. Use a bidirectional heartbeat to detect stuck pipeline.
commit eb2346ea7726c33715f3e651f4babcb6e6c3f0c2
Author: Hairong Kuang <[email protected]>
Date: Tue Nov 16 17:53:17 2010 +0000
Fix incorrect heartbeat ack serialization in HDFS-724 for append 0.20.
Combined with 7083b462
commit 0e996bdba420fd0031490457e27dda67f58de18f
Author: Eli Collins <[email protected]>
Date: Tue Nov 9 01:59:27 2010 +0000
HDFS-1240. svn merge -c 1032810 from branch-0.20.
Excluded
TestDFSShell not failing on working branch
Commit eb2346ea7726c33715f3e651f4babcb6e6c3f0c2
Author: Hairong Kuang <[email protected]>
Date: Tue Nov 16 17:53:17 2010 +0000
HDFS-895. Allow hflush/sync to occur in parallel with new writes to
the file
Required to mitigate HLog flush penalty
Need to raise default heap for tests for HDFS-895 unit tests to pass
commit 9a3094f3a085817d377ce69fb8eb002f966f992b
Author: Dhruba Borthakur <[email protected]>
Date: Fri Dec 10 05:38:06 2010 +0000
HDFS-1520. Lightweight NameNode operation recoverLease to trigger
lease recovery. (Hairong Kuang via dhruba)
Required by HBase for master to seize ownership of and quickly split
HLogs of failed regionservers
commit f9436bf6bd2f5bc4f8cebe2781c5ce628db31239
Author: Hairong Kuang <[email protected]>
Date: Fri Jan 7 20:11:38 2011 +0000
HDFS-1555. Disallow pipeline recovery if a file is already being lease
Excluded
Bad commit
commit 966ef38cb13d75ada40de2b9951496ddcb6918a4
Author: Hairong Kuang <[email protected]>
Date: Fri Jan 7 20:19:40 2011 +0000
Revert the change made to HDFS-1555:
Excluded
commit e499be8c74a871ef06aa8a74c621e68d50167d3d
Author: Hairong Kuang <[email protected]>
Date: Fri Jan 7 20:25:23 2011 +0000
HDFS-1555. Disallow pipelien recovery if a file is already being
lease recovered
commit df0d79cc2b09438c079fdf10b913936492117917
Author: Hairong Kuang <[email protected]>
Date: Mon Jan 10 19:01:36 2011 +0000
HDFS-1554. New semantics for recoverLease.
Required by HBase for master to seize ownership of and quickly split
HLogs of failed regionservers
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
