On Wed, Mar 21, 2012 at 1:57 PM, Sanjay Radia <san...@hortonworks.com> wrote:
> On Tue, Mar 20, 2012 at 5:37 PM, Eli Collins <e...@cloudera.com> wrote:
>
>>
>>
>> Append introduces non-trivial design and code complexity, which is not
>> worth the cost if we don't have real users.
>
> The bulk of the complexity of HDFS-265 ("the new Append") was around
> Hflush, concurrent readers, the pipeline etc. The code and complexity for
> appending to previously closed file was not that large.
>
And we'd still leverage that work. Which is not to say that append
isn't complicated. There were a fair number of append bugs that were
found in branch-20-append that we think are present in the new append
implementation (not sure if there are jiras for all of them).
Also, append + truncate removes the current invariant that we maintain
eg around visible length. So append opens the doors for lots of
additional complexity. We could decide to keep append but not add
truncate but I suspect that will be hard because once you open up the
doors to a lot of new use cases it's hard to close them. The larger
issue is how simple we'd like to keep HDFS, how many use cases we'd
like to grow it to.
>
>
>> Removing append means we
>> have the property that HDFS blocks, when finalized, are immutable.
>> This significantly simplifies the design and code, which significantly
>> simplifies the implementation of other features like snapshots,
>> HDFS-level caching, dedupe, etc.
>>
>
> While Snapshots are challenging with Append, it is solvable - the snapshot
> needs to remember the length of the file. (We have a working prototype - we
> will posting the design and the code soon).
>
Will check it out. When I read "Snapshots in Hadoop Distributed File
System" it looked like the bulk of the complexity was due to the
protocol for append:
http://www.cs.berkeley.edu/~sameerag/hdfs-snapshots.pdf
>
> I agree that the notion of an immutable file is useful since it lets the
> system and tools optimize certain things. A xerox-parc file system in the
> 80s had this feature that the system exploited. I would support adding the
> notion of an immutable file to Hadoop.
>
Good point, we could leverage this property on a per-file, rather than
per-filesystem basis.
Thanks,
Eli
