I feel that making the datanode talk to multiple namenodes is very valuable, especially when there is plenty of storage available on a single datanode machine (think 24 TB to 36 TB) and a single namenode does not have enough memory to hold all file metadata for such a large cluster in memory.
This is a feature that we are in dire need of, and could put it to good use starting "yesterday"! thanks, dhruba On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <c...@apache.org> wrote: > Sanjay, > > I assume the outlined changes won't an earlier version of HDFS from > upgrads to the federation version, right? > > Cos > > On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <sra...@yahoo-inc.com> wrote: > > > > Changes to the code base > > - The fundamental code change is to extend the notion of block id to now > > include a block pool id. > > - The NN had little change, the protocols did change to include the > block > > pool id. > > - The DN code did change. Each data structure is now indexed by the block > > pool id -- while this is a code change, it is architecturally very simple > > and low risk. > > - We also did a fair amount of cleanup of threads used to send block > reports > > - while it was not strictly necessary to do the cleanup we took the extra > > effort to pay the technical debt. As Dhruba recently noted, adding > support > > to send block reports to primary and secondary NN for HA will be now much > > easier to do. > -- Connect to me at http://www.facebook.com/dhruba
