Yes, I can talk about append as an example. Some differences with federation project are: - append had a comprehensive test plan document, which was designed an executed; - append was independently evaluated by HBase guys; - it introduced new benchmark for append; - We ran both DFSIO and NNThroughput. DFSIO was executed on a relatively small cluster. I couldn't find where I posted the results, my bad. But you may be able to find these tasks in our scrum records.
--Konstantin On Tue, Apr 26, 2011 at 11:55 PM, suresh srinivas <srini30...@gmail.com>wrote: > Konstantin, > > Could you provide me link to how this was done on a big feature, like say > append and how benchmark info was captured? I am planning to run dfsio > tests, btw. > > Regards, > Suresh > > On Tue, Apr 26, 2011 at 11:34 PM, suresh srinivas <srini30...@gmail.com > >wrote: > > > Konstantin, > > > > On Tue, Apr 26, 2011 at 10:26 PM, Konstantin Shvachko < > > shv.had...@gmail.com> wrote: > > > >> Suresh, Sanjay. > >> > >> 1. I asked for benchmarks many times over the course of different > >> discussions on the topic. > >> I don't see any numbers attached to jira, and I was getting the same > >> response, > >> Doug just got from you, guys: which is "why would the performance be > >> worse". > >> And this is not an argument for me. > >> > > > > We had done testing earlier and had found that performance had not > > degraded. We are waiting for out performance team to publish the official > > numbers to post it to the jira. Unfortunately they are busy qualifying > 2xx > > releases currently. I will get the perf numbers and post them. > > > > > >> > >> 2. I assume that merging requires a vote. I am sure people who know > bylaws > >> better than I do will correct me if it is not true. > >> Did I miss the vote? > >> > > > > > > As regards to voting, since I was not sure about the procedure, I had > > consulted Owen about it. He had indicated that voting is not necessary. > If > > the right procedure is to call for voting, I will do so. Owen any > comments? > > > > > >> > >> It feels like you are rushing this and are not doing what you would > expect > >> others to > >> do in the same position, and what has been done in the past for such > large > >> projects. > >> > > > > I am not trying to rush here and not follow the procedure required. I am > > not sure about what the procedure is. Any pointers to it is appreciated. > > > > > >> > >> Thanks, > >> --Konstantin > >> > >> > >> On Tue, Apr 26, 2011 at 9:43 PM, Doug Cutting <cutt...@apache.org> > wrote: > >> > >> > Suresh, Sanjay, > >> > > >> > Thank you very much for addressing my questions. > >> > > >> > Cheers, > >> > > >> > Doug > >> > > >> > On 04/26/2011 10:29 AM, suresh srinivas wrote: > >> > > Doug, > >> > > > >> > > > >> > >> 1. Can you please describe the significant advantages this approach > >> has > >> > >> over a symlink-based approach? > >> > > > >> > > Federation is complementary with symlink approach. You could choose > to > >> > > provide integrated namespace using symlinks. However, client side > >> mount > >> > > tables seems a better approach for many reasons: > >> > > # Unlike symbolic links, client side mount tables can choose to go > to > >> > right > >> > > namenode based on configuration. This avoids unnecessary RPCs to the > >> > > namenodes to discover the targer of symlink. > >> > > # The unavailability of a namenode where a symbolic link is > configured > >> > does > >> > > not affect reaching the symlink target. > >> > > # Symbolic links need not be configured on every namenode in the > >> cluster > >> > and > >> > > future changes to symlinks need not be propagated to multiple > >> namenodes. > >> > In > >> > > client side mount tables, this information is in a central > >> configuration. > >> > > > >> > > If a deployment still wants to use symbolic link, federation does > not > >> > > preclude it. > >> > > > >> > >> It seems to me that one could run multiple namenodes on separate > >> boxes > >> > > and run multile datanode processes per storage box > >> > > > >> > > There are several advantages to using a single datanode: > >> > > # When you have large number of namenodes (say 20), the cost of > >> running > >> > > separate datanodes in terms of process resources such as memory is > >> huge. > >> > > # The disk i/o management and storage utilization using a single > >> datanode > >> > is > >> > > much better, as it has complete view the storage. > >> > > # In the approach you are proposing, you have several clusters to > >> manage. > >> > > However with federation, all datanodes are in a single cluster; with > >> > single > >> > > configuration and operationally easier to manage. > >> > > > >> > >> The patch modifies much of the logic of Hadoop's central component, > >> upon > >> > > which the performance and reliability of most other components of > the > >> > > ecosystem depend. > >> > > That is not true. > >> > > > >> > > # Namenode is mostly unchanged in this feature. > >> > > # Read/write pipelines are unchanged. > >> > > # The changes are mainly in datanode: > >> > > #* the storage, FSDataset, Directory and Disk scanners now have > >> another > >> > > level to incorporate block pool ID into the hierarchy. This is not a > >> > > significant change that should cause performance or stability > >> concerns. > >> > > #* datanodes use a separate thread per NN, just like the existing > >> thread > >> > > that communicates with NN. > >> > > > >> > >> Can you please tell me how this has been tested beyond unit tests? > >> > > As regards to testing, we have passed 600+ tests. In hadoop, these > >> tests > >> > > are mostly integration tests and not pure unit tests. > >> > > > >> > > While these tests have been extensive, we have also been testing > this > >> > branch > >> > > for last 4 months, with QA validation that reflects our production > >> > > environment. We have found the system to be stable, performing well > >> and > >> > have > >> > > not found any blockers with the branch so far. > >> > > > >> > > HDFS-1052 has been open more than a year now. I had also sent an > email > >> > about > >> > > this merge around 2 months ago. There are 90 subtasks that have been > >> > worked > >> > > on last couple of months under HDFS-1052. Given that there was > enough > >> > time > >> > > to ask these questions, your email a day before I am planning to > merge > >> > the > >> > > branch into trunk seems late! > >> > > > >> > > >> > > > > > > > > -- > > Regards, > > Suresh > > > > > > > -- > Regards, > Suresh >
