+1 nice feature for HDFS
On Fri, Dec 6, 2013 at 7:32 PM, Arpit Agarwal <aagar...@hortonworks.com>wrote: > Hi Andrew, > > Our plan as stated back in August was to do this work principally in two > phases. > https://issues.apache.org/jira/browse/HDFS-2832?focusedCommentId=13739041 > > For the second phase which includes API support, we also need quota > management. For changes of this scope, to do all the work at once while > keeping the feature branch in sync with ongoing development in trunk is > unmanageable. Hence we'd like to stick with the initial plan and develop in > phases. > > Even for datanode caching the initial merge did not include the quota > management changes which are happening subsequently. > > Going forward, we will stabilize the current changes in trunk in the 2.4 > time frame. Next we will add quota management and API support which can > align with the 2.5 time frame, with the second merge potentially in > March/April. > > Arpit > > > On Fri, Dec 6, 2013 at 3:15 PM, Andrew Wang <andrew.w...@cloudera.com > >wrote: > > > Hi everyone, > > > > I'm still getting up to speed on the changes here (my fault for not > > following development more closely, other priorities etc etc), but the > > branch thus far is already quite impressive. It's quite an undertaking to > > turn the DN into a collection of Storages, along with the corresponding > > datastructure, tracking, and other changes in the NN and DN. > > > > Correct me if I'm wrong though, but this still leaves a substantial part > of > > the design doc to be implemented. Looking at the list of remaining > > subtasks, it seems like we still can't specify a storage type for a file > > (HDFS-5229) or write a file to a given storage type (HDFS-5391), along > with > > the corresponding client protocol changes. This leads me to two > questions: > > > > - If this is merged, what can I do with the new code? Without client > > changes or the ability to create a file on a different storage type, I > > don't know how (for example) I could hand this to our QA team to test. > I'm > > wondering why we want to merge now rather than when the branch is more > > feature complete. > > - What's the plan for the implementation of the remaining features? How > > many phases? What's the timeline for these phases? Particularly, related > to > > the use cases presented in section 2 of the design doc. > > > > I'm also going to post some design doc questions to the JIRA, there are a > > few technical q's I'd like to get clarification on. > > > > Thanks, > > Andrew > > > > > > On Wed, Dec 4, 2013 at 7:21 AM, Sirianni, Eric <eric.siria...@netapp.com > > >wrote: > > > > > +1 > > > > > > My team has been developing and testing against the HDFS-2832 branch > for > > > the past month. It has proven to be quite stable. > > > > > > Eric > > > > > > -----Original Message----- > > > From: Arpit Agarwal [mailto:aagar...@hortonworks.com] > > > Sent: Monday, December 02, 2013 7:07 PM > > > To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org > > > Subject: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk > > > > > > Hello all, > > > > > > I would like to call a vote to merge phase 1 of the Heterogeneous > Storage > > > feature into trunk. > > > > > > *Scope of the changes:* > > > The changes allow exposing the DataNode as a collection of storages and > > set > > > the foundation for subsequent work to present Heterogeneous Storages to > > > applications. This allows DataNodes to send block and storage reports > > > per-storage. In addition this change introduces the ability to add a > > > 'storage type' tag to the storage directories. This enables supporting > > > different types of storages in addition to disk storage. > > > > > > Development of the feature is tracked in the jira > > > https://issues.apache.org/jira/browse/HDFS-2832. > > > > > > *Details of development and testing:* > > > Development has been done in a separate branch - > > > https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The > > > updated design is posted at - > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf > > > . > > > The changes involve ~6K changed lines of code, with a third of those > > > changes being to tests. > > > > > > Please see the test plan > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor > > > the details. Once the feature is > > > merged into trunk, we will continue to test and fix any bugs that may > be > > > found on trunk as well as add further tests as outlined in the test > plan. > > > > > > The bulk of the design and implementation was done by Suresh Srinivas, > > > Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric > > > Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and > Todd > > > Lipcon for providing feedback on the Jiras and in discussions. > > > > > > This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT. > > > > > > Thanks, > > > Arpit > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
