On Thu, Mar 22, 2012 at 10:15 AM, Daryn Sharp <da...@yahoo-inc.com> wrote: > On Mar 20, 2012, at 7:37 PM, Eli Collins wrote: >> Hey gang, >> >> I'd like to get people's thoughts on the following proposal. I think >> we should consider removing append from HDFS. >> >> Where we are today.. append was added in the 0.17-19 releases >> (HADOOP-1700) and subsequently disabled (HADOOP-5224) due to quality >> issues. It and sync were re-designed, re-implemented, and shipped in >> 21.0 (HDFS-265). To my knowledge, there has been no real production >> use. Anecdotally people who worked on branch-20-append have told me >> they think the new trunk code is substantially less well-tested than >> the branch-20-append code (at least for sync, append was never well >> tested). It has certainly gotten way less pounding from HBase users. >> The design however, is much improved, and people think we can get >> hsync (and append) stabilized in trunk (mostly testing and bug >> fixing). > > Up front: I think append is a needed feature. >
Can you elaborate.. eg are there particular use cases at Yahoo! that have been running for years that are itching to start using append when 0.23 is deployed? Are you guys testing the new append implementation extensively because you have an app that's ready to use it when 0.23 is deployed? So far Milind has been the only one to chime in saying "we really need append, here's why". Which is great. > Politely speaking, I think the premise of the question is a bit dubious due > to circular nature. Ie. It's not used in production so is it worth it? No, I'm saying we've absorbed a lot of complexity for it, but I don't see downstream projects using it any time soon. Similarly there hasn't been a big push to get it working, eg there was a push for security and hbase support on 20, but not append (the append rewrite was invasive but so was security). It will have been several years from the time the rewrite was started until it gets deployed in production, which makes me think it's less of a priority. So much so that I wondered whether it was a big priority at all. > The stigma/perception that append has been unstable and is not well-tested is > a compelling reason to not be in production at major installations. The > situation is going to be akin to "You go first. No, you go first! No way, > you go first!". > > Downstream projects also aren't going to use something until it's stable, so > they either work around the limitation, or... they chose something other > hdfs. There's also the unanswerable question of how potential users have > been silently lost. We are unlikely to have heard the user demand from those > that chose another solution. Generally for every complaint/request, a large > N-many people didn't even bother. > > I envision a day where hdfs is a performant posix filesystem. I think that's unlikely. Posix compliance is a non-goal for HDFS. We are intentionally not-compliant in many cases to achieve scale and performance. Check out the Ceph file system paper (they manage the tradeoff between Posix and scale/performance explicitly). The primary motivation for Posix compliance is compatibility with existing Unix-like software. That's not HDFS' raison d'etre (which is the ecosystem of projects that run atop HDFS: MR, HBase, Pig, Hive, Flume, Sqoop, etc etc). HDFS' focus on it's core use case and simplicity is one of the reasons it's been as successful as it has been. That's not to say we don't need to do a lot more work to better integrate HDFS with existing software and tools. Fuse-DFS is just a start. We need to support a standard interface like NFS etc. These efforts do not require HDFS become fully Posix compliant. There's always a trade off between adding more features (increasing the size of the addressable market) and focusing on your core uses cases, quality, etc. In my mind append is on the boundary. I'm happy to be convinced that append is in HDFS' wheel house. Thanks, Eli
