one more thing, the "if" excludes object stores which don't offer consistency and atomic create-no-overwrite and rename. You can't run all hadoop apps directly on top of Amazon S3, without extra work (see netflix S3mper). Object stores do not always behave as filesystems, even if they implement the relevant Hadoop APIs (some do though, like google's and microsoft's)
HADOOP-9361 and the filesystem documentation attempt to formally specify what an FS should do; http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html Where "formally" means "try to rigorously define what HDFS does and how other filesystems (especially posix ones) differ" HADOOP-9565 looking at some explicit ObjectStore subclass of FileSystem to provide more details on object stores On 10 December 2014 at 20:20, Ari King <ari.brandeis.k...@gmail.com> wrote: > Hi, > > I'm doing a research paper on Hadoop -- specifically relating to its > dependency on HDFS. I need to determine if and how HDFS can be replaced. As > I understand it, there are a number of organizations that have produced > HDFS alternatives that support the Hadoop ecosystem, i.e. MapReduce, Hive, > HBase, etc. > > With the "if" part being answered, I'd appreciate insight/guidance on the > "how" part. Essentially, where can I find information on what MapReduce and > the other Hadoop subprojects require of the underlying file system and how > these subprojects expect to interact with the file system. > > Thanks! > > Best, > Ari > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
