Re: Hadoop without HDFS

2014-12-11 Thread Steve Loughran
one more thing, the "if" excludes object stores which don't offer consistency and atomic create-no-overwrite and rename. You can't run all hadoop apps directly on top of Amazon S3, without extra work (see netflix S3mper). Object stores do not always behave as filesystems, even if they implement the

Re: Hadoop without HDFS

2014-12-10 Thread Jay Vyas
Yup, that's a great summary. More details... The HCFS wiki page will give you insight into some tests you can run to test your FileSystem plugin class, which you will put in a jar file described below. In general, hadoop apps are written to the file system interface which is loaded runtime, s

Re: Hadoop without HDFS

2014-12-10 Thread Roman Shaposhnik
On Wed, Dec 10, 2014 at 12:20 PM, Ari King wrote: > Hi, > > I'm doing a research paper on Hadoop -- specifically relating to its > dependency on HDFS. I need to determine if and how HDFS can be replaced. As > I understand it, there are a number of organizations that have produced > HDFS alternativ

Hadoop without HDFS

2014-12-10 Thread Ari King
Hi, I'm doing a research paper on Hadoop -- specifically relating to its dependency on HDFS. I need to determine if and how HDFS can be replaced. As I understand it, there are a number of organizations that have produced HDFS alternatives that support the Hadoop ecosystem, i.e. MapReduce, Hive, HB