Thanks Colin: that's a good example of why we want To unify the hcfs test profile. So how can hcfs implementations use current hadoop-common tests?
In mind there are three ways. - one solution is to manually cobble together and copy tests , running them one by one and seeing which ones apply to their fs. this is what I think we do now (extending base contract, main operations tests, overriding some methods, ..). - another solution is that all hadoop filesystems should conform to one exact contract. Is that a pipe dream? Or is it possible? - a third solution. Is that we could use a declarative API where file system implementations declare which tests or groups of tests they don't want to run. That is basically hadoop-9361 - The third approach could be complimented by barebones, simple in-memory curated reference implementations that exemplify distilled filesystems with certain salient properties (I.e. Non atomic mkdirs) > On Mar 6, 2014, at 1:47 PM, Colin McCabe <cmcc...@alumni.cmu.edu> wrote: > > NetFlix's Apache-licensed S3mper system provides consistency for an > S3-backed store. > http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html > > It would be nice to see this or something like it integrated with > Hadoop. I fear that a lot of applications are not ready for eventual > consistency, and may never be, leading to the feeling that Hadoop on > S3 is buggy. > > Colin > >> On Thu, Mar 6, 2014 at 10:42 AM, Jay Vyas <jayunit...@gmail.com> wrote: >> do you consider that native S3 FS a real "reference implementation" for >> blob stores? or just something that , by mere chance, we are able to use as >> a ref. impl.
