On 7 March 2014 01:35, Jay Vyas wrote:
> Thanks steve. So i guess the conclusion is
>
> 1) Wait on HADOOP-9361.
>
"help" with is a better plan. I really don't look at it that often. I'll
try and get it ready to review this weekend
> 2) There definitively cannot be a strict contract for a sing
Thanks steve. So i guess the conclusion is
1) Wait on HADOOP-9361.
2) There definitively cannot be a strict contract for a single HCFS, based
on your examples shown.
In the meantime ill audit existing test coverage, and let me know if i can
lend a hand in the cleanup process.
On Thu, Mar 6,
Lets get the HADOOP-9361 stuff in (it lives alongside
FileSystemContractBaseTest) and you can work off that.
On 6 March 2014 18:57, Jay Vyas wrote:
> Thanks Colin: that's a good example of why we want To unify the hcfs test
> profile. So how can hcfs implementations use current hadoop-common
EMR's S3 does extra things, which is why netflix used injection tricks to
add theirs on top.
For blobstores, key use cases are
1. -general source of low-rate-of-change artifacts
2. -input for analysis jobs
3. -output from them
4. -chained operations
5. storage of data to outlive th
Thanks Colin: that's a good example of why we want To unify the hcfs test
profile. So how can hcfs implementations use current hadoop-common tests?
In mind there are three ways.
- one solution is to manually cobble together and copy tests , running them one
by one and seeing which ones apply
NetFlix's Apache-licensed S3mper system provides consistency for an
S3-backed store.
http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html
It would be nice to see this or something like it integrated with
Hadoop. I fear that a lot of applications are not ready for eventual
consiste
do you consider that native S3 FS a real "reference implementation" for
blob stores? or just something that , by mere chance, we are able to use as
a ref. impl.
On 6 March 2014 16:37, Jay Vyas wrote:
> As part of HADOOP-9361, im visioning this.
>
> 1) - We create In Memory FS implementation of different Reference
> FileSystems, each of which specifies appropriate tests , and passes those
> tests , i.e.
>
>InMemStrictlyConsistentFS (i.e. hdfs)
>
HDFS
