> A key innovation here is a partitioning layout algorithm that can support >> fast >> many to many recovery similar to HDFS but still support partitioned >> operation >> with deterministic key placement. >> > > Thanks for your contribution. > > Is here more detail info on this point? >
yes... our design document: http://peregrine_mapreduce.bitbucket.org/design/ I actually will probably write a paper on this... The more I started down the partitioned filesystem approach in terms of mapreduce the more I realized that there were some REALLY elegant imoplementation and design issues that I did not originally appreciate ... (so I partially got lucky). I think this approach could be generalized to work on normal map reduce jobs without much overhead. -- Founder/CEO Spinn3r.com <http://spinn3r.com/> Location: *San Francisco, CA* Skype: *burtonator* Skype-in: *(415) 871-0687*
