Dear Andy, > From: Andrew Purtell <apurt...@apache.org> > ...
> Do you intend this to be a joint project with the Hadoop community or > a technology competitor? As I had said in my email, we are looking for folks to colloborate with us to help get us integrated with Hadoop. So, to be explicitly clear, we are intending for this to be a joint project with the community. > Regrettably, KFS is not a "drop in replacement" for HDFS. > Hypothetically: I have several petabytes of data in an existing HDFS > deployment, which is the norm, and a continuous MapReduce workflow. > How do you propose I, practically, migrate to something like Sailfish > without a major capital expenditure and/or downtime and/or data loss? Well, we are not asking for KFS to replace HDFS. One path you could take is to experiment with Sailfish---use KFS just for the intermediate data and HDFS for everything else. There is no major capex :). While you get comfy with pushing intermediate data into a DFS, we get the ideas added to HDFS. This simplifies deployment considerations. > However, can the Sailfish I-files implementation be plugged in as an > alternate Shuffle implementation in MRv2 (see MAPREDUCE-3060 and > MAPREDUCE-4049), This'd be great! > with necessary additional plumbing for dynamic > adjustment of reduce task population? And the workbuilder could be > part of an alternate MapReduce Application Manager? It should be part of the AM. (Currently, with our implementation in Hadoop-0.20.2, the workbuilder serves the role of an AM). > The I-file concept could possibly be implemented here in a fairly self > contained way. One > could even colocate/embed a KFS filesystem with such an alternate > shuffle, like how MR task temporary space is usually colocated with > HDFS storage. Exactly. > Does this seem reasonable in any way? Great. Where do go from here? How do we get a colloborative effort going? Best, Sriram >> From: Sriram Rao <srirams...@gmail.com> >> To: common-dev@hadoop.apache.org >> Sent: Tuesday, May 8, 2012 10:32 AM >> Subject: Project announcement: Sailfish (also, looking for colloborators) >> >> Hi, >> >> I'd like to announce the release of a new open source project, Sailfish. >> >> http://code.google.com/p/sailfish/ >> >> Sailfish tries to improve Hadoop-performance, particularly for large-jobs >> which process TB's of data and run for hours. In building Sailfish, we >> modify how map-output is handled and transported from map->reduce. >> >> The project pages provide more information about the project. >> >> We are looking for colloborators who can help get some of the ideas into >> Apache Hadoop. A possible step forward could be to make "shuffle" phase of >> Hadoop pluggable. >> >> If you are interested in working with us, please get in touch with me. >> >> Sriram > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
