I would also like to point to work being done on PLFS-HDFS: http://institute.lanl.gov/isti/irhpit/presentations/PLFS-HDFS.pdf
This would be made much simpler by allowing appends. Checkpointing in MPI is a very common use-case, and after Hamster, PLFS-HDFS becomes an attractive way to do this. (Section 2 of the 2009 HotCloud paper by PDL: http://www.cs.cmu.edu/~svp/2009hotcloud-tablefs.pdf discusses the reasons for seeking commonalities between HPC and DISC file systems.) - Milind On 3/21/12 12:48 PM, "Bhandarkar, Milind" <milind.bhandar...@emc.com> wrote: >Eli, > >To clarify a little bit, I think HDFS-3120 is the right thing to do, to >disable appends, while still enabling hsync in branch-1. > >But, going forward, (say 0.23+) having appends working correctly will >definitely add value, and make HDFS more palatable for lots of other >workloads. > >Of course, I have a vested interest in this, because our team is working >on a project that requires append and truncate, and we will be testing it >thoroughly at scale in Q2 this year. Would it be okay to wait for the >results of this testing ? > >Thanks, > >- milind > >--- >Milind Bhandarkar >Greenplum Labs, EMC >(Disclaimer: Opinions expressed in this email are those of the author, and >do not necessarily represent the views of any organization, past or >present, the author might be affiliated with.) > >
