I was thinking to test YARN-hosted apps like MapReduce, we need to see how they handle filesystems with different consistency/atomicity models, and YARN -even MiniYARNCluster -forks things off.
If the MR commit logic is isolated, that could be tested in the JUnit JVM. But for other applications -example: Tez, its probably too complex to mock On 6 March 2014 16:17, Jay Vyas <jayunit...@gmail.com> wrote: > steve you mentioned: > > >> but to test YARN it has to be visible across processes. > > What do you mean by "test yarn"? I think for the FileSystem APIs unit > testing, we dont care about YARN, do we? > > > > > > On Thu, Mar 6, 2014 at 6:02 AM, Steve Loughran <ste...@hortonworks.com > >wrote: > > > On 5 March 2014 19:07, Jay Vyas <jayunit...@gmail.com> wrote: > > > > > Hi HCFS Community :) > > > > > > This is Jay... Some of you know me.... I hack on a broad range of file > > > system and hadoop ecosystem interoperability stuff. I just wanted to > > > introduce myself and let you folks know im going to be working to help > > > clean up the existing unit testing frameworks for the FileSystem and > > > FileContext APIs. I've listed some bullets below . > > > > > > - byte code inspection based code coverage for file system APIs with a > > tool > > > such as corbertura. > > > > > > - HADOOP-9361 points out that there are many different types of file > > > systems. > > > > > > > > It adds a lot more structure to the tests with an XML declaration of each > > FS (in the -test) JAR. > > > > It's pretty much complete except for some discrepancies between file:// > and > > hdfs that I need to fix in file: > > -handling of mkdirs if the destination exists and is a file (currently: > > returns 0) > > -seek() on a closed stream. Currently appears to work, at least on OS/X. > > > > > > > - Creating mock file systems which can be used to validate API tests, > > which > > > emulate different FS semantics (atomic directory creation, eventual > > > consistency, strict consistency, POSIX compliance, append support, > > etc...) > > > > > > > That's an interesting thought, adding some inconsistency semantics on top > > of an existing FS to emulate blobstore > > behaviour. How would you do this? A in-memory RAM FS could do some of > this, > > but to test YARN it has to be visible across processes. > > We'd really need an in-ram simulation of semantics that also offered an > RPC > > API of some form. > > > > > > > > > > > > Is anyone interested in the above issues or have any opinions on how / > > > where i should get started? > > > > > > Our end goal is to have a more transparent and portable set of test > APIs > > > for the hadoop file system implementors, across the board : so that we > > can > > > all test our individual implementations confidently. > > > > > > So, anywhere i can lend a hand - let me know. I think this effort will > > > require all of us in the file system community to join forces, and it > > will > > > benefit us all immensly in the long run as well. > > > > > > > > I should do another '9361 patch, once I get those final quirks in file:// > > sorted out so that it is consistent with HDFS. > > 1. HDFS is and continues to be, the definition of the semantics of all > > filesystem interfaces. > > 2. It'd be good if we understood more about what accidental features of > the > > FS code depends on. e.g. does anything rely on mkdirs() being atomic? Of > > 0x00 being a valid char in a filename? How do programs fail when > blocksize > > is too small (try setting it to 1 and see how pig reacts)? How much code > > depends on close() being near-instantaneous and never failing? Blobstores > > do their write then, and can break both these requirements -which is > > something a mock FS could add atop file: > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > > > -- > Jay Vyas > http://jayunit100.blogspot.com > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
