Jeff, ok, that makes more sense, I was under the mis-impression that it was creating and destroying mappers for each input record. I dont know why I had that in my head. My design suddenly became a lot clearer, and this provides a much more clean abstraction. Thanks for your help!
Josh Patterson TVA -----Original Message----- From: Jeff Eastman [mailto:j...@windwardsolutions.com] Sent: Tue 03/17/2009 6:02 PM To: core-user@hadoop.apache.org Subject: Re: RecordReader design heuristic Hi Josh, Well, I don't really see how you will get more mappers, just simpler logic in the mapper. The number of mappers is driven by how many input files you have and their sizes and not by any chunking you do in the record reader. Each record reader will get an entire split and will feed it to its mapper in a stream one record at a time. You can duplicate some of that logic in the mapper if you want but you already will have it in the reader so why bother? Jeff Patterson, Josh wrote: > Jeff, > So if I'm hearing you right, its "good" to send one point of data (10 > bytes here) to a single mapper? This mind set increases the number of > mappers, but keeps their logic scaled down to simply "look at this > record and emit/don't emit" --- which is considered more favorable? I'm > still getting the hang of the MR design tradeoffs, thanks for your > feedback. > > Josh Patterson > TVA > > -----Original Message----- > From: Jeff Eastman [mailto:j...@windwardsolutions.com] > Sent: Tuesday, March 17, 2009 5:11 PM > To: core-user@hadoop.apache.org > Subject: Re: RecordReader design heuristic > > If you send a single point to the mapper, your mapper logic will be > clean and simple. Otherwise you will need to loop over your block of > points in the mapper. In Mahout clustering, I send the mapper individual > > points because the input file is point-per-line. In either case, the > record reader will be iterating over a block of data to provide mapper > inputs. IIRC, splits will generally be an HDFS block or less, so if you > have files smaller than that you will get one mapper per. For larger > files you can get up to one mapper per split block. > > Jeff > > Patterson, Josh wrote: > >> I am currently working on a RecordReader to read a custom time series >> data binary file format and was wondering about ways to be most >> efficient in designing the InputFormat/RecordReader process. Reading >> through: >> >> http://wiki.apache.org/hadoop/HadoopMapReduce >> <http://wiki.apache.org/hadoop/HadoopMapReduce> >> >> gave me a lot of hints about how the various classes work together in >> order to read any type of file. I was looking at how the >> > TextInputFormat > >> uses the LineRecordReader in order to send individual lines to each >> mapper. My question is, what is a good heuristic in how to choose how >> much data to send to each mapper? With the stock LineRecordReader each >> mapper only gets to work with a single line which leads me to believe >> that we want to give each mapper very little work. Currently I'm >> > looking > >> at either sending each mapper a single point of data (10 bytes), which >> seems small, or sending a single mapper a block of data (around 819 >> points, at 10 bytes each, ---> 8190 bytes). I'm leaning towards >> > sending > >> the block to the mapper. >> >> These factors are based around dealing with a legacy file format (for >> now) so I'm just trying to make the best tradeoff possible for the >> > short > >> term until I get some basic stuff rolling, at which point I can >> > suggest > >> a better storage format, or just start converting the groups of stored >> points into a format more fitting for the platform. I understand that >> the InputFormat is not really trying to make much meaning out of the >> data, other than to help assist in getting the correct data out of the >> file based on the file split variables. Another question I have is, >> > with > >> a pretty much stock install, generally how big is each FileSplit? >> >> Josh Patterson >> TVA >> >> >> > > > >
