I was more concerned that the input to our input is from SQL databases and a proprietary EMC document store. And that our output is to a different SQL database.
I don't want to use any sort of file system at all. Adam Retter Software Developer Landmark Information Group T: 01392 685403 (x5403) 5-7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY www.landmark.co.uk -----Original Message----- From: Chuck Lam [mailto:[email protected]] Sent: 28 April 2009 20:25 To: [email protected] Subject: Re: Appropriate for Hadoop? HDFS is designed with Hadoop in mind, so there are certain advantages (e.g. performance, reliability, and ease of use) to using HDFS for Hadoop. However, it's not required. For example, when you run Hadoop in standalone mode, it just uses the file system on your local machine. When you run it on Amazon AWS, it can use S3 as a file system. On Tue, Apr 28, 2009 at 6:15 AM, Adam Retter <[email protected]>wrote: > > > Each document processing is independent and can be processed > > parallelly, so that part could be done in a map reduce job. > > Now whether it suits this use case depends on rate at which new > > URI's are discovered for processing and acceptable delay in processing > > of a document. The way I see it you can batch the URI's > > and input that to mapreduce job. Each mapper can work on sublist of > URIs. > > You can choose to make DB inserts from mapper itself. In that case > > you can set no of reducers to 0. Otherwise if batching of the queries > > is an option then you can consider making batch inserts in reducer. It > > will help in reducing load on DB. > > So I don't have to use HDFS at all when using Hadoop? > > > > Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY > Registered Number 2892803 Registered in England and Wales > > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > > The information contained in this e-mail is confidential and may be subject > to > legal privilege. If you are not the intended recipient, you must not use, > copy, > distribute or disclose the e-mail or any part of its contents or take any > action in reliance on it. If you have received this e-mail in error, please > e-mail the sender by replying to this message. All reasonable precautions > have > been taken to ensure no viruses are present in this e-mail. Landmark > Information > Group Limited cannot accept responsibility for loss or damage arising from > the > use of this e-mail or attachments and recommend that you subject these to > your virus checking procedures prior to use. > > Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY Registered Number 2892803 Registered in England and Wales This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email The information contained in this e-mail is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not use, copy, distribute or disclose the e-mail or any part of its contents or take any action in reliance on it. If you have received this e-mail in error, please e-mail the sender by replying to this message. All reasonable precautions have been taken to ensure no viruses are present in this e-mail. Landmark Information Group Limited cannot accept responsibility for loss or damage arising from the use of this e-mail or attachments and recommend that you subject these to your virus checking procedures prior to use.
