Hadoop-Common-trunk - Build # 566 - Failure

2011-01-04 Thread Apache Hudson Server
See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/566/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 20812 lines...] init: [touch] Creating /tmp/n

Re: Hadoop use direct I/O in Linux?

2011-01-04 Thread Christopher Smith
On Mon, Jan 3, 2011 at 7:15 PM, Brian Bockelman wrote: > The I/O pattern isn't truly random. To convert from physicist terms to CS > terms, the application is iterating through the rows of a column-oriented > store, reading out somewhere between 1 and 10% of the columns. The twist is > that the

RE: Hadoop use direct I/O in Linux?

2011-01-04 Thread Segel, Mike
All, While this is an interesting topic for debate, I think it's a moot point. A lot of DBAs (Especially Informix DBAs) don't agree with Linus. (I'm referring to an earlier post in this thread that referenced a quote from Linus T.) Direct I/O is a good thing. But if Linus is removing it from Li

Re: Hadoop use direct I/O in Linux?

2011-01-04 Thread Da Zheng
The most important reason for me to use direct I/O is that the Atom processor is too weak. If I wrote a simple program to write data to the disk, CPU is almost 100% but the disk hasn't reached its maximal bandwidth. When I write data to SSD, the difference is even larger. Even if the program ha

[jira] Resolved: (HADOOP-938) too many SequenceFile.createWriter() methods

2011-01-04 Thread Tom White (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-938. -- Resolution: Duplicate Fixed in HADOOP-6856. > too many SequenceFile.createWriter() methods > -

Review Request: HADOOP-7082. Configuration shouldn't hold its lock while outputting in writeXml

2011-01-04 Thread Todd Lipcon
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/207/ --- Review request for hadoop-common and Dhruba Borthakur. Summary --- See JIRA.

Re: Hadoop use direct I/O in Linux?

2011-01-04 Thread Christopher Smith
If you use direct I/O to reduce CPU time, that means you are saving CPU via DMA. If you are using Java's heap though, you can kiss that goodbye. That said, I'm surprised that the Atom can't keep up with magnetic disk unless you have a striped array. 100MB/s shouldn't be too taxing. Is it possible

[jira] Created: (HADOOP-7085) Cluster setup docs link to the wrong default config files

2011-01-04 Thread Eli Collins (JIRA)
Cluster setup docs link to the wrong default config files - Key: HADOOP-7085 URL: https://issues.apache.org/jira/browse/HADOOP-7085 Project: Hadoop Common Issue Type: Bug Comp

Re: Hadoop use direct I/O in Linux?

2011-01-04 Thread Da Zheng
On 1/4/11 5:17 PM, Christopher Smith wrote: > If you use direct I/O to reduce CPU time, that means you are saving CPU via > DMA. If you are using Java's heap though, you can kiss that goodbye. The buffer for direct I/O cannot be allocated from Java's heap anyway, I don't understand what you mean? >

Re: Hadoop use direct I/O in Linux?

2011-01-04 Thread Christopher Smith
On Tue, Jan 4, 2011 at 9:11 PM, Da Zheng wrote: > On 1/4/11 5:17 PM, Christopher Smith wrote: > > If you use direct I/O to reduce CPU time, that means you are saving CPU > via > > DMA. If you are using Java's heap though, you can kiss that goodbye. > The buffer for direct I/O cannot be allocated