Re: Beware sun's jvm version 1.6.0_05-b13 on linux
Allen Wittenauer wrote: On 5/15/09 11:38 AM, "Owen O'Malley" wrote: We have observed that the default jvm on RedHat 5 I'm sure some people are scratching their heads at this. The default JVM on at least RHEL5u0/1 is a GCJ-based 1.4, clearly incapable of running Hadoop. We [and, really, this is my doing... ^.^ ] replace it with the JVM from the JPackage folks. So while this isn't the default JVM that comes from RHEL, the warning should still be heeded. Presumably its one of those hard-to-reproduce race conditions that only surfaces under load on a big cluster so is hard to replicate in a unit test, right?
Re: Is there any performance issue with Jrockit JVM for Hadoop
Grace wrote: To follow up this question, I have also asked help on Jrockit forum. They kindly offered some useful and detailed suggestions according to the JRA results. After updating the option list, the performance did become better to some extend. But it is still not comparable with the Sun JVM. Maybe, it is due to the use case with short duration and different implementation in JVM layer between Sun and Jrockit. I would like to be back to use Sun JVM currently. Thanks all for your time and help. what about flipping the switch that says "run tasks in the TT's own JVM?". That should handle startup costs, and reduce the memory footprint
JobInProgress and TaskInProgress
Hi, how do i get the job progress n task progress information programmaticaly at any point of time using the API's there is a JobInProgress and TaskInProgress classes... but both of them are private any suggestions? Thanks, Raakhi
Re: Is there any performance issue with Jrockit JVM for Hadoop
On Mon, May 18, 2009 at 11:44 AM, Steve Loughran wrote: > Grace wrote: >> >> To follow up this question, I have also asked help on Jrockit forum. They >> kindly offered some useful and detailed suggestions according to the JRA >> results. After updating the option list, the performance did become better >> to some extend. But it is still not comparable with the Sun JVM. Maybe, it >> is due to the use case with short duration and different implementation in >> JVM layer between Sun and Jrockit. I would like to be back to use Sun JVM >> currently. Thanks all for your time and help. >> > > what about flipping the switch that says "run tasks in the TT's own JVM?". > That should handle startup costs, and reduce the memory footprint > The property mapred.job.reuse.jvm.num.tasks allows you to set how many tasks the JVM may be reused for (within a job), but it always runs in a separate JVM to the tasktracker. (BTW https://issues.apache.org/jira/browse/HADOOP-3675has some discussion about running tasks in the tasktracker's JVM). Tom
RE: proper method for writing files to hdfs
Sasha,
Connecting to the namenode is the proper way to establish the hdfs
connection. Afterwards the Hadoop client handler that is called by your
code will go directly to the datanodes. There is no reason for you to
communicate directly with a datanode nor is there a way for you to even know
where the data nodes are located. That is all done by the Hadoop client code
and done silently under the covers by Hadoop itself.
Bill
-Original Message-
From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy
Sent: Sunday, May 17, 2009 10:55 AM
To: core-user@hadoop.apache.org
Subject: proper method for writing files to hdfs
The following graphic outlines the architecture for HDFS:
http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif
If one is to write a client that adds data into HDFS, it needs to add it
through the Data Node. Now, from the graphic I am to understand that the
client doesn't communicate with the NameNode, and only the Data Node.
In the examples I've seen and the playing I am doing, I am connecting to the
hdfs url as a configuration parameter before I create a file. Is this the
incorrect way to create files in HDFS?
Configuration config = new Configuration();
config.set("fs.default.name","hdfs://foo.bar.com:9000/");
String path = "/tmp/i/am/a/path/to/a/file.name"
Path hdfsPath = new Path(path);
FileSystem fileSystem = FileSystem.get(config);
FSDataOutputStream os = fileSystem.create(hdfsPath, false);
os.write("something".getBytes());
os.close();
Should the client be connecting to a data node to create the file as
indicated in the graphic above?
If connecting to a data node is possible and suggested, where can I find
more details about this process?
Thanks in advance,
-sasha
--
Sasha Dolgy
sasha.do...@gmail.com
Re: proper method for writing files to hdfs
Hi Bill,
Thanks for that. If the NameNode is unavailable, how do we find the
secondary name node? Is there a way to deal with this in the code or
should a load balancer of some type sit above each and only direct
traffic to the name node if its listening?
-sd
On Mon, May 18, 2009 at 2:09 PM, Bill Habermaas wrote:
> Sasha,
>
> Connecting to the namenode is the proper way to establish the hdfs
> connection. Afterwards the Hadoop client handler that is called by your
> code will go directly to the datanodes. There is no reason for you to
> communicate directly with a datanode nor is there a way for you to even know
> where the data nodes are located. That is all done by the Hadoop client code
> and done silently under the covers by Hadoop itself.
>
> Bill
>
> -Original Message-
> From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy
> Sent: Sunday, May 17, 2009 10:55 AM
> To: core-user@hadoop.apache.org
> Subject: proper method for writing files to hdfs
>
> The following graphic outlines the architecture for HDFS:
> http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif
>
> If one is to write a client that adds data into HDFS, it needs to add it
> through the Data Node. Now, from the graphic I am to understand that the
> client doesn't communicate with the NameNode, and only the Data Node.
>
> In the examples I've seen and the playing I am doing, I am connecting to the
> hdfs url as a configuration parameter before I create a file. Is this the
> incorrect way to create files in HDFS?
>
> Configuration config = new Configuration();
> config.set("fs.default.name","hdfs://foo.bar.com:9000/");
> String path = "/tmp/i/am/a/path/to/a/file.name"
> Path hdfsPath = new Path(path);
> FileSystem fileSystem = FileSystem.get(config);
> FSDataOutputStream os = fileSystem.create(hdfsPath, false);
> os.write("something".getBytes());
> os.close();
>
> Should the client be connecting to a data node to create the file as
> indicated in the graphic above?
>
> If connecting to a data node is possible and suggested, where can I find
> more details about this process?
>
> Thanks in advance,
> -sasha
>
> --
> Sasha Dolgy
> sasha.do...@gmail.com
>
>
>
--
Sasha Dolgy
sasha.do...@gmail.com
Re: Is there any performance issue with Jrockit JVM for Hadoop
Tom White wrote: On Mon, May 18, 2009 at 11:44 AM, Steve Loughran wrote: Grace wrote: To follow up this question, I have also asked help on Jrockit forum. They kindly offered some useful and detailed suggestions according to the JRA results. After updating the option list, the performance did become better to some extend. But it is still not comparable with the Sun JVM. Maybe, it is due to the use case with short duration and different implementation in JVM layer between Sun and Jrockit. I would like to be back to use Sun JVM currently. Thanks all for your time and help. what about flipping the switch that says "run tasks in the TT's own JVM?". That should handle startup costs, and reduce the memory footprint The property mapred.job.reuse.jvm.num.tasks allows you to set how many tasks the JVM may be reused for (within a job), but it always runs in a separate JVM to the tasktracker. (BTW https://issues.apache.org/jira/browse/HADOOP-3675has some discussion about running tasks in the tasktracker's JVM). Tom Tom, that's why you are writing a book on Hadoop and I'm not ...you know the answers and I have some vague misunderstandings, -steve (returning to the svn book)
RE: proper method for writing files to hdfs
Sasha,
If the namenode is unavailable then you cannot communicate with Hadoop. It
is the single point of failure and once it is down then the system is
unusable. The secondary name node is not a failover substitute for the name
node. The name is misleading. It's purpose is simply to checkpoint the
namenode's data so you can recover from a namenode failure that has
corrupted data.
Bill
-Original Message-
From: Sasha Dolgy [mailto:sdo...@gmail.com]
Sent: Monday, May 18, 2009 9:34 AM
To: core-user@hadoop.apache.org
Subject: Re: proper method for writing files to hdfs
Hi Bill,
Thanks for that. If the NameNode is unavailable, how do we find the
secondary name node? Is there a way to deal with this in the code or
should a load balancer of some type sit above each and only direct
traffic to the name node if its listening?
-sd
On Mon, May 18, 2009 at 2:09 PM, Bill Habermaas wrote:
> Sasha,
>
> Connecting to the namenode is the proper way to establish the hdfs
> connection. Afterwards the Hadoop client handler that is called by your
> code will go directly to the datanodes. There is no reason for you to
> communicate directly with a datanode nor is there a way for you to even
know
> where the data nodes are located. That is all done by the Hadoop client
code
> and done silently under the covers by Hadoop itself.
>
> Bill
>
> -Original Message-
> From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy
> Sent: Sunday, May 17, 2009 10:55 AM
> To: core-user@hadoop.apache.org
> Subject: proper method for writing files to hdfs
>
> The following graphic outlines the architecture for HDFS:
> http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif
>
> If one is to write a client that adds data into HDFS, it needs to add it
> through the Data Node. Now, from the graphic I am to understand that the
> client doesn't communicate with the NameNode, and only the Data Node.
>
> In the examples I've seen and the playing I am doing, I am connecting to
the
> hdfs url as a configuration parameter before I create a file. Is this the
> incorrect way to create files in HDFS?
>
> Configuration config = new Configuration();
> config.set("fs.default.name","hdfs://foo.bar.com:9000/");
> String path = "/tmp/i/am/a/path/to/a/file.name"
> Path hdfsPath = new Path(path);
> FileSystem fileSystem = FileSystem.get(config);
> FSDataOutputStream os = fileSystem.create(hdfsPath, false);
> os.write("something".getBytes());
> os.close();
>
> Should the client be connecting to a data node to create the file as
> indicated in the graphic above?
>
> If connecting to a data node is possible and suggested, where can I find
> more details about this process?
>
> Thanks in advance,
> -sasha
>
> --
> Sasha Dolgy
> sasha.do...@gmail.com
>
>
>
--
Sasha Dolgy
sasha.do...@gmail.com
Re: proper method for writing files to hdfs
Ok, on the same page with that.
Going back to the original question. In our scenario we are trying to
stream data into HDFS and despite the posts and hints I've been
reading, it's still tough to crack this nut and this is why I thought
(and thankfully I wasn't right) that we were going about this the
wrong way:
We open up a new file and get the FSDataOutputStream and start to
write data and flush as concurrent information comes in:
2009-05-17 06:16:50,921 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2451 to 20
48 meta file offset to 23
2009-05-17 06:16:50,921 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0
adding seqno 3 to ack queue.
2009-05-17 06:16:50,921 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_5834867413110307425_1064 acking for packet 3
2009-05-17 06:16:51,111 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet
for block blk_5834867413110307425_1064 of length 735 seqno
4 offsetInBlock 2048 lastPacketInBlock false
2009-05-17 06:16:51,111 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2518 to 20
48 meta file offset to 23
2009-05-17 06:16:51,111 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0
adding seqno 4 to ack queue.
2009-05-17 06:16:51,112 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_5834867413110307425_1064 acking for packet 4
2009-05-17 06:16:51,297 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet
for block blk_5834867413110307425_1064 of length 509 seqno
5 offsetInBlock 2560 lastPacketInBlock false
2009-05-17 06:16:51,297 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2771 to 25
60 meta file offset to 27
The file gets bigger and bigger, but it is not commited to hdfs until
we close() the stream. We've waited for the block size to go above
64k and even higher, and it never writes itself out to hdfs. I've
seen the JIRA bug reports, etc.
Has no one done this? Is it bad to stream data into it? How do I
force it to flush the data to disk...
The POC is with environmental data every moment from multiple sources
for monitoring temperature in computers / facilities...
Suppose I'm just a little frustrated. I see that hadoop is brilliant
for large sets of data that you already have or are happy to move onto
HDFS ...
-sd
> On Mon, May 18, 2009 at 2:09 PM, Bill Habermaas wrote:
>> Sasha,
>>
>> Connecting to the namenode is the proper way to establish the hdfs
>> connection. Afterwards the Hadoop client handler that is called by your
>> code will go directly to the datanodes. There is no reason for you to
>> communicate directly with a datanode nor is there a way for you to even
> know
>> where the data nodes are located. That is all done by the Hadoop client
> code
>> and done silently under the covers by Hadoop itself.
>>
>> Bill
>>
>> -Original Message-
>> From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy
>> Sent: Sunday, May 17, 2009 10:55 AM
>> To: core-user@hadoop.apache.org
>> Subject: proper method for writing files to hdfs
>>
>> The following graphic outlines the architecture for HDFS:
>> http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif
>>
>> If one is to write a client that adds data into HDFS, it needs to add it
>> through the Data Node. Now, from the graphic I am to understand that the
>> client doesn't communicate with the NameNode, and only the Data Node.
>>
>> In the examples I've seen and the playing I am doing, I am connecting to
> the
>> hdfs url as a configuration parameter before I create a file. Is this the
>> incorrect way to create files in HDFS?
>>
>> Configuration config = new Configuration();
>> config.set("fs.default.name","hdfs://foo.bar.com:9000/");
>> String path = "/tmp/i/am/a/path/to/a/file.name"
>> Path hdfsPath = new Path(path);
>> FileSystem fileSystem = FileSystem.get(config);
>> FSDataOutputStream os = fileSystem.create(hdfsPath, false);
>> os.write("something".getBytes());
>> os.close();
>>
>> Should the client be connecting to a data node to create the file as
>> indicated in the graphic above?
>>
>> If connecting to a data node is possible and suggested, where can I find
>> more details about this process?
>>
>> Thanks in advance,
>> -sasha
>>
>> --
>> Sasha Dolgy
>> sasha.do...@gmail.com
RE: proper method for writing files to hdfs
-Original Message-
From: Sasha Dolgy [mailto:sdo...@gmail.com]
Sent: Monday, May 18, 2009 9:50 AM
To: core-user@hadoop.apache.org
Subject: Re: proper method for writing files to hdfs
Ok, on the same page with that.
Going back to the original question. In our scenario we are trying to
stream data into HDFS and despite the posts and hints I've been
reading, it's still tough to crack this nut and this is why I thought
(and thankfully I wasn't right) that we were going about this the
wrong way:
We open up a new file and get the FSDataOutputStream and start to
write data and flush as concurrent information comes in:
2009-05-17 06:16:50,921 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2451 to 20
48 meta file offset to 23
2009-05-17 06:16:50,921 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0
adding seqno 3 to ack queue.
2009-05-17 06:16:50,921 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_5834867413110307425_1064 acking for packet 3
2009-05-17 06:16:51,111 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet
for block blk_5834867413110307425_1064 of length 735 seqno
4 offsetInBlock 2048 lastPacketInBlock false
2009-05-17 06:16:51,111 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2518 to 20
48 meta file offset to 23
2009-05-17 06:16:51,111 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0
adding seqno 4 to ack queue.
2009-05-17 06:16:51,112 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for
block blk_5834867413110307425_1064 acking for packet 4
2009-05-17 06:16:51,297 DEBUG
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet
for block blk_5834867413110307425_1064 of length 509 seqno
5 offsetInBlock 2560 lastPacketInBlock false
2009-05-17 06:16:51,297 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file
offset of block blk_5834867413110307425_1064 from 2771 to 25
60 meta file offset to 27
The file gets bigger and bigger, but it is not commited to hdfs until
we close() the stream. We've waited for the block size to go above
64k and even higher, and it never writes itself out to hdfs. I've
seen the JIRA bug reports, etc.
Has no one done this? Is it bad to stream data into it? How do I
force it to flush the data to disk...
The POC is with environmental data every moment from multiple sources
for monitoring temperature in computers / facilities...
Suppose I'm just a little frustrated. I see that hadoop is brilliant
for large sets of data that you already have or are happy to move onto
HDFS ...
-sd
> On Mon, May 18, 2009 at 2:09 PM, Bill Habermaas wrote:
>> Sasha,
>>
>> Connecting to the namenode is the proper way to establish the hdfs
>> connection. Afterwards the Hadoop client handler that is called by your
>> code will go directly to the datanodes. There is no reason for you to
>> communicate directly with a datanode nor is there a way for you to even
> know
>> where the data nodes are located. That is all done by the Hadoop client
> code
>> and done silently under the covers by Hadoop itself.
>>
>> Bill
>>
>> -Original Message-
>> From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy
>> Sent: Sunday, May 17, 2009 10:55 AM
>> To: core-user@hadoop.apache.org
>> Subject: proper method for writing files to hdfs
>>
>> The following graphic outlines the architecture for HDFS:
>> http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif
>>
>> If one is to write a client that adds data into HDFS, it needs to add it
>> through the Data Node. Now, from the graphic I am to understand that the
>> client doesn't communicate with the NameNode, and only the Data Node.
>>
>> In the examples I've seen and the playing I am doing, I am connecting to
> the
>> hdfs url as a configuration parameter before I create a file. Is this the
>> incorrect way to create files in HDFS?
>>
>> Configuration config = new Configuration();
>> config.set("fs.default.name","hdfs://foo.bar.com:9000/");
>> String path = "/tmp/i/am/a/path/to/a/file.name"
>> Path hdfsPath = new Path(path);
>> FileSystem fileSystem = FileSystem.get(config);
>> FSDataOutputStream os = fileSystem.create(hdfsPath, false);
>> os.write("something".getBytes());
>> os.close();
>>
>> Should the client be connecting to a data node to create the file as
>> indicated in the graphic above?
>>
>> If connecting to a data node is possible and suggested, where can I find
>> more details about this process?
>>
>> Thanks in advance,
>> -sasha
>>
>> --
>> Sasha Dolgy
>> sasha.do...@gmail.com
Re: FSDataOutputStream flush() not working?
Hi Jason,
If the bufferSize is set when the stream is created, when the size is
reached, will it automatically write itself out to HDFS? What happens
when the buffer size is exceeded?
-sasha
On Mon, May 18, 2009 at 3:04 AM, jason hadoop wrote:
> When you open a file you have the option, blockSize
> /**
> * Opens an FSDataOutputStream at the indicated Path with write-progress
> * reporting.
> * @param f the file name to open
> * @param permission
> * @param overwrite if a file with this name already exists, then if true,
> * the file will be overwritten, and if false an error will be thrown.
> * @param bufferSize the size of the buffer to be used.
> * @param replication required block replication for the file.
> * @param blockSize
> * @param progress
> * @throws IOException
> * @see #setPermission(Path, FsPermission)
> */
> public abstract FSDataOutputStream create(Path f,
> FsPermission permission,
> boolean overwrite,
> int bufferSize,
> short replication,
> long blockSize,
> Progressable progress) throws IOException;
>
> On Fri, May 15, 2009 at 12:44 PM, Sasha Dolgy wrote:
>
>> Hi Todd,
>> Reading through the JIRA, my impression is that data will be written out to
>> hdfs only once it has reached a certain size in the buffer. Is it possible
>> to define the size of that buffer? Or is this a future enhancement?
>>
>> -sasha
>>
>> On Fri, May 15, 2009 at 6:14 PM, Todd Lipcon wrote:
>>
>> > Hi Sasha,
>> >
>> > What version are you running? Up until very recent versions, sync() was
>> not
>> > implemented. Even in the newest releases, sync isn't completely finished,
>> > and you may find unreliable behavior.
>> >
>> > For now, if you need this kind of behavior, your best bet is to close
>> each
>> > file and then open the next every N minutes. For example, if you're
>> > processing logs every 5 minutes, simply close log file log.00223 and
>> round
>> > robin to log.00224 right before you need the data to be available to
>> > readers. If you're collecting data at a low rate, these files may end up
>> > being rather small, and you should probably look into doing merges on the
>> > hour/day/etc to avoid small-file proliferation.
>> >
>> > If you want to track the work being done around append and sync, check
>> out
>> > HADOOP-5744 and the issues referenced therein:
>> >
>> > http://issues.apache.org/jira/browse/HADOOP-5744
>> >
>> > Hope that helps,
>> > -Todd
>> >
>> > On Fri, May 15, 2009 at 6:35 AM, Sasha Dolgy wrote:
>> >
>> > > Hi there, forgive the repost:
>> > >
>> > > Right now data is received in parallel and is written to a queue, then
>> a
>> > > single thread reads the queue and writes those messages to a
>> > > FSDataOutputStream which is kept open, but the messages never get
>> > flushed.
>> > > Tried flush() and sync() with no joy.
>> > >
>> > > 1.
>> > > outputStream.writeBytes(rawMessage.toString());
>> > >
>> > > 2.
>> > >
>> > > log.debug("Flushing stream, size = " + s.getOutputStream().size());
>> > > s.getOutputStream().sync();
>> > > log.debug("Flushed stream, size = " + s.getOutputStream().size());
>> > >
>> > > or
>> > >
>> > > log.debug("Flushing stream, size = " + s.getOutputStream().size());
>> > > s.getOutputStream().flush();
>> > > log.debug("Flushed stream, size = " + s.getOutputStream().size());
>> > >
>> > > The size() remains the same after performing this action.
>> > >
>> > > 2009-05-12 12:42:17,470 DEBUG [Thread-7] (FSStreamManager.java:28)
>> > > hdfs.HdfsQueueConsumer: Thread 19 getting an output stream
>> > > 2009-05-12 12:42:17,470 DEBUG [Thread-7] (FSStreamManager.java:49)
>> > > hdfs.HdfsQueueConsumer: Re-using existing stream
>> > > 2009-05-12 12:42:17,472 DEBUG [Thread-7] (FSStreamManager.java:63)
>> > > hdfs.HdfsQueueConsumer: Flushing stream, size = 1986
>> > > 2009-05-12 12:42:17,472 DEBUG [Thread-7] (DFSClient.java:3013)
>> > > hdfs.DFSClient: DFSClient flush() : saveOffset 1613 bytesCurBlock 1986
>> > > lastFlushOffset 1731
>> > > 2009-05-12 12:42:17,472 DEBUG [Thread-7] (FSStreamManager.java:66)
>> > > hdfs.HdfsQueueConsumer: Flushed stream, size = 1986
>> > > 2009-05-12 12:42:19,586 DEBUG [Thread-7] (HdfsQueueConsumer.java:39)
>> > > hdfs.HdfsQueueConsumer: Consumer writing event
>> > > 2009-05-12 12:42:19,587 DEBUG [Thread-7] (FSStreamManager.java:28)
>> > > hdfs.HdfsQueueConsumer: Thread 19 getting an output stream
>> > > 2009-05-12 12:42:19,588 DEBUG [Thread-7] (FSStreamManager.java:49)
>> > > hdfs.HdfsQueueConsumer: Re-using existing stream
>> > > 2009-05-12 12:42:19,589 DEBUG [Thread-7] (FSStreamManager.java:63)
>> > > hdfs.HdfsQueueConsumer: Flushing stream, size = 2235
>> > > 2009-05-12 12:42:19,589 DEBUG [Thread-7] (DFSClient.java:3013)
>> > > hdfs.DFSClient: DFSClient flush() : saveOffset 2125 bytesCurBlock 2235
>> > > lastFlushOffset 1986
>> > > 2009-05-12 12:42:19,590 DEBUG [Thread-7] (FSStreamManager.java:66)
>> > > hdfs.HdfsQueueConsumer: Flushed stream, size = 2235
>> > >
>> >
RE: proper method for writing files to hdfs
Hadoop writes data to the local filesystem, when the blocksize is reached it is written into hdfs. Think of hdfs as a block management system rather than a file system even though the end result is a series of blocks that constitute a file. You will not see the data in hdfs until the file is closed - that is the reality of the implementation. The only way to 'flush' is what you have already discovered - closing the file. To flush the data often in hdfs can be a very expensive operation when you consider that it will affect multiple nodes distributed over a network. I suspect that is why it isn't there. I believe there is a jira somewhere to have 'sync' force the data out to disk but I do not know the number or what its status is. Assuming you are collecting data as an unending process, you might consider closing the hdfs output at periodic intervals and/or collecting data locally (with your intervening flushes) and then moving it into hdfs so it can get processed by map/reduce. It is a prudent approach to minimize potential data loss if the hdfs connection gets broken. Every implementation is different so you gotta be creative. :o) Bill -Original Message- From: Sasha Dolgy [mailto:sdo...@gmail.com] Sent: Monday, May 18, 2009 9:50 AM To: core-user@hadoop.apache.org Subject: Re: proper method for writing files to hdfs Ok, on the same page with that. Going back to the original question. In our scenario we are trying to stream data into HDFS and despite the posts and hints I've been reading, it's still tough to crack this nut and this is why I thought (and thankfully I wasn't right) that we were going about this the wrong way: We open up a new file and get the FSDataOutputStream and start to write data and flush as concurrent information comes in: 2009-05-17 06:16:50,921 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file offset of block blk_5834867413110307425_1064 from 2451 to 20 48 meta file offset to 23 2009-05-17 06:16:50,921 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 adding seqno 3 to ack queue. 2009-05-17 06:16:50,921 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_5834867413110307425_1064 acking for packet 3 2009-05-17 06:16:51,111 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet for block blk_5834867413110307425_1064 of length 735 seqno 4 offsetInBlock 2048 lastPacketInBlock false 2009-05-17 06:16:51,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file offset of block blk_5834867413110307425_1064 from 2518 to 20 48 meta file offset to 23 2009-05-17 06:16:51,111 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 adding seqno 4 to ack queue. 2009-05-17 06:16:51,112 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_5834867413110307425_1064 acking for packet 4 2009-05-17 06:16:51,297 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving one packet for block blk_5834867413110307425_1064 of length 509 seqno 5 offsetInBlock 2560 lastPacketInBlock false 2009-05-17 06:16:51,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block file offset of block blk_5834867413110307425_1064 from 2771 to 25 60 meta file offset to 27 The file gets bigger and bigger, but it is not commited to hdfs until we close() the stream. We've waited for the block size to go above 64k and even higher, and it never writes itself out to hdfs. I've seen the JIRA bug reports, etc. Has no one done this? Is it bad to stream data into it? How do I force it to flush the data to disk... The POC is with environmental data every moment from multiple sources for monitoring temperature in computers / facilities... Suppose I'm just a little frustrated. I see that hadoop is brilliant for large sets of data that you already have or are happy to move onto HDFS ... -sd > On Mon, May 18, 2009 at 2:09 PM, Bill Habermaas wrote: >> Sasha, >> >> Connecting to the namenode is the proper way to establish the hdfs >> connection. Afterwards the Hadoop client handler that is called by your >> code will go directly to the datanodes. There is no reason for you to >> communicate directly with a datanode nor is there a way for you to even > know >> where the data nodes are located. That is all done by the Hadoop client > code >> and done silently under the covers by Hadoop itself. >> >> Bill >> >> -Original Message- >> From: sdo...@gmail.com [mailto:sdo...@gmail.com] On Behalf Of Sasha Dolgy >> Sent: Sunday, May 17, 2009 10:55 AM >> To: core-user@hadoop.apache.org >> Subject: proper method for writing files to hdfs >> >> The following graphic outlines the architecture for HDFS: >> http://hadoop.apache.org/core/docs/current/images/hdfsarchitecture.gif >> >> If one is to write a client that adds data into HDFS, it needs to add it >> through the Data Node. Now, from the graphic I am to understand th
Optimal Filesystem (and Settings) for HDFS
We are currently rebuilding our cluster - has anybody recommendations on the underlaying file system? Just standard Ext3? I could imagine that the block size could be larger than its default... Thx for any tips, Bob
Re: Beware sun's jvm version 1.6.0_05-b13 on linux
On May 18, 2009, at 3:42 AM, Steve Loughran wrote: Presumably its one of those hard-to-reproduce race conditions that only surfaces under load on a big cluster so is hard to replicate in a unit test, right? Yes. It reliably happens on a 100TB or larger sort, but almost never happens on a small scale. -- Owen
Re: Optimal Filesystem (and Settings) for HDFS
I believe Yahoo! uses ext3, though I know other people have said that XFS has performed better in various benchmarks. We use ext3, though we haven't done any benchmarks to prove its worth. This question has come up a lot, so I think it'd be worth doing a benchmark and writing up the results. I haven't been able to find a detailed analysis / benchmark writeup comparing various filesystems, unfortunately. Hope this helps, Alex On Mon, May 18, 2009 at 8:54 AM, Bob Schulze wrote: > We are currently rebuilding our cluster - has anybody recommendations on > the underlaying file system? Just standard Ext3? > > I could imagine that the block size could be larger than its default... > > Thx for any tips, > >Bob > >
Re: Optimal Filesystem (and Settings) for HDFS
Do not forget 'tune2fs -m 2'. By default this value gets set at 5%. With 1 TB disks we got 33 GB more usable space. Talk about instant savings! On Mon, May 18, 2009 at 1:31 PM, Alex Loddengaard wrote: > I believe Yahoo! uses ext3, though I know other people have said that XFS > has performed better in various benchmarks. We use ext3, though we haven't > done any benchmarks to prove its worth. > > This question has come up a lot, so I think it'd be worth doing a benchmark > and writing up the results. I haven't been able to find a detailed analysis > / benchmark writeup comparing various filesystems, unfortunately. > > Hope this helps, > > Alex > > On Mon, May 18, 2009 at 8:54 AM, Bob Schulze wrote: > >> We are currently rebuilding our cluster - has anybody recommendations on >> the underlaying file system? Just standard Ext3? >> >> I could imagine that the block size could be larger than its default... >> >> Thx for any tips, >> >> Bob >> >> >
Re: Optimal Filesystem (and Settings) for HDFS
On 5/18/09 11:33 AM, "Edward Capriolo" wrote: > Do not forget 'tune2fs -m 2'. By default this value gets set at 5%. > With 1 TB disks we got 33 GB more usable space. Talk about instant > savings! Yup. Although, I think we're using -m 1. > On Mon, May 18, 2009 at 1:31 PM, Alex Loddengaard wrote: >> I believe Yahoo! uses ext3, Yup. They won't buy me enough battery backed RAM to use a memory file system. ;)
unexpected overloaded mapper number
Hei everyone, I've updated my hadoop to 0.20. however, when I run my old version mapreduce program, it will report me " Task Id : attempt_200905181657_0002_m_000102_0, Status : FAILED Error initializing attempt_200905181818_0001_m_000102_0: java.io.FileNotFoundException: File file:/hadoop/hadoop-root/mapred/system/job_200905181818_0001/job.xml does not exist." I monitored the whole process, the file "file:/hadoop/hadoop-root/mapred/system/job_200905181818_0001/job.xml" really exists when initializing other tasks, after the system tried "attempt_200905181818_0001_m_000102_0", it has been deleted. Actually, I set the "mapred.map.tasks" equal to 101 in both mapred-site.xml amd mapred-default.xml. Also the parameter " conf.setNumMapTasks(101)". But the hadoop framework also gives me " Task Id : attempt_200905181657_0002_m_000102_0, Status : FAILED". Obviously, it tries to create the 102th tasks, and the setting does not allow. Then it failed Any suggestions will be appreciated! Thanks -- Chen
Re: A brief report of Second Hadoop in China Salon
Cheers! Thank you all organizers and speakers! On Sun, May 17, 2009 at 6:49 PM, Qingyan(Evan) Liu wrote: > yes, that's a great conference. thanks a lot to the organizers and > reporters. > > 2009/5/16 He Yongqiang > > > Hi, all > > In May 9, we held the second Hadoop In China salon. About 150 people > > attended, 46% of them are engineers/managers from industry companies, and > > 38% of them are students/professors from universities and institutes. > This > > salon was successfully held with great technical support from Yahoo! > > Beijing > > R&D, Zheng Shao from Facebook Inc., Wang Shouyan from Baidu Inc. and many > > other high technology companies in China. We got over one hundred > feedbacks > > from attendees, and most of them are interested in details and wants more > > discussions. And 1/3 of them want we to include more topics or more > > sessions > > for hadoop subprojects. And most students/professors want to be more > > familiar with hadoop and try to find new research topic on top of hadoop. > > Most students want to involve themselves and contribute to hadoop, but do > > not know how or find it is a little difficulty because of language/zone > > problems. > > Thank you all the attendees again. Without you, it would never success. > > > > We already put the slides on site: www.hadooper.cn, and the videos are > > coming soon. > > > > BTW, I insist on letting this event to be nonprofit. In the past two > > meetings, we did not charge anyone for anything. > > > -- My research interests are distributed systems, parallel computing and bytecode based virtual machine. My profile: http://www.linkedin.com/in/coderplay My blog: http://coderplay.javaeye.com
problems running filebench
Hi, everyone, I’m trying to do some performance test on HDFS with filebench which comes with hadoop-0.19.0-test.jar. I successfully run the command with –r attribute (read testing) and get the output. (the command :hadoop jar hadoop-0.19.0-test.jar filebench -r -txt -blk -pln -dir "hdfs://master:54310/dir1") But failed to run with –w attribute (write testing). The command and error message are below, any suggestions will be appreciated. BTW, is there any document about hadoop-xxx-test.jar? It seems I can only get information from tracing the source code? Regards, Hsiao-Feng, Yeh === [had...@164-22 hadoop]$ hadoop jar hadoop-0.19.0-test.jar filebench -w -txt -blk -pln -dir "hdfs://master:54310/dir1" DIR: hdfs://master:54310/dir1 W TXT_PLN: java.lang.NullPointerException at org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:201) at org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:227) at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:115) at org.apache.hadoop.io.FileBench.writeBench(FileBench.java:113) at org.apache.hadoop.io.FileBench$RW$1.exec(FileBench.java:295) at org.apache.hadoop.io.FileBench.run(FileBench.java:235) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.io.FileBench.main(FileBench.java:155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141) at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:165) at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Re: JobInProgress and TaskInProgress
Could you let us know what information are you looking to extract from these classes? You possibly could get them from other classes. Jothi On 5/18/09 6:23 PM, "Rakhi Khatwani" wrote: > Hi, > how do i get the job progress n task progress information > programmaticaly at any point of time using the API's > there is a JobInProgress and TaskInProgress classes... but both of them are > private > > any suggestions? > > Thanks, > Raakhi
Re: sort example
On Sun, May 17, 2009 at 3:33 PM, Chuck Lam wrote: > The mapred.text.key.comparator.options property is active only if you use > the KeyFieldBasedComparator. > > -D > mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator > Thanks Chuck. That was it. > There's a TotalOrderPartitioner class for doing sorting. This is what the > TeraSort uses. I haven't really looked at that class from the Streaming > angle, but that's where I'd start. Thanks for pointing me in the right direction. I am going to definitely look into it. -drd > On Sun, May 17, 2009 at 8:52 AM, David Rio wrote: > >> I thought about that.. but there has to be a better way. >> And it seems to work just fine in the streaming docs. Particulary the >> IPs example. >> >> -drd >> >> On Sun, May 17, 2009 at 10:39 AM, Ricky Ho wrote: >> > Is this a workaround ? >> > >> > If you know the max size of your key, can you make all keys the same size >> by prepending them with zeros ... >> > >> > So ... >> > 1324 becomes 001324 >> > 212 becomes 000212 >> > 123123 >> > >> > After you do the sorting, trim out the preceding zeros ... >> > >> > Rgds, >> > Ricky >> > -Original Message- >> > From: David Rio [mailto:driodei...@gmail.com] >> > Sent: Sunday, May 17, 2009 8:34 AM >> > To: core-user@hadoop.apache.org >> > Subject: Re: sort example >> > >> > On Sun, May 17, 2009 at 10:18 AM, Ricky Ho wrote: >> >> >> >> I think using a single reducer causes the sorting to be done >> sequentially and hence defeats the purpose of using Hadoop in the first >> place. >> > >> > I agree, but this is just for testing. >> > Actually I used two reducers in my example. >> > >> >> Perhaps you can use a different "partitioner" which partitions the key >> range > into different subranges, with a different reducer work on each >> subrange. >> > >> > Yes, but prior to that, I want to make the basic numerical sorting >> > work. It seems my args do not get passed to the partitioner class for >> > some reason. >> > >> > -drd >> > >> >
Re: JobInProgress and TaskInProgress
Hi, I am looking for the following: for each task: % completed for both map n reduce, exceptions (if encountered). for each job: % completed, status (RUNNING,FAILED,PAUSED etc). i would wanna write a program so that i can programatically access the above information at any point of time. Thanks, Raakhi On Mon, May 18, 2009 at 7:46 PM, Jothi Padmanabhan wrote: > Could you let us know what information are you looking to extract from > these > classes? You possibly could get them from other classes. > > Jothi > > > On 5/18/09 6:23 PM, "Rakhi Khatwani" wrote: > > > Hi, > > how do i get the job progress n task progress information > > programmaticaly at any point of time using the API's > > there is a JobInProgress and TaskInProgress classes... but both of them > are > > private > > > > any suggestions? > > > > Thanks, > > Raakhi > >
Re: JobInProgress and TaskInProgress
Look at JobClient -- There are some useful methods there. For example, displayTasks and monitorAndPrintJob might provide most of the information that you are looking for. Jothi On 5/19/09 10:14 AM, "Rakhi Khatwani" wrote: > Hi, > I am looking for the following: > for each task: % completed for both map n reduce, exceptions (if > encountered). > for each job: % completed, status (RUNNING,FAILED,PAUSED etc). > i would wanna write a program so that i can programatically access the above > information at any point of time. > > Thanks, > Raakhi > > On Mon, May 18, 2009 at 7:46 PM, Jothi Padmanabhan > wrote: > >> Could you let us know what information are you looking to extract from >> these >> classes? You possibly could get them from other classes. >> >> Jothi >> >> >> On 5/18/09 6:23 PM, "Rakhi Khatwani" wrote: >> >>> Hi, >>> how do i get the job progress n task progress information >>> programmaticaly at any point of time using the API's >>> there is a JobInProgress and TaskInProgress classes... but both of them >> are >>> private >>> >>> any suggestions? >>> >>> Thanks, >>> Raakhi >> >>
Re: JobInProgress and TaskInProgress
You can use RunningJob handle to query map/reduce progress. See api @ http://hadoop.apache.org/core/docs/r0.20.0/api/org/apache/hadoop/mapred/RunningJob.html Thanks Amareshwari Jothi Padmanabhan wrote: Look at JobClient -- There are some useful methods there. For example, displayTasks and monitorAndPrintJob might provide most of the information that you are looking for. Jothi On 5/19/09 10:14 AM, "Rakhi Khatwani" wrote: Hi, I am looking for the following: for each task: % completed for both map n reduce, exceptions (if encountered). for each job: % completed, status (RUNNING,FAILED,PAUSED etc). i would wanna write a program so that i can programatically access the above information at any point of time. Thanks, Raakhi On Mon, May 18, 2009 at 7:46 PM, Jothi Padmanabhan wrote: Could you let us know what information are you looking to extract from these classes? You possibly could get them from other classes. Jothi On 5/18/09 6:23 PM, "Rakhi Khatwani" wrote: Hi, how do i get the job progress n task progress information programmaticaly at any point of time using the API's there is a JobInProgress and TaskInProgress classes... but both of them are private any suggestions? Thanks, Raakhi
