Hi Todd,
Thank You for your input. Our data is like any apache log file(s). Basic
logging info which we are parsing.
Our data is alot which is why we are using HADOOP :).
I will look into running TT's on the hdfs clients just for job processing
and not to store any data locally. We can then also gage performance by
running the MAP/REDUCE both.
Thanks,
Usman
On Fri, May 1, 2009 at 4:22 AM, Usman Waheed <[email protected]> wrote:
Hi,
I just wanted to share a test we conducted in our small cluster of 3
datanodes and one namenode. Basically we have lots of data to process
and we
run a parsing script outside hadoop that creates the key,value pairs.
This
output which is plain txt files is then imported into hadoop using the
put/get etc commands.
Thanks for sharing, Usman. Some comments below:
In order to speed up things we run the parsing jobs on multiple
machines in
parallel which are not part of our cluster (3 datanodes + namenode) but
they
do have the same version of hadoop installed as the cluster which we
use to
perform the puts. This work flow has significantly improved our time to
import the data into HADOOP after which we run the reduce-only step to
aggregate.
Currently the way to insert data is through our namenode which all the
machines outsude the cluster call them hdfs clients connect to and are
not
part of the master/slave setup. I haven't tried but maybe we can perform
these puts via the datanodes themselves and not just through the
namenode?
Right now the namenode is the single point through which the hdfs client
machines insert the parsed data.
When you do an hdfs put "to the namenode" the actual data transfer goes
to
the datanodes anyway - the namenode isn't a data bottleneck. It's just
used
to allocate block locations for writers, and then DFSClient connects
directly to the DNs to transfer.
Secondly i would assume that this is a safe way to import parsed data
into
hadoop before we aggregate and will most likely not cause any data
corruption in HDFS. Granted anything can happen :).
Yep, this should be as safe as any other method.
It would be interesting to import our logs and perform the mapping step
inside HADOOP versus doing it outside. I wonder if the performance will
be
better, worse or the same. Yes this is dependent on many factors and
one of
them is the amount of datanodes, data to process, hardware etc we have
but
we are limited. We are trying to utilize machines outside the cluster
which
are idle and can process info and then insert the output into HADOOP
HDFS
via puts.
The performance is probably comparable on a small cluster like this,
depending on what the ratio of input/output data is. The advantage of
doing
the writes from within the cluster is that the namenode will try to
allocate
block locations on the local node, so there is less total transfer into
the
cluster. In a larger cluster, you might end up with a network bottleneck
going into the cluster, but with three nodes and any reasonable switch
you
shouldn't be running into that.
What does your input data look like? It might make more sense to upload
it
directly to the cluster and then use a MR job to perform the
transformation.
This way you don't have to worry about doing that distribution yourself.
If
you want to make use of those "extra" nodes that aren't part of your
cluster, you could probably just run TTs on them without running DNs.
-Todd
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
