So is ~ 1GB/minute transfer rate a reasonable performance benchmark?
Our test cluster consists of 4 quad core xeon machines with 2 non-raided
drives each. My initial tests show a transfer rate of around
1GB/minute, and that was slower that I expected it to be.
Thanks,
Scott
Brian Bockelman wrote:
Hey Sugandha,
Transfer rates depend on the quality/quantity of your hardware and the
quality of your client disk that is generating the data. I usually
say that you should expect near-hardware-bottleneck speeds for an
otherwise idle cluster.
There should be no "make it fast" required (though you should reviewi
the logs for errors if it's going slow). I would expect a 5GB file to
take around 3-5 minutes to write on our cluster, but it's a well-tuned
and operational cluster.
As Todd (I think) mentioned before, we can't help any when you say "I
want to make it faster". You need to provide diagnostic information -
logs, Ganglia plots, stack traces, something - that folks can look at.
Brian
On Jun 10, 2009, at 2:25 AM, Sugandha Naolekar wrote:
But if I want to make it fast, then??? I want to place the data in
HDFS and
reoplicate it in fraction of seconds. Can that be possible. and How?
On Wed, Jun 10, 2009 at 2:47 PM, kartik saxena <[email protected]>
wrote:
I would suppose about 2-3 hours. It took me some 2 days to load a
160 Gb
file.
Secura
On Wed, Jun 10, 2009 at 11:56 AM, Sugandha Naolekar
<[email protected]>wrote:It
Hello!
If I try to transfer a 5GB VDI file from a remote host(not a part of
hadoop
cluster) into HDFS, and get it back, how much time is it supposed to
take?
No map-reduce involved. Simply Writing files in and out from HDFS
through
a
simple code of java (usage of API's).
--
Regards!
Sugandha
--
Regards!
Sugandha
