On Wednesday, September 17, 2014 04:20:24 PM Alec Ten Harmsel wrote: > As far as HDFS goes, I would only set that up if you will use it for > Hadoop or related tools. It's highly specific, and the performance is > not good unless you're doing a massively parallel read (what it was > designed for). I can elaborate why if anyone is actually interested. > > We use Lustre for our high performance general storage. I don't have any > numbers, but I'm pretty sure it is *really* fast (10Gbit/s over IB > sounds familiar, but don't quote me on that).
I think any shared filesystem will be fast if you have a lot of bandwidth :) When comparing network filesystems it makes sense to keep the hardware identical reduce the overhead to a percentage. Eg. What is the theoretical maximum speed for the used network. (10Gbit/s) and what is the actual maximum speed you get with: 1) a single really large file (200GB) 2) a lot (100,000) smaller files (2MB) Then you can make an estimate on what to expect when using a 1Gbit/s network. I somehow don't expect James to have InfiniBand available for his research? Personally, when choosing between InfiniBand and Ethernet, I'm tempted to go with dedicated bonded 10Gbit/s links because of the price- difference. (A quick research shows me that Infiniband is about 3x as expensive for the same throughput) > > Personally, I would read up on these and see how they work. Then, > > based on that, decide if they are likely to assist in the specific > > situation you are interested in. > > Always good advice. It saves time to do some simple research (the reading type) before actually doing tests. -- Joost
