Hi,
I never messed with this options and don't have that bad results.
You have so low write performance withr 16 spindles. You should try to
change your controller, I have better write performance with half of the
spindles. Try areca controller, that's what we use.
Regarding to Infiniband, in the next cluster setup, I will mess up with
this options and share my results.
Thanks,
On 29-07-2011 12:50, Cédric Dufour - Idiap Research Institute wrote:
Hello DRBD users,
I wanted to share my experience with DRBD being used over (Infiniband)
SDP, for I obtained results that may be interesting to some of you.
I first started very frustated by the fact that while SDP
(re-)synching performances were better than TCP's (~5%), general write
operations (e.g. bonnie++ or dd if=/dev/zero ...) were horrible -
~50MB/s (SDP) vs. ~170MB/s (TCP) - the relevant DRBD configuration being:
net {
max-buffers 8192;
max-epoch-size 8192;
sndbuf-size 1024k;
}
After much playing around, I found that two parameters affect write
performances drastically:
1. DRBD's 'sdnbuf-size' parameter
While 'sdnbuf-size' does not affect TCP performances when set above a
value of 1024k, it does so sensibly for SDP connections.
The more you increase the 'sndbuf-size' value, the more SDP
performances increase: write (cf. dd if=/dev/zero ...) jumping from
~50MB/s (sndbuf-size=1024k) to ~120MB/s (sndbuf-size=10240k; the
maximum allowed).
Fortunately, increasing this parameter does not affect small writes
performances (cf. bonnie++ -n ... ), neither when using TCP or SDP.
2. SDP module's (ib_sdp) 'recv_poll' parameter
The 'recv_poll' parameter (default: 700usec) also affects SDP
performances sensibly.
I lowered it (step-by-step) down to 100usec and finally got ~230MB/s
raw write performances (cf. dd if=/dev/zero ...) and bonnie++ write
performances that are 10-15% above TCP performances.
Fortunately again, lowering this parameter does not affect small
writes performances (cf. bonnie++ -n ... ) nor general TCP-via-SDP
performances (iperf being stable at 9.1GBit/s via SDP vs. 3.2GBit/s
via TCP).
Also, CPU usage remains the same.
Finally, nothing amiss shows up in the kernel log.
In the end, I obtained performances that are consistent with what the
backing RAID array allows me to expect.
I hope this may be of interest to some of you.
If someone knowledgeable of DRBD and/or Infiniband modules parameters
were to read this post, I would happily have a feedback about why
those parameters affect DRBD (write) performances (but not (re-)synch's).
Eventually, maybe those results are only valid for the particular
hardware and/or software versions I have been playing with.
Speaking of...
Software version:
- Debian Squeeze 6.0.2 64-bit (kernel 2.6.32)
- Custom-built OFED 1.5.3.2 kernel modules (including patch related
to http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2240)
- Custom-built DRBD 8.3.11 kernel module (along with re-packaged DRBD
8.3.11 user-space utilities)
Hardware:
- Motherboard: Supermicro X7DWN+
- CPU: Intel Xeon L5410 (2x) [HT disabled, VT enabled]
- RAM: 32GB DDR2-667 (16x2GB)
- NIC: Mellanox MT25204 [InfiniHost III Lx HCA] (ib_rdma_bw ~1.5GB/s;
ib_rdma_lat ~2usec)
- HD/RAID: ICP 5165BR with 16x1TB HDs (RAID-6 - LVM2 - DRBD)
Cheers
--
Cédric Dufour @ Idiap Research Institute
"No complaint is enough praise"
_______________________________________________
drbd-user mailing list
[email protected]
http://lists.linbit.com/mailman/listinfo/drbd-user
--
Igor Neves<[email protected]>
3GNTW - Tecnologias de Informação, Lda
SIP: [email protected]
MSN: [email protected]
JID: [email protected]
PSTN: 00351 252377120
_______________________________________________
drbd-user mailing list
[email protected]
http://lists.linbit.com/mailman/listinfo/drbd-user
