What is the COPP? On Thu, Nov 6, 2014 at 1:53 PM, Wido den Hollander <w...@42on.com> wrote:
> On 11/06/2014 02:38 PM, Luis Periquito wrote: > > Hi Wido, > > > > What is the full topology? Are you using a north-south or east-west? So > far > > I've seen the east-west are slightly slower. What are the fabric modes > you > > have configured? How is everything connected? Also you have no > information > > on the OS - if I remember correctly there was a lot of improvements in > the > > latest kernels... > > The Nexus 3000s are connected with 40Gbit to the Nexus 7000. There are > two 7000 units and 8 3000s spread out over 4 racks. > > But the test I did was with two hosts connected to the same Nexus 3000 > switch using TwinAx cabling of 3m. > > The tests were performed with Ubuntu 14.04 (3.13) and RHEL 7 (3.10), but > that didn't make a difference. > > > > > And what about the bandwith? > > > > Just fine, no problems getting 10Gbit through the NICs. > > > The values you present don't seem awfully high, and the deviation seems > low. > > > > No, they don't seem high, but they are about 40% higher then the values > I see on other environments. 40% is a lot. > > This Ceph cluster is SSD-only, so the lower the latency, the more IOps > the system can do. > > Wido > > > On Thu, Nov 6, 2014 at 1:18 PM, Wido den Hollander <w...@42on.com> > wrote: > > > >> Hello, > >> > >> While working at a customer I've ran into a 10GbE latency which seems > >> high to me. > >> > >> I have access to a couple of Ceph cluster and I ran a simple ping test: > >> > >> $ ping -s 8192 -c 100 -n <ip> > >> > >> Two results I got: > >> > >> rtt min/avg/max/mdev = 0.080/0.131/0.235/0.039 ms > >> rtt min/avg/max/mdev = 0.128/0.168/0.226/0.023 ms > >> > >> Both these environment are running with Intel 82599ES 10Gbit cards in > >> LACP. One with Extreme Networks switches, the other with Arista. > >> > >> Now, on a environment with Cisco Nexus 3000 and Nexus 7000 switches I'm > >> seeing: > >> > >> rtt min/avg/max/mdev = 0.160/0.244/0.298/0.029 ms > >> > >> As you can see, the Cisco Nexus network has high latency compared to the > >> other setup. > >> > >> You would say the switches are to blame, but we also tried with a direct > >> TwinAx connection, but that didn't help. > >> > >> This setup also uses the Intel 82599ES cards, so the cards don't seem to > >> be the problem. > >> > >> The MTU is set to 9000 on all these networks and cards. > >> > >> I was wondering, others with a Ceph cluster running on 10GbE, could you > >> perform a simple network latency test like this? I'd like to compare the > >> results. > >> > >> -- > >> Wido den Hollander > >> 42on B.V. > >> Ceph trainer and consultant > >> > >> Phone: +31 (0)20 700 9902 > >> Skype: contact42on > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant > > Phone: +31 (0)20 700 9902 > Skype: contact42on >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
