Dear Openstack community,
Openstack environment:
2 compute nodes each has:
* 1x AMD Opteron(tm) Processor 6282 SE (56 cpus)
* 512GB RAM
* 1x dual port 25Gbps Mellanox connect-4 XOR bond
Physical network:
* Compute nodes connected through a mellanox 2700 switch (100Gbps)
* MTU 9000
Openstack network:
* OVS + VxLan
* Vxlan offloading setup on physical nics
* MTU on vms 8950
Compute nodes nics:
[root@hercules-23 ~]# ip a
...
6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0
state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0
state UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state
UP qlen 1000
link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff
inet 10.0.32.23/16 brd 10.0.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::3826:9daa:ef1b:43b0/64 scope link
valid_lft forever preferred_lft forever
...
VM nics:
[centos@centos2 ~]$ ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP
qlen 1000
link/ether fa:16:3e:a3:d4:85 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0
valid_lft 57885sec preferred_lft 57885sec
inet6 fe80::f816:3eff:fea3:d485/64 scope link
valid_lft forever preferred_lft forever
host to host bandwidth using muttcp (1 thread):
[root@hercules-22 ~]# nuttcp -l65536 -fparse 10.0.32.23
megabytes=19844.1759 real_seconds=10.00 rate_Mbps=16643.4189 tx_cpu=61
rx_cpu=99 retrans=0 rtt_ms=0.28
VMTP Output:
==================
Total Scenarios: 29
Passed Scenarios: 17 [100.00%]
Failed Scenarios: 0 [0.00%]
Skipped Scenarios: 12
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| Scenario | Scenario Name | Functional
Status | Data
|
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
| 1.1 | Same Network, Fixed IP, Intra-node, TCP | PASSED
| {'tp_kbps': '7613130', 'rtt_ms': '0.463333'}
|
| 1.2 | Same Network, Fixed IP, Intra-node, UDP | PASSED
| {128: {'tp_kbps': 59954, 'loss_rate': 0.02}, 1024: {'tp_kbps': 444264,
|
| | |
| 'loss_rate': 0.05}, 8192: {'tp_kbps': 2877674, 'loss_rate': 0.0}}
|
| 1.3 | Same Network, Fixed IP, Intra-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'0.376/0.285/0.439/0.047', |
| | |
| '64-byte': '0.617/0.361/1.513/0.388', '1500-byte':
'0.369/0.322/0.452/0.048'}} |
| 1.4 | Same Network, Fixed IP, Intra-node, Multicast | SKIPPED
| {}
|
| 2.1 | Same Network, Fixed IP, Inter-node, TCP | PASSED
| {'tp_kbps': '6864266', 'rtt_ms': '0.653333'}
|
| 2.2 | Same Network, Fixed IP, Inter-node, UDP | PASSED
| {128: {'tp_kbps': 80955, 'loss_rate': 0.46}, 1024: {'tp_kbps': 166186,
|
| | |
| 'loss_rate': 36.2}, 8192: {'tp_kbps': 2189389, 'loss_rate': 0.05}}
|
| 2.3 | Same Network, Fixed IP, Inter-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'0.484/0.315/0.561/0.068', |
| | |
| '64-byte': '0.719/0.421/2.714/0.666', '1500-byte':
'0.465/0.358/0.553/0.055'}} |
| 2.4 | Same Network, Fixed IP, Inter-node, Multicast | SKIPPED
| {}
|
| 3.1 | Different Network, Fixed IP, Intra-node, TCP | PASSED
| {'tp_kbps': '5207719', 'rtt_ms': '0.896667'}
|
| 3.2 | Different Network, Fixed IP, Intra-node, UDP | PASSED
| {128: {'tp_kbps': 76569, 'loss_rate': 0.9}, 1024: {'tp_kbps': 571847,
|
| | |
| 'loss_rate': 2.75}, 8192: {'tp_kbps': 1855166, 'loss_rate': 0.0}}
|
| 3.3 | Different Network, Fixed IP, Intra-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'0.859/0.658/1.103/0.133', |
| | |
| '64-byte': '0.976/0.576/1.946/0.361', '1500-byte':
'0.824/0.644/1.082/0.136'}} |
| 3.4 | Different Network, Fixed IP, Intra-node, Multicast | SKIPPED
| {}
|
| 4.1 | Different Network, Fixed IP, Inter-node, TCP | PASSED
| {'tp_kbps': '6730928', 'rtt_ms': '1.37667'}
|
| 4.2 | Different Network, Fixed IP, Inter-node, UDP | PASSED
| {128: {'tp_kbps': 31131, 'loss_rate': 4.84}, 1024: {'tp_kbps': 579859,
|
| | |
| 'loss_rate': 4.35}, 8192: {'tp_kbps': 1601092, 'loss_rate': 0.18}}
|
| 4.3 | Different Network, Fixed IP, Inter-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'0.915/0.740/1.138/0.144', |
| | |
| '64-byte': '1.134/0.713/2.210/0.539', '1500-byte':
'0.980/0.719/1.188/0.131'}} |
| 4.4 | Different Network, Fixed IP, Inter-node, Multicast | SKIPPED
| {}
|
| 5.1 | Different Network, Floating IP, Intra-node, TCP | PASSED
| {'tp_kbps': '5406554', 'rtt_ms': '0.836667'}
|
| 5.2 | Different Network, Floating IP, Intra-node, UDP | PASSED
| {128: {'tp_kbps': 71084, 'loss_rate': 4.26}, 1024: {'tp_kbps': 507900,
|
| | |
| 'loss_rate': 15.96}, 8192: {'tp_kbps': 2011414, 'loss_rate': 3.72}}
|
| 5.3 | Different Network, Floating IP, Intra-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'0.902/0.760/1.007/0.073', |
| | |
| '64-byte': '0.875/0.661/0.967/0.104', '1500-byte':
'0.889/0.754/0.988/0.082'}} |
| 5.4 | Different Network, Floating IP, Intra-node, Multicast | SKIPPED
| {}
|
| 6.1 | Different Network, Floating IP, Inter-node, TCP | PASSED
| {'tp_kbps': '6770995', 'rtt_ms': '0.973333'}
|
| 6.2 | Different Network, Floating IP, Inter-node, UDP | PASSED
| {128: {'tp_kbps': 81812, 'loss_rate': 2.25}, 1024: {'tp_kbps': 617485,
|
| | |
| 'loss_rate': 3.27}, 8192: {'tp_kbps': 1295351, 'loss_rate': 0.02}}
|
| 6.3 | Different Network, Floating IP, Inter-node, ICMP | PASSED
| {'rtt avg/min/max/stddev msec': {'391-byte':
'1.028/0.920/1.254/0.105', |
| | |
| '64-byte': '1.043/0.807/1.227/0.137', '1500-byte':
'0.988/0.759/1.194/0.146'}} |
| 6.4 | Different Network, Floating IP, Inter-node, Multicast | SKIPPED
| {}
|
| 7.1 | Native Throughput, TCP | SKIPPED
| {}
|
| 7.2 | Native Throughput, UDP | SKIPPED
| {}
|
| 7.3 | Native Throughput, ICMP | SKIPPED
| {}
|
| 7.4 | Native Throughput, Multicast | SKIPPED
| {}
|
| 8.1 | VM to Host Uploading | SKIPPED
| {}
|
| 8.2 | VM to Host Downloading | SKIPPED
| {}
|
+----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+
My concern is about east-west bandwidth:
Inter-host and intra-subnet (vmtp Scenario 2.1): I am getting ~6.8Gbps which I
found quite poor compared to 16.0 Gbps I am getting from the hosts
intra-host and intra-subnet (vmtp Scenario 1.1): In this example traffic
remains inside the compute node and no tunneling is required. I am getting
~7.6Gbps which I found quite poor compared to 16.0 Gbps I am getting from the
hosts
NOTE: I am not running anything except these tests on the machines so cpu usage
is quite low.
NOTE2: I know vmtp uses
I would like to understand what is stopping my vms to get full network
bandwidth. Is this related with an OVS configuration? ML2? Other part of
neutron? Something else?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: [email protected]
NOTICE
Please consider the environment before printing this email. This message and
any attachments are intended for the addressee named and may contain legally
privileged/confidential/copyright information. If you are not the intended
recipient, you should not read, use, disclose, copy or distribute this
communication. If you have received this message in error please notify us at
once by return email and then delete both messages. We accept no liability for
the distribution of viruses or similar in electronic communications. This
notice should not be removed.
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
