Re: [RFC] Enabling IPFIREWALL_FORWARD in run-time
> On 10/19/12 4:25 AM, Andrey V. Elsukov wrote: >> >> Hi All, >> >> Many years ago i have already proposed this feature, but at that time >> several people were against, because as they said, it could affect >> performance. Now, when we have high speed network adapters, SMP kernel >> and network stack, several locks acquired in the path of each packet, >> and i have an ability to test this in the lab. >> >> So, i prepared the patch, that removes IPFIREWALL_FORWARD option from >> the kernel and makes this functionality always build-in, but it is >> turned off by default and can be enabled via the sysctl(8) variable >> net.pfil.forward=1. >> >> http://people.freebsd.org/~ae/pfil_forward.diff >> >> Also we have done some tests with the ixia traffic generator connected >> via 10G network adapter. Tests have show that there is no visible >> difference, and there is no visible performance degradation. >> >> Any objections? Just another me-too mail - this is great news! I can't really comment on the quality of the patch or the performance results as I'm neither an expert in low-level coding nor do I have a test lab to give this patch a go, but if there are no concrete objections, I really hope this goes forward. Thanks for the good work. Regards, -- Nino ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: VIMAGE crashes on 9.x with hotplug net80211 devices
On Monday 22 October 2012 23:43:11 Adrian Chadd wrote: > Hi, > > I don't mind tackling the net80211 clone detach path. > > I do mind how the default for hotplug is "argh, it doesn't work." :-) > > So I'd like to come up with something to fix the basic device detach, > rather than having to actually add CURVNET_*() calls around each > if_free() in each device detach method. As already mentioned earlier, I don't terribly object if you'd place CURVNET_SET(ifp->if_vnet) inside if_free() and a limited number of similar functions, but I don't quite believe this is will enough to solve the device_detach() issue without having to touch any of the drivers... Marko ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
kern/92880: [libc] [patch] almost rewritten inet_network(3) function
The following reply was made to PR kern/92880; it has been noted by GNATS.
From: Andrey Simonenko
To: bug-follo...@freebsd.org
Cc:
Subject: kern/92880: [libc] [patch] almost rewritten inet_network(3) function
Date: Tue, 23 Oct 2012 11:36:04 +0300
I optimized inet_network() again.
Difference between implementation of inet_network(3) from 9.1-PRERELEASE
and my implementation.
STRING INET_NETWORKINET_NETWORK_NEW
"0x12" 0x0012 0x0012
"127.1"0x7f01 0x7f01
"127.1.2.3"0x7f010203 0x7f010203
"0x123456" INADDR_NONE INADDR_NONE
"0x12.0x34"0x1234 0x1234
"0x12.0x345" INADDR_NONE INADDR_NONE
"1.2.3.4.5"INADDR_NONE INADDR_NONE
"1..3.4" INADDR_NONE INADDR_NONE
"."INADDR_NONE INADDR_NONE
"1." INADDR_NONE INADDR_NONE
".1" INADDR_NONE INADDR_NONE
"0x" 0x INADDR_NONE <---
"0"0x 0x
"01.02.07.077" 0x0102073f 0x0102073f
"0x1.23.045.0" 0x01172500 0x01172500
"" INADDR_NONE INADDR_NONE
" "INADDR_NONE INADDR_NONE
" f" INADDR_NONE INADDR_NONE
"bar" INADDR_NONE INADDR_NONE
"1.2bar" INADDR_NONE INADDR_NONE
"1." INADDR_NONE INADDR_NONE
"=CA=C3=D5=CB=C5=CE" INADDR_NONE INADDR_NONE
"255.255.255.255" INADDR_NONE INADDR_NONE
"x"INADDR_NONE INADDR_NONE
"0X12" 0x0012 0x0012
"078" INADDR_NONE INADDR_NONE
"1 bar"0x0001 INADDR_NONE <---
"127.0xabcd" INADDR_NONE INADDR_NONE
"128" 0x0080 0x0080
"0.1.2"0x0102 0x0102
"0xff.010.23.0xa0" 0xff0817a0 0xff0817a0
"x10" 0x0010 INADDR_NONE <---
"X20" 0x0020 INADDR_NONE <---
"x10.x20" 0x1020 INADDR_NONE <---
"4294967297" 0x0001 INADDR_NONE <---
"0x1000f" 0x000f INADDR_NONE <---
"0403" 0x0003 INADDR_NONE <---
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
static in_addr_t
inet_network_new(const char *s)
{
u_int d, base, dots;
in_addr_t addr, byte;
u_char c;
bool flag;
addr =3D 0;
dots =3D 0;
for (;; ++s) {
byte =3D 0;
flag =3D false;
if (*s =3D=3D '0') {
++s;
if (*s =3D=3D 'x' || *s =3D=3D 'X') {
++s;
base =3D 16;
} else {
base =3D 8;
flag =3D true;
}
} else
base =3D 10;
for (; (c =3D *s) !=3D '\0'; ++s) {
d =3D digittoint(c);
if (c !=3D '0' && (d =3D=3D 0 || d >=3D base))
break;
byte =3D byte * base + d;
if (byte > UINT8_MAX)
return (INADDR_NONE);
flag =3D true;
}
if (!flag)
return (INADDR_NONE);
addr =3D (addr << 8) | byte;
if (c !=3D '.')
break;
if (++dots =3D=3D 4)
return (INADDR_NONE);
}
return (c =3D=3D '\0' ? addr : INADDR_NONE);
}
int
main(void)
{
const char *const addr_str_tbl[] =3D {
"0x12", "127.1", "127.1.2.3", "0x123456", "0x12.0x34",
"0x12.0x345", "1.2.3.4.5", "1..3.4", ".", "1.", ".1", "0x",
"0", "01.02.07.077", "0x1.23.045.0", "", " ", " f", "bar",
"1.2bar", "1.", "=CA=C3=D5=CB=C5=CE", "255.255.255.255", "x",
"0X12"=
, "078",
"1 bar", "127.0xabcd", "128", "0.1.2", "0xff.010.23.0xa0",
"x10", "X20", "x10.x20", "4294967297", "0x1000f",
"0403", NULL };
const char *const *addr_str;
size_t len;
in_addr_t addr1, addr2;
printf("STRING\t\t\tINET_NETWORK\tINET_NETWORK_NEW\n");
for (addr_str =3D addr_str_tbl; *addr_str !=3D NULL; ++addr_str) {
printf("\"%s\"", *addr_str);
len =3D strlen(*addr_str) + 2;
if (len < 8)
printf("\t\t\t");
else if (len < 16)
Re: VIMAGE crashes on 9.x with hotplug net80211 devices
On 23 October 2012 00:16, Marko Zec wrote: > As already mentioned earlier, I don't terribly object if you'd place > CURVNET_SET(ifp->if_vnet) inside if_free() and a limited number of similar > functions, but I don't quite believe this is will enough to solve the > device_detach() issue without having to touch any of the drivers... That's why I'm asking for more/better ideas. So far my ideas are: * for hotplug insert - do the same as what you're doing during the kldload and boot device enumeration pass - call CURVNET_SET(vnet0) * for device unload (hotplug or otherwise) - if vnet isn't set, implicitly set it to vnet0 * for the net80211 vaps, they get destroyed in a few places (ioctl path, device detach path, I'm sure I saw one more) so I have to use CURVNET_SET(ifp->if_vnet) on those. Now, that _should_ fix it for ath(4) and net80211, and it should fix it for all the other non-USB wireless devices out there. Now as for USB - Hans, what do you think? Should we do something similar? How does VIMAGE work right now with USB wireless and USB ethernet devices? Marko - thanks for persisting with this. I'd like to try and make this work for 10.0. Adrian ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: kern/92880: [libc] [patch] almost rewritten inet_network(3) function
... don't suppose you want to throw this into a test case somewhere in the tree?
The new ATF import would be ideal for this. :)
Adrian
On 23 October 2012 01:40, Andrey Simonenko wrote:
> The following reply was made to PR kern/92880; it has been noted by GNATS.
>
> From: Andrey Simonenko
> To: bug-follo...@freebsd.org
> Cc:
> Subject: kern/92880: [libc] [patch] almost rewritten inet_network(3) function
> Date: Tue, 23 Oct 2012 11:36:04 +0300
>
> I optimized inet_network() again.
>
> Difference between implementation of inet_network(3) from 9.1-PRERELEASE
> and my implementation.
>
> STRING INET_NETWORKINET_NETWORK_NEW
> "0x12" 0x0012 0x0012
> "127.1"0x7f01 0x7f01
> "127.1.2.3"0x7f010203 0x7f010203
> "0x123456" INADDR_NONE INADDR_NONE
> "0x12.0x34"0x1234 0x1234
> "0x12.0x345" INADDR_NONE INADDR_NONE
> "1.2.3.4.5"INADDR_NONE INADDR_NONE
> "1..3.4" INADDR_NONE INADDR_NONE
> "."INADDR_NONE INADDR_NONE
> "1." INADDR_NONE INADDR_NONE
> ".1" INADDR_NONE INADDR_NONE
> "0x" 0x INADDR_NONE <---
> "0"0x 0x
> "01.02.07.077" 0x0102073f 0x0102073f
> "0x1.23.045.0" 0x01172500 0x01172500
> "" INADDR_NONE INADDR_NONE
> " "INADDR_NONE INADDR_NONE
> " f" INADDR_NONE INADDR_NONE
> "bar" INADDR_NONE INADDR_NONE
> "1.2bar" INADDR_NONE INADDR_NONE
> "1." INADDR_NONE INADDR_NONE
> "=CA=C3=D5=CB=C5=CE" INADDR_NONE INADDR_NONE
> "255.255.255.255" INADDR_NONE INADDR_NONE
> "x"INADDR_NONE INADDR_NONE
> "0X12" 0x0012 0x0012
> "078" INADDR_NONE INADDR_NONE
> "1 bar"0x0001 INADDR_NONE <---
> "127.0xabcd" INADDR_NONE INADDR_NONE
> "128" 0x0080 0x0080
> "0.1.2"0x0102 0x0102
> "0xff.010.23.0xa0" 0xff0817a0 0xff0817a0
> "x10" 0x0010 INADDR_NONE <---
> "X20" 0x0020 INADDR_NONE <---
> "x10.x20" 0x1020 INADDR_NONE <---
> "4294967297" 0x0001 INADDR_NONE <---
> "0x1000f" 0x000f INADDR_NONE <---
> "0403" 0x0003 INADDR_NONE <---
>
> #include
> #include
>
> #include
> #include
>
> #include
> #include
> #include
> #include
> #include
> #include
>
> static in_addr_t
> inet_network_new(const char *s)
> {
> u_int d, base, dots;
> in_addr_t addr, byte;
> u_char c;
> bool flag;
>
> addr =3D 0;
> dots =3D 0;
> for (;; ++s) {
> byte =3D 0;
> flag =3D false;
> if (*s =3D=3D '0') {
> ++s;
> if (*s =3D=3D 'x' || *s =3D=3D 'X') {
> ++s;
> base =3D 16;
> } else {
> base =3D 8;
> flag =3D true;
> }
> } else
> base =3D 10;
> for (; (c =3D *s) !=3D '\0'; ++s) {
> d =3D digittoint(c);
> if (c !=3D '0' && (d =3D=3D 0 || d >=3D base))
> break;
> byte =3D byte * base + d;
> if (byte > UINT8_MAX)
> return (INADDR_NONE);
> flag =3D true;
> }
> if (!flag)
> return (INADDR_NONE);
> addr =3D (addr << 8) | byte;
> if (c !=3D '.')
> break;
> if (++dots =3D=3D 4)
> return (INADDR_NONE);
> }
> return (c =3D=3D '\0' ? addr : INADDR_NONE);
> }
>
> int
> main(void)
> {
> const char *const addr_str_tbl[] =3D {
> "0x12", "127.1", "127.1.2.3", "0x123456", "0x12.0x34",
> "0x12.0x345", "1.2.3.4.5", "1..3.4", ".", "1.", ".1", "0x",
> "0", "01.02.07.077", "0x1.23.045.0", "", " ", " f", "bar",
> "1.2bar", "1.", "=CA=C3=D5=CB=C5=CE", "255.255.255.255", "x",
> "0X12"=
> , "078",
> "1 bar", "127.0xabcd", "128", "0.1.2", "0xff.010.23.0xa0",
> "x10", "X20", "x10.x20", "4294967297", "0x1000f",
> "0403", NULL };
> const cha
fragmentation problem in FreeBSD 7
Hi folks, this is my first post to freebsd-net, and my first bug-fix submission... I hope this is the right mailing list for this issue, and the right format for sending in patches I'm working on a derivative of FreeBSD 7. I've run into a problem with IP header checksums when fragmenting to an e1000 (em) interface, and I've narrowed it down to a very simple test. The test setup is like this: [computer A]---(network 1)---[computer B]---(network 2)---[computer C] That gorgeous drawing shows computer A connected to computer B via network 1, and computer B connected to computer C via network 2. Computer B is set up to forward packets between networks 1 and 2. A can see B but not C. C can see B but not A. B forwards between A and C. Pretty simple. One of B's NICs is a Broadcom, handled by the bce driver; this one works fine in all my testing. B's other NIC is an Intel PRO/1000 handled by the em driver. This is the one giving me trouble. The test disables PMTUD on all three hosts. It then sets the MTU of the bce and em interfaces to the unrealistically low value of 72 bytes, and tries to pass TCP packets back and forth using nc on computers A and C (with computer B acting as a gateway). This is to force the B gateway to fragment the TCP frames it forwards. Receiving on the em and sending on the bce works just fine (as noted above). Small TCP frames that fit in the MTU, big TCP frames that get fragmented, no problems. Receiving on the bce and sending on the em interface works fine for small TCP frames that don't need fragmentation, but when B has to fragment the IP packets before sending them out the em, the IP header checksums in the IP packets that appear on the em's wires are wrong. I came to this conclusion by packet capture and by watching the 'bad header checksums' counter of 'netstat -s -p ip', both running on the computer receiving the fragments. Ok, those are all my observations, next comes thoughts about the cause & a proposed fix. The root of the problem is two-fold: 1. ip_output.c:ip_fragment() does not clear the CSUM_IP flag in the mbuf when it does software IP checksum computation, so the mbuf still looks like it needs IP checksumming. 2. The em driver does not advertise IP checksum offloading, but still checks the CSUM_IP flag in the mbuf and modifies the packet when that flag is set (this is in em_transmit_checksum_setup(), called by em_xmit()). Unfortunately the em driver gets the checksum wrong in this case, i guess that's why it doesn't advertise this capability in its if_hwassist! So the fragments that ip_fastfwd.c:ip_fastforward() gets from ip_output.c:ip_fragment() have ip->ip_sum set correctly, but the mbuf->m_pkthdr.csum_flags incorrectly has CSUM_IP still set, and this causes the em driver to emit incorrect packets. There are some other callers of ip_fragment(), notably ip_output(). ip_output() clears CSUM_IP in the mbuf csum_flags itself if it's not in if_hwassist, so avoids this problem. So, the fix is simple: clear the mbuf's CSUM_IP when computing ip->ip_sum in ip_fragment(). The first attached patch (against gitorious/svn_stable_7) does this. In looking at this issue, I noticed that ip_output()'s use of sw_csum is inconsistent. ip_output() splits the mbuf's csum_flags into two parts: the stuff that hardware will assist with (these flags get left in the mbuf) and the stuff that software needs to do (these get moved to sw_csum). But later ip_output() calls functions that don't get sw_csum, or that don't know to look in it and look in the mbuf instead. My second patch fixes these kinds of issues and (IMO) simplifies the code by leaving all the packet's checksumming needs in the mbuf, getting rid of sw_csum entirely. -- Sebastian Kuzminsky Linerate Systems 0001-Update-the-mbuf-csum_flags-of-IP-fragments-when-comp.patch Description: Binary data 0002-Simplify-the-tracking-of-mbuf-checksumming-needs.patch Description: Binary data ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: kern/92880: [libc] [patch] almost rewritten inet_network(3) function
On Tue, Oct 23, 2012 at 10:37 AM, Adrian Chadd wrote: > ... don't suppose you want to throw this into a test case somewhere in the > tree? > > The new ATF import would be ideal for this. :) Indeed. The only problem is that bsd.test.mk is missing so you can't really integrate testcases into the build *quite* yet, but I can work with you to write up/review the testcase; I have the missing ATF/test infrastructure pieces committed to perforce (but I broke things working on adding fixtures) and will be migrating everything over to github soon (so that way my hacking is more isolated and more collaboration can occur). Thanks! -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
Hi, Doing some poking around at the ixgb driver with a card I have at $work using netperf and two machines hooked up over crossover, I discovered that while ixgb's throughput performance was fantastic on 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO performance on the other hand is fantastic and doesn't degrade with the card across FreeBSD versions. Performance remains constant with ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. More details: The machines are hooked up in the following configuration: --- | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | ---- Machine Configuration: The card in Machine 2 is an 82599EB card according to pciconf -lv. /boot/loader.conf tunables (most of these are set according to 9.x defaults in order to establish a sane baseline): kern.ipc.nmbjumbo9=262144 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 kern.ipc.maxsockbuf=2097152 /etc/sysctl.conf tunables: net.inet.tcp.recvspace=65536 net.inet.tcp.recvspace_inc=16384 net.inet.tcp.recvspace_max=2097152 net.inet.tcp.sendspace=32768 net.inet.tcp.sendbuf_max=2097152 net.inet.tcp.sendbuf_inc=8192 Kernel Config: Machine 1 is running a custom version of FreeBSD. The version has been constant over the course of my testing. Can give vague details on the config, but can't give some specific details. Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. Networking configuration: - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. The interface mtu is 1500. - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. The interface mtu is 1500. Netperf configuration: - netserver is run on both machines; I don't add any additional arguments to the netserver invocation so it just goes off and forks. - netperf is run like: netperf -cCjt TCP_STREAM -H I was wondering if this was a known issue and/or others had seen similar problems with this card. I haven't gone into profiling the kernel yet with DTrace, but if no one gets back to me before sometime later on this week/next week that will be my next course of action for tracking down the source of the performance problem. Thanks! -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: > Hi, > > Doing some poking around at the ixgb driver with a card I have at > $work using netperf and two machines hooked up over crossover, I > discovered that while ixgb's throughput performance was fantastic on > 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 > by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO > performance on the other hand is fantastic and doesn't degrade with > the card across FreeBSD versions. Performance remains constant with > ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. > > More details: > > The machines are hooked up in the following configuration: > > --- > | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | > ---- > > Machine Configuration: > > The card in Machine 2 is an 82599EB card according to pciconf -lv. > > /boot/loader.conf tunables (most of these are set according to 9.x > defaults in order to establish a sane baseline): > > kern.ipc.nmbjumbo9=262144 > kern.ipc.nmbjumbo16=262144 > kern.ipc.nmbclusters=262144 > kern.ipc.nmbjumbop=262144 > kern.ipc.maxsockbuf=2097152 > > /etc/sysctl.conf tunables: > > net.inet.tcp.recvspace=65536 > net.inet.tcp.recvspace_inc=16384 > net.inet.tcp.recvspace_max=2097152 > net.inet.tcp.sendspace=32768 > net.inet.tcp.sendbuf_max=2097152 > net.inet.tcp.sendbuf_inc=8192 > > Kernel Config: > > Machine 1 is running a custom version of FreeBSD. The version has been > constant over the course of my testing. Can give vague details on the > config, but can't give some specific details. > Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. > > Networking configuration: > > - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. > The interface mtu is 1500. > - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. > The interface mtu is 1500. > > Netperf configuration: > > - netserver is run on both machines; I don't add any additional > arguments to the netserver invocation so it just goes off and forks. > - netperf is run like: netperf -cCjt TCP_STREAM -H > > I was wondering if this was a known issue and/or others had seen > similar problems with this card. I haven't gone into profiling the > kernel yet with DTrace, but if no one gets back to me before sometime > later on this week/next week that will be my next course of action for > tracking down the source of the performance problem. A couple more notes: 1. We're not using Intel SFP modules -- they're Finisar based (shouldn't matter, but I know that some Intel NICs are incredibly picky when it comes to SFP modules). 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. Thanks! -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
On 24.10.2012 01:13, Garrett Cooper wrote: On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: Hi, Doing some poking around at the ixgb driver with a card I have at $work using netperf and two machines hooked up over crossover, I discovered that while ixgb's throughput performance was fantastic on 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO performance on the other hand is fantastic and doesn't degrade with the card across FreeBSD versions. Performance remains constant with ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. More details: The machines are hooked up in the following configuration: --- | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | ---- Machine Configuration: The card in Machine 2 is an 82599EB card according to pciconf -lv. /boot/loader.conf tunables (most of these are set according to 9.x defaults in order to establish a sane baseline): kern.ipc.nmbjumbo9=262144 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 kern.ipc.maxsockbuf=2097152 /etc/sysctl.conf tunables: net.inet.tcp.recvspace=65536 net.inet.tcp.recvspace_inc=16384 net.inet.tcp.recvspace_max=2097152 net.inet.tcp.sendspace=32768 net.inet.tcp.sendbuf_max=2097152 net.inet.tcp.sendbuf_inc=8192 Kernel Config: Machine 1 is running a custom version of FreeBSD. The version has been constant over the course of my testing. Can give vague details on the config, but can't give some specific details. Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. Networking configuration: - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. The interface mtu is 1500. - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. The interface mtu is 1500. Netperf configuration: - netserver is run on both machines; I don't add any additional arguments to the netserver invocation so it just goes off and forks. - netperf is run like: netperf -cCjt TCP_STREAM -H I was wondering if this was a known issue and/or others had seen similar problems with this card. I haven't gone into profiling the kernel yet with DTrace, but if no one gets back to me before sometime later on this week/next week that will be my next course of action for tracking down the source of the performance problem. A couple more notes: 1. We're not using Intel SFP modules -- they're Finisar based (shouldn't matter, but I know that some Intel NICs are incredibly picky when it comes to SFP modules). 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. There have been a very large number of changes to our network stack between 7.4 and today. That makes it very difficult to pinpoint a specific change. First of all please test 10-current as well but make sure that you turn off WITNESS and INVARIANTS in your kernel config. Second there may be an issue with TSO and ixgb where TSO can cause packets that are a bit larger than 64K by the ethernet header size. ixgb DMA is set up for 64K all inclusive. While TSO did generate a IP valid packet it is arguably not very well digestable for the driver considering that 64K is a very magic number. TSO should reduce its largest packet size by max_linkhdr. A fix for that is in the works and will be available shortly. -- Andre ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
On Tue, Oct 23, 2012 at 4:32 PM, Andre Oppermann wrote: ... >>> Doing some poking around at the ixgb driver with a card I have at >>> $work using netperf and two machines hooked up over crossover, I >>> discovered that while ixgb's throughput performance was fantastic on >>> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >>> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >>> performance on the other hand is fantastic and doesn't degrade with >>> the card across FreeBSD versions. Performance remains constant with >>> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >>> >>> More details: >>> >>> The machines are hooked up in the following configuration: >>> >>> --- >>> >>> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >>> --- >>> - >>> >>> Machine Configuration: >>> >>> The card in Machine 2 is an 82599EB card according to pciconf -lv. >>> >>> /boot/loader.conf tunables (most of these are set according to 9.x >>> defaults in order to establish a sane baseline): >>> >>> kern.ipc.nmbjumbo9=262144 >>> kern.ipc.nmbjumbo16=262144 >>> kern.ipc.nmbclusters=262144 >>> kern.ipc.nmbjumbop=262144 >>> kern.ipc.maxsockbuf=2097152 >>> >>> /etc/sysctl.conf tunables: >>> >>> net.inet.tcp.recvspace=65536 >>> net.inet.tcp.recvspace_inc=16384 >>> net.inet.tcp.recvspace_max=2097152 >>> net.inet.tcp.sendspace=32768 >>> net.inet.tcp.sendbuf_max=2097152 >>> net.inet.tcp.sendbuf_inc=8192 >>> >>> Kernel Config: >>> >>> Machine 1 is running a custom version of FreeBSD. The version has been >>> constant over the course of my testing. Can give vague details on the >>> config, but can't give some specific details. >>> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >>> >>> Networking configuration: >>> >>> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >>> The interface mtu is 1500. >>> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >>> The interface mtu is 1500. >>> >>> Netperf configuration: >>> >>> - netserver is run on both machines; I don't add any additional >>> arguments to the netserver invocation so it just goes off and forks. >>> - netperf is run like: netperf -cCjt TCP_STREAM -H >>> >>> I was wondering if this was a known issue and/or others had seen >>> similar problems with this card. I haven't gone into profiling the >>> kernel yet with DTrace, but if no one gets back to me before sometime >>> later on this week/next week that will be my next course of action for >>> tracking down the source of the performance problem. >> >> >> A couple more notes: >> >> 1. We're not using Intel SFP modules -- they're Finisar based >> (shouldn't matter, but I know that some Intel NICs are incredibly >> picky when it comes to SFP modules). >> 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. > > There have been a very large number of changes to our network > stack between 7.4 and today. That makes it very difficult to > pinpoint a specific change. > > First of all please test 10-current as well but make sure that > you turn off WITNESS and INVARIANTS in your kernel config. > > Second there may be an issue with TSO and ixgb where TSO can > cause packets that are a bit larger than 64K by the ethernet > header size. ixgb DMA is set up for 64K all inclusive. While > TSO did generate a IP valid packet it is arguably not very well > digestable for the driver considering that 64K is a very magic > number. TSO should reduce its largest packet size by max_linkhdr. > A fix for that is in the works and will be available shortly. Ok. I'll give it a shot sometime this upcoming week to see if things improve (of course without INVARIANTS and WITNESS ;)..); I saw there were some changes in the works on CURRENT -- just haven't gotten around to testing that yet (melifaro and a few other folks have been working on performance and removing unnecessary locking too it seems). Thanks for the feedback! -Garrett ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
It you mean the ixgbe driver please call it that, the IS an ixgb driver which is for VERY old PCI cards, from the context I assume you mean the newer hardware :) Jack On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: > Hi, > > Doing some poking around at the ixgb driver with a card I have at > $work using netperf and two machines hooked up over crossover, I > discovered that while ixgb's throughput performance was fantastic on > 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 > by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO > performance on the other hand is fantastic and doesn't degrade with > the card across FreeBSD versions. Performance remains constant with > ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. > > More details: > > The machines are hooked up in the following configuration: > > --- > > | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | > --- > - > > Machine Configuration: > > The card in Machine 2 is an 82599EB card according to pciconf -lv. > > /boot/loader.conf tunables (most of these are set according to 9.x > defaults in order to establish a sane baseline): > > kern.ipc.nmbjumbo9=262144 > kern.ipc.nmbjumbo16=262144 > kern.ipc.nmbclusters=262144 > kern.ipc.nmbjumbop=262144 > kern.ipc.maxsockbuf=2097152 > > /etc/sysctl.conf tunables: > > net.inet.tcp.recvspace=65536 > net.inet.tcp.recvspace_inc=16384 > net.inet.tcp.recvspace_max=2097152 > net.inet.tcp.sendspace=32768 > net.inet.tcp.sendbuf_max=2097152 > net.inet.tcp.sendbuf_inc=8192 > > Kernel Config: > > Machine 1 is running a custom version of FreeBSD. The version has been > constant over the course of my testing. Can give vague details on the > config, but can't give some specific details. > Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. > > Networking configuration: > > - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. > The interface mtu is 1500. > - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. > The interface mtu is 1500. > > Netperf configuration: > > - netserver is run on both machines; I don't add any additional > arguments to the netserver invocation so it just goes off and forks. > - netperf is run like: netperf -cCjt TCP_STREAM -H > > I was wondering if this was a known issue and/or others had seen > similar problems with this card. I haven't gone into profiling the > kernel yet with DTrace, but if no one gets back to me before sometime > later on this week/next week that will be my next course of action for > tracking down the source of the performance problem. > > Thanks! > -Garrett > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ixgbe TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
On Tue, Oct 23, 2012 at 4:58 PM, Jack Vogel wrote: > It you mean the ixgbe driver please call it that, the IS an ixgb driver > which is for > VERY old PCI cards, from the context I assume you mean the newer hardware :) Yeah... I meant ixgbe. Subject line fixed :). Thanks! -Garrett > On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: >> >> Hi, >> >> Doing some poking around at the ixgb driver with a card I have at >> $work using netperf and two machines hooked up over crossover, I >> discovered that while ixgb's throughput performance was fantastic on >> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >> performance on the other hand is fantastic and doesn't degrade with >> the card across FreeBSD versions. Performance remains constant with >> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >> >> More details: >> >> The machines are hooked up in the following configuration: >> >> --- >> >> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >> --- >> - >> >> Machine Configuration: >> >> The card in Machine 2 is an 82599EB card according to pciconf -lv. >> >> /boot/loader.conf tunables (most of these are set according to 9.x >> defaults in order to establish a sane baseline): >> >> kern.ipc.nmbjumbo9=262144 >> kern.ipc.nmbjumbo16=262144 >> kern.ipc.nmbclusters=262144 >> kern.ipc.nmbjumbop=262144 >> kern.ipc.maxsockbuf=2097152 >> >> /etc/sysctl.conf tunables: >> >> net.inet.tcp.recvspace=65536 >> net.inet.tcp.recvspace_inc=16384 >> net.inet.tcp.recvspace_max=2097152 >> net.inet.tcp.sendspace=32768 >> net.inet.tcp.sendbuf_max=2097152 >> net.inet.tcp.sendbuf_inc=8192 >> >> Kernel Config: >> >> Machine 1 is running a custom version of FreeBSD. The version has been >> constant over the course of my testing. Can give vague details on the >> config, but can't give some specific details. >> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >> >> Networking configuration: >> >> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >> The interface mtu is 1500. >> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >> The interface mtu is 1500. >> >> Netperf configuration: >> >> - netserver is run on both machines; I don't add any additional >> arguments to the netserver invocation so it just goes off and forks. >> - netperf is run like: netperf -cCjt TCP_STREAM -H >> >> I was wondering if this was a known issue and/or others had seen >> similar problems with this card. I haven't gone into profiling the >> kernel yet with DTrace, but if no one gets back to me before sometime >> later on this week/next week that will be my next course of action for >> tracking down the source of the performance problem. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
