There's an option to control how many packets it'll process each pass through the isr thread, isn't there?
It'd be nicer if this stuff were able to be dynamically tuned. Adrian 2008/7/1 Paul <[EMAIL PROTECTED]>: > [Big list of testing , rebuilding kernel follows] > > Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk > in the config, added > options NO_ADAPTIVE_MUTEXES not sure if that makes any difference > or not, will test without. > Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf > 7.0-STABLE FreeBSD 7.0-STABLE #4: Tue Jul 1 01:22:18 CDT 2008 amd64 > Max input rate .. 587kpps? Take into consideration that these packets are > being forwarded out em1 interface which > causes a great impact on cpu usage. If I set up a firewall rule to block > the packets it can do over 1mpps on em0 input. > > input (em0) output > packets errs bytes packets errs bytes colls > 587425 67677 35435456 466 0 25616 0 > 587412 26629 35434766 453 0 24866 0 > 587043 26874 35412442 410 0 22544 0 > 536117 30264 32347300 440 0 24164 0 > 546240 61521 32951060 459 0 25350 0 > 563568 66881 33998676 435 0 23894 0 > 572766 43243 34550840 440 0 24164 0 > 572336 44411 34525836 445 0 24558 0 > 572539 37013 34536222 457 0 25136 0 > 571340 39512 34459008 440 0 24110 0 > 572673 55137 34540576 438 0 24056 0 > 555506 49918 33505764 457 0 25330 0 > 545744 69010 32916908 461 0 25298 0 > 559472 75650 33745636 429 0 23694 0 > 564358 60130 34039104 433 0 23786 0 > > last pid: 1134; load averages: 1.04, 0.94, 0.59 > up 0+00:14:13 01:49:59 > 70 processes: 6 running, 46 sleeping, 17 waiting, 1 lock > CPU: 0.0% user, 0.0% nice, 25.6% system, 0.0% interrupt, 74.4% idle > Mem: 11M Active, 6596K Inact, 45M Wired, 156K Cache, 9072K Buf, 1917M Free > Swap: 8192M Total, 8192M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root 171 ki31 0K 16K RUN 1 12:40 97.56% idle: cpu1 > 36 root -68 - 0K 16K *em1 2 9:44 85.06% em0 taskq > 10 root 171 ki31 0K 16K CPU3 3 11:10 82.47% idle: cpu3 > 13 root 171 ki31 0K 16K CPU0 0 12:25 73.88% idle: cpu0 > 11 root 171 ki31 0K 16K RUN 2 6:43 50.10% idle: cpu2 > 37 root -68 - 0K 16K CPU3 3 1:58 16.46% em1 taskq > > > I noticed.. em0 taskq isn't using 100% cpu like it was on the generic > kernel.. What's up with that? Why do I still have all 4 CPUs pretty idle and > em0 taskq isn't near 100%? I'm going to try 4bsd and see > if that makes it go back to the other way. > > em0: Excessive collisions = 0 > em0: Sequence errors = 0 > em0: Defer count = 0 > em0: Missed Packets = 45395545 > em0: Receive No Buffers = 95916690 > em0: Receive Length Errors = 0 > em0: Receive errors = 0 > em0: Crc errors = 0 > em0: Alignment errors = 0 > em0: Collision/Carrier extension errors = 0 > em0: RX overruns = 2740181 > em0: watchdog timeouts = 0 > em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 > em0: XON Rcvd = 0 > em0: XON Xmtd = 0 > em0: XOFF Rcvd = 0 > em0: XOFF Xmtd = 0 > em0: Good Packets Rcvd = 450913688 > em0: Good Packets Xmtd = 304777 > em0: TSO Contexts Xmtd = 94 > em0: TSO Contexts Failed = 0 > > -----Rebooting with: > kern.hz=2000 > hw.em.rxd=512 > hw.em.txd=512 > > Seems maybe a little bit slower but it's hard to tell since i'm generating > random packets the pps varies about 50k +/- probably depending > on the randomness.. About the same PPS/errors.. here's a vmstat 1 > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52276K 1922M 286 0 1 0 277 0 0 0 7686 838 19436 0 > 15 85 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13431 127 33430 0 > 27 73 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13406 115 33222 0 > 27 73 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13430 115 33393 0 > 26 74 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13411 115 33322 0 > 26 74 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13576 123 33415 0 > 25 75 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13842 115 33354 0 > 26 74 > > ------Trying kern.kz=250 > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52288K 1923M 607 1 2 0 582 0 0 0 4885 789 12073 0 > 8 92 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13793 119 33552 0 > 27 73 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13959 115 33446 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13861 115 33707 0 > 30 70 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13784 115 33602 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13886 123 33843 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13913 115 33711 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13920 115 33766 0 > 27 73 > > pps still no major difference.. > jumps between 530k-580k > > -----Putting HZ back to 1000, > recompiling kernel with 4BSD SCHED.. > many minutes later.. (can't do make -j with the kernel or it errors) > Well, I have to say.. 4BSD is less pps, it will not go over 530k however it > seems much, > more consistent and not jumping around as much it stays between 520-530 most > of the time and i see some ticks > at 480's in netstat.. > em0 taskq still not using 100%, max around 75-80 > > -----Building same as above but with preemption off > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 0 52288K 1922M 563 1 2 0 540 0 0 0 6724 725 22195 0 > 12 88 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13200 119 48075 0 > 27 73 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13243 123 49137 0 > 24 76 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13260 115 48633 0 > 26 74 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13247 115 48625 0 > 25 75 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13248 115 48687 0 > 24 76 > > hmm more context switches.. > pps same, maybe a shade lower.. > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 171 ki31 0K 16K RUN 2 3:39 97.12% idle: cpu2 > 12 root 171 ki31 0K 16K CPU1 1 3:45 95.70% idle: cpu1 > 36 root -68 - 0K 16K CPU0 0 2:18 82.67% em0 taskq > 10 root 171 ki31 0K 16K CPU3 3 3:24 82.57% idle: cpu3 > 13 root 171 ki31 0K 16K RUN 0 2:01 20.07% idle: cpu0 > 37 root -68 - 0K 16K - 3 0:31 15.58% em1 taskq > > > -------rebuilding with ULE, keeping preemption off > Hmm.. what the? > 450-480kpps seems to be max here. That's.. weird.. > I'm going to have to rebuild with Preemption on again just to double check > this.. > input (em0) output > packets errs bytes packets errs bytes colls > 464020 95690 28009004 434 0 23728 0 > 455318 90105 27484456 469 0 25778 0 > 455720 99914 27511970 462 0 25384 0 > 465019 86021 28071946 428 0 23392 0 > 456024 78336 27528862 440 0 24040 0 > 455018 93526 27468908 440 0 24040 0 > 461235 91218 27841604 464 0 25336 0 > 454345 89812 27427262 424 0 23176 0 > 452661 96937 27327392 441 0 24094 0 > 456584 90393 27561138 459 0 25222 0 > 455021 97441 27470158 450 0 24736 0 > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52276K 1655M 456 1 1 0 441 0 0 0 9775 3598 26256 0 > 20 80 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12817 119 33056 0 > 25 75 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12700 123 32975 0 > 27 73 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12659 115 32897 0 > 27 73 > > > ------OK I'm stumped now.. Rebuilt with preemption and ULE and preemption > again and it's not doing what it did before.. > How could that be? Now about 500kpps.. > > That kind of inconsistency almost invalidates all my testing.. why would it > be so much different after trying a bunch of kernel options and rebooting a > bunch of times and then going back to the original config doesn't get you > what it did in the beginning.. > > I'll have to dig into this further.. never seen anything like it :) > > Hopefully the ip_input fix will help free up a few cpu cycles. > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
