On Wed, 7 Oct 2009, rihad wrote:
Robert Watson wrote:
On Wed, 7 Oct 2009, rihad wrote:
snapshot of the top -SH output in the steady state? Let top run for a
few minutes and then copy/paste the first 10-20 lines into an e-mail.
Sure. Mind you: now there's only 1800 entries in each of the two ipfw
tables, so any drops have stopped. But it only takes another 200-300
entries to start dropping.
Could you do the same in the net.isr.direct=1 configuration so we can
compare?
net.isr.direct=1:
So it seems that CPU exhaustion is likely not the source of drops -- what I
was looking for in both configurations were signs that any individual thread
was approaching 80% utilization, which in a peak load situation might mean it
hitting 100% and therefore leading to packet loss for that reason.
The statistic you're monitoring has a couple of interpretations, but the most
likely interpretation is overfilling the output queue on the network interface
you're transmitting on. In turn there are various possible reasons for this
happening, but the two most common would be:
(1) Average load is exceeding the transmit capacity of the driver/hardware
pipeline -- the pipe is just too small.
(2) Peak capacity (burstiness) is exceeding the transmit capacity of the
driver/hardware pipeline.
The questions that Luigi and others have been asking about your dummynet
configuration are to some extent oriented around determining whether the
burstiness introduced by dummynet could be responsible for that. Suggestions
like increasing timer resolution are intended to spread out the injection of
packets by dummynet to attempt to reduce the peaks of burstiness that occur
when multiple queues inject packets in a burst that exceeds the queue depth
supported by combined hardware descriptor rings and software transmit queue.
The two solutions, then are (a) to increase the timer resolution significantly
so that packets are injected in smaller bursts and (b) increase the queue
capacities. The hardware queue limits likely can't be raised w/o new
hardware, but the ifnet transmit queue sizes can be increased. Timer
resolution going up is almost certainly not a bad idea in your configuration,
although does require a reboot as you have observed.
On a side note: one other possible interpretation of that statistic is that
you're seeing fragmentation problems. Usually in forwarding scenarios this is
unlikely. However, it wouldn't hurt to make sure you have LRO turned off on
the network interfaces you're using, assuming it's supported by the driver.
Robert N M Watson
Computer Laboratory
University of Cambridge
last pid: 92152; load averages: 0.99, 1.18, 1.15
up 1+01:42:28 14:53:09
162 processes: 9 running, 136 sleeping, 17 waiting
CPU: 2.1% user, 0.0% nice, 5.4% system, 7.0% interrupt, 85.5% idle
Mem: 1693M Active, 1429M Inact, 447M Wired, 197M Cache, 214M Buf, 170M Free
Swap: 2048M Total, 12K Used, 2048M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12 root 171 ki31 0K 16K CPU6 6 24.3H 100.00% idle: cpu6
13 root 171 ki31 0K 16K CPU5 5 23.8H 95.95% idle: cpu5
14 root 171 ki31 0K 16K CPU4 4 23.4H 93.12% idle: cpu4
16 root 171 ki31 0K 16K CPU2 2 23.0H 90.19% idle: cpu2
11 root 171 ki31 0K 16K CPU7 7 24.2H 87.26% idle: cpu7
15 root 171 ki31 0K 16K CPU3 3 22.8H 86.18% idle: cpu3
18 root 171 ki31 0K 16K RUN 0 20.6H 84.96% idle: cpu0
17 root 171 ki31 0K 16K CPU1 1 933:23 47.85% idle: cpu1
29 root -68 - 0K 16K WAIT 1 522:02 46.88% irq256: bce0
465 root -68 - 0K 16K - 7 55:15 12.65% dummynet
31 root -68 - 0K 16K WAIT 2 57:29 4.74% irq257: bce1
21 root -44 - 0K 16K WAIT 0 34:55 4.64% swi1: net
19 root -32 - 0K 16K WAIT 4 51:41 3.96% swi4: clock
sio
30 root -64 - 0K 16K WAIT 6 5:43 0.73% irq16: mfi0
Almost 2000 entries in the table, traffic load= 420-430 mbps, drops haven't
yet started.
Previous net.isr.direct=0:
155 processes: 10 running, 129 sleeping, 16 waiting
CPU: 2.4% user, 0.0% nice, 2.0% system, 9.3% interrupt, 86.2% idle
Mem: 1691M Active, 1491M Inact, 454M Wired, 130M Cache, 214M Buf, 170M
Free
Swap: 2048M Total, 12K Used, 2048M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15 root 171 ki31 0K 16K CPU3 3 22.4H 97.85% idle: cpu3
14 root 171 ki31 0K 16K CPU4 4 23.0H 96.29% idle: cpu4
12 root 171 ki31 0K 16K CPU6 6 23.8H 94.58% idle: cpu6
16 root 171 ki31 0K 16K CPU2 2 22.5H 90.72% idle: cpu2
13 root 171 ki31 0K 16K CPU5 5 23.4H 90.58% idle: cpu5
18 root 171 ki31 0K 16K RUN 0 20.3H 85.60% idle: cpu0
17 root 171 ki31 0K 16K CPU1 1 910:03 78.37% idle: cpu1
11 root 171 ki31 0K 16K CPU7 7 23.8H 65.62% idle: cpu7
21 root -44 - 0K 16K CPU7 7 19:03 48.34% swi1: net
29 root -68 - 0K 16K WAIT 1 515:49 19.63% irq256: bce0
31 root -68 - 0K 16K WAIT 2 56:05 5.52% irq257: bce1
19 root -32 - 0K 16K WAIT 5 50:05 3.86% swi4: clock
sio
983 flowtools 44 0 12112K 6440K select 0 13:20 0.15% flow-capture
465 root -68 - 0K 16K - 3 51:19 0.00% dummynet
3 root -8 - 0K 16K - 1 7:41 0.00% g_up
4 root -8 - 0K 16K - 2 7:14 0.00% g_down
30 root -64 - 0K 16K WAIT 6 5:30 0.00% irq16: mfi0
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
