2011/2/7 Pyun YongHyeon <pyu...@gmail.com> > On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > > > To: Lev Serebryakov <l...@serebryakov.spb.ru> > > > Cc: freebsd-net@freebsd.org > > > > > > > > > 2011/2/7 Lev Serebryakov <l...@serebryakov.spb.ru> > > > > > > Hello, Karim. > > >> You wrote 7 февраля 2011 г., 19:58:04: > > >> > > >> > > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver > > >> runs > > >> > out of mbuf clusters it simply stops receiving even after the > clusters > > >> have > > >> > been freed. > > >> It looks like my problems with em0 (see thread "em0 hangs without > > >> any messages like "Watchdog timeout", only down/up reset it.")... > > >> Codebase for em and igb is somewhat common... > > >> > > >> -- > > >> // Black Lion AKA Lev Serebryakov <l...@serebryakov.spb.ru> > > >> > > >> I agree. > > > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > > > I might not have mentioned but I can also 'fix' the problem by doing > > > ifconfig igb0 down/up. > > > > > > I will try using POLLING to 'automatize' the reset as you mentioned in > your > > > thread. > > > > > > Karim. > > > > > > > > Follow up on tests with POLLING: The problem is still occurring although > it > > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will > follow: > > > > 9219/99426/108645 mbufs in use (current/cache/total) > > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) > > Do you see network processes are stuck in keglim state? If you see > that I think that's not trivial to solve. You wouldn't even kill > that process if it is under keglim state unless some more mbuf > clusters are freed from other places. >
No keglim state, here is a snapshot of top -SH while the problem is happening: 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% idle: cpu5 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% idle: cpu7 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% idle: cpu3 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% idle: cpu6 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% idle: cpu4 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% idle: cpu2 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% idle: cpu1 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% idle: cpu0 18 root -32 - 0K 8K WAIT 6 0:04 0.10% swi4: clock s 20 root -44 - 0K 8K WAIT 4 0:08 0.00% swi1: net 29 root -68 - 0K 8K - 0 0:02 0.00% igb0 que 35 root -68 - 0K 8K - 2 0:02 0.00% em1 taskq 28 root -68 - 0K 8K WAIT 5 0:01 0.00% irq256: igb0 keep in mind that num_queues has been forced to 1. > > I think both igb(4) and em(4) pass received frame to upper stack > before allocating new RX buffer. If driver fails to allocate new RX > buffer driver will try to refill RX buffers in next run. Under > extreme resource shortage case, this situation can produce no more > RX buffers in RX descriptor ring and this will take the box out of > network. Other drivers avoid that situation by allocating new RX > buffer before passing received frame to upper stack. If RX buffer > allocation fails driver will just reuse old RX buffer without > passing received frame to upper stack. That does not completely > solve the keglim issue though. I think you should have enough mbuf > cluters to avoid keglim. > > However the output above indicates you have enough free mbuf > clusters. So I guess igb(4) encountered zero available RX buffer > situation in past but failed to refill the RX buffer again. I guess > driver may be able to periodically check available RX buffers. > Jack may have better idea if this was the case.(CCed) > That is exactly the pattern. The driver runs out of clusters but they eventually get consumed and freed although the driver refuses to process any new frames. It is, on the other hand, perfectly capable of sending out packets. > > 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) > > 0/12800/12800/12800 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 20738K/257622K/278361K bytes allocated to network (current/cache/total) > > 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/5/6656 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > > dev.igb.0.%driver: igb > > dev.igb.0.%location: slot=0 function=0 > > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 > > subdevice=0x0000 class=0x020000 > > dev.igb.0.%parent: pci7 > > dev.igb.0.nvm: -1 > > dev.igb.0.flow_control: 3 > > dev.igb.0.enable_aim: 1 > > dev.igb.0.rx_processing_limit: 100 > > dev.igb.0.link_irq: 4 > > dev.igb.0.dropped: 0 > > dev.igb.0.tx_dma_fail: 0 > > dev.igb.0.rx_overruns: 464 > > dev.igb.0.watchdog_timeouts: 0 > > dev.igb.0.device_control: 1490027073 > > dev.igb.0.rx_control: 67141658 > > dev.igb.0.interrupt_mask: 0 > > dev.igb.0.extended_int_mask: 0 > > dev.igb.0.tx_buf_alloc: 14 > > dev.igb.0.rx_buf_alloc: 34 > > dev.igb.0.fc_high_water: 29488 > > dev.igb.0.fc_low_water: 29480 > > dev.igb.0.queue0.interrupt_rate: 111111 > > dev.igb.0.queue0.txd_head: 877 > > dev.igb.0.queue0.txd_tail: 877 > > dev.igb.0.queue0.no_desc_avail: 0 > > dev.igb.0.queue0.tx_packets: 92013 > > dev.igb.0.queue0.rxd_head: 570 > > dev.igb.0.queue0.rxd_tail: 570 > > dev.igb.0.queue0.rx_packets: 163386 > > dev.igb.0.queue0.rx_bytes: 240260310 > > dev.igb.0.queue0.lro_queued: 0 > > dev.igb.0.queue0.lro_flushed: 0 > > dev.igb.0.mac_stats.excess_coll: 0 > > dev.igb.0.mac_stats.single_coll: 0 > > dev.igb.0.mac_stats.multiple_coll: 0 > > dev.igb.0.mac_stats.late_coll: 0 > > dev.igb.0.mac_stats.collision_count: 0 > > dev.igb.0.mac_stats.symbol_errors: 0 > > dev.igb.0.mac_stats.sequence_errors: 0 > > dev.igb.0.mac_stats.defer_count: 0 > > dev.igb.0.mac_stats.missed_packets: 3104 > > dev.igb.0.mac_stats.recv_no_buff: 4016 > > dev.igb.0.mac_stats.recv_undersize: 0 > > dev.igb.0.mac_stats.recv_fragmented: 0 > > dev.igb.0.mac_stats.recv_oversize: 0 > > dev.igb.0.mac_stats.recv_jabber: 0 > > dev.igb.0.mac_stats.recv_errs: 0 > > dev.igb.0.mac_stats.crc_errs: 0 > > dev.igb.0.mac_stats.alignment_errs: 0 > > dev.igb.0.mac_stats.coll_ext_errs: 0 > > dev.igb.0.mac_stats.xon_recvd: 0 > > dev.igb.0.mac_stats.xon_txd: 346 > > dev.igb.0.mac_stats.xoff_recvd: 0 > > dev.igb.0.mac_stats.xoff_txd: 3450 > > dev.igb.0.mac_stats.total_pkts_recvd: 166515 > > dev.igb.0.mac_stats.good_pkts_recvd: 163411 > > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > > dev.igb.0.mac_stats.mcast_pkts_recvd: 51 > > dev.igb.0.mac_stats.rx_frames_64: 10 > > dev.igb.0.mac_stats.rx_frames_65_127: 1601 > > dev.igb.0.mac_stats.rx_frames_128_255: 53 > > dev.igb.0.mac_stats.rx_frames_256_511: 42 > > dev.igb.0.mac_stats.rx_frames_512_1023: 18 > > dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 > > dev.igb.0.mac_stats.good_octets_recvd: 240948229 > > dev.igb.0.mac_stats.good_octets_txd: 5947150 > > dev.igb.0.mac_stats.total_pkts_txd: 95809 > > dev.igb.0.mac_stats.good_pkts_txd: 92013 > > dev.igb.0.mac_stats.bcast_pkts_txd: 1516 > > dev.igb.0.mac_stats.mcast_pkts_txd: 1817 > > dev.igb.0.mac_stats.tx_frames_64: 90302 > > dev.igb.0.mac_stats.tx_frames_65_127: 1711 > > dev.igb.0.mac_stats.tx_frames_128_255: 0 > > dev.igb.0.mac_stats.tx_frames_256_511: 0 > > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > > dev.igb.0.mac_stats.tso_txd: 0 > > dev.igb.0.mac_stats.tso_ctx_fail: 0 > > dev.igb.0.interrupts.asserts: 5584 > > dev.igb.0.interrupts.rx_pkt_timer: 163411 > > dev.igb.0.interrupts.rx_abs_timer: 163386 > > dev.igb.0.interrupts.tx_pkt_timer: 92013 > > dev.igb.0.interrupts.tx_abs_timer: 0 > > dev.igb.0.interrupts.tx_queue_empty: 92013 > > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > > dev.igb.0.interrupts.rx_desc_min_thresh: 19 > > dev.igb.0.interrupts.rx_overrun: 0 > > dev.igb.0.host.breaker_tx_pkt: 0 > > dev.igb.0.host.host_tx_pkt_discard: 0 > > dev.igb.0.host.rx_pkt: 0 > > dev.igb.0.host.breaker_rx_pkts: 0 > > dev.igb.0.host.breaker_rx_pkt_drop: 0 > > dev.igb.0.host.tx_good_pkt: 0 > > dev.igb.0.host.breaker_tx_pkt_drop: 0 > > dev.igb.0.host.rx_good_bytes: 240948229 > > dev.igb.0.host.tx_good_bytes: 5947150 > > dev.igb.0.host.length_errors: 0 > > dev.igb.0.host.serdes_violation_pkt: 0 > > dev.igb.0.host.header_redir_missed: 0 > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
