Hi Jason
Have you tried:
hw.em.fc_setting="0" (in loader.conf)
ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol
with MSIX and no multiqueue.
Advanced features has always been a source of problem.
It is worth a try and help to narrow down possibilities.
It would also be helpful if you provide 'ifconfig' output
when the problem happens.
And a question: Does interface RX also hangs or it is just TX?
On 10/26/2011 12:25 AM, Jason Wolfe wrote:
On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe<nitrobo...@gmail.com> wrote:
Bumping rx/tx descriptors to 2048 was actually for performance reasons and
not to try to get around the issue. I did some fairly in depth testing and
found under heavy load it performed the best with those settings.
As mentioned on the other thread I'll re enable MSI-X on a few servers here
and collect uptime and the kernel msgbuf in addition. I'll bump the
descriptors down to 512 to try and increase our chances and compile the
driver with EM_MULTIQUEUE also.
Jason
Hi there,
So I have a small pool of server running EM_MULTIQUEUE with lower
descriptors as promised and just received an alert of an event. I have a
fairly large pool of servers on the same hardware running the same OS/driver
sans MSI-X and multiqueue with not a single 'wedge' event in about 2 months
now, and it seems multiqueue has not changed the commonality of the issue.
Here is my loader.conf followed by everything collected:
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.syncache.hashsize="1024"
net.inet.tcp.syncache.bucketlimit="512"
net.inet.tcp.syncache.cachelimit="65536"
net.inet.tcp.hostcache.hashsize="1024"
net.inet.tcp.hostcache.bucketlimit="512"
net.inet.tcp.hostcache.cachelimit="65536"
hw.em.rxd="512"
hw.em.txd="512"
cc_cubic_load="YES"
I bounced em1 because dropped packets incremented 1386169 to 1386355 and the
interface is not incrementing packets out.
1:30PM up 4 days, 6:19, 0 users, load averages: 0.18, 0.38, 0.42
interrupt total rate
irq3: uart1 5816 0
cpu0: timer 736655476 2000
irq256: em0:rx 0 38122306 103
irq257: em0:tx 0 1605535054 4359
irq258: em0:link 1 0
irq259: em1:rx 0 2192460862 5952
irq260: em1:tx 0 1599049303 4341
irq261: em1:link 4172 0
irq262: mps0 212448927 576
cpu2: timer 736647277 2000
cpu3: timer 736647302 2000
cpu1: timer 736647302 2000
Total 8594223798 23333
27653/6022/33675 mbufs in use (current/cache/total)
3054/3196/6250/5700670 mbuf clusters in use (current/cache/total/max)
3054/1041 mbuf+clusters out of packet secondary zone in use (current/cache)
23266/1642/24908/2850335 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
106085K/14465K/120550K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
22 requests for I/O initiated by sendfile
0 calls to protocol drain routines
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
em0 1500<Link#1> 00:25:90:1f:f5:7d 38575296 0 0 6300959828 0 0 706638
em0 1500 fe80:1::225:9 fe80:1::225:90ff: 0 - - 3 - - -
em1 1500<Link#2> 00:25:90:1f:f5:7d 6091053202 22415 0 6327642657 0 0
1386797
em1 1500 fe80:2::225:9 fe80:2::225:90ff: 0 - - 1 - - -
lagg0 1500<Link#5> 00:25:90:1f:f5:7d 6129556798 0 0 12627493094 2093435 0 0
lagg0 1500 69.164.38.0/2 69.164.38.93 5429109508 - - 12630422599 - - -
lagg0 1500 fe80:5::225:9 fe80:5::225:90ff: 12 - - 17 - - -
lagg0 1500 2607:f4e8:310 2607:f4e8:310:12: 13655 - - 13663 - - -
kern.msgbuf:
Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE
Oct 25 13:30:04 cds1043 kernel: em0: hw tdh = 105, hw tdt = 158
Oct 25 13:30:04 cds1043 kernel: em0: hw rdh = 191, hw rdt = 190
Oct 25 13:30:04 cds1043 kernel: em0: Tx Queue Status = 0
Oct 25 13:30:04 cds1043 kernel: em0: TX descriptors avail = 422
Oct 25 13:30:04 cds1043 kernel: em0: Tx Descriptors avail failure = 0
Oct 25 13:30:04 cds1043 kernel: em0: RX discarded packets = 0
Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Check = 192
Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Refresh = 191
Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE
Oct 25 13:30:04 cds1043 kernel: em1: hw tdh = 159, hw tdt = 159
Oct 25 13:30:04 cds1043 kernel: em1: hw rdh = 193, hw rdt = 191
Oct 25 13:30:04 cds1043 kernel: em1: Tx Queue Status = 0
Oct 25 13:30:04 cds1043 kernel: em1: TX descriptors avail = 512
Oct 25 13:30:04 cds1043 kernel: em1: Tx Descriptors avail failure = 0
Oct 25 13:30:04 cds1043 kernel: em1: RX discarded packets = 0
Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Check = 407
Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Refresh = 436
net.inet.ip.intr_queue_maxlen: 512
net.inet.ip.intr_queue_drops: 0
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.0.%parent: pci1
dev.em.0.nvm: -1
dev.em.0.debug: -1
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.rx_processing_limit: 100
dev.em.0.flow_control: 3
dev.em.0.eee_control: 0
dev.em.0.link_irq: 1
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.rx_overruns: 0
dev.em.0.watchdog_timeouts: 0
dev.em.0.device_control: 1074790984
dev.em.0.rx_control: 67141634
dev.em.0.fc_high_water: 18432
dev.em.0.fc_low_water: 16932
dev.em.0.queue0.txd_head: 208
dev.em.0.queue0.txd_tail: 208
dev.em.0.queue0.tx_irq: 1605545961
dev.em.0.queue0.no_desc_avail: 0
dev.em.0.queue0.rxd_head: 348
dev.em.0.queue0.rxd_tail: 347
dev.em.0.queue0.rx_irq: 38122461
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd: 38588018
dev.em.0.mac_stats.good_pkts_recvd: 38588018
dev.em.0.mac_stats.bcast_pkts_recvd: 38572739
dev.em.0.mac_stats.mcast_pkts_recvd: 1868
dev.em.0.mac_stats.rx_frames_64: 38572811
dev.em.0.mac_stats.rx_frames_65_127: 2762
dev.em.0.mac_stats.rx_frames_128_255: 12186
dev.em.0.mac_stats.rx_frames_256_511: 236
dev.em.0.mac_stats.rx_frames_512_1023: 23
dev.em.0.mac_stats.rx_frames_1024_1522: 0
dev.em.0.mac_stats.good_octets_recvd: 2470666426
dev.em.0.mac_stats.good_octets_txd: 8361514233625
dev.em.0.mac_stats.total_pkts_txd: 6301004697
dev.em.0.mac_stats.good_pkts_txd: 6301004695
dev.em.0.mac_stats.bcast_pkts_txd: 80
dev.em.0.mac_stats.mcast_pkts_txd: 2425
dev.em.0.mac_stats.tx_frames_64: 36413025
dev.em.0.mac_stats.tx_frames_65_127: 648631416
dev.em.0.mac_stats.tx_frames_128_255: 7701802
dev.em.0.mac_stats.tx_frames_256_511: 11499983
dev.em.0.mac_stats.tx_frames_512_1023: 56954995
dev.em.0.mac_stats.tx_frames_1024_1522: 5539803478
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 3
dev.em.0.interrupts.rx_pkt_timer: 0
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.tx_abs_timer: 0
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 0
dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3
dev.em.1.%driver: em
dev.em.1.%location: slot=0 function=0
dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.1.%parent: pci2
dev.em.1.nvm: -1
dev.em.1.debug: -1
dev.em.1.rx_int_delay: 0
dev.em.1.tx_int_delay: 66
dev.em.1.rx_abs_int_delay: 66
dev.em.1.tx_abs_int_delay: 66
dev.em.1.rx_processing_limit: 100
dev.em.1.flow_control: 3
dev.em.1.eee_control: 0
dev.em.1.link_irq: 4172
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.dropped: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.rx_overruns: 0
dev.em.1.watchdog_timeouts: 0
dev.em.1.device_control: 1074790984
dev.em.1.rx_control: 67141634
dev.em.1.fc_high_water: 18432
dev.em.1.fc_low_water: 16932
dev.em.1.queue0.txd_head: 159
dev.em.1.queue0.txd_tail: 159
dev.em.1.queue0.tx_irq: 1599049292
dev.em.1.queue0.no_desc_avail: 0
dev.em.1.queue0.rxd_head: 122
dev.em.1.queue0.rxd_tail: 121
dev.em.1.queue0.rx_irq: 2190215040
dev.em.1.mac_stats.excess_coll: 0
dev.em.1.mac_stats.single_coll: 0
dev.em.1.mac_stats.multiple_coll: 0
dev.em.1.mac_stats.late_coll: 0
dev.em.1.mac_stats.collision_count: 0
dev.em.1.mac_stats.symbol_errors: 0
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.defer_count: 0
dev.em.1.mac_stats.missed_packets: 22415
dev.em.1.mac_stats.recv_no_buff: 11223
dev.em.1.mac_stats.recv_undersize: 0
dev.em.1.mac_stats.recv_fragmented: 0
dev.em.1.mac_stats.recv_oversize: 0
dev.em.1.mac_stats.recv_jabber: 0
dev.em.1.mac_stats.recv_errs: 0
dev.em.1.mac_stats.crc_errs: 0
dev.em.1.mac_stats.alignment_errs: 0
dev.em.1.mac_stats.coll_ext_errs: 0
dev.em.1.mac_stats.xon_recvd: 0
dev.em.1.mac_stats.xon_txd: 0
dev.em.1.mac_stats.xoff_recvd: 0
dev.em.1.mac_stats.xoff_txd: 0
dev.em.1.mac_stats.total_pkts_recvd: 6091088292
dev.em.1.mac_stats.good_pkts_recvd: 6091065877
dev.em.1.mac_stats.bcast_pkts_recvd: 38569835
dev.em.1.mac_stats.mcast_pkts_recvd: 1860
dev.em.1.mac_stats.rx_frames_64: 2329378133
dev.em.1.mac_stats.rx_frames_65_127: 2592499514
dev.em.1.mac_stats.rx_frames_128_255: 7898056
dev.em.1.mac_stats.rx_frames_256_511: 15815777
dev.em.1.mac_stats.rx_frames_512_1023: 44494959
dev.em.1.mac_stats.rx_frames_1024_1522: 1100979438
dev.em.1.mac_stats.good_octets_recvd: 2043621877185
dev.em.1.mac_stats.good_octets_txd: 8381779145365
dev.em.1.mac_stats.total_pkts_txd: 6327642657
dev.em.1.mac_stats.good_pkts_txd: 6327642657
dev.em.1.mac_stats.bcast_pkts_txd: 2149
dev.em.1.mac_stats.mcast_pkts_txd: 11
dev.em.1.mac_stats.tx_frames_64: 36904932
dev.em.1.mac_stats.tx_frames_65_127: 662019693
dev.em.1.mac_stats.tx_frames_128_255: 7256854
dev.em.1.mac_stats.tx_frames_256_511: 11840333
dev.em.1.mac_stats.tx_frames_512_1023: 57343575
dev.em.1.mac_stats.tx_frames_1024_1522: 5552277270
dev.em.1.mac_stats.tso_txd: 0
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.interrupts.asserts: 3934
dev.em.1.interrupts.rx_pkt_timer: 3
dev.em.1.interrupts.rx_abs_timer: 0
dev.em.1.interrupts.tx_pkt_timer: 0
dev.em.1.interrupts.tx_abs_timer: 0
dev.em.1.interrupts.tx_queue_empty: 0
dev.em.1.interrupts.tx_queue_min_thresh: 0
dev.em.1.interrupts.rx_desc_min_thresh: 0
dev.em.1.interrupts.rx_overrun: 1
Jason
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
