Re: Reproducible Kernel Panic on 8.1-STABLE [SEC=UNCLASSIFIED]
On 14.10.2010 09:26, Wilkinson, Alex wrote: I have come across a bug that triggers a kernel panic on 8.1-STABLE(r213395) through the use of /usr/ports/sysutils/fusefs-sshfs. Typically i do an sshfs mount as such: #sshfs usern...@hostname:/home/username local_mountpoint/ This mounts the remote filesystem fine. However, when i edit and save a file in say vi on the remote sshfs i get the following panic everytime: Try this out http://www.freebsd.org/cgi/query-pr.cgi?pr=149674 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Reproducible Kernel Panic on 8.1-STABLE [SEC=UNCLASSIFIED]
0n Thu, Oct 14, 2010 at 02:13:27PM +0600, Sergey Nikolenko wrote: >On 14.10.2010 09:26, Wilkinson, Alex wrote: >> I have come across a bug that triggers a kernel panic on 8.1-STABLE(r213395) through the >> use of /usr/ports/sysutils/fusefs-sshfs. Typically i do an sshfs mount as such: >> >> #sshfs usern...@hostname:/home/username local_mountpoint/ >> >> This mounts the remote filesystem fine. However, when i edit and save a file in >> say vi on the remote sshfs i get the following panic everytime: > >Try this out >http://www.freebsd.org/cgi/query-pr.cgi?pr=149674 Yes! GREAT! This patch fixes the kernel panic! Can we get this committed ASAP ? -Alex IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Problem with security log
On Wed, 13 Oct 2010 01:17:58 -0700 Jeremy Chadwick wrote: > There isn't a 100% reliable way to get rid of this problem. I've been > harping about this for years (sorry to sound like a jerk, but this > really is a major problem that keeps coming up and annoys users/admins > to no end. Or the problem might not be so major, it only shows up on the mailing lists about every six months. So maybe those who have the problem learns how to live with it. If it was a major problem, the mailing lists would be flooded. Just my 0.02 eurocents. -- Regards, Torfinn Ingolfsen ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Problem with security log
On Thu, Oct 14, 2010 at 01:46:38PM +0200, Torfinn Ingolfsen wrote: > On Wed, 13 Oct 2010 01:17:58 -0700 > Jeremy Chadwick wrote: > > > There isn't a 100% reliable way to get rid of this problem. I've been > > harping about this for years (sorry to sound like a jerk, but this > > really is a major problem that keeps coming up and annoys users/admins > > to no end. > > Or the problem might not be so major, it only shows up on the mailing > lists about every six months. So maybe those who have the problem learns > how to live with it. > If it was a major problem, the mailing lists would be flooded. > > Just my 0.02 eurocents. I respect your opinion, and you're right, it doesn't come up too often. But I classify this problem as high/severe because it makes a mess of things at the worst time possible -- situations where you absolutely need reliable output due to the nature of the problem you're dealing with (debugging a kernel, figuring out how/why something broke in the kernel, devices complaining about issues, disks reporting problems, etc.). Last time I think this came up was in July or August, and prior to that, March (at least on the lists I'm sub'd to). John Baldwin had some useful things to say back in March about the issue. Relevant thread posts: http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70744.html http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70745.html http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70746.html -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Problem with security log
on 14/10/2010 15:07 Jeremy Chadwick said the following: > On Thu, Oct 14, 2010 at 01:46:38PM +0200, Torfinn Ingolfsen wrote: >> On Wed, 13 Oct 2010 01:17:58 -0700 >> Jeremy Chadwick wrote: >> >>> There isn't a 100% reliable way to get rid of this problem. I've been >>> harping about this for years (sorry to sound like a jerk, but this >>> really is a major problem that keeps coming up and annoys users/admins >>> to no end. >> >> Or the problem might not be so major, it only shows up on the mailing >> lists about every six months. So maybe those who have the problem learns >> how to live with it. >> If it was a major problem, the mailing lists would be flooded. >> >> Just my 0.02 eurocents. > > I respect your opinion, and you're right, it doesn't come up too > often. > > But I classify this problem as high/severe because it makes a mess of > things at the worst time possible -- situations where you absolutely > need reliable output due to the nature of the problem you're dealing > with (debugging a kernel, figuring out how/why something broke in the > kernel, devices complaining about issues, disks reporting problems, > etc.). > > Last time I think this came up was in July or August, and prior to that, > March (at least on the lists I'm sub'd to). John Baldwin had some > useful things to say back in March about the issue. Relevant thread > posts: > > http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70744.html > http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70745.html > http://www.mail-archive.com/freebsd-hack...@freebsd.org/msg70746.html > Just to clarify. This still talks about printing to console. During panics and traps. Because of one logical message being printed with multiple printf calls. And so on. But still this is about printing to console. Which is related. Which is a problem too. The other problem is that all writing to message buffer is always done character by character only. Even when writing special tokens like log levels. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Problem with security log
On Thu, 14 Oct 2010 05:07:48 -0700 Jeremy Chadwick wrote: > But I classify this problem as high/severe because it makes a mess of > things at the worst time possible -- situations where you absolutely > need reliable output due to the nature of the problem you're dealing > with (debugging a kernel, figuring out how/why something broke in the > kernel, devices complaining about issues, disks reporting problems, > etc.). And that I can agree on; the problem is severe, and should be fixed. Hopefully that will happen someday. -- Regards, Torfinn Ingolfsen ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Bogus "igb1: Could not setup receive structures" in 8-STABLE
I've run across a strange problem with the igb driver in 8-STABLE - when I try to do anything with the second igb interface, I get one or more "igb1: Could not setup receive structures" error messages. This can be reproduced as simply as booting in single-user mode with an empty /boot/loader.conf and doing: ifconfig igb0 up ifconfig igb1 up I've tried to track this down, and as far as I can see, this is from some change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I can bring up both interfaces. When I try it with an 8-STABLE kernel, I get the error and igb1 is missing the "RUNNING" flag: igb0: flags=8843 metric 0 mtu 9000 options=1bb ether 00:25:90:xx:xx:bc inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 nd6 options=3 media: Ethernet 1000baseT status: active igb1: flags=8803 metric 0 mtu 9000 options=1bb ether 00:25:90:xx:xx:bd inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid 0x2 nd6 options=3 media: Ethernet 1000baseT status: active I see 3 mbuf_jumbo_page allocation failures from: (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$ ITEM SIZE LIMIT USED FREE REQUESTS FAILURES 64 Bucket:536,0, 263,3, 263, 106 128 Bucket: 1048,0, 523,2, 525, 139 mbuf_jumbo_page: 4096,12800,12307, 493,30343,3 which correspond to the 3 "igb1: Could not setup receive structures" mess- ages. If I try another "ifconfig igb1 up", I get another console message and the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger value, like 15000, I get the same error message when trying to work with the igb1 device, so I don't think it is a "real" error but indicates a problem in the driver. This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at- tached. Note that most of the igb1 values are zero: (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0" dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x0400 class=0x02 dev.igb.1.%parent: pci1 dev.igb.1.nvm: -1 dev.igb.1.flow_control: 3 dev.igb.1.enable_aim: 1 dev.igb.1.rx_processing_limit: 100 dev.igb.1.link_irq: 1 dev.igb.1.device_control: 13632065 dev.igb.1.extended_int_mask: 2147483648 dev.igb.1.fc_high_water: 47488 dev.igb.1.fc_low_water: 47472 (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0" dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 subdevice=0x0400 class=0x02 dev.igb.0.%parent: pci1 dev.igb.0.nvm: -1 dev.igb.0.flow_control: 3 dev.igb.0.enable_aim: 1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.link_irq: 4 dev.igb.0.device_control: 1087373889 dev.igb.0.rx_control: 67338274 dev.igb.0.interrupt_mask: 4 dev.igb.0.extended_int_mask: 2147484671 dev.igb.0.fc_high_water: 47488 dev.igb.0.fc_low_water: 47472 dev.igb.0.queue0.txd_head: 823 dev.igb.0.queue0.txd_tail: 823 dev.igb.0.queue0.tx_packets: 1402 dev.igb.0.queue0.rxd_head: 319 dev.igb.0.queue0.rxd_tail: 318 dev.igb.0.queue0.rx_packets: 319 dev.igb.0.queue0.rx_bytes: 69075 dev.igb.0.queue1.txd_head: 2 dev.igb.0.queue1.txd_tail: 2 dev.igb.0.queue1.tx_packets: 1 dev.igb.0.queue1.rxd_head: 52 dev.igb.0.queue1.rxd_tail: 51 dev.igb.0.queue1.rx_packets: 52 dev.igb.0.queue1.rx_bytes: 6369 dev.igb.0.queue2.txd_head: 27 dev.igb.0.queue2.txd_tail: 27 dev.igb.0.queue2.tx_packets: 13 dev.igb.0.queue2.rxd_head: 72 dev.igb.0.queue2.rxd_tail: 71 dev.igb.0.queue2.rx_packets: 72 dev.igb.0.queue2.rx_bytes: 8789 dev.igb.0.queue3.txd_head: 177 dev.igb.0.queue3.txd_tail: 177 dev.igb.0.queue3.tx_packets: 64 dev.igb.0.queue3.rxd_head: 88 dev.igb.0.queue3.rxd_tail: 87 dev.igb.0.queue3.rx_packets: 88 dev.igb.0.queue3.rx_bytes: 16454 dev.igb.0.queue4.rxd_head: 21 dev.igb.0.queue4.rxd_tail: 20 dev.igb.0.queue4.rx_packets: 21 dev.igb.0.queue4.rx_bytes: 3547 dev.igb.0.queue5.txd_head: 19 dev.igb.0.queue5.txd_tail: 19 dev.igb.0.queue5.tx_packets: 9 dev.igb.0.queue5.rxd_head: 120 dev.igb.0.queue5.rxd_tail: 119 dev.igb.0.queue5.rx_packets: 120 dev.igb.0.queue5.rx_bytes: 35518 dev.igb.0.queue6.txd_head: 21 dev.igb.0.queue6.txd_tail: 21 dev.igb.0.queue6.tx_packets: 10 dev.igb.0.queue6.rxd_head: 21 dev.igb.0.queue6.rxd_tail: 20 dev.igb.0.queue6.rx_packets: 21 dev.igb.0.queue6.rx_bytes: 2876 dev.igb.0.queue7.txd_head: 100 dev.igb.0.queue7.txd_tail: 100 dev.igb.0.queue7.tx_packets: 50 dev.igb.0.queue
Re: Bogus "igb1: Could not setup receive structures" in 8-STABLE
The problem is mbuf resources, the driver is autoconfiguring the number of queues based on the number of cores, on newer systems with lots of them this is outstripping the mbuf resource pool. I have decided to hard limit the queues to 8, you can fix the number manually by searching for num_queues in if_igb.c and setting it to something other than 0 for now. I am at work on a number of issues with igb and em right now which is why there has not been an MFC yet. Questions to me, Jack On Thu, Oct 14, 2010 at 3:26 PM, Terry Kennedy wrote: > I've run across a strange problem with the igb driver in 8-STABLE - when > I try to do anything with the second igb interface, I get one or more > "igb1: > Could not setup receive structures" error messages. > > This can be reproduced as simply as booting in single-user mode with an > empty /boot/loader.conf and doing: > >ifconfig igb0 up >ifconfig igb1 up > > I've tried to track this down, and as far as I can see, this is from some > change introduced between 8.1-RELEASE (igb 1.9.5) and the current 8-STABLE > (igb 2.0.1). When I try it when booting from the 8.1-RELEASE amd64 DVD, I > can bring up both interfaces. When I try it with an 8-STABLE kernel, I get > the error and igb1 is missing the "RUNNING" flag: > > igb0: flags=8843 metric 0 mtu 9000 > > options=1bb >ether 00:25:90:xx:xx:bc >inet6 fe80::225:90ff:fe02:xxbc%igb0 prefixlen 64 scopeid 0x1 >nd6 options=3 >media: Ethernet 1000baseT >status: active > igb1: flags=8803 metric 0 mtu 9000 > > options=1bb >ether 00:25:90:xx:xx:bd >inet6 fe80::225:90ff:fe02:xxbd%igb1 prefixlen 64 tentative scopeid > 0x2 >nd6 options=3 >media: Ethernet 1000baseT >status: active > > I see 3 mbuf_jumbo_page allocation failures from: > > (1:20) rz1m:/sys/dev/e1000# vmstat -z | grep -v 0\$ > ITEM SIZE LIMIT USED FREE REQUESTS > FAILURES > > 64 Bucket:536,0, 263,3, 263, > 106 > 128 Bucket: 1048,0, 523,2, 525, > 139 > mbuf_jumbo_page: 4096,12800,12307, 493,30343, >3 > > which correspond to the 3 "igb1: Could not setup receive structures" mess- > ages. If I try another "ifconfig igb1 up", I get another console message > and > the counter goes to 4. If I bump the kern.ipc.nmbjumbop sysctl to a larger > value, like 15000, I get the same error message when trying to work with > the > igb1 device, so I don't think it is a "real" error but indicates a problem > in the driver. > > This is on a Supermicro X8DTH-iF, BIOS 2.0a (latest) with a dual on-board > 82576. The dev.igb sysctl's for the two ports (excluding 0 values) are at- > tached. Note that most of the igb1 values are zero: > > (0:34) rz1m:/sys/dev/e1000# sysctl -a | grep igb.1 | grep -v ": 0" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x02 > dev.igb.1.%parent: pci1 > dev.igb.1.nvm: -1 > dev.igb.1.flow_control: 3 > dev.igb.1.enable_aim: 1 > dev.igb.1.rx_processing_limit: 100 > dev.igb.1.link_irq: 1 > dev.igb.1.device_control: 13632065 > dev.igb.1.extended_int_mask: 2147483648 > dev.igb.1.fc_high_water: 47488 > dev.igb.1.fc_low_water: 47472 > > (0:35) rz1m:/sys/dev/e1000# sysctl -a | grep igb.0 | grep -v ": 0" > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.1 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 > subdevice=0x0400 class=0x02 > dev.igb.0.%parent: pci1 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.device_control: 1087373889 > dev.igb.0.rx_control: 67338274 > dev.igb.0.interrupt_mask: 4 > dev.igb.0.extended_int_mask: 2147484671 > dev.igb.0.fc_high_water: 47488 > dev.igb.0.fc_low_water: 47472 > dev.igb.0.queue0.txd_head: 823 > dev.igb.0.queue0.txd_tail: 823 > dev.igb.0.queue0.tx_packets: 1402 > dev.igb.0.queue0.rxd_head: 319 > dev.igb.0.queue0.rxd_tail: 318 > dev.igb.0.queue0.rx_packets: 319 > dev.igb.0.queue0.rx_bytes: 69075 > dev.igb.0.queue1.txd_head: 2 > dev.igb.0.queue1.txd_tail: 2 > dev.igb.0.queue1.tx_packets: 1 > dev.igb.0.queue1.rxd_head: 52 > dev.igb.0.queue1.rxd_tail: 51 > dev.igb.0.queue1.rx_packets: 52 > dev.igb.0.queue1.rx_bytes: 6369 > dev.igb.0.queue2.txd_head: 27 > dev.igb.0.queue2.txd_tail: 27 > dev.igb.0.queue2.tx_packets: 13 > dev.igb.0.queue2.rxd_head: 72 > dev.igb.0.queue2.rxd_tail: 71 > dev.igb.0.queue2.rx_packets: 72 > dev.igb.0.queue2.rx_bytes: 8789 > dev.igb.0.queue3.txd_head: 177 > dev.igb.0.queue3.txd_tail: 177 > dev.igb.0.queue3.tx_packets: 64 > dev.igb.0.queue3.rxd_head: 88 > dev.igb.0.queue3.rxd_tail: 87 >
Re: Reproducible Kernel Panic on 8.1-STABLE [SEC=UNCLASSIFIED]
0n Thu, Oct 14, 2010 at 04:51:10PM +0800, Wilkinson, Alex wrote: >0n Thu, Oct 14, 2010 at 02:13:27PM +0600, Sergey Nikolenko wrote: > >>On 14.10.2010 09:26, Wilkinson, Alex wrote: >>> I have come across a bug that triggers a kernel panic on 8.1-STABLE(r213395) through the >>> use of /usr/ports/sysutils/fusefs-sshfs. Typically i do an sshfs mount as such: >>> >>> #sshfs usern...@hostname:/home/username local_mountpoint/ >>> >>> This mounts the remote filesystem fine. However, when i edit and save a file in >>> say vi on the remote sshfs i get the following panic everytime: >> >>Try this out >>http://www.freebsd.org/cgi/query-pr.cgi?pr=149674 > >Yes! GREAT! This patch fixes the kernel panic! Can we get this committed ASAP ? Committed! |--+--+--| | [ 12:44 beat ] Original commit message # | | | | | | | | ports/sysutils/fusefs-kmod/Makefile 1.31 diff | | | | ports/sysutils/fusefs-kmod/files/patch-fuse_module__fuse_main.c 1.1 | | | | | | | | - Fix panic on FreeBSD 8.x and newer | | | | - Bump PORTREVISION | | | | | | | | PR: ports/149674 | | | | Submitted by: Dmitrij Tejblum | | | | Approved by:Anish Mistry (maintainer) | | | |--+--+--| -Alex IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Bogus "igb1: Could not setup receive structures" in 8-STABLE
> The problem is mbuf resources, the driver is autoconfiguring the number of > queues based on the number of cores, on newer systems with lots of them > this is outstripping the mbuf resource pool. That would make sense, as these systems have 16 cores (dual E5520's). > I have decided to hard limit the queues to 8, you can fix the number > manually > by searching for num_queues in if_igb.c and setting it to something other > than > 0 for now. I changed it to 8, and saw the same problem. I noted that the igb boot messages changed from: Oct 14 18:28:02 rz1m kernel: igb0: Using MSIX interrupts with 10 vectors Oct 14 18:28:02 rz1m kernel: igb1: Using MSIX interrupts with 10 vectors to: Oct 14 21:53:44 rz1m kernel: igb0: Using MSIX interrupts with 9 vectors Oct 14 21:53:44 rz1m kernel: igb1: Using MSIX interrupts with 9 vectors So I dropped the value to 3 (on the assumption that the system uses one more than the specified value per interface), and got: igb0: Using MSIX interrupts with 4 vectors igb1: Using MSIX interrupts with 4 vectors and both igb interfaces came up. I didn't try to find the maximum number of queues that would work. > I am at work on a number of issues with igb and em right now which is why > there has not been an MFC yet. Understood. Thanks for the quick response and workaround. Terry Kennedy http://www.tmk.com te...@tmk.com New York, NY USA ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
