Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
Hi John On 22/03/2012 14:16, "John Baldwin" wrote: >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: >> Hi List >> >> I've been seeing the following in the messages log of my freebsd syslog >> server for quite some time now: >> >> --- >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> writing; fd='12', error='No buffer space available (55)' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> time_reopen='60' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> writing; fd='13', error='No buffer space available (55)' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> time_reopen='60' >> --- >> >> These happen at a frequency of about 7 per minute on average. See >>attached >> trend graphs for an idea of the volume of traffic we're doing, as well >>as >> the memory and cpu utilisation trends on this server during this period. >> As can be seen from the graphs, load does not seem to be the issue. >> Occasionally during the week, the system freezes and requires a reboot, >>I >> think it's related to the above message, though I'm not sure. >> >> My question is: What does this error mean, and how can I resolve it? >> >> I have tried to frame this as an operating system kernel resource issue, >> and experimented with increasing the freebsd kernel sysctls for UDP >> performance: > >It means that the network driver has "filled" up with packets. Are you >using >igb(4)? We have to crank the number of descriptors assigned to igb to >the max >to workaround this at work (we get DNS timeouts during a simple boot >otherwise). hw.igb.maxtxd is the tunable you would set. The max value >you >can set it to is 4096. This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core (see attached graph), load average is now 1, which I would guess means that the system is now processing a lot more syslog data now that "more packets are making it through the network driver". I'll keep monitoring over a 24 hour period though, to see how effective this is. > >-- >John Baldwin Many Thanks! Traiano > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ifconfig meshforward command bug
On Thu, Mar 22, 2012 at 10:23 PM, Adrian Chadd wrote:
> I've fixed the one that Monthadar has submitted.
>
> Monthadar, would you mind reviewing the rest of the mesh commands and
> see which others need fixing?
from what I can see the bug is for flags that we can either turn on or
off. Another one I found,
static
DECL_CMD_FUNC(set80211meshpeering, val, d)
{
- set80211(s, IEEE80211_IOC_MESH_AP, atoi(val), 0, NULL);
+ set80211(s, IEEE80211_IOC_MESH_AP, d, 0, NULL);
I took a quick look and think this is last one
>
>
> Adrian
thnx
--
Monthadar Al Jaberi
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ifconfig meshforward command bug
On Thu, Mar 22, 2012 at 7:56 PM, Adrian Chadd wrote:
> On 22 March 2012 11:56, Adrian Chadd wrote:
>> On 22 March 2012 11:28, Bernhard Schmidt wrote:
>>
>>> Seems about right, those declared with DEF_CMD() should use d.
>>
>> Shall I commit this?
>
> .. a lot of these mesh functions use DECL_CMD_FUNC(), should they all
> be using d?
No I dont think all of them should use d. If you send a value like for
meshttl for example you should not use d, d is only for enable/disable
flags, for example this is CORRECT:
static
DECL_CMD_FUNC(set80211meshttl, val, d)
{
set80211(s, IEEE80211_IOC_MESH_TTL, atoi(val), 0, NULL);
}
br
>
>
> adrian
--
Monthadar Al Jaberi
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
On Thursday, March 22, 2012 11:00:35 am Traiano Welcome wrote: > Hi John > > > > On 22/03/2012 14:16, "John Baldwin" wrote: > > >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: > >> Hi List > >> > >> I've been seeing the following in the messages log of my freebsd syslog > >> server for quite some time now: > >> > >> --- > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while > >> writing; fd='12', error='No buffer space available (55)' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; > >> time_reopen='60' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while > >> writing; fd='13', error='No buffer space available (55)' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; > >> time_reopen='60' > >> --- > >> > >> These happen at a frequency of about 7 per minute on average. See > >>attached > >> trend graphs for an idea of the volume of traffic we're doing, as well > >>as > >> the memory and cpu utilisation trends on this server during this period. > >> As can be seen from the graphs, load does not seem to be the issue. > >> Occasionally during the week, the system freezes and requires a reboot, > >>I > >> think it's related to the above message, though I'm not sure. > >> > >> My question is: What does this error mean, and how can I resolve it? > >> > >> I have tried to frame this as an operating system kernel resource issue, > >> and experimented with increasing the freebsd kernel sysctls for UDP > >> performance: > > > >It means that the network driver has "filled" up with packets. Are you > >using > >igb(4)? We have to crank the number of descriptors assigned to igb to > >the max > >to workaround this at work (we get DNS timeouts during a simple boot > >otherwise). hw.igb.maxtxd is the tunable you would set. The max value > >you > >can set it to is 4096. > > > Thanks for the clue. It seems igb is indeed being used. However, I not > that when I try to list this tunable, I don't get anything: > > --- > [root@syslog2]# sysctl -a | grep igb > [root@syslog2]# > --- > > > --- > [root@syslog2 /boot]# sysctl hw.igb.maxtxd=4000 > sysctl: unknown oid 'hw.igb.maxtxd' > --- > > > Is this due to the way igb is compiled into the kernel, or am I tuning in > the wrong place :-) You need to put it in /boot/loader.conf and then reboot. Do try using 4096 rather than 4000. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
Hi John On 22/03/2012 19:49, "John Baldwin" wrote: >On Thursday, March 22, 2012 11:00:35 am Traiano Welcome wrote: >> Hi John >> >> >> >> On 22/03/2012 14:16, "John Baldwin" wrote: >> >> >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: >> >> Hi List >> >> >> >> I've been seeing the following in the messages log of my freebsd >>syslog >> >> server for quite some time now: >> >> >> >> --- >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> >> writing; fd='12', error='No buffer space available (55)' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> >> time_reopen='60' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> >> writing; fd='13', error='No buffer space available (55)' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> >> time_reopen='60' >> >> --- >> >> >> >> These happen at a frequency of about 7 per minute on average. See >> >>attached >> >> trend graphs for an idea of the volume of traffic we're doing, as >>well >> >>as >> >> the memory and cpu utilisation trends on this server during this >>period. >> >> As can be seen from the graphs, load does not seem to be the issue. >> >> Occasionally during the week, the system freezes and requires a >>reboot, >> >>I >> >> think it's related to the above message, though I'm not sure. >> >> >> >> My question is: What does this error mean, and how can I resolve it? >> >> >> >> I have tried to frame this as an operating system kernel resource >>issue, >> >> and experimented with increasing the freebsd kernel sysctls for UDP >> >> performance: >> > >> >It means that the network driver has "filled" up with packets. Are you >> >using >> >igb(4)? We have to crank the number of descriptors assigned to igb to >> >the max >> >to workaround this at work (we get DNS timeouts during a simple boot >> >otherwise). hw.igb.maxtxd is the tunable you would set. The max value >> >you >> >can set it to is 4096. >> >> >> Thanks for the clue. It seems igb is indeed being used. However, I not >> that when I try to list this tunable, I don't get anything: >> >> --- >> [root@syslog2]# sysctl -a | grep igb >> [root@syslog2]# >> --- >> >> >> --- >> [root@syslog2 /boot]# sysctl hw.igb.maxtxd=4000 >> sysctl: unknown oid 'hw.igb.maxtxd' > >> --- >> >> >> Is this due to the way igb is compiled into the kernel, or am I tuning >>in >> the wrong place :-) > >You need to put it in /boot/loader.conf and then reboot. Do try using >4096 >rather than 4000. I'm an idiot :-) Looks like the correct driver is bce: --- [root@syslog2]# sysctl dev.bce | grep -v stat dev.bce.0.%desc: Broadcom NetXtreme II BCM5708 1000Base-T (B2) dev.bce.0.%driver: bce dev.bce.0.%location: slot=0 function=0 dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x164c subvendor=0x1028 subdevice=0x01b3 class=0x02 dev.bce.0.%parent: pci7 dev.bce.0.l2fhdr_error_count: 0 dev.bce.0.mbuf_alloc_failed_count: 0 dev.bce.0.fragmented_mbuf_count: 0 dev.bce.0.dma_map_addr_rx_failed_count: 0 dev.bce.0.dma_map_addr_tx_failed_count: 0 dev.bce.0.unexpected_attention_count: 0 dev.bce.0.com_no_buffers: 0 dev.bce.1.%desc: Broadcom NetXtreme II BCM5708 1000Base-T (B2) dev.bce.1.%driver: bce dev.bce.1.%location: slot=0 function=0 dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x164c subvendor=0x1028 subdevice=0x01b3 class=0x02 dev.bce.1.%parent: pci3 dev.bce.1.l2fhdr_error_count: 0 dev.bce.1.mbuf_alloc_failed_count: 0 dev.bce.1.fragmented_mbuf_count: 0 dev.bce.1.dma_map_addr_rx_failed_count: 0 dev.bce.1.dma_map_addr_tx_failed_count: 0 dev.bce.1.unexpected_attention_count: 0 dev.bce.1.com_no_buffers: --- And it looks like the buffer full errors have returned, and the perceived improvement may just have been due to traffic backoff due to the reboot But I can't seem to find any tunables for the bce driver. The man page only indicates one: --- SYSCTL VARIABLES The following variables are available as both sysctl(8) variables and loader(8) tunables: hw.bce.msi_enable Whether or not MSI support is enabled in the driver. The default value is 1. --- The only other solution I can think of is an upgrade. Is there an alternative to this ? > >-- >John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE
On 3/20/2012 2:57 PM, John Baldwin wrote: >>> TX when link becomes active. I've also updated it to fix resume for em >>> and igb to DTRT when buf_ring is used, and to not include old-style start >>> routines at all when using multiq. It is at >>> http://www.freebsd.org/~jhb/patches/e1000_txeof2.patch >> Thank for the patch sirs, so far it does look like it did the trick. >> I'll know for certain here in a few days if I'm still in the clear. >> I'm guessing after it goes through some more testing it'll be too late >> to slip it into 8.3? > > Yes, this is too late for 8.3, but thanks for testing! Hi, Is there a RELENG_8 version of this patch ? I have a server that used to shows this issue quite a bit, but has not since 7.3.2. I would be happy to stress it on the box. The patch above does not apply cleanly due to the netmap diffs, but I can manually merge if thats the only difference. em1: port 0x2000-0x201f mem 0xb410-0xb411,0xb412-0xb4123fff irq 16 at device 0.0 on pci11 em1: Using MSIX interrupts with 3 vectors em1: [ITHREAD] em1: [ITHREAD] em1: [ITHREAD] em1: Ethernet address: 00:15:17:ed:68:a4 em1@pci0:11:0:0:class=0x02 card=0x34ec8086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xb410, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled bar [1c] = type Memory, range 32, base 0xb412, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 0003[140] = Serial 1 001517ed68a4 -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ifconfig meshforward command bug
On Friday 23 March 2012 10:11:07 Monthadar Al Jaberi wrote:
> On Thu, Mar 22, 2012 at 10:23 PM, Adrian Chadd wrote:
> > I've fixed the one that Monthadar has submitted.
> >
> > Monthadar, would you mind reviewing the rest of the mesh commands and
> > see which others need fixing?
>
> from what I can see the bug is for flags that we can either turn on or
> off. Another one I found,
>
> static
> DECL_CMD_FUNC(set80211meshpeering, val, d)
> {
> - set80211(s, IEEE80211_IOC_MESH_AP, atoi(val), 0, NULL);
> + set80211(s, IEEE80211_IOC_MESH_AP, d, 0, NULL);
>
> I took a quick look and think this is last one
Correct, and committed, thanks!
--
Bernhard
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE
On Fri, Mar 23, 2012 at 11:09 AM, Mike Tancsa wrote: > On 3/20/2012 2:57 PM, John Baldwin wrote: TX when link becomes active. I've also updated it to fix resume for em and igb to DTRT when buf_ring is used, and to not include old-style start routines at all when using multiq. It is at http://www.freebsd.org/~jhb/patches/e1000_txeof2.patch >>> Thank for the patch sirs, so far it does look like it did the trick. >>> I'll know for certain here in a few days if I'm still in the clear. >>> I'm guessing after it goes through some more testing it'll be too late >>> to slip it into 8.3? >> >> Yes, this is too late for 8.3, but thanks for testing! > > Hi, > Is there a RELENG_8 version of this patch ? I have a server that used to > shows this issue quite a bit, but has not since 7.3.2. I would be happy > to stress it on the box. The patch above does not apply cleanly due to > the netmap diffs, but I can manually merge if thats the only difference. > > em1: port 0x2000-0x201f mem > 0xb410-0xb411,0xb412-0xb4123fff irq 16 at device 0.0 on pci11 > em1: Using MSIX interrupts with 3 vectors > em1: [ITHREAD] > em1: [ITHREAD] > em1: [ITHREAD] > em1: Ethernet address: 00:15:17:ed:68:a4 > > em1@pci0:11:0:0: class=0x02 card=0x34ec8086 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xb410, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xb412, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 0003[140] = Serial 1 001517ed68a4 > > > -- > --- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, m...@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ Mike, Attached is my patch with the small issues you mention cleaned up. It worked for me against RELENG_8 (8.3-PRERELEASE) as of 4 days ago. On the testing front, I've been stable for those 4 days across the pool of test machines. Prior I couldn't get past 48 hours without an interface 'wedge'. Thanks again! Jason if_em_fix-fbsd83.diff Description: Binary data ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Intel 82574L interface wedging - em7.3.2/8.2-STABLE
On Friday, March 23, 2012 2:09:29 pm Mike Tancsa wrote: > On 3/20/2012 2:57 PM, John Baldwin wrote: > >>> TX when link becomes active. I've also updated it to fix resume for em > >>> and igb to DTRT when buf_ring is used, and to not include old-style start > >>> routines at all when using multiq. It is at > >>> http://www.freebsd.org/~jhb/patches/e1000_txeof2.patch > >> Thank for the patch sirs, so far it does look like it did the trick. > >> I'll know for certain here in a few days if I'm still in the clear. > >> I'm guessing after it goes through some more testing it'll be too late > >> to slip it into 8.3? > > > > Yes, this is too late for 8.3, but thanks for testing! > > Hi, > Is there a RELENG_8 version of this patch ? I have a server that used to > shows this issue quite a bit, but has not since 7.3.2. I would be happy > to stress it on the box. The patch above does not apply cleanly due to > the netmap diffs, but I can manually merge if thats the only difference. I suspect netmap is the only difference. I have not yet tried to make a patch for 8.x. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
