Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-12 Thread Andre Oppermann
Tim Gebbett wrote: Hi Andre, did some careful testing yesterday and last night. I seem to be still hitting an unknown buffer although the probem is much alleviated. The system achieved a 7hour run at 500mbit where ETIMEDOUT occured. I was feeding 11 other streams to the server whos counters sho

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-10 Thread Tim Gebbett
Hi Andre, did some careful testing yesterday and last night. I seem to be still hitting an unknown buffer although the probem is much alleviated. The system achieved a 7hour run at 500mbit where ETIMEDOUT occured. I was feeding 11 other streams to the server whos counters show an uninterrupted

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-09 Thread Andre Oppermann
Tim Gebbett wrote: Hi all, applied the patch, Well before a ETIMEDOUT error occurred (around 60secs), the tcp debug started venting massive quantities of tcp_output error 55 while sending with syncache noise: The error seems to be coming from the interface send queue which hits the limit. If

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Deng XueFeng
hi, applied the patch for 6.2, rebuild & install kernel, but nothing changed. ETIMEDOUT still occur. > Hi Tim, > > looking at the ip_output() path there are some places that can > return ENOBUFS: > > a) interface queue length check > > b) packet filter > > c) destination address rewr

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Tim Gebbett
Hi Andre, Applied the patch, I could not see anything different to the last test. No packet filtering or NAT are enabled, the test is running over a switch. Many thanks - Tim 258/6657/6915 mbufs in use (current/cache/total) 256/1084/1340/25600 mbuf clusters in use (current/cache/total/max) 256/1

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Andre Oppermann
Hi Tim, looking at the ip_output() path there are some places that can return ENOBUFS: a) interface queue length check b) packet filter c) destination address rewrite through NAT d) if_output() call e) IP fragmentation if DF was not set The first one of those is the most likely to be t

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Tim Gebbett
Hi all, applied the patch, Well before a ETIMEDOUT error occurred (around 60secs), the tcp debug started venting massive quantities of tcp_output error 55 while sending with syncache noise: y 8 12:14:26 timtest kernel: :63859 to [192.168.5.40]:80; tcp_output: error 55 whilTeC Ps:e n[d1i9n2g

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Mark Hills
On Thu, 8 May 2008, Andre Oppermann wrote: Deng XueFeng wrote: hi, the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ? The logging function is not (yet) present in RELENG_6. I'll post the patch when I've backported the functionality. However it's an important information t

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-08 Thread Andre Oppermann
Deng XueFeng wrote: hi, the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ? The logging function is not (yet) present in RELENG_6. I'll post the patch when I've backported the functionality. However it's an important information that it happens on 6.2 too. That means the so

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-07 Thread Deng XueFeng
hi, the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ? > I've looked at the code paths again. There are two possibilities: > > a) the mbuf allocator has some anomaly where it rejects memory requests > but doesn't update the statistics (the code is there however). > >

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-07 Thread Andre Oppermann
I've looked at the code paths again. There are two possibilities: a) the mbuf allocator has some anomaly where it rejects memory requests but doesn't update the statistics (the code is there however). b) the error doesn't come from the mbuf allocation but from ip_output() and further

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-05 Thread Deng XueFeng
hi I'am also meet this problem in my mss server(missey streaming server). one encoder push stream to mss, then run 100 client player playing the sream, as the client number increase, mss will occur this error sooner or later like this: I'am using kqueue, and will got a event with EV_EOF and ffl

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-03 Thread Tim Gebbett
Hi Andre, Just to introduce myself, I am now helping Mark Hills with testing. Thank you for your suggestion, here are the results from a similar system (RELENG-7) with increasing kern.ipc.nmbjumbop to 25600. at 1600 streams using approx 340mbit, netstat -m was reporting 12550/250/12800/128

Re: read() returns ETIMEDOUT on steady TCP connection

2008-05-02 Thread Andre Oppermann
Mark Hills wrote: On Wed, 23 Apr 2008, Andre Oppermann wrote: http://people.freebsd.org/~andre/tcp_output-error-log.diff Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1 and report any output. You likely get some (normal) noise from syncache. What we are looking for is r

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-24 Thread Andre Oppermann
Mark Hills wrote: On Wed, 23 Apr 2008, Andre Oppermann wrote: http://people.freebsd.org/~andre/tcp_output-error-log.diff Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1 and report any output. You likely get some (normal) noise from syncache. What we are looking for is r

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-23 Thread Mark Hills
On Wed, 23 Apr 2008, Andre Oppermann wrote: http://people.freebsd.org/~andre/tcp_output-error-log.diff Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1 and report any output. You likely get some (normal) noise from syncache. What we are looking for is reports from tcp_out

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-22 Thread Andre Oppermann
Andre Oppermann wrote: Mark Hills wrote: On Mon, 21 Apr 2008, Andre Oppermann wrote: Mark Hills wrote: On Sun, 20 Apr 2008, Peter Jeremy wrote: I can't explain the problem but it definitely looks like a resource starvation issue within the kernel. I've traced the source of the ETIMEDOUT

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-21 Thread Andre Oppermann
Mark Hills wrote: On Mon, 21 Apr 2008, Andre Oppermann wrote: Mark Hills wrote: On Sun, 20 Apr 2008, Peter Jeremy wrote: I can't explain the problem but it definitely looks like a resource starvation issue within the kernel. I've traced the source of the ETIMEDOUT within the kernel to tc

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-20 Thread Mark Hills
On Mon, 21 Apr 2008, Andre Oppermann wrote: Mark Hills wrote: On Sun, 20 Apr 2008, Peter Jeremy wrote: I can't explain the problem but it definitely looks like a resource starvation issue within the kernel. I've traced the source of the ETIMEDOUT within the kernel to tcp_timer_rexmt() in

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-20 Thread Sten Daniel Soersdal
Mark Hills wrote: On Sun, 20 Apr 2008, Peter Jeremy wrote: Can you give some more detail about your hardware (speed, CPU, available RAM, UP or SMP) and the application (roughly what does the core of the code look like and is it single-threaded/multi-threaded and/or multi-process). The current

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-20 Thread Andre Oppermann
Mark Hills wrote: On Sun, 20 Apr 2008, Peter Jeremy wrote: Can you give some more detail about your hardware (speed, CPU, available RAM, UP or SMP) and the application (roughly what does the core of the code look like and is it single-threaded/multi-threaded and/or multi-process). The current

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-20 Thread Mark Hills
On Sun, 20 Apr 2008, Peter Jeremy wrote: Can you give some more detail about your hardware (speed, CPU, available RAM, UP or SMP) and the application (roughly what does the core of the code look like and is it single-threaded/multi-threaded and/or multi-process). The current test is a Dell 265

Re: read() returns ETIMEDOUT on steady TCP connection

2008-04-19 Thread Peter Jeremy
On Sat, Apr 19, 2008 at 03:27:28PM +0100, Mark Hills wrote: >I'm are having a trouble with TCP connections being dropped with "read: >Operation timed out". What is unusual is that this is happening right in >the middle of sending a steady stream of data with no network congestion. Can you give s

read() returns ETIMEDOUT on steady TCP connection

2008-04-19 Thread Mark Hills
Hello, I'm are having a trouble with TCP connections being dropped with "read: Operation timed out". What is unusual is that this is happening right in the middle of sending a steady stream of data with no network congestion. The system is FreeBSD 7 and a bespoke streaming server with 1Gbit