On Sun, Oct 26, 2014 at 11:05:51AM +0200, Ergin Ozekes wrote: > Hi james; > > In the beginning of my investigation I've set the all interfaces ring > buffer size max value of the ethernet cards ( which is 4078 by using > ethtool -g). And also I've increased the socket buffer size by using kernel > parameters as is below. > net.core.rmem_default=536870912 > net.core.wmem_default=536870912 > net.core.rmem_max=536870912 > net.core.wmem_max=536870912 > net.ipv4.tcp_mem=268435456 268435456 268435456 > net.ipv4.tcp_rmem=268435456 268435456 268435456 > net.ipv4.tcp_wmem=268435456 268435456 268435456
You really don't need to do that, and it could make things worse rather than better. > I've increased the some of the queue size as is below: > net.core.netdev_max_backlog = 300000 > net.core.somaxconn = 65535 > net.ipv4.tcp_max_orphans = 268435456 > net.ipv4.tcp_max_syn_backlog = 268435456 Why raise these so high? > Also kernel shared memory size. > > kernel.shmmax = 16000000000 > kernel.shmall = 4000000000 > > After these setting I saw the increasing "segment retransmited" and > "packets collapsed in receive queue due to low socket buffer" lines on the > "netstat -st" output. Probably because you're running out of RAM. net.ipv4.tcp_rmem, and net.ipv4.tcp_wmem are per connection. autosizing mostly works well, take advantage of it. > I think there are timeout values or other parameter settings which triggers > the above failures. > > Do you have any idea? > Best Regards, > Ergin > > > On Sat, Oct 25, 2014 at 1:46 AM, James Peach <jpe...@apache.org> wrote: > > > > > > On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <ergin.oze...@gmail.com> > > wrote: > > > > > > Hi All; > > > > > > My problem's solution is as below: > > > > > > > > ############################################################################## > > > # Buffer size > > > > > ############################################################################## > > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760 > > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760 > > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760 > > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760 So you're saying that you need 10 megabytes buffer size on input and output to server and client. I can do over 10 gigabit/sec with a single client with 512k buffer easily on LAN. You only need high values if you have a really fast connection with a really high latency. Like 10 megabytes at 100 msec ping would be 100 megabytes/sec to a single client. For normal web loads window sizes never even get very high if you use autosizing as most objects are small etc. > > Great! You might want to check the buffers on your network adaptors as > > well (ethtool -g IIRC) > > > > > > > > Best Regards, > > > Ergin > > > > > > On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <ergin.oze...@gmail.com> > > > wrote: > > > > > >> Hi all; > > >> > > >> In according to my investigation on the ats read > > >> performance,NetHandler::mainNetEvent is the top function. On my system > > >> lots of buffer over run is exist. In my system net input interface has > > 250 > > >> Mbps traffic but on the client output only 125 Mbps is exist. Half of > > the > > >> downloaded content is returning to client. > > >> > > >> Does anyone knows the reason of this case? > > >> And how can I fixe it? > > >> > > >> This is the "perf top" output: > > >> 4.82% libpcre.so.3.13.1 [.] match > > >> > > >> > > >> 3.27% [kernel] [k] native_write_msr_safe > > >> > > >> > > >> 2.09% [kernel] [k] ipt_do_table > > >> > > >> > > >> 1.60% [kernel] [k] __schedule > > >> > > >> > > >> 1.57% [kernel] [k] menu_select > > >> > > >> > > >> 1.43% libc-2.13.so [.] 0x00000000000e92b0 > > >> > > >> > > >> 1.35% [kernel] [k] find_busiest_group > > >> > > >> > > >> 1.16% traffic_server [.] EThread::execute() > > >> > > >> > > >> 1.12% [kernel] [k] copy_user_generic_string > > >> > > >> > > >> 1.10% [kernel] [k] nf_iterate > > >> > > >> > > >> 1.04% [kernel] [k] _raw_spin_lock_irqsave > > >> > > >> > > >> 1.03% [kernel] [k] int_sqrt > > >> > > >> > > >> *1.02% traffic_server [.] NetHandler::mainNetEvent(int, > > >> Event*)* > > >> > > >> 0.96% [bnx2x] [k] bnx2x_rx_int > > >> > > >> > > >> 0.93% [kernel] [k] _raw_spin_lock > > >> > > >> > > >> 0.90% [kernel] [k] htable_selective_cleanup > > >> > > >> > > >> 0.86% [kernel] [k] cpuidle_enter_state > > >> > > >> > > >> 0.83% [kernel] [k] enqueue_task_fair > > >> > > >> > > >> 0.83% [kernel] [k] tcp_packet > > >> > > >> > > >> 0.76% [kernel] [k] apic_timer_interrupt > > >> > > >> > > >> 0.76% [kernel] [k] timerqueue_add > > >> > > >> > > >> 0.76% [kernel] [k] idle_cpu > > >> > > >> > > >> 0.71% [bnx2x] [k] bnx2x_start_xmit > > >> > > >> > > >> 0.67% [kernel] [k] rb_erase > > >> > > >> > > >> *0.64% traffic_server [.] read_from_net(NetHandler*, > > >> UnixNetVConnection*, EThread*) * > > >> > > >> > > >> > > >> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <ergin.oze...@gmail.com> > > >> wrote: > > >> > > >>> Hi all; > > >>> > > >>> My current working network bandwidth is 350 Mbps. > > >>> My netstat -st output prints out below value increasingly. > > >>> > > >>> 1555903887 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555908175 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555912925 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555920054 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555929162 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555938162 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555945682 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555951783 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555959318 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555962474 packets collapsed in receive queue due to low socket > > buffer > > >>> 1555969574 packets collapsed in receive queue due to low socket > > buffer > > >>> > > >>> I've increased to socket buffer size, backlog queue size and memlock > > >>> value was set to unlimited for ats user. How can I fix it completely? > > >>> BTW, How can I increase read from socket(connection) performance of the > > >>> ATS? > > >>> > > >>> > > >>> Best Regards, > > >>> Ergin Ozekes > > >>> > > >>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <ergin.oze...@gmail.com > > > > > >>> wrote: > > >>> > > >>>> Hi all; > > >>>> > > >>>> My current working network bandwidth is 350 Mbps. > > >>>> My netstat -st output prints out below value increasingly. > > >>>> > > >>>> 1555903887 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555908175 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555912925 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555920054 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555929162 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555938162 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555945682 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555951783 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555959318 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555962474 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> 1555969574 packets collapsed in receive queue due to low socket > > >>>> buffer > > >>>> > > >>>> I've increased to socket buffer size, backlog queue size and memlock > > >>>> value was set to unlimited for ats user. How can I fix it completely? > > >>>> > > >>>> Best Regards, > > >>>> Ergin Ozekes > > >>>> > > >>> > > >>> > > >> > > > >
