After changing the following line on our edge Firewalls PC.conf the Centos server that was unusable is now usable. I've done another tcp dump and there are still lot's of TCP ACT DUP's but not as many as there were before,
match on $ExtIf scrub (random-id min-ttl 64 set-tos lowdelay reassemble tcp max-mss 1472) label "Scrubbing" to... match in on $ExtIf scrub (random-id min-ttl 64 set-tos lowdelay reassemble tcp max-mss 1472) label "Scrubbing" I will have to do some reading so see exactly why the above rule is causing issue with Centos VM's but for now everything seems back to normal :>) Keith On Tue, Apr 23, 2013 at 12:11 AM, Keith <ke...@scott-land.net> wrote: > Hi, we recently switched our squid server from a OBSD server on VMware a > Centos server on XEN but there appears to be an issue somewhere between the > centos server and our OBSD Routers (DMZ) or our external OBSD firewalls. > > If I log into the Centos server and run either wget or curl to an > exnternal http server I get a kind of random 1 in 3 chance or it working or > taking upto 30 seconds to complete. I've run tcpdump on the Centos box and > on the router and have imported the results into wireshare and they both > show lots of TCP Dup ACK's as shown below. > > We don't have any issues with any of our other servers that are also on > the same lan as this squid server so I think it's either a Centos, > Centos/Xen, or a OBSD issue. does anyone have any ideas what might be going > on here ? > > This dump was captured on our OBSD router. > > No. Time Source Destination Protocol Length Info > 3917 2.797310 10.0.0.X 20.0.0.X TCP 74 35247 > > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=2936085 > TSecr=0 WS=64 > 3922 2.799411 10.0.0.X 20.0.0.X TCP 66 35247 > > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2936087 TSecr=0 > 3923 2.799543 10.0.0.X 20.0.0.X HTTP 175 GET > / HTTP/1.0 > 3926 2.801331 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 3923#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936089 TSecr=0 > 3927 2.801333 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 3923#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936089 TSecr=0 > 3930 2.802423 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 3923#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936090 TSecr=0 > 3931 2.802425 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 3923#4] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936090 TSecr=0 > 4140 3.002585 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 4142 3.003391 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 4140#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936291 TSecr=0 > 4663 3.410632 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 4665 3.411451 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 4663#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2936699 TSecr=0 > 5538 4.226611 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 5541 4.227445 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 5538#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2937515 TSecr=0 > 9846 5.843961 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 5538#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2939132 TSecr=0 > 9851 5.844811 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 5538#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2939133 TSecr=0 > 9861 5.858633 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 9863 5.859432 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 9861#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2939147 TSecr=0 > 14821 9.122718 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 14823 9.123526 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 14821#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2942411 TSecr=0 > 17858 11.859699 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 14821#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2945148 TSecr=0 > 17863 11.860531 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 14821#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2945148 TSecr=0 > 25393 15.650790 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 25395 15.651626 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 25393#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2948939 TSecr=0 > 45327 23.890899 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 25393#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2957178 TSecr=0 > 48330 25.906963 10.0.0.X 20.0.0.X TCP 74 35248 > > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=2959194 > TSecr=0 WS=64 > 48337 25.908983 10.0.0.X 20.0.0.X TCP 66 35248 > > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2959197 TSecr=0 > 48338 25.909077 10.0.0.X 20.0.0.X HTTP 175 GET > / HTTP/1.0 > 48342 25.911184 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 48338#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959199 TSecr=0 > 48343 25.911186 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 48338#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959199 TSecr=0 > 48346 25.912272 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 48338#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959200 TSecr=0 > 48347 25.912274 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 48338#4] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959200 TSecr=0 > 48788 26.112919 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 48794 26.113718 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 48788#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959401 TSecr=0 > 49385 26.520920 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 49387 26.521745 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 49385#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2959809 TSecr=0 > 50594 27.336952 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 50596 27.337765 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 50594#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2960625 TSecr=0 > 52574 28.921899 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 50594#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2962210 TSecr=0 > 52576 28.922743 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 50594#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2962210 TSecr=0 > 52639 28.968964 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 52641 28.969752 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 52639#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2962257 TSecr=0 > 55547 32.233026 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 55549 32.233851 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 55547#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2965521 TSecr=0 > 59833 34.937494 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 55547#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2968225 TSecr=0 > 59835 34.938503 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 55547#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2968226 TSecr=0 > 66466 38.761131 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 66468 38.761969 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 66466#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2972049 TSecr=0 > 82253 46.859463 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 66466#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2980147 TSecr=0 > 97032 51.906615 10.0.0.X 20.0.0.X TCP 74 35249 > > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=2985194 > TSecr=0 WS=64 > 97034 51.908763 10.0.0.X 20.0.0.X TCP 66 35249 > > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2985196 TSecr=0 > 97035 51.908849 10.0.0.X 20.0.0.X HTTP 175 GET > / HTTP/1.0 > 97038 51.910955 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97035#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985198 TSecr=0 > 97039 51.910957 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97035#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985198 TSecr=0 > 97042 51.912054 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97035#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985199 TSecr=0 > 97043 51.912056 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97035#4] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985199 TSecr=0 > 97301 52.112305 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 97304 52.113105 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97301#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985400 TSecr=0 > 97833 52.520290 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 97835 52.521102 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 97833#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2985808 TSecr=0 > 99547 53.336306 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 99549 53.337113 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 99547#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2986624 TSecr=0 > 103240 54.952961 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 99547#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2988240 TSecr=0 > 103242 54.953807 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 99547#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2988241 TSecr=0 > 103256 54.968334 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 103260 54.969125 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 103256#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2988256 TSecr=0 > 109179 58.232391 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 109181 58.233424 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 109179#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2991521 TSecr=0 > 114534 60.968474 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 109179#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2994256 TSecr=0 > 114536 60.969325 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 109179#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2994256 TSecr=0 > 122863 64.760491 10.0.0.X 20.0.0.X HTTP 175 [TCP > Retransmission] GET / HTTP/1.0 > 122865 64.761316 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 122863#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=2998048 TSecr=0 > 140956 72.999821 10.0.0.X 20.0.0.X TCP 66 [TCP > Dup ACK 122863#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0 > TSval=3006287 TSecr=0 > 149188 77.906440 10.0.0.X 20.0.0.X TCP 74 35250 > > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3011193 > TSecr=0 WS=64 > 149190 77.908726 10.0.0.X 20.0.0.X TCP 66 35250 > > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=3011196 TSecr=0 > 149191 77.908820 10.0.0.X 20.0.0.X HTTP 175 GET > / HTTP/1.0 > 149194 77.910921 10.0.0.X 20.0.0.X TCP 66 35250 > > http [ACK] Seq=110 Ack=1449 Win=17536 Len=0 TSval=3011198 TSecr=1465368908 > 149195 77.910923 10.0.0.X 20.0.0.X TCP 66 35250 > > http [ACK] Seq=110 Ack=1794 Win=20416 Len=0 TSval=3011198 TSecr=1465368908 > 149196 77.912997 10.0.0.X 20.0.0.X TCP 66 35250 > > http [FIN, ACK] Seq=110 Ack=1794 Win=20416 Len=0 TSval=3011200 > TSecr=1465368908 > 149199 77.914014 10.0.0.X 20.0.0.X TCP 66 35250 > > http [ACK] Seq=111 Ack=1795 Win=20416 Len=0 TSval=3011201 TSecr=1465368908 > > > Cheers > Keith
