Charles - thank you for your excellent investigation! I'm pretty sure
that my colo provider isn't running a firewall (I've asked them not to,
anyhow). I am running IPFW on that box, with the standard "allow tcp
from any to any established" followed by the "allow tcp any to my_ip 80
setup". I've done that on other servers without it being a problem like
this. I'm going to have the colo double check for router acl's or
something like that in the morning.
Since this is such an intermittent problem, I can't yet say that it's
fixed, but I ran with the "disks being idled" theory and wrote a small
script that creates a file and deletes a file every minute, and since
that's been running, I've not seeing the issue repeat - but then this is
not a very repeatable problem.
Thanks again for your great assistance.
Jerry
Charles Swiger wrote:
On Feb 13, 2006, at 3:12 PM, Jerry Bell wrote:
I didn't want to spam the link out, but it's www.musiclodge.com. I will
gather the capture data from working and non working sessions and
send it
out.
Well, I can confirm the behavior you've described.
It looks somewhat like a stateful firewall or is in the way and is
generating an RST, even while your webserver tries to generate a
response. However, once the firewall sees the outbound traffic, it
seems to create a dynamic rule which lets the traffic from subsequent
connections through:
5-pan# tcpdump -tnXs 0 host www.musiclodge.com
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
IP 199.103.21.238.50740 > 63.175.100.44.80: S 2282569549:2282569549(0)
win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 1159441862 0>
0x0000: 4510 003c 4653 4000 4006 7328 c767 15ee
E..<[EMAIL PROTECTED]@.s(.g..
0x0010: 3faf 642c c634 0050 880d 3f4d 0000 0000
?.d,.4.P..?M....
0x0020: a002 ffff 815f 0000 0204 05b4 0103 0300
....._..........
0x0030: 0101 080a 451b adc6 0000 0000 ....E.......
IP 63.175.100.44.80 > 199.103.21.238.50740: S 2634350592:2634350592(0)
ack 2282569550 win 65535
0x0000: 4500 0028 0000 4000 2506 d49f 3faf 642c
E..([EMAIL PROTECTED],
0x0010: c767 15ee 0050 c634 9d05 0000 880d 3f4e
.g...P.4......?N
0x0020: 5012 ffff 03bc 0000 0000 0000 0000 1b60
P..............`
0x0030: 2678 &x
IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 1 win 65535
0x0000: 4510 0028 4655 4000 4006 733a c767 15ee
E..([EMAIL PROTECTED]@.s:.g..
0x0010: 3faf 642c c634 0050 880d 3f4e 9d05 0001
?.d,.4.P..?N....
0x0020: 5010 ffff 03bd 0000 P.......
3-way handshake is completed here, next traffic should be from my
machine making the "GET /", request, but instead your machine sends
another ACK:
IP 63.175.100.44.80 > 199.103.21.238.50740: S 2238145710:2238145710(0)
ack 2282569550 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp
1453026167 1159441862>
0x0000: 4500 003c 57fa 4000 3206 6f91 3faf 642c
E..<[EMAIL PROTECTED],
0x0010: c767 15ee 0050 c634 8567 64ae 880d 3f4e
.g...P.4.gd...?N
0x0020: a012 ffff 9cdb 0000 0204 05b4 0103 0301
................
0x0030: 0101 080a 569b 6b77 451b adc6 9345 1153
....V.kwE....E.S
Interesting that the previous ack had no TCP options set, whereas this
one does include a timestamp in response.
IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 396204883 win 65535
<nop,nop,timestamp 1159441863 1453026167>
0x0000: 4510 0034 4656 4000 4006 732d c767 15ee
[EMAIL PROTECTED]@.s-.g..
0x0010: 3faf 642c c634 0050 880d 3f4e 9d05 0001
?.d,.4.P..?N....
0x0020: 8010 ffff 8157 0000 0101 080a 451b adc7
.....W......E...
0x0030: 569b 6b77 V.kw
Where did sequence # 396204883 come from? And your side follows up
with a pair of connection resets, and a normal ACK packet, too.
IP 63.175.100.44.80 > 199.103.21.238.50740: R 2634350593:2634350593(0)
win 0
0x0000: 4500 0028 b6f6 4000 3206 10a9 3faf 642c
E..([EMAIL PROTECTED],
0x0010: c767 15ee 0050 c634 9d05 0001 0000 0000
.g...P.4........
0x0020: 5004 0000 cb24 0000 0000 0000 0000 f3fa
P....$..........
0x0030: 5489 T.
IP 63.175.100.44.80 > 199.103.21.238.50740: R 2634350593:2634350593(0)
win 0
0x0000: 4500 0028 4bfc 4000 3206 7ba3 3faf 642c
E..([EMAIL PROTECTED],
0x0010: c767 15ee 0050 c634 9d05 0001 0000 0000
.g...P.4........
0x0020: 5004 0000 cb24 0000 0000 0000 0000 abb8
P....$..........
0x0030: c9be ..
IP 63.175.100.44.80 > 199.103.21.238.50740: S 2238145710:2238145710(0)
ack 2282569550 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp
1453026467 1159441862>
0x0000: 4500 003c 3a9d 4000 3206 8cee 3faf 642c
E..<:[EMAIL PROTECTED],
0x0010: c767 15ee 0050 c634 8567 64ae 880d 3f4e
.g...P.4.gd...?N
0x0020: a012 ffff 9baf 0000 0204 05b4 0103 0301
................
0x0030: 0101 080a 569b 6ca3 451b adc6 bdd6 d7c9
....V.l.E.......
...and my side closes, too. Something is badly confused.
IP 199.103.21.238.50740 > 63.175.100.44.80: R 2282569550:2282569550(0)
win 0
0x0000: 4500 0028 465a 4000 4006 7345 c767 15ee
E..([EMAIL PROTECTED]@.sE.g..
0x0010: 3faf 642c c634 0050 880d 3f4e 0000 0000
?.d,.4.P..?N....
0x0020: 5004 0000 a0cf 0000 P.......
-------------------
When I repeat the connection attempt a few seconds later:
IP 199.103.21.238.50743 > 63.175.100.44.80: S 262625798:262625798(0)
win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 1159442517 0>
0x0000: 4510 003c 46c8 4000 4006 72b3 c767 15ee
E..<[EMAIL PROTECTED]@.r..g..
0x0010: 3faf 642c c637 0050 0fa7 5a06 0000 0000
?.d,.7.P..Z.....
0x0020: a002 ffff 815f 0000 0204 05b4 0103 0300
....._..........
0x0030: 0101 080a 451b b055 0000 0000 ....E..U....
IP 63.175.100.44.80 > 199.103.21.238.50743: S 362624500:362624500(0)
ack 262625799 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp
1453058903 1159442517>
0x0000: 4500 003c e034 4000 3206 e756 3faf 642c
E..<[EMAIL PROTECTED],
0x0010: c767 15ee 0050 c637 159d 35f4 0fa7 5a07
.g...P.7..5...Z.
0x0020: a012 ffff 169b 0000 0204 05b4 0103 0301
................
0x0030: 0101 080a 569b eb57 451b b055 55d6 9ceb
....V..WE..UU...
IP 199.103.21.238.50743 > 63.175.100.44.80: . ack 1 win 65535
<nop,nop,timestamp 1159442517 1453058903>
0x0000: 4510 0034 46c9 4000 4006 72ba c767 15ee
[EMAIL PROTECTED]@.r..g..
0x0010: 3faf 642c c637 0050 0fa7 5a07 159d 35f5
?.d,.7.P..Z...5.
0x0020: 8010 ffff 8157 0000 0101 080a 451b b055
.....W......E..U
0x0030: 569b eb57 V..W
3-way handshake finishes OK, this time your initial ACK is OK and
includes a timestamp back.
IP 199.103.21.238.50743 > 63.175.100.44.80: P 1:17(16) ack 1 win 65535
<nop,nop,timestamp 1159442525 1453058903>
0x0000: 4510 0044 46cd 4000 4006 72a6 c767 15ee
[EMAIL PROTECTED]@.r..g..
0x0010: 3faf 642c c637 0050 0fa7 5a07 159d 35f5
?.d,.7.P..Z...5.
0x0020: 8018 ffff 8167 0000 0101 080a 451b b05d
.....g......E..]
0x0030: 569b eb57 4745 5420 2f20 4854 5450 2f31
V..WGET./.HTTP/1
0x0040: 2e30 0d0a .0..
Here was my GET, which you then ACK....
IP 63.175.100.44.80 > 199.103.21.238.50743: . ack 17 win 33304
<nop,nop,timestamp 1453059309 1159442525>
0x0000: 4500 0034 052a 4000 3206 c269 3faf 642c
[EMAIL PROTECTED],
0x0010: c767 15ee 0050 c637 159d 35f5 0fa7 5a17
.g...P.7..5...Z.
0x0020: 8010 8218 be99 0000 0101 080a 569b eced
............V...
0x0030: 451b b05d db4e 4827 E..].NH'
IP 199.103.21.238.50743 > 63.175.100.44.80: P 17:19(2) ack 1 win 65535
<nop,nop,timestamp 1159442526 1453059309>
0x0000: 4510 0036 46ce 4000 4006 72b3 c767 15ee
[EMAIL PROTECTED]@.r..g..
0x0010: 3faf 642c c637 0050 0fa7 5a17 159d 35f5
?.d,.7.P..Z...5.
0x0020: 8018 ffff 8159 0000 0101 080a 451b b05e
.....Y......E..^
0x0030: 569b eced 0d0a V.....
...followed by a correct data response.
IP 63.175.100.44.80 > 199.103.21.238.50743: . 1:1449(1448) ack 19 win
33304 <nop,nop,timestamp 1453059329 1159442526>
0x0000: 4500 05dc c865 4000 3206 f985 3faf 642c
[EMAIL PROTECTED],
0x0010: c767 15ee 0050 c637 159d 35f5 0fa7 5a19
.g...P.7..5...Z.
0x0020: 8010 8218 2e5f 0000 0101 080a 569b ed01
....._......V...
0x0030: 451b b05e 4854 5450 2f31 2e31 2032 3030
E..^HTTP/1.1.200
0x0040: 204f 4b0d 0a44 6174 653a 204d 6f6e 2c20
.OK..Date:.Mon,.
[ ... ]
Check your firewall, or see whether your ISP or whoever has put a HTTP
reverse proxy in place which is breaking these connections.
---Chuck
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
