On Mon, Jul 13, 2015 at 10:52:46PM +0930, Jack Burton wrote:
> >
> > I don't pretend to know httpd (at all), but I'm wondering, what should
> > fstat(1) say, over time, for the httpd processes?
>
> Thanks Tor -- that was exactly the clue I needed to isolate the
> problem.
>
> [snip]
>
> admin talks to a custom FastCGI daemon, which is most likely the culprit
> -- I'll debug it tomorrow.
>
> "portal" (the other HTTPS server) also talks to a (different) custom
> FastCGI daemon, but carries orders of magnitude more traffic and didn't
> have any stale sockets -- so clearly our problem is at the other end of
> admin's FastCGI socket (not with httpd itself). Sorry for the noise.
>
> Ted -- similarly, you may want to look into whatever is at the other end
> of your "server1"'s FastCGI socket. If your issue is the same as ours,
> that's likely where you'll find the cause.
>
I am not sure you should conclude yet. I don't use FastCGI. ;-}
Now, as I write, I have 218 open fd's, compared to the 206 or whatever I had
in my previous post. I've got a few "dangling" :443 streams (the :80 ones
seem to disappear like they should), and then a bunch of these:
www httpd 17244 213* internet stream tcp 0x0 *:0
While I have been writing this, the recent (since this morning) fd's have
looked like this ('$' denotes end of list):
www httpd 17244 206* internet stream tcp 0x0 193.214.208.180:443 <--
193.214.208.185:57311
www httpd 17244 207* internet stream tcp 0x0 *:0
www httpd 17244 208* internet stream tcp 0x0 *:0
www httpd 17244 209* internet stream tcp 0x0 *:0
www httpd 17244 210* internet stream tcp 0x0 *:0
www httpd 17244 211* internet stream tcp 0x0 *:0
www httpd 17244 212* internet stream tcp 0x0 *:0
www httpd 17244 213* internet stream tcp 0x0 *:0
www httpd 17244 214* internet stream tcp 0x0 193.214.208.180:80 <--
66.249.78.231:59307
$
www httpd 17244 206* internet stream tcp 0x0 193.214.208.180:443 <--
193.214.208.185:57311
www httpd 17244 207* internet stream tcp 0x0 *:0
www httpd 17244 208* internet stream tcp 0x0 *:0
www httpd 17244 209* internet stream tcp 0x0 *:0
www httpd 17244 210* internet stream tcp 0x0 *:0
www httpd 17244 211* internet stream tcp 0x0 *:0
www httpd 17244 212* internet stream tcp 0x0 *:0
www httpd 17244 213* internet stream tcp 0x0 *:0
$
Notice how 214 (:80) closed and went away.
A few minutes later, I have these:
www httpd 17244 206* internet stream tcp 0x0 193.214.208.180:443 <--
193.214.208.185:57311
www httpd 17244 207* internet stream tcp 0x0 *:0
www httpd 17244 208* internet stream tcp 0x0 *:0
www httpd 17244 209* internet stream tcp 0x0 *:0
www httpd 17244 210* internet stream tcp 0x0 *:0
www httpd 17244 211* internet stream tcp 0x0 *:0
www httpd 17244 212* internet stream tcp 0x0 *:0
www httpd 17244 213* internet stream tcp 0x0 *:0
www httpd 17244 214* internet stream tcp 0x0 193.214.208.180:443 <--
86.129.139.178:60804
$
www httpd 17244 206* internet stream tcp 0x0 193.214.208.180:443 <--
193.214.208.185:57311
www httpd 17244 207* internet stream tcp 0x0 *:0
www httpd 17244 208* internet stream tcp 0x0 *:0
www httpd 17244 209* internet stream tcp 0x0 *:0
www httpd 17244 210* internet stream tcp 0x0 *:0
www httpd 17244 211* internet stream tcp 0x0 *:0
www httpd 17244 212* internet stream tcp 0x0 *:0
www httpd 17244 213* internet stream tcp 0x0 *:0
www httpd 17244 214* internet stream tcp 0x0 *:0
www httpd 17244 215* internet stream tcp 0x0 *:0
www httpd 17244 216* internet stream tcp 0x0 193.214.208.180:443 <--
86.129.139.178:61345
$
FWIW, the following is a dump from some earlier connections from the same
client (they look too short):
07:23:48.292311 193.214.208.180.443 > 86.129.139.178.51968: S
4293888040:4293888040(0) ack 147006770 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 3> (DF)
0000: 4500 0034 2113 4000 4006 a4f2 c1d6 d0b4 E..4!.@.@.......
0010: 5681 8bb2 01bb cb00 ffef 8828 08c3 2532 V..........(..%2
0020: 8012 4000 377d 0000 0204 05b4 0101 0402 ..@.7}..........
0030: 0103 0303 ....
07:23:48.345674 86.129.139.178.51968 > 193.214.208.180.443: . ack 1 win 16698
(DF)
0000: 4500 0028 5a8a 4000 7206 3987 5681 8bb2 E..(Z.@.r.9.V...
0010: c1d6 d0b4 cb00 01bb 08c3 2532 ffef 8829 ..........%2...)
0020: 5010 413a 7711 0000 dd2d 0000 0000 P.A:w....-....
07:23:48.346721 86.129.139.178.51968 > 193.214.208.180.443: P 1:116(115) ack 1
win 16698 (DF)
0000: 4500 009b 5a8b 4000 7206 3913 5681 8bb2 E...Z.@.r.9.V...
0010: c1d6 d0b4 cb00 01bb 08c3 2532 ffef 8829 ..........%2...)
0020: 5018 413a 0813 0000 1603 0100 6e01 0000 P.A:........n...
0030: 6a03 0155 9e05 48fa 033a 70a9 351e 8015 j..U..H..:p.5...
0040: 97b8 4deb ad29 538c effc 13be 7c2d eea5 ..M..)S.....|-..
0050: c00a d400 0018 002f 0035 0005 000a c009 ......./.5......
0060: c00a c013 c014 0032 0038 0013 0004 0100 .......2.8......
0070: 0029 0000 000e 000c 0000 0962 6f67 7573 .).........bogus
0080: 2e6e 6574 000a 0008 0006 0017 0018 0019 .net............
0090: 000b 0002 0100 ff01 0001 00 ...........
07:23:48.350817 193.214.208.180.443 > 86.129.139.178.51968: P 1:8(7) ack 116
win 2178 (DF)
0000: 4500 002f 368b 4000 4006 8f7f c1d6 d0b4 E../6.@.@.......
0010: 5681 8bb2 01bb cb00 ffef 8829 08c3 25a5 V..........)..%.
0020: 5018 0882 74e0 0000 1503 0100 0202 28 P...t.........(
07:23:48.403677 86.129.139.178.51969 > 193.214.208.180.443: S
3771038199:3771038199(0) win 8192 <mss 1452,nop,wscale 2,nop,nop,sackOK> (DF)
0000: 4500 0034 5a8d 4000 7206 3978 5681 8bb2 E..4Z.@.r.9xV...
0010: c1d6 d0b4 cb01 01bb e0c5 79f7 0000 0000 ..........y.....
0020: 8002 2000 b2e5 0000 0204 05ac 0103 0302 .. .............
0030: 0101 0402 ....
07:23:48.403881 193.214.208.180.443 > 86.129.139.178.51969: S
3568087258:3568087258(0) ack 3771038200 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 3> (DF)
0000: 4500 0034 fd68 4000 4006 c89c c1d6 d0b4 E..4.h@.@.......
0010: 5681 8bb2 01bb cb01 d4ac b0da e0c5 79f8 V.............y.
0020: 8012 4000 0d44 0000 0204 05b4 0101 0402 ..@..D..........
0030: 0103 0303 ....
07:23:48.405379 86.129.139.178.51968 > 193.214.208.180.443: F 116:116(0) ack 8
win 16696 (DF)
0000: 4500 0028 5a8c 4000 7206 3985 5681 8bb2 E..(Z.@.r.9.V...
0010: c1d6 d0b4 cb00 01bb 08c3 25a5 ffef 8830 ..........%....0
0020: 5011 4138 7698 0000 e78b 0000 0000 P.A8v.........
07:23:48.405496 193.214.208.180.443 > 86.129.139.178.51968: . ack 117 win 2178
(DF)
0000: 4500 0028 3f45 4000 4006 86cc c1d6 d0b4 E..(?E@.@.......
0010: 5681 8bb2 01bb cb00 ffef 8830 08c3 25a6 V..........0..%.
0020: 5010 0882 74d9 0000 P...t...
07:23:48.455731 86.129.139.178.51969 > 193.214.208.180.443: . ack 1 win 16698
(DF)
0000: 4500 0028 5a8e 4000 7206 3983 5681 8bb2 E..(Z.@.r.9.V...
0010: c1d6 d0b4 cb01 01bb e0c5 79f8 d4ac b0db ..........y.....
0020: 5010 413a 4cd8 0000 b5c4 0000 0000 P.A:L.........
07:23:48.456175 86.129.139.178.51969 > 193.214.208.180.443: F 1:1(0) ack 1 win
16698 (DF)
0000: 4500 0028 5a8f 4000 7206 3982 5681 8bb2 E..(Z.@.r.9.V...
0010: c1d6 d0b4 cb01 01bb e0c5 79f8 d4ac b0db ..........y.....
0020: 5011 413a 4cd7 0000 d046 0000 0000 P.A:L....F....
07:23:48.456402 193.214.208.180.443 > 86.129.139.178.51969: . ack 2 win 2178
(DF)
0000: 4500 0028 92e2 4000 4006 332f c1d6 d0b4 E..(..@.@.3/....
0010: 5681 8bb2 01bb cb01 d4ac b0db e0c5 79f9 V.............y.
0020: 5010 0882 74d9 0000 P...t...
07:25:48.453812 86.129.139.178.51968 > 193.214.208.180.443: R 117:117(0) ack 8
win 0 (DF)
0000: 4500 0028 5b05 4000 7206 390c 5681 8bb2 E..([.@.r.9.V...
0010: c1d6 d0b4 cb00 01bb 08c3 25a6 ffef 8830 ..........%....0
0020: 5014 0000 b7cc 0000 e4c6 0000 0000 P.............
07:25:48.502315 86.129.139.178.51969 > 193.214.208.180.443: R 2:2(0) ack 1 win
0 (DF)
0000: 4500 0028 5b06 4000 7206 390b 5681 8bb2 E..([.@.r.9.V...
0010: c1d6 d0b4 cb01 01bb e0c5 79f9 d4ac b0db ..........y.....
0020: 5014 0000 8e0d 0000 9c74 0000 0000 P........t....
Kind regards,
Tor
