Hi Patrick,
Few month back, even I got the same error. At that time
I googled the internet
and came across this
https://issues.apache.org/bugzilla/show_bug.cgi?id=37770
I think (but not sure) this problem occurs because httpd makes long
lived connection to the
downstream application server. Do you have firewall between httpd and
application server?
Try making it non persistent with disablereuse on in
proxypass, still you get errors? of course it may degrade your
performance, so be careful
Prashant Keshvani
www.merceworld.com
mumbai
Patrick Moriarty wrote:
Hi,
We've deployed Apache 2.2 as a reverse ssl proxy in front of
Jetty-6.1.11. The number of users has increaded in the last couple of
weeks and we are now seeing proxy errors in the Apache error log 3-5
times daily:
[Wed Jul 29 10:45:40 2009] [error] [client xxx.xxx.xxx.xxx] proxy:
error reading status line from remote server 172.16.101.156, referer:
https://xxx.xxxxx.com/app/xxx
[Wed Jul 29 10:45:40 2009] [error] [client xxx.xxx.xxx.xxx] proxy:
Error reading from remote server returned by /xxx,
referer:https://xxx.xxxxx.com/app/xxx
The request in question doesn't appear in Jetty's request log. We
captured the TCP stream using Wireshark and found that the request is
forwarded by Apache but coincides with a FIN,ACK from Jetty to close
the connection because 30 seconds have passed without activity. Jetty
responds to the request with a RST, but Apache continues to use the
connection. Here is the entire connection from Jetty's point of view:
No. Time Source Destination
Dest Port Protocol Info
21379 2009-07-29 10:44:40.685505 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [SYN] Seq=0
Win=5840 Len=0 MSS=1460 TSV=4182231540 TSER=0 WS=7
21380 2009-07-29 10:44:40.685517 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=4182015031 TSER=4182231540
WS=7
21381 2009-07-29 10:44:40.685624 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1
Ack=1 Win=5888 Len=0 TSV=4182231541 TSER=4182015031
21382 2009-07-29 10:44:40.685868 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1
(application/x-www-form-urlencoded)
21383 2009-07-29 10:44:40.685876 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [ACK] Seq=1
Ack=1173 Win=8192 Len=0 TSV=4182015031 TSER=4182231541
21384 2009-07-29 10:44:40.736972 172.16.101.156
172.16.101.155 47295 HTTP HTTP/1.1 200 OK
[Unreassembled Packet [incorrect TCP checksum]]
21385 2009-07-29 10:44:40.736987 172.16.101.156
172.16.101.155 47295 HTTP Continuation or non-HTTP
traffic
21386 2009-07-29 10:44:40.737323 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=1173 Ack=1449 Win=8832 Len=0 TSV=4182231592 TSER=4182015082
21387 2009-07-29 10:44:40.737343 172.16.101.156
172.16.101.155 47295 HTTP Continuation or non-HTTP
traffic
21388 2009-07-29 10:44:40.737579 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=1173 Ack=2897 Win=11648 Len=0 TSV=4182231592 TSER=4182015082
21389 2009-07-29 10:44:40.737584 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=1173 Ack=4345 Win=14592 Len=0 TSV=4182231592 TSER=4182015082
21390 2009-07-29 10:44:40.737605 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=1173 Ack=4634 Win=17536 Len=0 TSV=4182231593 TSER=4182015083
21391 2009-07-29 10:44:40.738115 172.16.101.156
172.16.101.155 47295 HTTP Continuation or non-HTTP
traffic
21392 2009-07-29 10:44:40.738210 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=1173 Ack=4639 Win=17536 Len=0 TSV=4182231593 TSER=4182015084
21519 2009-07-29 10:44:48.710869 172.16.101.155
172.16.101.156 8080 HTTP GET /xxx/xxxx HTTP/1.1
21520 2009-07-29 10:44:48.710876 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [ACK]
Seq=4639 Ack=2081 Win=10496 Len=0 TSV=4182023056 TSER=4182239565
21521 2009-07-29 10:44:48.714462 172.16.101.156
172.16.101.155 47295 HTTP HTTP/1.1 304
21522 2009-07-29 10:44:48.714580 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=2081 Ack=4716 Win=17536 Len=0 TSV=4182239569 TSER=4182023060
21578 2009-07-29 10:45:09.878681 172.16.101.155
172.16.101.156 8080 TCP [TCP segment of a reassembled
PDU]
21579 2009-07-29 10:45:09.878794 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx.xxxx HTTP/1.1
(application/x-www-form-urlencoded)
21580 2009-07-29 10:45:09.878803 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [ACK]
Seq=4716 Ack=4862 Win=16384 Len=0 TSV=4182044223 TSER=4182260733
21581 2009-07-29 10:45:11.237149 172.16.101.156
172.16.101.155 47295 HTTP HTTP/1.1 302 Found
21582 2009-07-29 10:45:11.237304 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [ACK]
Seq=4862 Ack=5013 Win=20352 Len=0 TSV=4182262091 TSER=4182045582
21701 2009-07-29 10:45:40.877019 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [FIN, ACK]
Seq=5013 Ack=4862 Win=16384 Len=0 TSV=4182075221 TSER=4182262091
21702 2009-07-29 10:45:40.880273 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx.xxxx HTTP/1.1
(application/x-www-form-urlencoded)
21703 2009-07-29 10:45:40.880305 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [RST]
Seq=5014 Win=0 Len=0
21704 2009-07-29 10:45:40.880312 172.16.101.155
172.16.101.156 8080 TCP 47295 > http-alt [FIN, ACK]
Seq=6107 Ack=5014 Win=20352 Len=0 TSV=4182291733 TSER=4182075221
21705 2009-07-29 10:45:40.880317 172.16.101.156
172.16.101.155 47295 TCP http-alt > 47295 [RST]
Seq=5014 Win=0 Len=0
and here's another connection with the same error but a slightly
different conclusion:
No. Time Source Destination
Dest Port Protocol Info
259350 2009-07-28 12:25:34.242754 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [SYN] Seq=0
Win=5840 Len=0 MSS=1460 TSV=4101887388 TSER=0 WS=7
259351 2009-07-28 12:25:34.242767 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=4101671152 TSER=4101887388
WS=7
259352 2009-07-28 12:25:34.242856 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=1
Ack=1 Win=5888 Len=0 TSV=4101887388 TSER=4101671152
...
306028 2009-07-28 12:32:28.458704 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1
(application/x-www-form-urlencoded)
306038 2009-07-28 12:32:28.471810 172.16.101.156
172.16.101.155 42530 HTTP HTTP/1.1 200 OK
(application/json)
306039 2009-07-28 12:32:28.471961 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=34931 Ack=237709 Win=81280 Len=0 TSV=4102301605 TSER=4102085368
306040 2009-07-28 12:32:28.472982 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
306041 2009-07-28 12:32:28.473078 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=34931 Ack=237714 Win=81280 Len=0 TSV=4102301606 TSER=4102085369
307087 2009-07-28 12:32:51.321792 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1
(application/x-www-form-urlencoded)
307097 2009-07-28 12:32:51.334466 172.16.101.156
172.16.101.155 42530 HTTP HTTP/1.1 200 OK
(application/json)
307098 2009-07-28 12:32:51.334615 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36015 Ack=237956 Win=81280 Len=0 TSV=4102324467 TSER=4102108230
307099 2009-07-28 12:32:51.335616 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
307100 2009-07-28 12:32:51.335713 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36015 Ack=237961 Win=81280 Len=0 TSV=4102324468 TSER=4102108231
307512 2009-07-28 12:33:09.577293 172.16.101.155
172.16.101.156 8080 HTTP GET /xxx/xxxxx HTTP/1.1
307564 2009-07-28 12:33:09.616699 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [ACK]
Seq=237961 Ack=36892 Win=64128 Len=0 TSV=4102126512 TSER=4102342709
307582 2009-07-28 12:33:09.626212 172.16.101.156
172.16.101.155 42530 HTTP HTTP/1.1 200 OK
307583 2009-07-28 12:33:09.626369 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36892 Ack=238260 Win=81280 Len=0 TSV=4102342758 TSER=4102126521
307584 2009-07-28 12:33:09.626425 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
307585 2009-07-28 12:33:09.626428 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
307587 2009-07-28 12:33:09.626768 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36892 Ack=239708 Win=81280 Len=0 TSV=4102342759 TSER=4102126521
307588 2009-07-28 12:33:09.626842 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
307589 2009-07-28 12:33:09.627013 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36892 Ack=240880 Win=81280 Len=0 TSV=4102342759 TSER=4102126521
307590 2009-07-28 12:33:09.627019 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=36892 Ack=240885 Win=81280 Len=0 TSV=4102342759 TSER=4102126522
309125 2009-07-28 12:33:30.245862 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1
(application/x-www-form-urlencoded)
309126 2009-07-28 12:33:30.245870 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [ACK]
Seq=240885 Ack=37985 Win=64128 Len=0 TSV=4102147140 TSER=4102363377
309137 2009-07-28 12:33:30.258721 172.16.101.156
172.16.101.155 42530 HTTP HTTP/1.1 200 OK
(application/json)
309138 2009-07-28 12:33:30.258873 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=37985 Ack=241140 Win=81280 Len=0 TSV=4102363390 TSER=4102147153
309139 2009-07-28 12:33:30.259867 172.16.101.156
172.16.101.155 42530 HTTP Continuation or non-HTTP
traffic
309140 2009-07-28 12:33:30.259961 172.16.101.155
172.16.101.156 8080 TCP 42530 > http-alt [ACK]
Seq=37985 Ack=241145 Win=81280 Len=0 TSV=4102363391 TSER=4102147154
310858 2009-07-28 12:34:00.245039 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [FIN, ACK]
Seq=241145 Ack=37985 Win=64128 Len=0 TSV=4102177138 TSER=4102363391
310860 2009-07-28 12:34:00.259377 172.16.101.155
172.16.101.156 8080 TCP [TCP segment of a reassembled
PDU]
310861 2009-07-28 12:34:00.259394 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [RST]
Seq=241146 Win=0 Len=0
310862 2009-07-28 12:34:00.259485 172.16.101.155
172.16.101.156 8080 TCP [TCP segment of a reassembled
PDU]
310863 2009-07-28 12:34:00.259491 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [RST]
Seq=241146 Win=0 Len=0
310864 2009-07-28 12:34:00.259628 172.16.101.155
172.16.101.156 8080 TCP [TCP segment of a reassembled
PDU]
310865 2009-07-28 12:34:00.259632 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [RST]
Seq=241146 Win=0 Len=0
310866 2009-07-28 12:34:00.259634 172.16.101.155
172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1
(application/x-www-form-urlencoded)
310867 2009-07-28 12:34:00.259639 172.16.101.156
172.16.101.155 42530 TCP http-alt > 42530 [RST]
Seq=241146 Win=0 Len=0
Does anyone know what's going on here?
Regards, thanks,
Patrick
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
" from the digest: users-digest-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
" from the digest: users-digest-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org
