On Mon, Mar 23, 2020 at 8:03 PM Frank Millman <fr...@chagford.com> wrote: > > On 2020-03-22 12:11 PM, Chris Angelico wrote: > > On Sun, Mar 22, 2020 at 8:30 PM Frank Millman <fr...@chagford.com> wrote: > >> > >> On 2020-03-22 10:45 AM, Chris Angelico wrote: > > > > If you can recreate the problem with a single socket and multiple > > requests, that would be extremely helpful. I also think it's highly > > likely that this is the case. > > > > I am working on a stripped-down version, but I realise there are a few > things I have not grasped. > > Hope you don't mind, but can you scan through what follows and tell me > if I am on the right lines?
No probs! > Both the client and the server can send a header with 'Keep-alive', but > what does it actually mean? > > If the client sends it, does that mean that it wants the server to keep > the connection open, and only close it when the client closes it from > the other end? > > Conversely, if the server sends it, does it mean that it wants the > client to keep the connection open? If so, under what condition would > the server close the connection (other than a timeout). Should the > server only send 'Keep-alive' if it receives 'Keep-alive'? I'm pretty sure the server should never say "keep-alive" if the client said "close". (There's a bit of an oddity in that the default changed between, I think, HTTP 1.0 and 1.1. But that's a minor discrepancy.) Generally, you'll see one of these options: 1) Client says "Connection: close". This means "I'm only gonna be doing this one request, then we're done". Good for non-browser usage, automated scripts etc. 2) Client says "Keep-Alive", server says "Close". This means the client would be okay with connection reuse, but the server's saying no (maybe it doesn't support it, or maybe it's in a soft shutdown state where it wants to end connections, or whatever). After the current request, the connection will be closed. 3) Both ends say "Connection: Keep-Alive". After the request, the socket isn't closed, and can be reused for some other request. I'm not sure how a client would (or should) handle it if the server says "Connection: Keep-Alive" after the client said "Connection: Close". Probably the client would just close its end of the socket and that'd be that. > In my program, when I send a file in response to a GET, I send a header > with 'Keep-alive', then I send the file, then I close the connection. > This now seems wrong. It could even be the cause of my bug, but then why > has it only appeared now? Both Edge and Chrome send 'Keep-alive' headers. > It mightn't be fundamentally WRONG (there needs to be *some* way to signal that you're now done), but it would probably be suboptimal. If you're always going to close the connection after one request, just send Connection: Close. > If I am thinking along the right lines, then the exchange should go like > this - > > Client sends request, with 'Keep-alive' header. > > Server sends response, but does not close the connection. The server > should reply with a 'Keep-alive' header. If it does not, the client will > close the connection, in which case the server will also close. > > Assuming that they both send 'Keep-alive', the onus is on the client to > close the connection when it has no more requests. The server should > have a timeout in case the client goes away. I think either end is allowed to close the connection. > Assuming that the above is correct, the client will rely on > 'Content-length' to determine when it has received the entire request. > If the client has more than one request, it will send the first, wait > for the response, when fully received as per the 'Content-length' it > will send the next one, until all requests have been sent and all > responses received, at which point it will close the connection. Yep. Which makes things very interesting (for serene values of "interesting", such as "we're all gonna die") when you want to use connection pooling with something that fundamentally doesn't have a length, such as a chunked transfer. Which you are doing. That may or may not actually be significant. > All this assumes only one connection. Alternatively the client could > open multiple connections for the requests. In that case it would make > sense to use 'Connection: Close', so that the server can close the > connection straight away, making it available for reuse. What you'll often see is that the client has some sort of connection limit, and reuses those same connections. For instance, it might have a maximum of four connections per server (where "server" has to include the name as well as its IP and port, due to TLS SNI), so it requests an HTML page on one connection (with keep-alive), parses that, and then fires off one request on the reused connection and three others on new connections. As those requests complete, it sends new requests on the same sockets. > If this all makes sense, I should write two versions of the client > program, one using a single connection, and one using a pool of connections. > Possibly! I think you'll most likely see that one of those behaves perfectly normally, and you only trigger the issue in the other. So you could move forward with just one test program. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
