Hi, That sounds great, much appreciated. I'll be available all week to test any patches you might propose.
B. > On 24 Jun 2023, at 21:35, Willy Tarreau <w...@1wt.eu> wrote: > > Hi Robert, > > On Sat, Jun 24, 2023 at 08:39:22PM +0100, Robert Newson wrote: >> So, the behaviour of the _changes endpoint when used with the feed=continuous >> and heartbeat=X (where X is number of milliseconds) is as follows; >> >> 1) when _changes is invoked, couchdb opens its internal "docs in update >> order" btree at the position indicated by the 'since' parameter, or the start >> of the btree if 'since' is not supplied. >> 2) for every key/value in the index from that point until the end of the >> btree, a json object is written to the http response as a chunk (we might >> pack multiple objects in one chunk but an object never spans multiple >> chunks). >> 3) once all of those are written to the response, couchdb then waits for >> notifications that new updates have been made (a document >> create/update/delete operation in a separate request), and then writes those >> updates to the http response as chunks too. >> 4) in the absence of updates, a timeout occurs, which causes us to write the >> heartbeat chunk (which consists of a single newline character). the timer is >> then reset. so, for a database that is receiving no writes, the response >> would consist of periodic newlines and nothing else. >> 5) steps 3 and 4 could conceivably continue (in any order) indefinitely; >> until a server crash or the client disconnects, etc. > > OK so that's clearly a long-polling type of processing. > >> So, yes, clearly there is no HTTP-compliant solution. > > Indeed. > >> The motivating case for me is we encountered a user that was particularly >> sensitive to the delayed heartbeats, in that it triggered poor connection >> handling logic on their side with much greater probability than if the 'keep >> alive' was working. It is ironic, to say the least, that the fundamental >> issue is that CouchDB is, and always has been, dependent on something (timely >> delivery of the partial http chunks in the response) that the HTTP spec does >> not require any client or proxy to honour. > > Indeed, but in parallel that matches stuff we've already met in the > past, and while we're not particularly trying to optimize for such > use cases, we try to remain transparent enough so that these continue > to work well enough. > > I think we should study an option. Since we already have "option > http-no-delay" for such use cases, we should probably make the > compression trigger a systematic flush when the option is set. This > would follow the spirit of this option and extend it to compression. > > I have not looked at the compression code for a while so I'll first > need to check if we can act on the queue to forcefully flush it, but > with that granted, the rest could possibly be simple enough and easy > to setup. I'll try to check that next week. > > Willy
