So I have a simple question from all this. Is there a twisted idiom I can use so that deferred returned from a Klein route are not canceled if the client resets the connection? It's fine if it's before I've gotten the request body, but once I've gotten the request body I want all deferreds to succeed. I only wan't the streaming of the final result to fail.
On Sun, Jan 24, 2021 at 7:40 PM Robert DiFalco <robert.difa...@gmail.com> wrote: > Well, I've dealt with this issue with other languages. Not sure how to > deal with it in Klein/Twisted. This operation is idempotent so I suppose > what I'd like to happen is have the whole chain of deferred's succeed but > then just not be able to write the response to the socket -- but not > interrupt the chain. Unfortunately, what I really need is my operations to > be atomic -- introduce a two phase commit or some such. But that's too big > of a job on this legacy code base for now. Right now I'd be content if I > could save the database record and then send the SQS message to AWS and not > have that SQS send interrupted if the client closed the socket.. Is there a > simple way to achieve that? If I got the request body I'm good, I don't > care if I can't write the response. > > One other thing that would be nice is to know why a deferred was canceled. > If the client close a connection I might like to ignore the cancel, but I > think what is happening is that Twisted is pretty smart. So it either knows > I'm going to make a write to SQS using Boto that is inbound, so it is > somehow able to cancel that operation -- perhaps more likely the client > closed the connection and the connection canceled defers after I called > deferToThread but before a thread was available to run on. Either way, I'd > like them all to run, and just fail to write the final response from the > end of the chain. > > Sorry, too many words. > > > > > On Sun, Jan 24, 2021 at 3:22 PM Glyph <gl...@twistedmatrix.com> wrote: > >> If you're dealing with lots of clients on the public internet, sometimes >> this is just gonna happen, for a variety of reasons; it's normal. We would >> welcome better error reporting for this scenario so it doesn't require the >> kind of debugging you just did :-). >> >> -g >> >> On Jan 24, 2021, at 2:58 PM, Robert DiFalco <robert.difa...@gmail.com> >> wrote: >> >> That makes sense, thank you. A timeout seems unlikely but maybe the >> client is closing the connection due to a network issue. This is an >> extremely rare occurrence. >> >> On Sun, Jan 24, 2021 at 2:41 PM Glyph <gl...@twistedmatrix.com> wrote: >> >>> While a socket is open and receiving data, recv() will either give you a >>> non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) >>> if no bytes are ready. A result of zero bytes (the empty string) means >>> "end of file" - the other end has closed the socket. >>> >>> So what's happening here is your client is timing out or otherwise >>> canceling its request by closing the socket, and this is the correct, >>> intentional response to that scenario. >>> >>> -g >>> >>> On Jan 24, 2021, at 11:57 AM, Robert DiFalco <robert.difa...@gmail.com> >>> wrote: >>> >>> You're absolutely right, I meant "cancel the deferred". I don't grok >>> server sockets very well so maybe someone can help. But apparently, klein >>> does a .doRead from our server socket (getting the request from the >>> client?). This returns a "why" of "connection done" so that closes the >>> connection before we have written our response to the client, and that >>> cancels the deferred SQS write. >>> >>> >>> https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155 >>> >>> The method above is "doRead". Which calls this: >>> >>> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239 >>> >>> I guess if If socket.rcv() returns an empty string it simply closes the >>> connection. >>> >>> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250 >>> >>> Is that normal? I mean I guess it must be but then why is the read >>> getting an empty string and closing the connection? I can't really account >>> for it? Some kind of back pressure due to load? >>> >>> Thanks for any thoughts. >>> >>> >>> >>> On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau <colin.dunk...@gmail.com> >>> wrote: >>> >>>> >>>> >>>> On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco < >>>> robert.difa...@gmail.com> wrote: >>>> >>>>> Hi, I apologize this question is a little vague. I'm looking for >>>>> pointers. I have a klein route that makes an underlying deferToThread call >>>>> with a simple single thread (an IO based sync call I can't change, a boto3 >>>>> sqs write). The thread pool is simple, just a couple of threads, nothing >>>>> fancy. >>>>> >>>>> VERY rarely it appears that Klein cancels the thread. What techniques >>>>> can I use to figure out why my thread is being Canceled? There's nothing >>>>> in >>>>> the failure to tell me "who, why, or where" it was canceled. Also, I >>>>> cannot >>>>> get this down to a reproducible case, but here's the boto3 sqs wrapper, >>>>> this fall back works fine, but it's a band-aide for an error I can't track >>>>> down.: >>>>> >>>>> def write(self, payload): >>>>> """ >>>>> Write message to SQS async from thread pool. If twisted cancels the >>>>> thread, instead write synchronously. >>>>> >>>>> def _retrySynchronously(error): >>>>> if error.type != CancelledError: >>>>> return error >>>>> >>>>> log.warn("Async SQS write cancelled. Calling synchronously.") >>>>> return defer.succeed(self._writeSyncFallback(payload)) >>>>> >>>>> deferredCall = self._deferToThread(self.sqs.write, payload) >>>>> deferredCall.addErrback(_retrySynchronously) >>>>> return deferredCall >>>>> >>>>> def _writeSyncFallback(self, payload): >>>>> return self.sqs.write(payload) >>>>> >>>>> The _deferToThread call just uses my own thread pool with 2 threads, >>>>> but is otherwise stock. >>>>> >>>>> Is there a level of logging I'm missing or some other thing that would >>>>> tell me why the thread is being canceled? The retry works great and Klein >>>>> does not return an error from the route. >>>>> >>>>> Thanks in advance. >>>>> >>>>> >>>> I think we'll need to see more code for this, specifically the caller >>>> of that `write` method, and its callers, etc. Note that the thread itself >>>> isn't being cancelled, the Deferred you get from _deferToThread is... so >>>> you'll most likely need to find out what code interacts with that object to >>>> progress in isolating this. >>>> >>>> In my quick skim of the deferToThread and ThreadPool source, I can't >>>> find any explicit cancellations. While that certainly doesn't rule it out, >>>> it does make me think you're more likely to find the issue by inspecting >>>> the callers involved. >>>> _______________________________________________ >>>> Twisted-Python mailing list >>>> Twisted-Python@twistedmatrix.com >>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>>> >>> _______________________________________________ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >>> >>> _______________________________________________ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
