[Twisted-Python] If the errbacks of a canceled Deferred are called with error other than CancelledError, is this acceptable?

[zhang kai]

If I cancel a Deferred, the errbacks of the Deferred are not called with
CancelledError but with other errors like ConnectionDone, is this
acceptable?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] TestInternet2.testPickledTimer Failed

[zhang kai]

Hi,

I'm a Google Summer of Code intern working on "Deferred Cancellation"
project. I'm recently working on adding cancellation support to
twisted.internet.task.LoopingCall.

However, after I added the canceller to LoopingCall.deferred,
the twisted.test.test_application.TestInternet2.testPickledTimer failed due
to a PicklingError.

My branch is loopingcall-deferred-cancellation-6656. Here is the diff of my
code: http://twistedmatrix.com/~diffresource.twistd/6656

Here is the error message:

===
[ERROR]
Traceback (most recent call last):
  File
"/home/kai/Projects/GSoC/DeferredCancellation/Twisted/branches/loopingcall-deferred-cancellation-6656/twisted/test/test_application.py",
line 487, in testPickledTimer
s = pickle.dumps(t0)
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 725, in save_inst
save(stuff)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 725, in save_inst
save(stuff)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle  at 0x8f1fb8c>: it's
not found as twisted.internet.posixbase.

twisted.test.test_application.TestInternet2.testPickledTimer
===

I thought the reason was the circular references. However I searched about
it and found that pickle could handle the circular reference cases. But the
only significant change is that after I added the canceller, there is a
circular reference between LoopingCall and LoopingCall.deferred. So I don't
know what's the problem. How can I fix this?

Thanks,
-Kai
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] TestInternet2.testPickledTimer Failed

[zhang kai]

On Thu, Aug 1, 2013 at 7:50 PM,  wrote:

> I suggest you take a look at TimerService and figure out why pickling one
> of those ever tries to pickle a LoopingCall (take a look around
> `__getstate__` and `volatile`, I think that's where the problem is).  I
> think you'll find an existing bug that the unit test previously failed to
> reveal but which your changes have revealed.
>

Thanks for your help. I've found that the bug is due to a typo. The
"_loopFinshed" in `TimerService.volatile` should be "_loopFinished". So
when pickling a `TimerService` it will actually try to pickle
"_loopFinished"(the Deferred returned by `LoopingCall.start`). When the
Deferred has a reference to the `LoopingCall` instance it will then try to
pickle the `LoopingCall`, thus causes the problem.

I've searched and there is no ticket about this. So I will file a new
ticket and fix this.

-Kai
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Raising exception from a Deferred canceller.

[zhang kai]

Hi,

As itamar mentioned in ticket #6676 , If a cancellation
function for a Deferred throws an exception(the cancel() method of
Deferred won’t
throw exceptions, but the canceller may), behavior is undefined. If the
cancellation function throws an exception it is currently not caught, and
cancellation does not occur.

We can catch the exception and log it, and fallback to just firing Deferred
 withCancelledError. This won’t break any old code. But an exception
raising from the cancellation function often means the cancellation is
failed.

Another option we have is taking this opportunity to make the cancellation
being able to fail. There is the motivation:

There are cases where a Deferred is uncancellable. For example, we can call
twisted.mail.imap4.IMAP4Client.delete to delete a mailbox. When the
operation is waiting in the queue, we can cancel it by removing it from the
queue. However, when the operation is already sent and is waiting for the
response, it becomes uncancellable.

If we allow the canceller(NOT the cancel() method of the Deferred) to raise
an exception, we can tell the user the cancellation is failed and the
Deferredwon’t be fired with a CancelledError.

Raising an exception from cancel() may break the old code. So we can catch
the exception raised by the canceller, then return a False without firing
theDeferred to tell the user that the cancellation is failed.

In order to avoid missing unexpected exceptions, we can create a
CancellationFailedError. When the canceller raises CancellationFailedError,
we catch it and return False. When the canceller raises others exceptions,
we catch it, log it then return False.

Something like this:

def cancel(self):
if not self.called:
canceller = self._canceller
if canceller:
try:
canceller(self)
except CancellationFailedError:
return False
except Exception:
log.err(None, "Unexpected exception from canceller.")
return False
else:
# Arrange to eat the callback that will eventually be fired
# since there was no real canceller.
self._suppressAlreadyCalled = True
if not self.called:
# There was no canceller, or the canceller didn't call
# callback or errback.
self.errback(failure.Failure(CancelledError()))
return True
elif isinstance(self.result, Deferred):
# Waiting for another deferred -- cancel it instead.
return self.result.cancel()
else:
return False

This won’t break any code by raising an exception from cancel(), although
some code may rely on cancel() not returning any value.

So, what’s your opinion on raising an exception from the canceller?


Regards,

-Kai
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python