#37033: Psycopg2 has different semantics than psycopg3 and should not be
deprecated
--------------------------+-----------------------------------------
Reporter: sastraxi | Type: Uncategorized
Status: new | Component: Uncategorized
Version: 6.0 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------+-----------------------------------------
The context: we have been trialling psycopg3 in our (async-free!) Django
5.2 codebase. I noted that
https://docs.djangoproject.com/en/6.0/releases/4.2/#psycopg-3-support says
"Support for psycopg2 is likely to be deprecated and removed at some point
in the future." -- this ticket has my observations that make me believe
this is not a good idea.
We have many Celery (5.6.2) tasks that run on idempotent queue workers
with Django fully loaded. When we re-deploy, we kill the worker containers
and re-queue the tasks that were in-flight / ready to process on that
worker. It does this by raising exceptions from signal handlers:
https://github.com/celery/billiard/blob/main/billiard/common.py#L106
ultimately raises a `SystemExit` exception from inside the signal handler
when it receives the SIGQUIT signal.
As outlined in [Safe Asynchronous Exceptions for
Python](https://www.cs.williams.edu/~freund/papers/python.pdf), raising
exceptions in this way can arbitrarily interrupt execute at any bytecode
boundary.
=== Example case
Here's a simplified task handler so you can visualize the problem:
{{{#!python
@celery_app.task(MY_IDEMPOTENT_QUEUE)
def my_task():
with transaction.atomic():
[...]
my_model.save()
transaction.on_commit(lambda: my_other_task.delay())
}}}
Sometimes, when a signal is delivered while commit() is running, the
following happens:
1. PostgreSQL has applied the COMMIT (we can later verify DB state).
2. psycopg3 is interrupted during its Python‑level wait loop; an exception
bubbles out of commit() (or _set_autocommit) rather than a normal return.
3. Django sees an exception in `atomic.__exit__` and therefore does not
run `on_commit` callbacks, even though the transaction actually committed.
=== Root cause
* psycopg2's COMMIT path uses `PQexec("COMMIT")` inside a
Py_BEGIN_ALLOW_THREADS region, so the interpreter will not run the Python
signal handler until that call returns; this effectively makes COMMIT
atomic w.r.t. Python async exceptions.
* psycopg3's design explicitly avoids blocking libpq calls, using a Python
generator + wait functions to drive the COMMIT protocol step-by-step. That
means Python bytecode is executing throughout, so signal handlers can
raise exceptions mid‑protocol.
I do not believe this will be possible to solve with psycopg3's approach,
so developers using Django's on_commit inside of celery worker handlers
need to be aware of this behaviour.
--
Ticket URL: <https://code.djangoproject.com/ticket/37033>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019d8729a824-d4716301-ae32-4b0f-b4c0-3ad97f877f27-000000%40eu-central-1.amazonses.com.
