#37033: Psycopg2 has different semantics than psycopg3 and should not be
deprecated
--------------------------------+--------------------------------------
Reporter: Cameron Gorrie | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------+--------------------------------------
Comment (by Simon Charette):
I think the real problem at play is that your worker deployment workflow
is not truly idempotent; I don't think it's fair to assume that
`SIGKILL`'ing a process without gracefully handling it on the receiving
end will do the right thing by default.
The way Celery deals with `SoftTimeLimitExceeded` and worker termination
signals by surfacing exception and allowing the flow of the program to be
interrupted at any time is naive and error prone (it also causes cursor
corruption on the `mysqlclient` MySQL library). The correct way of
gracefully handling non-fatal error signal is usually to set a flag on the
execution context that the application can check at a safe checkpoint
(`self.terminate = True`).
> When we re-deploy, we kill the worker containers and re-queue the tasks
that were in-flight / ready to process on that worker.
Any reason not to
[https://docs.celeryq.dev/en/stable/userguide/workers.html#warm-shutdown
perform a warm shutdown] first to limit the cases when this can happen
I'm afraid there's is little Django can do here, if `psycopg2` development
halts and it is no longer supported upstream we will have to pull the plug
on it. I'm not sure this is the right place to advocate for a problem at
the intersection of Celery and psycopg to be solved, have you raised this
problem with any of these third party library maintainers? Surely Celery
is aware that this signalling approach is problematic (e.g. what if a task
is the middle of performing a critical HTTP request when the signal
surfaces?) and has documented patterns on how to deal with this problem.
--
Ticket URL: <https://code.djangoproject.com/ticket/37033#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019d87c0141d-fec80a86-bbf9-4209-87e0-52ad7be83e1d-000000%40eu-central-1.amazonses.com.
