#28334: contrib.postgresql overwhelms database with "select from pg_type"
queries
on each request
-------------------------------------+-------------------------------------
Reporter: Igor Gumenyuk | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: contrib.postgres | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Igor Gumenyuk:
Old description:
> On each newly created database connection `django.contrib.postgresql`
> tries to register hstore type for this connection via
> `connection_created` signal and `register_hstore` function.
>
> https://github.com/django/django/blob/master/django/contrib/postgres/apps.py#L20
> https://github.com/django/django/blob/stable/1.11.x/django/contrib/postgres/signals.py#L16
>
> Behind the scenes `register_hstore` runs sql query to get `oid` of
> `hstore` type:
> https://github.com/psycopg/psycopg2/blob/master/lib/extras.py#L885
>
> This happens on every(!) request (unless persistent connections enabled)
> SELECT t.oid, %s
> FROM pg_type t JOIN pg_namespace ns
> ON typnamespace = ns.oid
> WHERE typname = 'hstore';
>
> This is just huge overhead, since `register_hstore` accepts `oid`
> argument to avoid hitting database every call.
> We have seen significant latency spikes because of this.
>
> Possible solution would be have configurable `GET_HSTORE_OID_FUNC` in
> `DATABASES->OPTIONS` setting which can be cached in any way.
> This will also maintain backwards compatibility.
>
> I can send patch for this if proposed solution is acceptable to be merged
> in master.
New description:
On each newly created database connection `django.contrib.postgresql`
tries to register hstore type for this connection via `connection_created`
signal and `register_hstore` function.
https://github.com/django/django/blob/master/django/contrib/postgres/apps.py#L20
https://github.com/django/django/blob/stable/1.11.x/django/contrib/postgres/signals.py#L16
Behind the scenes `register_hstore` runs sql query to get `oid` of
`hstore` type:
https://github.com/psycopg/psycopg2/blob/master/lib/extras.py#L885
This happens on every(!) request (unless persistent connections enabled)
SELECT t.oid, %s
FROM pg_type t JOIN pg_namespace ns
ON typnamespace = ns.oid
WHERE typname = 'hstore';
This is just huge overhead, since `register_hstore` accepts `oid` argument
to avoid hitting database every call.
We have seen significant latency spikes because of this.
Possible solution would be to have configurable `GET_HSTORE_OID_FUNC` in
`DATABASES->OPTIONS` setting which can be cached in any way.
This will also maintain backwards compatibility.
I can send patch for this if proposed solution is acceptable to be merged
in master.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28334#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.5ba32aff8c0e066e634c6937ade4f493%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
