#26522: Non-deterministic crash in django.db.models.sql.Query.combine()
-------------------------------------+-------------------------------------
Reporter: Ole Laursen | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.9
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mikalai Radchuk):
It's a very tricky issue. I think, it can damage data in some cases.
In my case Django ORM didn't fail with assertion error: it generated wrong
SQL query so I got wrong data from my DB. On development environment I
didn't get this issue because it appears randomly. I'm a lucky person - in
my case I query for some data for UI, but if you perform some calculations
using data from a queryset that silently returns wrong data - you are in
trouble. It's quite possible that 4/5 of your processes work properly, but
one gives wrong results.
Here is simplified version of my issue.
Models:
{{{
#!python
from django.db import models
class Category(models.Model):
title = models.CharField(max_length=255, unique=True)
class Meta:
verbose_name_plural = 'categories'
def __str__(self):
return self.title
class Document(models.Model):
title = models.CharField(blank=True, max_length=255)
categories = models.ManyToManyField('Category', blank=True,
related_name='+')
def __str__(self):
return self.title
}}}
Build a query set that could return wrong data:
{{{
#!python
from app_with_models.models import Document, Category
queryset = Document.objects.all()
# Exclude documents without taxonomy (Category) tags
queryset = queryset.filter(categories__in=Category.objects.all())
# Apply predefined taxonomy filters
category_predefined = [1] # Should be in fixtures
queryset = queryset.filter(categories__in=category_predefined)
queryset = queryset.distinct()
used_category_ids = queryset.values_list('categories__pk', flat=True)
print(used_category_ids.query)
}}}
This code should generate the following SQL query (formatted):
{{{
#!sql
SELECT DISTINCT
"app_with_models_document_categories"."category_id"
FROM "app_with_models_document"
LEFT OUTER JOIN "app_with_models_document_categories" ON
("app_with_models_document"."id" =
"app_with_models_document_categories"."document_id")
INNER JOIN "app_with_models_document_categories" T4 ON
("app_with_models_document"."id" = T4."document_id")
WHERE (
"app_with_models_document_categories"."category_id" IN (SELECT U0."id"
AS Col1 FROM "app_with_models_category" U0)
AND T4."category_id" IN (1)
);
}}}
But in some cases it generates this:
{{{
#!sql
SELECT DISTINCT
T4."category_id"
FROM "app_with_models_document"
LEFT OUTER JOIN "app_with_models_document_categories" ON
("app_with_models_document"."id" =
"app_with_models_document_categories"."document_id")
INNER JOIN "app_with_models_document_categories" T4 ON
("app_with_models_document"."id" = T4."document_id")
WHERE (
"app_with_models_document_categories"."category_id" IN (SELECT U0."id"
AS Col1 FROM "app_with_models_category" U0)
AND T4."category_id" IN (1)
);
}}}
This query generates absolutely wrong results. In my example it will
always return `1` as `category_id`.
Some more details:
* Can be reproduced on `Django==1.10.7`, `Django==1.11.4` and, most
likely, on order versions too. Fixed in
9bbb6e2d2536c4ac20dc13a94c1f80494e51f8d9
* Can be reproduced on Python 3.5, but not on Python 3.6 (at least, I
didn't manage to do that). I guess it's because of
[https://docs.python.org/3/whatsnew/3.6.html#new-dict-implementation new
dict implementation].
* I've tested with Postgres 9.6.3 and `psycopg2==2.6.1` and
`psycopg2==2.7.3.1`, but it looks like the issue is not DB engine/adapter
specific.
Hope it helps.
Can it be backported into supported (or at least into LTS)? I would really
appreciate it. Me an my colleague have spent 4 days debugging our code an
I'm afraid that it could cause more serious issues for other people.
--
Ticket URL: <https://code.djangoproject.com/ticket/26522#comment:12>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/068.dce1a80774207ba0e215cacb9c884167%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
