#25095: Fields annotated for filtering which are not included in values are
included in GROUP BY clause in SQL
-------------------------------------+-------------------------------------
Reporter: mitchelljkotler | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> I am running a complex query, which annotates certain fields solely to
> filter on them. They are not included in a call to values. Another
> annotation is included, but it is incorrect due to the previous
> annotations being included in the GROUP BY clause.
>
> Here is a not so simple example:
>
> {{{
> # We are filtering on the journey date, using the datediff function
> # We annotate the reported tds fields, since we need two aliases of them
> for the different filterings applied
>
> print EpTag.objects.filter(statuschanges__status=attached)\
> .annotate(attach_date=F('statuschanges__reported_tds'))\
> .filter(statuschanges__status=detached)\
> .annotate(detach_date=F('statuschanges__reported_tds'))\
> .annotate(journey_days=Func(F('detach_date'), F('attach_date'),
> function='DATEDIFF', output_field=IntegerField()))\
> .filter(journey_days__lt=10)\
> .annotate(name=F('object__producer__name'))\
> .values('object__producer', 'name')\
> .annotate(count=Count('pk', distinct=True))\
> .order_by().query
>
> SELECT `ep_object`.`cid`, `ep_entity`.`name` AS `name`, COUNT(DISTINCT
> `ep_tag`.`tid`) AS `count`
> FROM `ep_tag`
> INNER JOIN `ep_statuschange` ON ( `ep_tag`.`tid` =
> `ep_statuschange`.`tid` )
> INNER JOIN `ep_statuschange` T4 ON ( `ep_tag`.`tid` = T4.`tid` )
> LEFT OUTER JOIN `ep_object` ON ( `ep_tag`.`oid` = `ep_object`.`oid` )
> LEFT OUTER JOIN `ep_entity` ON ( `ep_object`.`cid` = `ep_entity`.`eid` )
> WHERE (`ep_statuschange`.`statusid` = 201 AND T4.`statusid` = 1000 AND
> DATEDIFF(T4.`reported_tds`, `ep_statuschange`.`reported_tds`) < 10)
> # Group by still includes reported_tds and DATEDIFF fields, even though
> they were not selected. This breaks my count
> GROUP BY `ep_object`.`cid`, `ep_statuschange`.`reported_tds`,
> T4.`reported_tds`, DATEDIFF(T4.`reported_tds`,
> `ep_statuschange`.`reported_tds`), `ep_entity`.`name` ORDER BY NULL
> }}}
>
> In django/db/models/sql/query.py, line 1751 - all annotations are
> included in the group by. I believe this should be using
> annotation_select instead of annotations.
New description:
I am running a complex query, which annotates certain fields solely to
filter on them. They are not included in a call to values. Another
annotation is included, but it is incorrect due to the previous
annotations being included in the GROUP BY clause.
Here is a not so simple example:
{{{
# We are filtering on the journey date, using the datediff function
# We annotate the reported tds fields, since we need two aliases of them
for the different filterings applied
print EpTag.objects.filter(statuschanges__status=attached)\
.annotate(attach_date=F('statuschanges__reported_tds'))\
.filter(statuschanges__status=detached)\
.annotate(detach_date=F('statuschanges__reported_tds'))\
.annotate(journey_days=Func(F('detach_date'), F('attach_date'),
function='DATEDIFF', output_field=IntegerField()))\
.filter(journey_days__lt=10)\
.annotate(name=F('object__producer__name'))\
.values('object__producer', 'name')\
.annotate(count=Count('pk', distinct=True))\
.order_by().query
SELECT `ep_object`.`cid`, `ep_entity`.`name` AS `name`, COUNT(nnn
`ep_tag`.`tid`) AS `count`
FROM `ep_tag`
INNER JOIN `ep_statuschange` ON ( `ep_tag`.`tid` = `ep_statuschange`.`tid`
)
INNER JOIN `ep_statuschange` T4 ON ( `ep_tag`.`tid` = T4.`tid` )
LEFT OUTER JOIN `ep_object` ON ( `ep_tag`.`oid` = `ep_object`.`oid` )
LEFT OUTER JOIN `ep_entity` ON ( `ep_object`.`cid` = `ep_entity`.`eid` )
WHERE (`ep_statuschange`.`statusid` = 201 AND T4.`statusid` = 1000 AND
DATEDIFF(T4.`reported_tds`, `ep_statuschange`.`reported_tds`) < 10)
# Group by still includes reported_tds and DATEDIFF fields, even though
they were not selected. This breaks my count
GROUP BY `ep_object`.`cid`, `ep_statuschange`.`reported_tds`,
T4.`reported_tds`, DATEDIFF(T4.`reported_tds`,
`ep_statuschange`.`reported_tds`), `ep_entity`.`name` ORDER BY NULL
}}}
In django/db/models/sql/query.py, line 1751 - all annotations are included
in the group by. I believe this should be using annotation_select instead
of annotations.
--
Comment (by mitchelljkotler):
I created a pull request, 5002.
It adds one test to test for this issue.
@akaariai, I am not sure I understand what you want to test with that test
case.
--
Ticket URL: <https://code.djangoproject.com/ticket/25095#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/073.dfe870a5acaea881de43a6f6b10d61eb%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
