Hello Raphael, Thanks for a great analysis. I can confirm this is a bug in Django.
I didn’t realize SQLITE_MAX_COLUMN would be the limit here. Would you mind opening a ticket on https://code.djangoproject.com/? (Alternatively I can do it.) Best regards, -- Aymeric. > On 8 janv. 2016, at 10:55, Raphael Hertzog <hert...@debian.org> wrote: > > [ CCing upstream Django developer to have its opinion ] > > Hello, > > On Fri, 08 Jan 2016, Christophe Siraut wrote: >> Sorry for the confusion, I have the same libsqlite3 version as you, and >> it is that one I recompiled with SQLITE_MAX_COLUMN 32767. > > I can't reproduce the issue with Django 1.8. It looks like a regression of > Django 1.9 that uses an SQL request to generate quoted values for the > parameters fed into requests of the from "WHERE foo IN (...value > list...)". > > Also does this also happen if you drop debug_toolbar from INSTALLED_APPS ? > I have the feeling that this code path is only used when you actually want > to print the SQL query that has been executed and this is usally only > needed during development. > > The regression has been introduced by this commit: > > https://github.com/django/django/commit/4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5 > > commit 4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5 > Author: Aymeric Augustin <aymeric.augus...@m4x.org> > Date: Sun Sep 13 09:30:35 2015 +0200 > > Refs #14091 -- Fixed connection.queries on SQLite. > > It's supposed to fix https://code.djangoproject.com/ticket/14091 for > SQLite. Christophe, can you verify that the issue is gone if you revert > that change in your Django ? > > Aymeric, the above change is problematic because for a request that > looks like MyModel.objects.filter(foo__in=my_array) it will break as soon > as my_array goes above 2000 entries in length since that's the default > limit for SQLITE_MAX_COLUMN and you will trigger it with your huge > "SELECT QUOTE(?),...." query. > > I know the query can also trigger the limit set with > SQLITE_MAX_VARIABLE_NUMBER which defaults to 999 but at least on Debian > we increased that limit significantly (to 250000) so that limit > is not a practical problem. The above limit is more problematic > as we can't increase it to the same value (the max is 32767) and > while the limit on the variable number was unreasonably low, I can > understand the limit on the number of columns much better. > > See https://sqlite.org/limits.html > > Aymeric, what do you think ? Shall we open a ticket for this regression ? > > Can you update the code to process parameters by batch of less than 2000 > entries ? > > You can look at https://bugs.debian.org/809211 for the former discussion > in this bug report. > > Cheers, > -- > Raphaël Hertzog ◈ Debian Developer > > Support Debian LTS: http://www.freexian.com/services/debian-lts.html > Learn to master Debian: http://debian-handbook.info/get/
