[ CCing upstream Django developer to have its opinion ] Hello,
On Fri, 08 Jan 2016, Christophe Siraut wrote: > Sorry for the confusion, I have the same libsqlite3 version as you, and > it is that one I recompiled with SQLITE_MAX_COLUMN 32767. I can't reproduce the issue with Django 1.8. It looks like a regression of Django 1.9 that uses an SQL request to generate quoted values for the parameters fed into requests of the from "WHERE foo IN (...value list...)". Also does this also happen if you drop debug_toolbar from INSTALLED_APPS ? I have the feeling that this code path is only used when you actually want to print the SQL query that has been executed and this is usally only needed during development. The regression has been introduced by this commit: https://github.com/django/django/commit/4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5 commit 4f6a7663bcddffb114f2647f9928cbf1fdd8e4b5 Author: Aymeric Augustin <aymeric.augus...@m4x.org> Date: Sun Sep 13 09:30:35 2015 +0200 Refs #14091 -- Fixed connection.queries on SQLite. It's supposed to fix https://code.djangoproject.com/ticket/14091 for SQLite. Christophe, can you verify that the issue is gone if you revert that change in your Django ? Aymeric, the above change is problematic because for a request that looks like MyModel.objects.filter(foo__in=my_array) it will break as soon as my_array goes above 2000 entries in length since that's the default limit for SQLITE_MAX_COLUMN and you will trigger it with your huge "SELECT QUOTE(?),...." query. I know the query can also trigger the limit set with SQLITE_MAX_VARIABLE_NUMBER which defaults to 999 but at least on Debian we increased that limit significantly (to 250000) so that limit is not a practical problem. The above limit is more problematic as we can't increase it to the same value (the max is 32767) and while the limit on the variable number was unreasonably low, I can understand the limit on the number of columns much better. See https://sqlite.org/limits.html Aymeric, what do you think ? Shall we open a ticket for this regression ? Can you update the code to process parameters by batch of less than 2000 entries ? You can look at https://bugs.debian.org/809211 for the former discussion in this bug report. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: http://www.freexian.com/services/debian-lts.html Learn to master Debian: http://debian-handbook.info/get/
