#27940: Migrations fail when when a model's parent class is changed to abstract
--
Wrong SQL being generated
-------------------------------------+-------------------------------------
Reporter: Sawan Vithlani | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 1.10
Severity: Normal | Resolution:
Keywords: SQLite migrations | Triage Stage:
abstract inheritance | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> Consider the following class hierarchy:
>
> {{{
> #!python
> class TestCase(models.Model):
> name = models.CharField(max_length=10)
>
> class TerminalTest(TestCase):
> pass
>
> class BrowserTest(TestCase):
> pass
>
> $ ./manage.py makemigrations testbed
> Migrations for 'testbed':
> testbed\migrations\0001_initial.py:
> - Create model TestCase
> - Create model BrowserTest
> - Create model TerminalTest
>
> $ ./manage.py migrate testbed
> Operations to perform:
> Apply all migrations: testbed
> Running migrations:
> Applying testbed.0001_initial... OK
> }}}
>
> Now we want to make the parent an abstract class:
>
> {{{
> #!python
> class TestCase(models.Model):
> name = models.CharField(max_length=10)
>
> class Meta:
> abstract = True
>
> class TerminalTest(TestCase):
> pass
>
> class BrowserTest(TestCase):
> pass
>
> ./manage.py makemigrations testbed
> You are trying to add a non-nullable field 'id' to browsertest without a
> default; we can't do that (the database needs something to populate
> existing rows).
> Please select a fix:
> 1) Provide a one-off default now (will be set on all existing rows with
> a null value for this column)
> 2) Quit, and let me add a default in models.py
> Select an option: 1
> Please enter the default value now, as valid Python
> The datetime and django.utils.timezone modules are available, so you can
> do e.g. timezone.now
> Type 'exit' to exit this prompt
> >>> 1
> You are trying to add a non-nullable field 'name' to browsertest without
> a default; we can't do that (the database needs something to populate
> existing rows).
> Please select a fix:
> 1) Provide a one-off default now (will be set on all existing rows with
> a null value for this column)
> 2) Quit, and let me add a default in models.py
> Select an option: 1
> Please enter the default value now, as valid Python
> The datetime and django.utils.timezone modules are available, so you can
> do e.g. timezone.now
> Type 'exit' to exit this prompt
> >>> ' '
> You are trying to add a non-nullable field 'id' to terminaltest without a
> default; we can't do that (the database needs something to populate
> existing rows).
> Please select a fix:
> 1) Provide a one-off default now (will be set on all existing rows with
> a null value for this column)
> 2) Quit, and let me add a default in models.py
> Select an option: 1
> Please enter the default value now, as valid Python
> The datetime and django.utils.timezone modules are available, so you can
> do e.g. timezone.now
> Type 'exit' to exit this prompt
> >>> 1
> You are trying to add a non-nullable field 'name' to terminaltest without
> a default; we can't do that (the database needs something to populate
> existing rows).
> Please select a fix:
> 1) Provide a one-off default now (will be set on all existing rows with
> a null value for this column)
> 2) Quit, and let me add a default in models.py
> Select an option: 1
> Please enter the default value now, as valid Python
> The datetime and django.utils.timezone modules are available, so you can
> do e.g. timezone.now
> Type 'exit' to exit this prompt
> >>> ' '
>
> Migrations for 'testbed':
> testbed\migrations\0002_auto_20170315_1711.py:
> - Remove field testcase_ptr from browsertest
> - Remove field testcase_ptr from terminaltest
> - Add field id to browsertest
> - Add field name to browsertest
> - Add field id to terminaltest
> - Add field name to terminaltest
> - Delete model TestCase
>
> $ ./manage.py migrate testbed
> Operations to perform:
> Apply all migrations: testbed
> Running migrations:
> Applying testbed.0002_auto_20170315_1600...Traceback (most recent call
> last):
> File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
> 64, in execute
> return self.cursor.execute(sql, params)
> File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py",
> line 337, in execute
> return Database.Cursor.execute(self, query, params)
> sqlite3.OperationalError: near ")": syntax error
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "./manage.py", line 10, in <module>
> execute_from_command_line(sys.argv)
> File "C:\my-venv\lib\site-packages\django\core\management\__init__.py",
> line 367, in execute_from_command_line
> utility.execute()
> File "C:\my-venv\lib\site-packages\django\core\management\__init__.py",
> line 359, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File "C:\my-venv\lib\site-packages\django\core\management\base.py",
> line 294, in run_from_argv
> self.execute(*args, **cmd_options)
> File "C:\my-venv\lib\site-packages\django\core\management\base.py",
> line 345, in execute
> output = self.handle(*args, **options)
> File "C:\my-venv\lib\site-
> packages\django\core\management\commands\migrate.py", line 204, in handle
> fake_initial=fake_initial,
> File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
> line 115, in migrate
> state = self._migrate_all_forwards(state, plan, full_plan, fake=fake,
> fake_initial=fake_initial)
> File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
> line 145, in _migrate_all_forwards
> state = self.apply_migration(state, migration, fake=fake,
> fake_initial=fake_initial)
> File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
> line 244, in apply_migration
> state = migration.apply(state, schema_editor)
> File "C:\my-venv\lib\site-packages\django\db\migrations\migration.py",
> line 129, in apply
> operation.database_forwards(self.app_label, schema_editor, old_state,
> project_state)
> File "C:\my-venv\lib\site-
> packages\django\db\migrations\operations\fields.py", line 147, in
> database_forwards
> schema_editor.remove_field(from_model,
> from_model._meta.get_field(self.name))
> File "C:\my-venv\lib\site-
> packages\django\db\backends\sqlite3\schema.py", line 249, in remove_field
> self._remake_table(model, delete_fields=[field])
> File "C:\my-venv\lib\site-
> packages\django\db\backends\sqlite3\schema.py", line 199, in
> _remake_table
> self.quote_name(model._meta.db_table),
> File "C:\my-venv\lib\site-packages\django\db\backends\base\schema.py",
> line 112, in execute
> cursor.execute(sql, params)
> File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
> 79, in execute
> return super(CursorDebugWrapper, self).execute(sql, params)
> File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
> 64, in execute
> return self.cursor.execute(sql, params)
> File "C:\my-venv\lib\site-packages\django\db\utils.py", line 94, in
> __exit__
> six.reraise(dj_exc_type, dj_exc_value, traceback)
> File "C:\my-venv\lib\site-packages\django\utils\six.py", line 685, in
> reraise
> raise value.with_traceback(tb)
> File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
> 64, in execute
> return self.cursor.execute(sql, params)
> File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py",
> line 337, in execute
> return Database.Cursor.execute(self, query, params)
> django.db.utils.OperationalError: near ")": syntax error
> }}}
>
> The problem is that the SQL query generated by this
> ([https://github.com/django/django/blob/320ec4ed27c254a87e09a70601b1b27ae0a0456e/django/db/backends/sqlite3/schema.py#L195-L198
> code block]) in ''**django/db/backends/sqlite3/schema.py**'':
>
> {{{#!python
> # Copy data from the old table into the new table
> field_maps = list(mapping.items())
> self.execute("INSERT INTO %s (%s) SELECT %s FROM %s" % (
> self.quote_name(temp_model._meta.db_table),
> ', '.join(self.quote_name(x) for x, y in field_maps),
> ', '.join(y for x, y in field_maps),
> self.quote_name(model._meta.db_table),
> ))
> }}}
> reads:
>
> {{{#!sql
> 'INSERT INTO "testbed_browsertest" () SELECT FROM
> "testbed_browsertest__old" '
> }}}
>
> This is a wrong query in SQLite syntax. Correct query is:
>
> {{{#!sql
> 'INSERT INTO "testbed_browsertest" SELECT * FROM
> "testbed_browsertest__old" '
> }}}
>
> This has been verified by monkey-patching the query generated and having
> the migration succeed.
New description:
Consider the following class hierarchy:
{{{
#!python
class TestCase(models.Model):
name = models.CharField(max_length=10)
class TerminalTest(TestCase):
pass
class BrowserTest(TestCase):
pass
$ ./manage.py makemigrations testbed
Migrations for 'testbed':
testbed\migrations\0001_initial.py:
- Create model TestCase
- Create model BrowserTest
- Create model TerminalTest
$ ./manage.py migrate testbed
Operations to perform:
Apply all migrations: testbed
Running migrations:
Applying testbed.0001_initial... OK
}}}
Now we want to make the parent an abstract class:
{{{
#!python
class TestCase(models.Model):
name = models.CharField(max_length=10)
class Meta:
abstract = True
class TerminalTest(TestCase):
pass
class BrowserTest(TestCase):
pass
./manage.py makemigrations testbed
You are trying to add a non-nullable field 'id' to browsertest without a
default; we can't do that (the database needs something to populate
existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a
null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can
do e.g. timezone.now
Type 'exit' to exit this prompt
>>> 1
You are trying to add a non-nullable field 'name' to browsertest without a
default; we can't do that (the database needs something to populate
existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a
null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can
do e.g. timezone.now
Type 'exit' to exit this prompt
>>> ' '
You are trying to add a non-nullable field 'id' to terminaltest without a
default; we can't do that (the database needs something to populate
existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a
null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can
do e.g. timezone.now
Type 'exit' to exit this prompt
>>> 1
You are trying to add a non-nullable field 'name' to terminaltest without
a default; we can't do that (the database needs something to populate
existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows with a
null value for this column)
2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can
do e.g. timezone.now
Type 'exit' to exit this prompt
>>> ' '
Migrations for 'testbed':
testbed\migrations\0002_auto_20170315_1711.py:
- Remove field testcase_ptr from browsertest
- Remove field testcase_ptr from terminaltest
- Add field id to browsertest
- Add field name to browsertest
- Add field id to terminaltest
- Add field name to terminaltest
- Delete model TestCase
$ ./manage.py migrate testbed
Operations to perform:
Apply all migrations: testbed
Running migrations:
Applying testbed.0002_auto_20170315_1600...Traceback (most recent call
last):
File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
64, in execute
return self.cursor.execute(sql, params)
File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py",
line 337, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: near ")": syntax error
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "C:\my-venv\lib\site-packages\django\core\management\__init__.py",
line 367, in execute_from_command_line
utility.execute()
File "C:\my-venv\lib\site-packages\django\core\management\__init__.py",
line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\my-venv\lib\site-packages\django\core\management\base.py", line
294, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\my-venv\lib\site-packages\django\core\management\base.py", line
345, in execute
output = self.handle(*args, **options)
File "C:\my-venv\lib\site-
packages\django\core\management\commands\migrate.py", line 204, in handle
fake_initial=fake_initial,
File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake,
fake_initial=fake_initial)
File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake,
fake_initial=fake_initial)
File "C:\my-venv\lib\site-packages\django\db\migrations\executor.py",
line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "C:\my-venv\lib\site-packages\django\db\migrations\migration.py",
line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state,
project_state)
File "C:\my-venv\lib\site-
packages\django\db\migrations\operations\fields.py", line 147, in
database_forwards
schema_editor.remove_field(from_model,
from_model._meta.get_field(self.name))
File "C:\my-venv\lib\site-
packages\django\db\backends\sqlite3\schema.py", line 249, in remove_field
self._remake_table(model, delete_fields=[field])
File "C:\my-venv\lib\site-
packages\django\db\backends\sqlite3\schema.py", line 199, in _remake_table
self.quote_name(model._meta.db_table),
File "C:\my-venv\lib\site-packages\django\db\backends\base\schema.py",
line 112, in execute
cursor.execute(sql, params)
File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
64, in execute
return self.cursor.execute(sql, params)
File "C:\my-venv\lib\site-packages\django\db\utils.py", line 94, in
__exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "C:\my-venv\lib\site-packages\django\utils\six.py", line 685, in
reraise
raise value.with_traceback(tb)
File "C:\my-venv\lib\site-packages\django\db\backends\utils.py", line
64, in execute
return self.cursor.execute(sql, params)
File "C:\my-venv\lib\site-packages\django\db\backends\sqlite3\base.py",
line 337, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: near ")": syntax error
}}}
The problem is that the SQL query generated by this
([https://github.com/django/django/blob/320ec4ed27c254a87e09a70601b1b27ae0a0456e/django/db/backends/sqlite3/schema.py#L195-L198
code block]) in ''**django/db/backends/sqlite3/schema.py**'':
{{{#!python
# Copy data from the old table into the new table
field_maps = list(mapping.items())
self.execute("INSERT INTO %s (%s) SELECT %s FROM %s" % (
self.quote_name(temp_model._meta.db_table),
', '.join(self.quote_name(x) for x, y in field_maps),
', '.join(y for x, y in field_maps),
self.quote_name(model._meta.db_table),
))
}}}
reads:
{{{#!sql
'INSERT INTO "testbed_browsertest" () SELECT FROM
"testbed_browsertest__old" '
}}}
This is a wrong query in SQLite syntax. Correct query is:
{{{#!sql
'INSERT INTO "testbed_browsertest" SELECT * FROM
"testbed_browsertest__old" '
}}}
This has been verified by monkey-patching the query generated and having
the migration move to the next stage -- where it fails for a completely
different reason (nothing to do with SQL). See bug
[https://code.djangoproject.com/ticket/27941 27941]
--
Comment (by Sawan Vithlani):
Please see bug [https://code.djangoproject.com/ticket/27941 27941] for
related bug.
--
Ticket URL: <https://code.djangoproject.com/ticket/27940#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/063.5d7672e3d6b39b33fbba638f85264de1%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
