#24628: Squash migration is not marked as applied when the migrations it
replaces
are
----------------------------+------------------------------------
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------+------------------------------------
Old description:
> After some time in the chat with MarkusH, we decided that this should be
> a bug report.
>
> Some context: I have quite some migrations (33) of which the first
> already was a squash of several migrations. It did get it's 'replaced'
> field removed though, as the original migrations it replaced are long
> gone already. The goal now was to get that long list of migrations down
> to one again, for peace of mind.
>
> This led to the following symptoms: When I created the squashed migration
> everything seemed fine. ./manage.py migrate did say that nothing was to
> be done (but then again, all the migrations to be squashed where already
> applied).
>
> But adding another migration after that ended in tears - ./manage migrate
> didn't want to apply it and told me so in no uncertain terms.
>
> {{{
> (pycess)dwt@atlan ~/Code/Projekte/pycess/pycess (git)-[master] %
> ./manage.py migrate
> Traceback (most recent call last):
> File "./manage.py", line 10, in <module>
> execute_from_command_line(sys.argv)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
> line 330, in execute_from_command_line
> utility.execute()
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
> line 322, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
> line 347, in run_from_argv
> self.execute(*args, **cmd_options)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
> line 398, in execute
> output = self.handle(*args, **options)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/core/management/commands/migrate.py",
> line 86, in handle
> executor = MigrationExecutor(connection,
> self.migration_progress_callback)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/executor.py",
> line 19, in __init__
> self.loader = MigrationLoader(self.connection)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
> line 47, in __init__
> self.build_graph()
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
> line 281, in build_graph
> _reraise_missing_dependency(migration, parent, e)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
> line 264, in _reraise_missing_dependency
> raise exc
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
> line 274, in build_graph
> self.graph.add_dependency(migration, key, parent)
> File
> "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/graph.py",
> line 124, in add_dependency
> parent
> django.db.migrations.graph.NodeNotFoundError: Migration
> process.0002_auto_20150411_1005 dependencies reference nonexistent parent
> node ('process', '0001_squashed_initial_2')
> }}}
>
> Here we had quite some discussion in #django-dev with MarkusH, of which
> the result to me was that a) django doesn't seem to ever record that a
> squashed migration is applied, at least as long as it is recognizable by
> django as a squashed migration (i.e. it has a .replaced property). b)
> there seems to be no way to tell django that the replacing squashed
> migration really is already applied for this database.
>
> The last one turned out to be achievable if you remove the old migrations
> and the .replaces property from the squashed migration and then apply it
> with --fake
>
> MarkusH might want to say more here that he can describe better.
>
> For ease of reproduction I'm attaching the project where this occurred
> for me.
New description:
(I am attempting a shorter and clearer description of this bug, but
leaving the original description intact below - carljm).
In Django 1.8, consider an app `A` with migrations `1` and `2` and a
squashed migration `1_squashed_2` that replaces both `1` and `2`. Consider
a deployment of this app which has only `1` applied. When it receives the
update including `2` and `1_squashed_2` and is migrated, `2` is marked as
applied in the database but `1_squashed_2` is not.
This does not immediately appear to be a problem, because as long as
migrations `1` and `2` exist and `1_squashed_2` is marked as replacing
them, Django automatically considers `1_squashed_2` to be applied (and
shows it as such in `showmigrations`). But Django never actually records
`1_squashed_2` itself as applied in the database.
At some point, once all deployments have migrated through `2`, the idea is
that `1` and `2` can be removed, and the `replaces` tag removed from
`1_squashed_2`. At this point we have a problem, because now Django
considers `1_squashed_2` to not be applied, and tries to apply it again,
causing errors because tables already exist, etc. The only solution is to
manually `--fake` squashed migrations on all deployments when their
`replaces` tag is removed, which is a pain and eliminates much of the
convenience of the migration system; `--fake` should not be necessary in
the normal course of things.
The solution appears simple: anytime a migration is marked as having been
applied, Django should check if it is part of a replaced set, and if that
replaced set is now fully applied, the replacing migration should also be
marked as applied.
It may be that this check should be performed by `migrate` even when it
hasn't applied any migrations in the current run, as that would help to
correct the corruption already caused by this bug in many existing
databases, but not yet noticed because the replaced migrations haven't yet
been removed. (This correction should probably be mentioned to the user so
it doesn't happen silently.)
Original report follows:
After some time in the chat with MarkusH, we decided that this should be a
bug report.
Some context: I have quite some migrations (33) of which the first already
was a squash of several migrations. It did get it's 'replaced' field
removed though, as the original migrations it replaced are long gone
already. The goal now was to get that long list of migrations down to one
again, for peace of mind.
This led to the following symptoms: When I created the squashed migration
everything seemed fine. ./manage.py migrate did say that nothing was to be
done (but then again, all the migrations to be squashed where already
applied).
But adding another migration after that ended in tears - ./manage migrate
didn't want to apply it and told me so in no uncertain terms.
{{{
(pycess)dwt@atlan ~/Code/Projekte/pycess/pycess (git)-[master] %
./manage.py migrate
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
line 330, in execute_from_command_line
utility.execute()
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
line 322, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
line 347, in run_from_argv
self.execute(*args, **cmd_options)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
line 398, in execute
output = self.handle(*args, **options)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/commands/migrate.py",
line 86, in handle
executor = MigrationExecutor(connection,
self.migration_progress_callback)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/executor.py",
line 19, in __init__
self.loader = MigrationLoader(self.connection)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 47, in __init__
self.build_graph()
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 281, in build_graph
_reraise_missing_dependency(migration, parent, e)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 264, in _reraise_missing_dependency
raise exc
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 274, in build_graph
self.graph.add_dependency(migration, key, parent)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/graph.py",
line 124, in add_dependency
parent
django.db.migrations.graph.NodeNotFoundError: Migration
process.0002_auto_20150411_1005 dependencies reference nonexistent parent
node ('process', '0001_squashed_initial_2')
}}}
Here we had quite some discussion in #django-dev with MarkusH, of which
the result to me was that a) django doesn't seem to ever record that a
squashed migration is applied, at least as long as it is recognizable by
django as a squashed migration (i.e. it has a .replaced property). b)
there seems to be no way to tell django that the replacing squashed
migration really is already applied for this database.
The last one turned out to be achievable if you remove the old migrations
and the .replaces property from the squashed migration and then apply it
with --fake
MarkusH might want to say more here that he can describe better.
For ease of reproduction I'm attaching the project where this occurred for
me.
--
Comment (by carljm):
I just ran into this as well. I think it is definitely a bug and should be
fixed in code, not just in documentation.
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/061.f6f4179d70a11a21f8b2e5e739654385%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
