Corrupted Migration?
Hi all, A migration I was running (manage.py migrate) died in the middle. I deleted the migration file, fixed the problem (wrong default value), recreated the migration, and re-ran it. But now, I'm getting an inconsistent state (error message: [Model] has no field named [field]). When I check the database, all the old fields are there. However, when I hop into a Django shell (manage.py shell) and check the model fields ([Model]._meta.fields), they reflect the new, post-migration state. What is going on, and how do I fix this? Thanks! -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/95f8-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Corrupted Migration?
Thanks for the reply. I should have mentioned that I tried all of that: the problem is the model state is ahead of the database state, not behind it. It seems that the database migration was atomic, so everything was rolled back as that transaction failed, but the same thing didn't happen at the model level. So, now I'm stuck with a model state that the previous migrations don't capture. On Wednesday, September 30, 2015 at 5:08:22 AM UTC-7, Mike Dewhirst wrote: > > On 30/09/2015 4:16 PM, Daniel Chen wrote: > > Hi all, > > > > A migration I was running (manage.py migrate) died in the middle. I > > deleted the migration file, fixed the problem (wrong default value), > > recreated the migration, and re-ran it. But now, I'm getting an > > inconsistent state (error message: [Model] has no field named [field]). > > When I check the database, all the old fields are there. However, when I > > hop into a Django shell (manage.py shell) and check the model fields > > ([Model]._meta.fields), they reflect the new, post-migration state. What > > is going on, and how do I fix this? > > Do it again. Delete the migration. Take note of the last valid > migration. Open the django_migrations table and delete the migration > record(s) after the last valid migration. > > makemigrations again and then migrate again to bring the database into > line with your models. > > BTW, the shell is looking at the model not the database. > > Mike > > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Django users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to django-users...@googlegroups.com > > <mailto:django-users+unsubscr...@googlegroups.com >. > > To post to this group, send email to django...@googlegroups.com > > > <mailto:django...@googlegroups.com >. > > Visit this group at http://groups.google.com/group/django-users. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/django-users/95f8-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com > > > < > https://groups.google.com/d/msgid/django-users/95f8-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Corrupted Migration?
I have tried that (by the way, the offending migration didn't show up in the django_migrations table because the migration failed). I think we might be talking past each other. Sorry if I'm being unclear. Here's an example of what's going on: 1. Before migration, my database is at state X, and my migrations files reflect that. 2. I change my model file, updating the state (let's call it state Y). Then, I run make migrations, generating a new migration file that corresponds to state Y. 3. I run the migrate command to get my database to state Y. It fails, so now my database is back at state X. There is nothing in the database to delete (not even from the django_migrations table) because the transaction was rolled back. 4. I delete the migration file that corresponds to state Y. My database is back in state X, and my migration files again reflect that. 5. I run the migrate command again, and it fails (same problem). The error is django.db.models.fields.FieldDoesNotExist: [Model] has no field named [field]. 6. [field] corresponds to a new field in state Y, not in an existing field in state X. I'm confused - why is it looking for a field that doesn't exist in the pre-migrated state (state X)? 7. So, I assume that the model state somehow is de-synced from the database/migration state. I hop into the Django shell and check the model metadata. The fields exactly match state Y. My database is still in state X. Does that make sense? On Wednesday, September 30, 2015 at 4:10:18 PM UTC-7, Mike Dewhirst wrote: > > On 1/10/2015 6:51 AM, Daniel Chen wrote: > > Thanks for the reply. I should have mentioned that I tried all of that: > > the problem is the model state is ahead of the database state, not > > behind it. It seems that the database migration was atomic, so > > everything was rolled back as that transaction failed, but the same > > thing didn't happen at the model level. So, now I'm stuck with a model > > state that the previous migrations don't capture. > > The model is the master. It is outside the transaction and will never be > touched by migrations. > > Your task is to identify which previous migration matches the current > database structure and delete all migrations after that point. That > means deleting the migration files after the one you identified AND > deleting the migration records in django_migrations in the database. > > Thereafter, the next migration will pick up all the changes in the model > and bring the database back into alignment. > > This is all much the same as I said last time so if this doesn't work it > means I haven't understood your problem. > > > > > On Wednesday, September 30, 2015 at 5:08:22 AM UTC-7, Mike Dewhirst > wrote: > > > > On 30/09/2015 4:16 PM, Daniel Chen wrote: > > > Hi all, > > > > > > A migration I was running (manage.py migrate) died in the middle. > I > > > deleted the migration file, fixed the problem (wrong default > value), > > > recreated the migration, and re-ran it. But now, I'm getting an > > > inconsistent state (error message: [Model] has no field named > > [field]). > > > When I check the database, all the old fields are there. However, > > when I > > > hop into a Django shell (manage.py shell) and check the model > fields > > > ([Model]._meta.fields), they reflect the new, post-migration > > state. What > > > is going on, and how do I fix this? > > > > Do it again. Delete the migration. Take note of the last valid > > migration. Open the django_migrations table and delete the migration > > record(s) after the last valid migration. > > > > makemigrations again and then migrate again to bring the database > into > > line with your models. > > > > BTW, the shell is looking at the model not the database. > > > > Mike > > > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the > Google > > > Groups "Django users" group. > > > To unsubscribe from this group and stop receiving emails from it, > > send > > > an email to django-users...@googlegroups.com > > > <mailto:django-users+unsubscr...@googlegroups.com > >. > > > To post to this group, send email to django...@googlegroups.com > > > > > <mailto:django...@googleg
Re: Corrupted Migration?
@Mike: I'm using Python 2.7.10 and Django 1.7. I'm not sure the ticket applies, but I appreciate the link! @James: Sorry, I misspoke! [field] actually corresponds to a old field in state X. Before adding the new fields in state Y, the migration is trying to remove some old fields in state X but isn't finding them. I think this means that I'm unable to re-run that migration since it's looking for state X fields to remove, but can't find them (the models is already in state Y). My confusion is in the fact that the model state wouldn't roll back to state X after the first failed transaction (like the DB did). This would be analogous to my writing you a check that will bounce. Once the check bounces, both your personal wealth and your bank statement should be back to what they were originally. But in this particular case, it looks like your personal wealth is stuck in a state that assumes the check has been cashed and your account has been credited, but your bank account disagrees :) By the way, I talk about model state under the assumption that the Django ORM somehow internally keeps track of the models - I'm not sure exactly how it does this or if it does this at all. On Thursday, October 1, 2015 at 1:17:43 AM UTC-7, James Schneider wrote: > > > > I have tried that (by the way, the offending migration didn't show up in > the django_migrations table because the migration failed). > > > > I think we might be talking past each other. Sorry if I'm being unclear. > Here's an example of what's going on: > > Before migration, my database is at state X, and my migrations files > reflect that. > > I change my model file, updating the state (let's call it state Y). > Then, I run make migrations, generating a new migration file that > corresponds to state Y. > > I run the migrate command to get my database to state Y. It fails, so > now my database is back at state X. There is nothing in the database to > delete (not even from the django_migrations table) because the transaction > was rolled back. > > I delete the migration file that corresponds to state Y. My database is > back in state X, and my migration files again reflect that. > > I run the migrate command again, and it fails (same problem). The error > is django.db.models.fields.FieldDoesNotExist: [Model] has no field named > [field]. > > After you delete Y migration, have you searched your migration files to > verify that there are no references to that field? > > > [field] corresponds to a new field in state Y, not in an existing field > in state X. I'm confused - why is it looking for a field that doesn't exist > in the pre-migrated state (state X)? > > Do you have any RunPython or other custom modifications in your migrations > that would cause an object based on your modified model to be created (and > saved), possibly relying on a default value for the new 'field' specified > in your Y-version model, maybe as part of a data migration? > > > So, I assume that the model state somehow is de-synced from the > database/migration state. I hop into the Django shell and check the model > metadata. The fields exactly match state Y. My database is still in state X. > > Does that make sense? > > Makes perfect sense when you think about it. Being out of sync in this > manner happens constantly, and is exactly why migrations exist. > > Django makes the assumption that the current values in your models.py > files exactly match the layout of the DB, since you should be syncing > everything using migrations and validating via tests. > > Any time you modify a model, the DB and you model definitions become out > of sync, with your models being 'ahead' of your DB since you haven't > informed the DB of your model changes via migrations yet. > > Think of it like someone writing you a check. Your relative personal > wealth increases as soon as you receive the check, but your bank statement > won't agree with you until you submit the check to them so that they can > credit your account accordingly. > > Feel free to write me a check (for a large amount, bigger the better) so I > can demonstrate the effect it has on my bank account. ;-) > > -James > On Thursday, October 1, 2015 at 1:17:43 AM UTC-7, James Schneider wrote: > > > > I have tried that (by the way, the offending migration didn't show up in > the django_migrations table because the migration failed). > > > > I think we might be talking past each other. Sorry if I'm being unclear. > Here's an example of what's going on: > > Before migration, my database is at state X, and my migrations files > reflect that. > > I change my model file, updating the state (let's call it state Y). > Then, I run make migrations, generating a new migration file that > corresponds to state Y. > > I run the migrate command to get my database to state Y. It fails, so > now my database is back at state X. There is nothing in the database to > delete (not even from the django_migrations table
Re: Corrupted Migration?
@Mike: I'm using Python 2.7.10 and Django 1.7. I'm not sure the ticket applies, but I appreciate the link! @James: Sorry, I misspoke! [field] actually corresponds to a old field in state X. Before adding the new fields in state Y, the migration is trying to remove some old fields in state X but isn't finding them. I think this means that I'm unable to re-run that migration since it's looking for state X fields to remove, but can't find them (the model is already in state Y). My confusion is in the fact that the model state wouldn't roll back to state X after the first failed transaction (like the DB did). This would be analogous to my writing you a check that will bounce. Once the check bounces, both your personal wealth and your bank statement should be back to what they were originally. But in this particular case, it looks like your personal wealth is stuck in a state that assumes the check has been cashed and your account has been credited, but your bank account disagrees :) By the way, I talk about model state under the assumption that the Django ORM somehow internally keeps track of the models - I'm not sure exactly how it does this or if it does this at all. If you could tell me more about this, that'd be much appreciated! On Thursday, October 1, 2015 at 1:17:43 AM UTC-7, James Schneider wrote: > > > > I have tried that (by the way, the offending migration didn't show up in > the django_migrations table because the migration failed). > > > > I think we might be talking past each other. Sorry if I'm being unclear. > Here's an example of what's going on: > > Before migration, my database is at state X, and my migrations files > reflect that. > > I change my model file, updating the state (let's call it state Y). > Then, I run make migrations, generating a new migration file that > corresponds to state Y. > > I run the migrate command to get my database to state Y. It fails, so > now my database is back at state X. There is nothing in the database to > delete (not even from the django_migrations table) because the transaction > was rolled back. > > I delete the migration file that corresponds to state Y. My database is > back in state X, and my migration files again reflect that. > > I run the migrate command again, and it fails (same problem). The error > is django.db.models.fields.FieldDoesNotExist: [Model] has no field named > [field]. > > After you delete Y migration, have you searched your migration files to > verify that there are no references to that field? > > > [field] corresponds to a new field in state Y, not in an existing field > in state X. I'm confused - why is it looking for a field that doesn't exist > in the pre-migrated state (state X)? > > Do you have any RunPython or other custom modifications in your migrations > that would cause an object based on your modified model to be created (and > saved), possibly relying on a default value for the new 'field' specified > in your Y-version model, maybe as part of a data migration? > > > So, I assume that the model state somehow is de-synced from the > database/migration state. I hop into the Django shell and check the model > metadata. The fields exactly match state Y. My database is still in state X. > > Does that make sense? > > Makes perfect sense when you think about it. Being out of sync in this > manner happens constantly, and is exactly why migrations exist. > > Django makes the assumption that the current values in your models.py > files exactly match the layout of the DB, since you should be syncing > everything using migrations and validating via tests. > > Any time you modify a model, the DB and you model definitions become out > of sync, with your models being 'ahead' of your DB since you haven't > informed the DB of your model changes via migrations yet. > > Think of it like someone writing you a check. Your relative personal > wealth increases as soon as you receive the check, but your bank statement > won't agree with you until you submit the check to them so that they can > credit your account accordingly. > > Feel free to write me a check (for a large amount, bigger the better) so I > can demonstrate the effect it has on my bank account. ;-) > > -James > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1ec20d88-1c82-4d28-9561-4065b2635288%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Corrupted Migration?
Hi all,
Thanks for all the help! Sorry for the incredibly slow response, but I just
wanted to give an update:
The problem was that I was trying to remove a foreign key (let's call that
foreign key "book", referencing a "book" table). I had to manually go into
the migration and add '_id' (e.g., "book_id") in every reference to that
model (odd that you need to manually update the migrations, but whatever).
Afterwards,
I was able to successfully dump the DML - which looked more or less correct.
However, the next problem was that I had unique_together constraints on
that foreign key and another existing key (e.g., unique_together("book_id",
"author_id")). I got the corresponding error message: "ValueError: Found
wrong number (0) of constraints for main_tablename(book_id, author_id)", so
I still wasn't able to complete the migration.
I worked around this by not dropping that column ("book_id" is still in the
table), and the migration finally worked. However, I'd love to get rid of
that column once and for all. So, if anyone knows a solution, please let me
know!
Dan
On Friday, October 2, 2015 at 11:26:50 PM UTC-7, James Schneider wrote:
>
> @James: Sorry, I misspoke! [field] actually corresponds to a old field in
>>> state X. Before adding the new fields in state Y, the migration is trying
>>> to remove some old fields in state X but isn't finding them. I think this
>>> means that I'm unable to re-run that migration since it's looking for state
>>> X fields to remove, but can't find them (the model is already in state Y).
>>> My confusion is in the fact that the model state wouldn't roll back to
>>> state X after the first failed transaction (like the DB did).
>>>
>>
>> That makes even less sense than your original issue. Are you sure that
>> migration X was applied properly? What versions of migrations are stored in
>> your django_migrations table? Is the latest migration listed in there for
>> your app? You should be able to find it with a raw SQL statement, something
>> like "select * from django_migrations where app = '';".
>>
>> Does [field] currently exist in your database so that it can be removed?
>> If not, you've had issues with migrations beyond this particular i
>>
>> No. The model is all your own work. Anything you write stays written. The
>> migration system tries to implement your work. If it fails, it will roll
>> the database back to its previous state. It does nothing to your work. You
>> need to take note of the failure, decide how to fix your work and try again.
>> ssue. Were any migration files removed? Does the database match migration
>> X?
>>
>> Even better, can you also post the full traceback that you receive during
>> the failed migrations? That might lead to more clues.
>>
>
> Whoops, looks like part of my response got wrapped around Mike's by
> accident. Here's what it was supposed to say:
>
> "That makes even less sense than your original issue. Are you sure that
> migration X was applied properly? What versions of migrations are stored in
> your django_migrations table? Is the latest migration listed in there for
> your app? You should be able to find it with a raw SQL statement, something
> like "select * from django_migrations where app = '';".
>
> Does [field] currently exist in your database so that it can be removed?
> If not, you've had issues with migrations beyond this particular issue.
> Were any migration files removed? Does the database match migration X?
>
> Even better, can you also post the full traceback that you receive during
> the failed migrations? That might lead to more clues."
>
> -James
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/0482bf2e-47eb-408e-90c3-0a654b7a68c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Corrupted Migration?
Another update:
The actual problem was the automatically generated migrations from the
makemigrations command removed the database fields before removing the
constraints on those fields (leading to a FieldDoesNotExist error when
trying to drop the constraint). The solution was just to move the
RemoveField operation below the AlterUniqueTogether operation in the
migration file. (Also, it turns out that removing the "_id" suffix didn't
matter after all).
Thanks again for the help, everyone!
On Tuesday, October 20, 2015 at 10:18:02 PM UTC-7, Daniel Chen wrote:
>
> Hi all,
>
> Thanks for all the help! Sorry for the incredibly slow response, but I
> just wanted to give an update:
>
> The problem was that I was trying to remove a foreign key (let's call that
> foreign key "book", referencing a "book" table). I had to manually go into
> the migration and add '_id' (e.g., "book_id") in every reference to that
> model (odd that you need to manually update the migrations, but whatever).
> Afterwards,
> I was able to successfully dump the DML - which looked more or less correct.
> However, the next problem was that I had unique_together constraints on
> that foreign key and another existing key (e.g., unique_together("book_id",
> "author_id")). I got the corresponding error message: "ValueError: Found
> wrong number (0) of constraints for main_tablename(book_id, author_id)", so
> I still wasn't able to complete the migration.
>
> I worked around this by not dropping that column ("book_id" is still in
> the table), and the migration finally worked. However, I'd love to get rid
> of that column once and for all. So, if anyone knows a solution, please let
> me know!
>
> Dan
>
>
> On Friday, October 2, 2015 at 11:26:50 PM UTC-7, James Schneider wrote:
>>
>> @James: Sorry, I misspoke! [field] actually corresponds to a old field in
>>>> state X. Before adding the new fields in state Y, the migration is trying
>>>> to remove some old fields in state X but isn't finding them. I think this
>>>> means that I'm unable to re-run that migration since it's looking for
>>>> state
>>>> X fields to remove, but can't find them (the model is already in state Y).
>>>> My confusion is in the fact that the model state wouldn't roll back to
>>>> state X after the first failed transaction (like the DB did).
>>>>
>>>
>>> That makes even less sense than your original issue. Are you sure that
>>> migration X was applied properly? What versions of migrations are stored in
>>> your django_migrations table? Is the latest migration listed in there for
>>> your app? You should be able to find it with a raw SQL statement, something
>>> like "select * from django_migrations where app = '';".
>>>
>>> Does [field] currently exist in your database so that it can be removed?
>>> If not, you've had issues with migrations beyond this particular i
>>>
>>> No. The model is all your own work. Anything you write stays written.
>>> The migration system tries to implement your work. If it fails, it will
>>> roll the database back to its previous state. It does nothing to your work.
>>> You need to take note of the failure, decide how to fix your work and try
>>> again.
>>> ssue. Were any migration files removed? Does the database match
>>> migration X?
>>>
>>> Even better, can you also post the full traceback that you receive
>>> during the failed migrations? That might lead to more clues.
>>>
>>
>> Whoops, looks like part of my response got wrapped around Mike's by
>> accident. Here's what it was supposed to say:
>>
>> "That makes even less sense than your original issue. Are you sure that
>> migration X was applied properly? What versions of migrations are stored in
>> your django_migrations table? Is the latest migration listed in there for
>> your app? You should be able to find it with a raw SQL statement, something
>> like "select * from django_migrations where app = '';".
>>
>> Does [field] currently exist in your database so that it can be removed?
>> If not, you've had issues with migrations beyond this particular issue.
>> Were any migration files removed? Does the database match migration X?
>>
>> Even better, can you also post the full traceback that you receive during
>> the failed migrations? That might lead to more clues."
>>
>> -James
>>
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/886186b5-737e-447e-b53b-86185316e3cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
