Re: Django migrations taking all the memory on Django 1.9

2016-09-08 Thread andrea crotti
Ah ok thanks, well generally it's not a blocker, except that we first have to upgrade to 1.10. We have done a lot of migrations in the last few months and given the size of the project it always takes a long time from one version to the other, so just not sure when I'll be able to test these ch

Re: Django migrations taking all the memory on Django 1.9

2016-09-06 Thread Tim Graham
Per our support versions policy, Django 1.9.x is only receiving fixes for security and data loss issues. Migrations optimizations are unlikely to qualify for a backport to Django 1.10.x either. Hopefully that doesn't demotivate you from contributing an improvement. https://docs.djangoproject.co

Re: Django migrations taking all the memory on Django 1.9

2016-09-05 Thread andrea crotti
Thanks Marcus and others, this issue is in general getting worse and worse. We did some more investigation and we should probably just help out the work you are doing. Do you think it would be possible to apply these changes also on Django 1.9.9 or it has to be from master/1.10 only? I also tr

Re: Django migrations taking all the memory on Django 1.9

2016-08-08 Thread Tim Graham
I think that error suggests that you disabled migrations for an app (something_else) that's a dependency of another app that doesn't have migrations disabled (something). That isn't permitted. I guess there's no good solution for your requirements as of now. On Monday, August 8, 2016 at 6:02:48

Re: Django migrations taking all the memory on Django 1.9

2016-08-08 Thread andrea crotti
Since we also use --keepdb I thought we could simply avoid running all the migrations, and maybe dynamically even skip all the apps that were not changed at all. So I just tried something silly like this: NO_MIGRATIONS = os.environ.get('NO_MIGRATIONS', 'false').lower() in ('true', '1') if NO

Re: Django migrations taking all the memory on Django 1.9

2016-08-06 Thread Tim Graham
As long as your tests don't rely on any data migrations, you can set MIGRATIONS_MODULES['app'] = None for all apps in a test settings file as described in https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-MIGRATION_MODULES to disable migrations while testing. There's a ticket

Re: Django migrations taking all the memory on Django 1.9

2016-08-06 Thread andrea crotti
Ok great thanks for the answer Markus. And yes I can try to help, it's quite a big issue for us.. Anything else we can do to improve the situation in the meanwhile? In general I was wondering why do we need to handle all the "baggage" of supporting backward migrations and doing things 100% prope

Re: Django migrations taking all the memory on Django 1.9

2016-08-05 Thread Markus Holtermann
Thanks Andrea, this is a known issue. Let me give you a bit of a history on this: In 1.7 Django didn't do any caching. That was awfully slow for huge projects like yours. In 1.8 Django introduced heavy caching. Things improved speed wise but failed horrifically when you were altering relations