For some reason the text is white… Here it is in black: Hey Marcin,
The problem is that data migration based on app layer (python objects, ie. Models and Managers here) will cause troubles after some time (when app is changing). In the other words - you cannot rely on your app layer when doing database changes. You should never do that, especially for projects requiring LTS. In theory I understand the idea, but in practice, migrations, the ORM and the content type model are all part of Django and I don’t really see how the app changing could cause troubles. Do you have an example in mind? Maybe "automatic" is not a proper word. They should be created automatically, but should be applied "at the right time". Ok, that we agree on! CT opt-in should be connected to a signal related to the creation of migration files (Autodetector?), not to a signal related to their execution. I.e. pre/post_autodection signals should be introduced. I think we agree that the solution would be some sort of signal triggered when a model creation/deletion is detected (in the `makemigrations` phase as opposed to `migrate`) so that some code is added to the generated migration. The use of a signal is important to keep things decoupled and flexible. The “some code is added to the migration” part still needs to be determined, either in the shape of insert/delete statements or a RunPython leveraging the ORM. In both cases, the values to insert (Adding a model) or to lookup for delete (Removing a model) are just 2 strings, the app label and the model class name. After adding contrib.contenttypes, Django should check existence of django_content_type table. If exists, Django should only check for data changes and generate series of inserts/deletes. If not, Django should generate inserts for all models. If CT is removed later, Django should remove all CT data . It’s a good idea but I don’t know offhand how we can keep migrations and content type decoupled to do that (especially the removal). Finally, I also think the concept could be extended to the permission model which faces similar issues. Regards -- Arthur On October 5, 2018 at 9:30:58 AM, Arthur Rio ([email protected]) wrote: Hey Marcin, The problem is that data migration based on app layer (python objects, ie. Models and Managers here) will cause troubles after some time (when app is changing). In the other words - you cannot rely on your app layer when doing database changes. You should never do that, especially for projects requiring LTS. In theory I understand the idea, but in practice, migrations, the ORM and the content type model are all part of Django and I don’t really see how the app changing could cause troubles. Do you have an example in mind? Maybe "automatic" is not a proper word. They should be created automatically, but should be applied "at the right time". Ok, that we agree on! CT opt-in should be connected to a signal related to the creation of migration files (Autodetector?), not to a signal related to their execution. I.e. pre/post_autodection signals should be introduced. I think we agree that the solution would be some sort of signal triggered when a model creation/deletion is detected (in the `makemigrations` phase as opposed to `migrate`) so that some code is added to the generated migration. The use of a signal is important to keep things decoupled and flexible. The “some code is added to the migration” part still needs to be determined, either in the shape of insert/delete statements or a RunPython leveraging the ORM. In both cases, the values to insert (Adding a model) or to lookup for delete (Removing a model) are just 2 strings, the app label and the model class name. After adding contrib.contenttypes, Django should check existence of django_content_type table. If exists, Django should only check for data changes and generate series of inserts/deletes. If not, Django should generate inserts for all models. If CT is removed later, Django should remove all CT data . It’s a good idea but I don’t know offhand how we can keep migrations and content type decoupled to do that (especially the removal). Finally, I also think the concept could be extended to the permission model which faces similar issues. Regards -- Arthur On October 4, 2018 at 4:47:19 PM, Marcin Nowak ([email protected]) wrote: Hi Arthur. BTW: RunPython() is another thing, which can break your migrations, and should not be used (especially not by Django internally), because it relies on the application layer. How else can you do a data migration? There is no > `migrations.InsertIntoTable`, > You're right. That's why "Insert" should be (in my opinion) introduced. Together with "migrations.Delete". The problem is that data migration based on app layer (python objects, ie. Models and Managers here) will cause troubles after some time (when app is changing). In the other words - you cannot rely on your app layer when doing database changes. You should never do that, especially for projects requiring LTS. RunPython() should be used only when you cannot do anything else. In such case you must accept all consequences. I'm not against RunPython(), but against doing data migrations using it. The problem with hypothetical "Insert" operation is with mapping field types. Insert cannot use Models directly (app layer is changing over a time), but should "know" how to map arguments (python types, values) to a database literals. It can be achieved by introducing field mapping for a every insert or per migration file (something like "model freeze", but only for used fields). Also Insert should not accept variables calculated in the app layer (within a migration) - it should contain only plain/direct data. But using Python as a language for migrations, will be hard to avoid such possibility. This is important, because database management should not rely on app layer. Using variables (i.e. from settings) would result in inconsistent data changes between different environments. the only other way currently would be to run a `migrations.RunSql` query > which may look different based on the database used. > RunSQL is not the solution for db agnostic operations. It may be only used within a project, because db engine used changes rarely (due to the nature and importance of the data and relational databases, and systems dependent on the db). But RunSQL is a handful operation, because SQL is a natural language for db management. I'm doing many raw sqls in migrations. > Two things are wrong: > > - automatic creation of content types > > Why is it wrong to automatically create a content type? > Maybe "automatic" is not a proper word. They should be created automatically, but should be applied "at the right time". > Content type is an opt-in feature since you can remove it from > `INSTALLED_APPS`. > I know, but it is required by contrib.auth. I saw no project depending on something else, so CT is optional.. but theoretically ;) > > - creation of content types after running migrations > > That’s the only real problem for me. Maybe using a signal for > `migrations.CreateModel` (e.g. post_create_model), instead of using > `post_migrate` would fix it, but there may be other scenarios I’m not > thinking about. > Because signals are related to the app layer and mixing app and db layers together is generally wrong (sorry for wording, please read as "not so good"), changing one signal to another is not a good solution. Before I wrote about the issue, I was doing diagnosis and I was thinking about the issue for a while. So I think that the best place for applying data changes is a migration. Django can detect model changes (adding and removals of models, too), so it can generate series of inserts or deletes in the right place, to run them *at the right time*. Or if `django.contrib.contenttypes` is only added later on to > `INSTALLED_APPS`? After adding contrib.contenttypes, Django should check existence of django_content_type table. If exists, Django should only check for data changes and generate series of inserts/deletes. If not, Django should generate inserts for all models. If CT is removed later, Django should remove all CT data . CT opt-in should be connected to a signal related to the creation of migration files (Autodetector?), not to a signal related to their execution. I.e. pre/post_autodection signals should be introduced. Kind Regards, Marcin -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/af22034d-3983-4b19-87b2-256adc1cb11a%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/af22034d-3983-4b19-87b2-256adc1cb11a%40googlegroups.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADOBPEGi9Zz_E_%3DcjULE0SBsas7j%3DwZRmmK603iwD0X1%2BEQ3Ug%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
