I think we should re-open ticket 31007[1] as there was a consensus in 2017 during the previous attempt at this[2] and things have not really changed - a 32bit primary key is still not a sensible default given the calamity that can occur when you hit the end of it’s range.
The sticky point is the implementation - how do we migrate new projects whilst minimising disruption to existing projects (who either won’t ever hit the limit, or have too much data to migrate wholesale). I think special casing the migrations framework is an avenue to explore, so I created this today and to my surprise it seems to work quite well: https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839 <https://github.com/orf/django/commit/0a335f208cee1376c25eff55c6f866de122c7839>. The obvious effect of this is that for old projects the Django model thinks its primary key is 64 bit while the underlying table is only 32 bit. I can’t think of much practical impact of this approach: - Python 3 doesn’t distinguish between 32/64 bit numbers so code will continue to work as-is. - Third party migrations that ship with 32 bit columns won’t be impacted, and it’s up to them to create migrations if they want to. - Adding `id = BigAutoField(primary_key=True)` to a model will create an explicit AlterField migration, when projects are able to migrate. Am I missing something? > I haven't thought this through, but maybe it's possible to restrict the > change to inside AutoField, rather than create a new field class. If its > db_parameters() method / db_type() could receive enough context from the > migration history, it could potentially determine its own database type. I’m really not sure how we would do that: wouldn’t it require a pretty big refactor? It would definitely be cleaner. 1. https://code.djangoproject.com/ticket/31007 <https://code.djangoproject.com/ticket/31007> 2. https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ <https://groups.google.com/forum/#!msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ> > On 12 Jun 2020, at 12:11, Adam Johnson <[email protected]> wrote: > > I haven't thought this through, but maybe it's possible to restrict the > change to inside AutoField, rather than create a new field class. If its > db_parameters() method / db_type() could receive enough context from the > migration history, it could potentially determine its own database type. For > those who want to fixed field sizes, BigAutoField and SmallAutoField could be > left, and MediumAutoField added. > > (Whilst poking around I think I spotted a small cleanup: > https://code.djangoproject.com/ticket/31698 > <https://code.djangoproject.com/ticket/31698> ) > > On Fri, 12 Jun 2020 at 11:40, Tom Forbes <[email protected] > <mailto:[email protected]>> wrote: >> I think we should restrict the setting between normal and big auto fields >> only. Allowing UUID's would be changing the type, with the potential for >> havoc with code incompalities throughout django. It's also not possible to >> migrate tables over to the new type. > > That’s a really good point, I got a bit swept up with the idea to realise > that it would break a lot of things if applied generally! > >> The autodetector knows if a model is new. It could be that during one >> version Django outputs BigAutoField for fields added in CreateModel only. > > We could make every automatic PK become a BigAutoField but include some logic > in the migrations framework to ignore changes between an auto_created > AutoField -> BigAutoField? This would ignore all existing models in your > migration history, but all completely new models would receive BigAutoField > PKs from the start. Users who explicitly add a `BigAutoField` to a model with > a previously created `AutoField` would see a migration created as the > `auto_created` flag is not set. It would also play nicely with foreign keys > which also need to be 8 bytes. > > This would take care of third party apps with migrations and not require a > setting but the downside is that model state will be slightly different from > the database. All models would have `BigAutoField`’s whereas the database > would only have an `AutoField`. This feels slightly iffy to me even though I > cannot think of a case where it would have any practical effect. > >> On 11 Jun 2020, at 19:22, Adam Johnson <[email protected] >> <mailto:[email protected]>> wrote: >> >> Big +1 on solving this from me. >> >> - The setting would take any dotted path to a class, or a single class name >> for a build in field. This would potentially solve [3], and could be useful >> to people who want to default to other fields like UUIDs (or a custom >> BigAutoField) for whatever reason >> >> I think we should restrict the setting between normal and big auto fields >> only. Allowing UUID's would be changing the type, with the potential for >> havoc with code incompalities throughout django. It's also not possible to >> migrate tables over to the new type. >> >> What do you think the solution is for third-party apps? They ship their own >> migrations and can't really be tied to project state. >> >> As Django migrations are derived from the current model state so there’s no >> way I can think of to express “make this auto-generated field a BigAutoField >> only if this model is new”. >> >> The autodetector knows if a model is new. It could be that during one >> version Django outputs BigAutoField for fields added in CreateModel only. >> >> On Thu, 11 Jun 2020 at 16:28, Tom Forbes <[email protected] >> <mailto:[email protected]>> wrote: >> I’d like to re-propose switching Django to use BigAutoField’s rather than >> the current AutoField. This has been proposed[1] before (and a MR made[2]) >> but it was closed due to implementation issues and not much else has >> happened since then. >> >> As many of you are aware the max value a standard AutoField can hold is >> 2,147,483,647 (2.1 billion) which sounds like more than you can ever need. >> But it’s often not, and you only find out at the worst possible time - out >> of the blue at night and during a period of rapid growth. The process for >> fixing this issue also becomes a lot harder as your data grows - when you’ve >> hit the limit you’re looking at a multi-hour `ALTER TABLE` on Postgres >> during which writes and reads are blocked, or a risky operation to create a >> new table with the correct primary key and copy old data over in batches. >> Basically if you’ve experienced this before you wouldn’t wish it on your >> worst enemy. >> >> I’m proposing that we add a `MODELS_PRIMARY_KEY` (name improvements >> welcome!) setting that _defaults_ to `BigAutoField`, with prominent >> docs/release notes saying that to preserve the existing behaviour this >> should be set to `AutoField`. I think this the only realistic way we can >> implement this for new projects in a way that ensures it will be used >> meaningfully and not forgotten about until it’s too late. >> >> Rails managed to do this migration somewhat painlessly due the big >> differences between Rails and Django models. As Django migrations are >> derived from the current model state so there’s no way I can think of to >> express “make this auto-generated field a BigAutoField only if this model is >> new”. The way I see it is that a global setting is very easy to toggle and >> there is little chance of missing the large numbers of migrations that would >> be generated during the Django update. Smaller applications could apply the >> migrations with little issue and larger applications would be able to >> opt-out (as well as be reminded that this is a problem they could face!). >> >> Some specifics: >> - The setting would take any dotted path to a class, or a single class name >> for a build in field. This would potentially solve [3], and could be useful >> to people who want to default to other fields like UUIDs (or a custom >> BigAutoField) for whatever reason >> - The setting would also be used for GenericForeignKeys, which right now are >> backed by a PositiveIntegerField and so suffer from the same AutoField >> limitations >> >> Any thoughts on this? >> >> Tom >> >> 1. >> https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ >> <https://groups.google.com/d/msg/django-developers/imBJwRrtJkk/P4g0Y87lAgAJ> >> >> 2. https://github.com/django/django/pull/8924/ >> <https://github.com/django/django/pull/8924/> >> >> 3. https://code.djangoproject.com/ticket/56 >> <https://groups.google.com/d/msg/django-developers/VFXZpHnuEJc/bbefjX9yCQAJ> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/435EC704-3EF6-4EF4-BF85-175AE29C01F5%40tomforb.es >> >> <https://groups.google.com/d/msgid/django-developers/435EC704-3EF6-4EF4-BF85-175AE29C01F5%40tomforb.es?utm_medium=email&utm_source=footer>. >> >> >> -- >> Adam >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/CAMyDDM1EbbffnydYkVZELcvX3d5y%3DprOCi-b0naYDsP0PRSOwA%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/django-developers/CAMyDDM1EbbffnydYkVZELcvX3d5y%3DprOCi-b0naYDsP0PRSOwA%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/23F63785-EE11-4E2B-9DBB-91DB5393DB11%40tomforb.es > > <https://groups.google.com/d/msgid/django-developers/23F63785-EE11-4E2B-9DBB-91DB5393DB11%40tomforb.es?utm_medium=email&utm_source=footer>. > > > -- > Adam > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CAMyDDM3wYPFp00fS2FpESNQCX1ARGbZsuL9Kzy8SSp87JQK75w%40mail.gmail.com > > <https://groups.google.com/d/msgid/django-developers/CAMyDDM3wYPFp00fS2FpESNQCX1ARGbZsuL9Kzy8SSp87JQK75w%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/F48A7A88-D9DD-496D-A50B-A63621F41546%40tomforb.es.
