#34697: Migration serializer for sets results in indeterministic output due
unstable iteration order / hash randomization
---------------------------------+--------------------------------------
Reporter: Yury V. Zaytsev | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+--------------------------------------
Comment (by Yury V. Zaytsev):
Hi, thanks for the feedback and the reference to an earlier ticket.
Actually, we've also hit the flip-flops with the dependencies, which are
annoying, but we do hit them so rarely, that I actually have forgotten
about them. However, we are hitting the set-caused issues every couple of
days and this finally motioned me to try to do something about it. I have
salvaged the old part of the patch concerning the dependencies, rebased
it, fixed and it seems to work. Very nice. The result is uploaded to the
ticket.
Yes, I know about the "workaround" of setting the `PYTHONHASHSEED`, but
this is not a practical solution for us. Maybe you can do that if you are
a sole developer working alone on some Django project. You can write a
wrapper script around `manage.py`, or set this variable globally, or use
the IDE configuration. But if you are a team of more than a dozen of
developers working on a large number of Django projects collaboratively,
things are getting very annoying and difficult.
We can't easily patch `manage.py` to set it for `makemigrations`, because
it has to be set before interpreter startup. Wrapper scripts are also
problematic, because each member would have to integrate it in his
workflow. Finally, forcing people to set `PYTHONHASHSEED` globally on
their machine is a questionable solution with security implications, which
is why it was introduced in the first place...
Making migrations completely deterministic is easy ;-) You just have to
sort in `BaseSequenceSerializer` for all sequence types, and not just for
set types like I did. Only this will of course have a huge number of
unwanted side-effects like force-sorted choices and so on, and it will
cause hugely complicated and unnecessary code mess to write opt-outs
explicitly.
Right now, all concerns raised by Simon in the past are addressed:
1. Sets are (now) globally deterministic
2. Dicts are serialized in an fixed insertion order
3. Dependencies are now sorted explicitly as an exception^*
4. Other stuff (lists, tuples) are ordered, because the order has a
semantic
^* Or maybe a better approach would be to switch dependencies to a set if
the order is not important, how would you like that?
So I think that fixing it for set types and dependencies will solve 99% of
the cases for everyone without any bad side effects and no effort on the
part of the users, and will not cause much maintenance & code complexity
burden. For now you've got 2 tickets in 5 years for this, and if these
patches are committed and we still missed a thing, then it's no problem to
fix another edge case 10 years later, in as far as I'm concerned... :-)
I would be curious to hear Simon's thoughts on that.
--
Ticket URL: <https://code.djangoproject.com/ticket/34697#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/01070189319a47f2-8e80773b-340f-4d42-ae7d-c1b7a1f5e99d-000000%40eu-central-1.amazonses.com.
