Hello, As the test suite is growing, it’s getting slower. I’ve tried to make it faster by running tests in parallel.
The current state of my experiment is here: https://github.com/django/django/pull/4063 <https://github.com/django/django/pull/4063> I’m distributing the tests to workers with the multiprocessing module. While the idea is simple, the unittest APIs make its implementation painful. ** Results ** Without the patch: Ran 9016 tests in 350.610s ./runtests.py 355,86s user 20,48s system 92% cpu 6:48,23 total With the patch Ran 9016 tests in 125.778s ./runtests.py --parallel 512,31s user 29,92s system 300% cpu 3:00,73 total Since it takes almost one minute to create databases, parallelization makes the execution of tests go from 6 minutes to 2 minutes. This isn’t bad, but the x3 speedup is a bit disappointing given that I have 4 physical / 8 logical cores. Perhaps the IPC is expensive. Does anyone have insights about scaling with multiprocessing? ** Limitations ** This technique works well with in-memory SQLite databases. Each process gets its own copy of the database in its memory space. It fails with on-disk SQLite databases. SQLite can’t cope with this level of concurrency. It timeouts while attempting to lock the database. It fails with PostgreSQL (and, I assume, other databases) because tests collide, for instance when they attempt to load the same fixture. ** Next steps ** At this point, the patch improves the common use case of running `./runtests.py` locally to check a database-independent change, and little else. Do you think it would be useful to include it in Django anyway? Do you have concerns about the implementation? Charitably, I’ll say that “it works”… Releasing it separately as a custom test runner may be more appropriate. What do you think? -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/639C2955-7AAB-4BC6-940D-EA69F7F51280%40polytechnique.org. For more options, visit https://groups.google.com/d/optout.
