#27734: Parallel test workers can be assigned an unexpected index when
recovering
from errors
-----------------------------------+------------------------------------
Reporter: Adam Wróbel | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: parallel | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by Jacob Walls):
* has_patch: 1 => 0
* summary:
Add a helpful error message when a parallel test worker is assigned an
unexpected index
=>
Parallel test workers can be assigned an unexpected index when
recovering from errors
* type: Cleanup/optimization => Bug
Comment:
Thanks Aarav Sharma for alerting me that we had an issue for this. I've
been meaning to open this exact issue for a while.
It's easiest to simulate the ''occasional'' failure described here by just
forcing workers to respawn with `maxtasksperchild`:
{{{#!diff
diff --git a/django/test/runner.py b/django/test/runner.py
index 56eda9d98d..ddbbeee510 100644
--- a/django/test/runner.py
+++ b/django/test/runner.py
@@ -576,6 +576,9 @@ class ParallelTestSuite(unittest.TestSuite):
self.debug_mode,
self.used_aliases,
],
+ # For testing only,
+ # Emulate the occasional failures that happen (under resource
contention?)
+ maxtasksperchild=1,
) as pool:
test_results =
pool.imap_unordered(self.run_subsuite.__func__, args)
}}}
Then run a subsuite of tests that make heavy use of the database:
`./runtests.py queries`
{{{#!py
File "/Users/jwalls/django/django/db/backends/sqlite3/creation.py", line
146, in setup_worker_connection
source_db = self.connection.Database.connect(
f"file:{source_db_name}?mode=ro", uri=True
)
sqlite3.OperationalError: unable to open database file
}}}
----
I'd like to reframe this ticket to actually fix the underlying issue
instead of adjusting the error message. We could do that by making the
database clones a pool that that workers pull from instead of trying to
map ephemeral workers 1:1 with fixed resources. All of this functionality
in the parallel runner is private, so we shouldn't be shy about fixing the
design.
--
Ticket URL: <https://code.djangoproject.com/ticket/27734#comment:15>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019d21c15aa7-aca50fc8-be54-4658-8367-a7d3575bb206-000000%40eu-central-1.amazonses.com.
