#27890: runtests.py cleanup exception on Python 3.6
---------------------------------------------+------------------------
Reporter: Vytis Banaitis | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Per [https://github.com/django/django/pull/8127#issuecomment-283322945
Tim's suggestion], I've looked into a cleanup exception that happens when
running tests on Python 3.6:
{{{
$ ./tests/runtests.py basic
Testing against Django installed in '/home/tim/code/django/django' with up
to 3 processes
....
Destroying test database for alias 'other'...
Traceback (most recent call last):
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 254,
in _run_finalizers
finalizer()
File "/opt/python3.6.0/lib/python3.6/multiprocessing/util.py", line 186,
in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
465, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/home/tim/.virtualenvs/django36/lib/python3.6/shutil.py", line
463, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/django_k0xziymh/pymp-i4s112bj'
}}}
What I found out:
- `multiprocessing` is involved. Indeed, running with `--parallel=1` does
not trigger this error.
- Jinja2 is involved. Running tests without Jinja2 installed does not
trigger this error.
What happens:
1. `multiprocessing` registers `atexit` handler.
1. `runtests.py` creates a temp dir (e.g. `/tmp/django_k0xziymh`) and
registers `atexit` handler to remove it.
1. `multiprocessing` creates a temp dir (e.g. `/tmp/django_k0xziymh/pymp-
i4s112bj`) which will be deleted in the handler it registered earlier.
1. Tests happen.
1. `runtests.py` exit handler deletes the temp dir.
1. `multiprocessing` exit handler tries to delete the inner temp dir but
it is already gone.
On earlier Python versions 1 and 2 are swapped and 5 and 6 are swapped, so
the error does not happen.
Jinja2 is imported by a chain of imports starting with `from django.test
import TestCase, TransactionTestCase` (full chain below).
On Python 3.6 Jinja2
[https://github.com/pallets/jinja/blob/master/jinja2/__init__.py#L74-L78
patches async support] which eventually imports `multiprocessing.util`
which registers the exit handler.
Possible solutions:
- Create the temp dir before importing `django` modules.
- Move some imports in `runtests.py` into functions, thereby delaying the
indirect import of Jinja2.
- Move some imports somewhere else into functions to break the import
chain.
- ...
The import chain leading up to the import of `multiprocessing.util` (with
uninteresting import machinery stack frames removed):
{{{
File "./runtests.py", line 18, in <module>
from django.test import TestCase, TransactionTestCase
File "/home/vytis/src/django/django/test/__init__.py", line 5, in
<module>
from django.test.client import Client, RequestFactory
File "/home/vytis/src/django/django/test/client.py", line 12, in
<module>
from django.core.handlers.base import BaseHandler
File "/home/vytis/src/django/django/core/handlers/base.py", line 7, in
<module>
from django.urls import get_resolver, set_urlconf
File "/home/vytis/src/django/django/urls/__init__.py", line 1, in
<module>
from .base import (
File "/home/vytis/src/django/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/home/vytis/src/django/django/urls/exceptions.py", line 1, in
<module>
from django.http import Http404
File "/home/vytis/src/django/django/http/__init__.py", line 5, in
<module>
from django.http.response import (
File "/home/vytis/src/django/django/http/response.py", line 13, in
<module>
from django.core.serializers.json import DjangoJSONEncoder
File "/home/vytis/src/django/django/core/serializers/__init__.py", line
23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/home/vytis/src/django/django/core/serializers/base.py", line 6,
in <module>
from django.db import models
File "/home/vytis/src/django/django/db/models/__init__.py", line 3, in
<module>
from django.db.models.aggregates import * # NOQA
File "/home/vytis/src/django/django/db/models/aggregates.py", line 5, in
<module>
from django.db.models.expressions import Func, Star
File "/home/vytis/src/django/django/db/models/expressions.py", line 6,
in <module>
from django.db.models import fields
File "/home/vytis/src/django/django/db/models/fields/__init__.py", line
11, in <module>
from django import forms
File "/home/vytis/src/django/django/forms/__init__.py", line 6, in
<module>
from django.forms.boundfield import * # NOQA
File "/home/vytis/src/django/django/forms/boundfield.py", line 5, in
<module>
from django.forms.widgets import Textarea, TextInput
File "/home/vytis/src/django/django/forms/widgets.py", line 21, in
<module>
from .renderers import get_default_renderer
File "/home/vytis/src/django/django/forms/renderers.py", line 11, in
<module>
from django.template.backends.jinja2 import Jinja2
File "/home/vytis/src/django/django/template/backends/jinja2.py", line
1, in <module>
import jinja2
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 81, in <module>
_patch_async()
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/__init__.py", line 77, in _patch_async
from jinja2.asyncsupport import patch_all
File "/home/vytis/src/env/django-py36/lib/python3.6/site-
packages/jinja2/asyncsupport.py", line 13, in <module>
import asyncio
File "/opt/python/lib/python3.6/asyncio/__init__.py", line 21, in
<module>
from .base_events import *
File "/opt/python/lib/python3.6/asyncio/base_events.py", line 17, in
<module>
import concurrent.futures
File "/opt/python/lib/python3.6/concurrent/futures/__init__.py", line
17, in <module>
from concurrent.futures.process import ProcessPoolExecutor
File "/opt/python/lib/python3.6/concurrent/futures/process.py", line 55,
in <module>
from multiprocessing.connection import wait
File "/opt/python/lib/python3.6/multiprocessing/connection.py", line 23,
in <module>
from . import util
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27890>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/049.2d87343e217ad30e741ed0dbf2345bb1%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
