Hi all,
Can anyone confirm that the following is a bug in Django (and that I'm not
simply missing the correct way to do this)?
I'm using Django 1.7 on Python 3.4.
I'm new to Django but a longtime Python programmer. I've been digging into
a ValueError I get when I supply a BytesIO instance as the `file` parameter
to an ImageFile instance, then supply that ImageFile instance as the
`content` parameter to `FieldFile.save`.
In summary: The method `FieldFile.save` saves `content` to `self.storage`
(which causes `content` to be closed) but then attempts to access the size
of `content`. This causes a ValueError("I/O operation on closed file.") to
be raised if the internal file inside the ImageFile instance is a BytesIO
instance. Here is `FieldFile.save`:
https://github.com/django/django/blob/master/django/db/models/fields/files.py#L86
----
This appears to be an interaction between the following two changesets:
- 8 months ago:
https://github.com/django/django/commit/30fc49a7ca0d030c7855f31ed44395903fa6abdd
- 3 years ago:
https://github.com/django/django/commit/88ff6567310d809abddb97adebf04d5e9403ca8a
This changeset also touches relevant code, but I don't think it altered any
logic relevant to this problem:
- 2 months ago:
https://github.com/django/django/commit/96fc3908ad25f40b4c6e2f76be2639f665223227
----
Here's the detailed line-by-line trace:
In "django/db/models/fields/files.py", in method `FieldFile.save(self,
name, content, save=True)`:
https://github.com/django/django/blob/master/django/db/models/fields/files.py#L86
this function invokes `self.storage.save(name, content)` [line 88],
then assigns `content.size` to `self._size` [line 92].
In "django/core/files/storage.py", in method `Storage.save(self, name,
content)`:
https://github.com/django/django/blob/master/django/core/files/storage.py#L37
this function invokes `self._save(name, content)` [line 51].
In "django/core/files/storage.py", in method `FileSystemStorage._save(self,
name, content)`:
https://github.com/django/django/blob/master/django/core/files/storage.py#L175
this function invokes `content.close()` [line 231]:
https://github.com/django/django/blob/master/django/core/files/storage.py#L231
At this point, `content` is now closed.
But then back to method `FieldFile.save(self, name, content, save=True)`,
line 92:
https://github.com/django/django/blob/master/django/db/models/fields/files.py#L92
we next access the `size` attribute of `content`.
The type of `content` is ImageFile, which inherits File:
- class ImageFile:
https://github.com/django/django/blob/master/django/core/files/images.py#L11
- class File:
https://github.com/django/django/blob/master/django/core/files/base.py#L13
Accessing the `size` attribute of `content` accesses the `size` property:
https://github.com/django/django/blob/master/django/core/files/base.py#L64
which invokes the `File._get_size(self)` method:
https://github.com/django/django/blob/master/django/core/files/base.py#L55
At this point, `content` (a recently-created ImageFile instance) doesn't
have a `_size` attribute, so `self._get_size_from_underlying_file()` is
invoked [line 58]:
In method `File._get_size_from_underlying_file()`, line 39:
https://github.com/django/django/blob/master/django/core/files/base.py#L39
the function checks for various attributes of the underlying Python file in
a chained if-statement.
In my case, the underlying Python file is a BytesIO instance:
https://docs.python.org/3/library/io.html#io.BytesIO
which inherits io.BufferedIOBase:
https://docs.python.org/3/library/io.html#io.BufferedIOBase
which in turn inherits io.IOBase:
https://docs.python.org/3/library/io.html#io.IOBase
The attributes checked, and the results when the file is a BytesIO instance:
- `size`: no
- `name`: no
- `tell` and `seek`: yes, in io.IOBase
Because the `tell` and `seek` attributes were found, the function attempts
to determine the file size using `self.file.tell()` [line 48]:
https://github.com/django/django/blob/master/django/core/files/base.py#L48
However, at this point, because the BytesIO instance is closed, this will
result in a ValueError("I/O operation on closed file.") being raised.
This ValueError exception is raised in the Python 3.4 source, in the file
"Python-3.4.0/Modules/_io/bytesio.c", when the macro `CHECK_CLOSED(self)`
[line 22] is invoked by function `bytesio_tell(bytesio *self)` [line 254].
----
At this point, my work-around is to set the `_size` attribute of the
ImageFile manually (to the BytesIO size calculated using the same method as
in `File._get_size_from_underlying_file()`) before invoking
`FieldFile.save`.
def get_bytesio_size(buf):
"""Calculate the size of BytesIO instance `buf`.
Note: Ensure `buf` is not already closed!
"""
# This approach was copied from "django/core/files/base.py"
pos = buf.tell()
buf.seek(0, os.SEEK_END)
size = buf.tell()
buf.seek(pos)
return size
img_file._size = get_bytesio_size(bytesio)
# Then img_field.save(img_name, img_file), etc...
It works, but obviously it's hacky to do this.
Can anyone confirm that this problem is indeed a bug in Django (and that
I'm not simply missing the correct way to do this)?
Thanks,
jb
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/CABtX16GWKggatron%3Dm1R%2B2La4-m8Gu%2B%2BP7ZjZtWAyN205Z1ihA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
