#20584: Django's Memcached backend get_many() doesn't handle iterators
-------------------------------------+-------------------------------------
Reporter: guyon.moree@… | Owner: nobody
Type: Bug | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Christian Barcenas):
* status: closed => new
* version: 1.5 => master
* has_patch: 0 => 1
* resolution: needsinfo =>
* stage: Unreviewed => Ready for checkin
Old description:
> When the "keys" parameter to get_many() is a generator, the values will
> be lost in the zip function.
>
> https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py#L93
>
> Here's a simplified code example:
>
> {{{
> def make_key(k):
> return k
>
> user_ids = (11387, 1304318)
>
> keys = ('user_%d' % x for x in user_ids)
>
> new_keys = map(lambda x: make_key(x), keys)
>
> m = dict(zip(new_keys, keys))
>
> assert( m == {} )
> }}}
>
> I believe this is related to this zip() behaviour:
> http://stackoverflow.com/questions/11210300/why-does-zip-drop-the-values-
> of-my-generator
>
> I encountered this bug when upgrading from django 1.3 to django 1.5.1
New description:
When the `keys` parameter to `get_many()` is an iterator, its values will
be consumed in a list comprehension, but then later the already-consumed
iterator is passed to `zip`.
https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py#L90
This causes a very confusing `ValueError` which is raised when the cache
backend attempts to map lower-level Memcache-backend cache keys back to
higher-level Djanco cache keys.
{{{
Traceback (most recent call last):
File "[snip]/django/tests/cache/tests.py", line 1345, in
test_get_many_accepts_iterator
values = cache.get_many(iter(['fizz']))
File "[snip]/django/lib/python3.6/site-
packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py",
line 91, in get_many
return {m[k]: v for k, v in ret.items()}
File "[snip]/django/lib/python3.6/site-
packages/Django-2.2.dev20180706090656-py3.6.egg/django/core/cache/backends/memcached.py",
line 91, in <dictcomp>
return {m[k]: v for k, v in ret.items()}
KeyError: ':1:fizz'
}}}
--
Comment:
Re-opening this ticket, and updating the description with more details.
The Memcache backend's `get_many()` function definitely fails when the key
inputs are iterables which can be consumed only once, such as generators.
--
Ticket URL: <https://code.djangoproject.com/ticket/20584#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/079.1f21d4c203d8ac20229f3b5acc1c5d6e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
