New submission from Anton Afanasyev:
This issue results in redundant memory consumption for e.g. in this case:
================================================
from itertools import *
def test_islice():
items, lookahead = tee(repeat(1, int(1e9)))
lookahead = islice(lookahead, 10)
for item in lookahead:
pass
for item in items:
pass
if __name__ == "__main__":
test_islice()
================================================
This demo is taken from real case where one needs to look ahead input stream
before processing it. For my PC this demo stops with 'Segmentation fault'
message after exhausting all PC memory, while running it with patched python
consumes only 0.1% of memory till the end.
When one uses pure pythonic implementation of itertools.islice() (taken from
docs), the issue goes away as well, since this implementation doesn't store
redundant reference to source iterator.
================================================
def islice(iterable, *args):
s = slice(*args)
it = iter(xrange(s.start or 0, s.stop or sys.maxint, s.step or 1))
nexti = next(it)
for i, element in enumerate(iterable):
if i == nexti:
yield element
nexti = next(it)
================================================
Attaching patch for this issue. Have to change '__reduce__()' method since now
unpickling of exhausted 'islice()' object cannot return old source iterator.
----------
components: Extension Modules
files: patch_3.4_8c8315bac6a8.diff
keywords: patch
messages: 216939
nosy: Anton.Afanasyev, rhettinger
priority: normal
severity: normal
status: open
title: itertools.islice() doesn't release reference to the source iterator when
the slice is exhausted
type: resource usage
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file34990/patch_3.4_8c8315bac6a8.diff
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue21321>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
