[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Changes by Arfrever Frehtes Taifersar Arahesis : -- stage: -> committed/rejected ___ Python tracker ___ ___ Python-bugs-list mailing

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: Thanks Serhiy, I merged your last patch with mine. This issue should now be fixed. As Martin wrote, an enhancement would be to reimplement these functions in C without such table. -- resolution: -> fixed status: open -> closed __

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Roundup Robot added the comment: New changeset 7093d5758954 by Victor Stinner in branch '3.4': Issue #20879: Delay the initialization of encoding and decoding tables for http://hg.python.org/cpython/rev/7093d5758954 New changeset 06d646935c9a by Victor Stinner in branch 'default': (Merge 3.4) Is

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Serhiy Storchaka added the comment: Well, our patches are almost same so your patch LGTM (and it has more comments). But note that my initialization of _b32tab2 is a little faster (bytes() is called 32 times instead of 1024). -- ___ Python tracker

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Serhiy Storchaka added the comment: I don't like base64_urlparse_lazy_init-3.patch at all. It waste memory and time for initializing of non-needed tables in case when only encoding or only decoding is used. Here is new patch based on base64_urlparse_lazy_init-2.patch. Unlike to my patch, it d

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: Sorry, it took me many tries to write the perfect patch :-) The current code has artificial dependencies between variables and has references to variables when they are not needed in fact. base64_urlparse_lazy_init-4.patch should be even simpler. -- A

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: Serhiy wrote on Rietveld: "As far as this constant is repeated twice, it wastes memory and errorprone." Oh, I expected marshal to be smart and use references, but it does not. Here is a simpler patch which initialize all base85 tables at once using a "b85alpha

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: I combined the two patches and I tried to address all comments: base64_urlparse_lazy_init-2.patch. -- Added file: http://bugs.python.org/file34418/base64_urlparse_lazy_init-2.patch ___ Python tracker

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: urlparse_lazy_init.patch looks good, but you should add a comment explaining your change. See my review: http://bugs.python.org/review/20879/ -- ___ Python tracker ___

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Antoine Pitrou added the comment: Victor's approach looks fine to me. -- type: performance -> resource usage versions: +Python 3.5 -Python 3.3, Python 3.4 ___ Python tracker ___

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Serhiy Storchaka added the comment: Yes, because small tables consume two orders less memory. -- ___ Python tracker ___ ___ Python-bug

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: > Here is simpler patch. Your patch only changes _*tab2, not the smaller tables. -- ___ Python tracker ___

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Serhiy Storchaka added the comment: And here is similar patch for urllib.parse. -- Added file: http://bugs.python.org/file34329/urlparse_lazy_init.patch ___ Python tracker ___ __

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Serhiy Storchaka added the comment: I was more interested in the import time, and it has slightly increased. Memory consumption ((32**2+2*85**2)*sys.getsizeof(b'xx')/2**10 + sys.getsizeof(dict.fromkeys(range(32**2))) + 2*sys.getsizeof(dict.fromkeys(range(85**2))) + 2*(85**2-256)*sys.getsizeof

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Changes by R. David Murray : -- nosy: +r.david.murray ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: base64_on_demand.patch: create tables the first time they are needed. b85decode() already uses such trick to build _b85dec. Note: a85decode() doesn't use a precomputed table. > If people think that these functions are too slow, they should propose an > extens

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

STINNER Victor added the comment: > I'd personally be in favor of such a change (drop the tables). I didn't propose to drop the table, but create a table the first time that it is needed. I never used ascii85 nor base85 nor base32 in my applications. Maybe because ascii85 and base85 are new i

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Martin v. Löwis added the comment: I'm also -0 on delayed creation of the tables. If these encodings are used, the memory overhead will still occur. -- ___ Python tracker ___ __

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

Martin v. Löwis added the comment: It's a classical time-space trade-off. I'm not sure this is a bug; if it is, the reasonable change (IMO) is to just revert the introduction of these *2 tables, and loop through the output character-by-character. I'd personally be in favor of such a change (dr

[issue20879] base64 module of Python 3.4 uses 920 kB of memory

New submission from STINNER Victor: According to tracemalloc, "import base64" allocates 920.6 kB of memory. The 3 top locations are: Lib/base64.py:414: size=420 KiB, count=7226, average=59 B _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] Lib/base64.py:306: size=420 KiB, count=