[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: And here is alternative patch which uses a hashtable. Both patches have about the same performance for *.pyc files, but marshal_hashtable.patch is much faster for duplicated values. Marshalling [1000]*10**6, [1000.0]*10**6 and [1000.0j]*10**6 with version 3

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: Here is a patch which adds separate dict for interned strings (otherwise they can be uninterned) and for bytes. It also slightly simplify the code. -- Added file: http://bugs.python.org/file38012/marshal_refs_by_value_3.patch

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: Here are results of the benchmark which measures dump and load time for all pyc files in the stdlib (including tests). https://bitbucket.org/storchaka/cpython-stuff/src/default/marshal/marshalbench.py $ find * -name __pycache__ -exec rm -rf '{}' + $ ./python

[issue20416] Marshal: special case int and float, don't use references

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file37895/bench_issue20416.py ___ Python tracker ___ ___ Python-bugs-list mai

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: Here is more general solution. For simple values (ints, floats, complex numbers, short strings) it is faster to use the value itself as a key than create new integer object (id). Without the patch: data ver. dumps(ms) loads(ms) size(KiB) ge

[issue20416] Marshal: special case int and float, don't use references

Larry Hastings added the comment: This is not a release blocker. You guys can play with this for 3.5. FWIW I prefer Serhiy's approach. -- priority: release blocker -> normal versions: +Python 3.5 -Python 3.4 ___ Python tracker

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: Here is a patch which special cases small integers. It decreases v3 and v4 dump time of bench.py by 10% without affecting load time. Of course in real cases the benefit will be much less. -- Added file: http://bugs.python.org/file34071/marshal_small_

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: > As a compromise, we could suppress the sharing for small int objects, since > they are singletons, anyway. This is implementation detail. But we can use more efficient way to memoizating small int objects. I also suppose than efficient C implementation of

[issue20416] Marshal: special case int and float, don't use references

Antoine Pitrou added the comment: > Integers, without the patch: > > dumps v3: 62.8 ms > data size v3: 4882.8 kB > loads v3: 10.7 ms > > Integers, with the patch: > > dumps v3: 18.6 ms (-70%) > data size v3: 4882.8 kB (same size) > loads v3: 27.7 ms (+158%) As I wrote

[issue20416] Marshal: special case int and float, don't use references

Martin v. Löwis added the comment: For the record, format 3 was added through issue16475, format 4 was added through issue19219. In msg175962, Kristjan argued that there is no reason _not_ to share int objects, e.g. across multiple code objects. Now it seems that this argument is flawed: ther

[issue20416] Marshal: special case int and float, don't use references

STINNER Victor added the comment: > Did you tested for numerous shared int and floats? [1000] * 100 and > [1000.0] * 100? AFAIK this was important use cases for adding 3 or 4 > versions. Here are new benchmarks on Python 3.4 with: Integers: [1000] * 100 Floats: [1000.0] *

[issue20416] Marshal: special case int and float, don't use references

Serhiy Storchaka added the comment: Did you tested for numerous shared int and floats? [1000] * 100 and [1000.0] * 100? AFAIK this was important use cases for adding 3 or 4 versions. -- ___ Python tracker

[issue20416] Marshal: special case int and float, don't use references

Changes by Serhiy Storchaka : -- nosy: +kristjan.jonsson, pitrou, serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing

[issue20416] Marshal: special case int and float, don't use references

STINNER Victor added the comment: > I am doing clinic conversion for marshal module so I am adding myself to nosy > list to make sure both tickets are synchronized. The Derby is suspended until the release of Python 3.4 final. I consider this issue as an important performance regression that s

[issue20416] Marshal: special case int and float, don't use references

Vajrasky Kok added the comment: I am doing clinic conversion for marshal module so I am adding myself to nosy list to make sure both tickets are synchronized. http://bugs.python.org/issue20185 -- nosy: +vajrasky ___ Python tracker

[issue20416] Marshal: special case int and float, don't use references

STINNER Victor added the comment: Oh by the way, on Python 3.4, the file size (on version 3 and 4) is unchanged with my patch. Writing a reference produces takes exactly the same size than an integer. -- ___ Python tracker

[issue20416] Marshal: special case int and float, don't use references

STINNER Victor added the comment: Performance of Python 3.3.3+: --- dumps v0: 374.8 ms data size v0: 45582.9 kB loads v0: 625.3 ms dumps v1: 374.6 ms data size v1: 45582.9 kB loads v1: 605.1 ms dumps v2: 152.9 ms data size v2: 41395.4 kB loads v2: 556.5 ms --- So with the patch, the Python 3.4

[issue20416] Marshal: special case int and float, don't use references

Changes by STINNER Victor : -- nosy: +loewis ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue20416] Marshal: special case int and float, don't use references

STINNER Victor added the comment: Use attached bench.py to compare performances. Without the patch: --- dumps v0: 389.6 ms data size v0: 45582.9 kB loads v0: 573.3 ms dumps v1: 391.4 ms data size v1: 45582.9 kB loads v1: 558.0 ms dumps v2: 166.9 ms data size v2: 41395.4 kB loads v2: 482.2 ms

[issue20416] Marshal: special case int and float, don't use references

New submission from STINNER Victor: Attached patched disables references for int and float types. -- files: marshal3_numbers.patch keywords: patch messages: 209524 nosy: haypo priority: normal severity: normal status: open title: Marshal: special case int and float, don't use references