[issue30468] Propagate zipfile.py pypy issue #905 patch to CPython 3.7
Alecsandru Patrascu added the comment: Serhiy, I am curious why did you have chosen to compute the CRC32 table everytime? It is standard (the generator polynomial does not change) and always will output the same values. And it is also less computational intensive to loading a precomputed array vs calculating it each time a zip archive is extracted. -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue30468> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: Hello Brett, I saw that there was no other activity on this issue so far. Can you please update this, as I do not have rights? -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: I've updated the patch for 2.7 branch to contain the LTO part merged by Gregory. -- Added file: http://bugs.python.org/file43142/cpython2_with_optimizations_v02.patch ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26359] CPython build options for out-of-the box performance
Alecsandru Patrascu added the comment: The patches in discussions are the v02 ones. -- ___ Python tracker <http://bugs.python.org/issue26359> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
New submission from Alecsandru Patrascu: Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a request to turn-on Profile Guided Optimization or PGO as the default build option for Python (both 2.7 and 3.6), given its performance benefits on a wide variety of workloads and hardware. For instance, as shown from attached sample performance results from the Grand Unified Python Benchmark, >20% speed up was observed. In addition, we are seeing 2-9% performance boost from OpenStack/Swift where more than 60% of the codes are in Python 2.7. Our analysis indicates the performance gain was mainly due to reduction of icache misses and CPU front-end stalls. Attached is the Makefile patches that modify the all build target and adds a new one called "disable-profile-opt". We built and tested this patch for Python 2.7 and 3.6 on our Linux machines (CentOS 7/Ubuntu Server 14.04, Intel Xeon Haswell/Broadwell with 18/8 cores). We use "regrtest" suite for training as it provides the best performance improvement. Some of the test programs in the suite may fail which leads to build fail. One solution is to disable the specific failed test using the "-x " flag (as shown in the patch) Steps to apply the patch: 1. hg clone https://hg.python.org/cpython cpython 2. cd cpython 3. hg update 2.7 (needed for 2.7 only) 4. Copy *.patch to the current directory 5. patch < python2.7-pgo.patch (or patch < python3.6-pgo.patch) 6. ./configure 7. make To disable PGO 7b. make disable-profile-opt In the following, please find our sample performance results from latest XEON machine, XEON Broadwell EP. Hardware (HW): Intel XEON (Broadwell) 8 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false Operating System: Ubuntu 14.04.3 LTS trusty OS configuration: CPU freq set at fixed: 2.6GHz by echo 260 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq echo 260 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq Address Space Layout Randomization (ASLR) disabled (to reduce run to run variation) by echo 0 > /proc/sys/kernel/randomize_va_space GCC version:gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) Benchmark: Grand Unified Python Benchmark (GUPB) GUPB Source: https://hg.python.org/benchmarks/ Python2.7 results: Python source: hg clone https://hg.python.org/cpython cpython Python Source: hg update 2.7 hg id: 0511b1165bb6 (2.7) hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10 hg --debug id -i: 0511b1165bb6cf40ada0768a7efc7ba89316f6a5 Benchmarks Speedup(%) simple_logging 20 raytrace20 silent_logging 19 richards19 chaos 16 formatted_logging 16 json_dump 15 hexiom2 13 pidigits12 slowunpickle12 django_v2 12 unpack_sequence 11 float 11 mako11 slowpickle 11 fastpickle 11 django 11 go 10 json_dump_v210 pathlib 10 regex_compile 10 pybench 9.9 etree_process 9 regex_v88 bzr_startup 8 2to38 slowspitfire8 telco 8 pickle_list 8 fannkuch8 etree_iterparse 8 nqueens 8 mako_v2 8 etree_generate 8 call_method_slots 7 html5lib_warmup 7 html5lib7 nbody 7 spectral_norm 7 spambayes 7 fastunpickle6 meteor_contest 6 chameleon 6 rietveld6 tornado_http5 unpickle_list 5 pickle_dict 4 regex_effbot3 normal_startup 3 startup_nosite 3 etree_parse 2 call_method_unknown 2 call_simple 1 json_load 1 call_method 1 Python3.6 results Python source: hg clone https://hg.python.org/cpython cpython hg id: 96d016f78726 tip hg id -r 'ancestors(.) and tag()': 1a58b1227501 (3.5) v3.5.0rc1 hg --debug id -i: 96d016f78726afbf66d396f084b291ea43792af1 Benchmark Speedup(%) fastunpickle22.94
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file40231/python2.7-pgo.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40232/python3.6-pgo.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40226/python2.7-3.6-pgo.zip ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Alecsandru Patrascu added the comment: I added the patches as individual files and removed the zip file. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Alecsandru Patrascu added the comment: The patches are tested on Linux machines, with GNU GCC >4.8.3. From your output I see that you are using the CLANG compiler. CLANG uses a different set of flags for PGO that are not compatible with GCC's, therefore the compilation will fail. Can you please use the GNU GCC compiler to test the patches? -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Alecsandru Patrascu added the comment: I received the review and will post new patch versions as soon as I update them. Regarding PGO on clang, I will need a bit more time to edit the Makefile and will post it just for clang, to be easier for us to see the differences. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40236/python2.7-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40239/python3.6-pgo-v02-mac.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40238/python2.7-pgo-v02-mac.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Alecsandru Patrascu added the comment: I modified the patches after the review made by Brett (python2.7-pgo-v02.patch and python3.6-pgo-v02.patch): - removed the call to pybench - left the PGO steps as optional. To use it we run "make profile-opt" - in the initial patches, I left out test_hashlib because in our benchmarks we did not see any gain by applying PGO to the hash functions. It is not harmful and we can let it there or skip it. Nevertheless, in order not to create confusions about it, I removed that parameter from the patch. I also added the patches for Mac exclusively (python2.7-pgo-v02-mac.patch and python3.6-pgo-v02-mac.patch). You must have llvm-profdata installed and in your path (in /Library/Developer/CommandLineTools/usr/bin/) to use it. I tested on Yosemite and it compiles fine with clang. I am working on a generic version (configure and Makefile patches) to merge all these platforms and will post them as soon as it is done. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization active by-default
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40237/python3.6-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40255/python3.5-pgo-v03.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I modified the patches to be compatible with both environments. The new versions modify the configure.ac file also, therefore you will need to run "autoconf" by hand. Also, in case of MaOS you will need to have llvm-profdata installed and in your path. I kept the expanded form of regrtest (/Lib/test/regrtest.py) because this way it is clearer to the user what is the main file that runs the training workload. Also, the "|| true" is necessary also, due to the nature of regrtest. This test suite is designed to return a fail code if a test is not ok, even for tests that do not comply with certain dependencies (meaning users that didn't installed any other libraries). -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40254/python2.7-pgo-v03.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Sorry, it was a typo. I made a correction to it. I will also modify to -m flag, instead of the explicit file execution. Regarding the clang/gcc support, in v03 version of patches, GCC is supported. On Linux is straightforward. On Mac I see that the default development environment also has the "gcc" command, but it is a binary stub that calls clang in backend, so the flags are adjusted for clang-in-gcc-clothing. You say to support clang explicitly as a compiler in 2.7 and 3.6? -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Thank you for the clarifications! Your point make sense, we don't want to exclude clang environments. I will analyze this and post some patches once I'm done with it. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I modified the patches with clang support. Also, I added an important check for the architecture on which PGO is running. Our proposal targets x86 platforms, since our measurements are made only on x86 hardware. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40262/python2.7-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40263/python3.6-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40262/python2.7-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40263/python3.6-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40268/python3.6-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40267/python2.7-pgo-v04.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40274/python3.6-pgo-v05.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40273/python2.7-pgo-v05.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I fixed the files after the review. Regarding the PROFILE_OPT_OLD line, I think that it is better to keep also the old task used for PGO, until clear evidence and measurements that regrtest is performing better on other architectures exists. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: The profile merging is necessary in case you want to use a pure clang compiler or you use GCC in OSX. For example, a general profiling action using clang will result in at least one binary profile. For our case, when using regrtest, we will have multiple profiles as the test is a multi-process one. The application llvm-profdata has the ability to merge the information collected from multiple processes, thus having a more precise map of what is executed from the profiled application. This step is mandatory even if we train on a single threaded or single process workload and have just one profile. More information about the entire process can be found here: http://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I've updated the patches after review and implemented the checkup for llvm-profdata for both Linux and OSX. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40289/python3.6-pgo-v06.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40288/python2.7-pgo-v06.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: That's a good point Skip. I added another set of patches, just for the README files, explaining the entire procedure, so now anyone reading it will see that PGO is available, what are the steps involved and a brief comment about the warning. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40299/README2.7-pgo-v01.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40300/README3.6-pgo-v01.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I've updated the patches with redirect to /dev/null, as is it is more clearer to the user what is our intent, without having him to necessarily read the regrtest documentation. I've also added a warning message regarding the output and ported all these lines to 3.6 and to the README files also. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40389/README2.7-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40390/README3.6-pgo-v02.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I named this task PROFILE_TASK_X86 because it is rigorously tested and we have proven that regrtest performs better on this architecture. Until any other clear evidence and solid measurements that regrtest is performing better on other architectures exists, I'd keep it this way. Even though this does not threaten the stability of the interpreter in any way, the dual aspect you mentioned appears because CPython 2 and 3 have slightly different makefile rule format. To create a common patch working cross-versions will create a very tangled Makefile. If you all agree that having an unified patch for both versions is acceptable, I will work on that. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40391/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40392/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Removed file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: If you are talking just about the 3.6 patch, it is called this way to emphasize the fact that it is intended for the development branch. It is perfectly compatible with 3.5, therefore it is not needed for packagers to maintain two distinct versions. I've tested with: hg update 3.5 ; hg import --no-commit python3.6-pgo-v07.patch ; ./configure ; make profile-opt I also renamed the profile task makefile name. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
New submission from Alecsandru Patrascu:
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation.
I would like to submit a patch that improves the performance of the hash
computation code on stringobject, bufferobject and unicodeobject. As can be
seen from the attached sample performance results from the Grand Unified Python
Benchmark, speedups up to 40% were observed. Furthermore, we see a 5-7%
performance on OpenStack/Swift, where most of the code is in Python 2.7.
Attached is the patch that modifies Object/stringobject.c,
Object/bufferobject.c and Object/unicodeobject.c files. We built and tested
this patch for Python 2.7 on our Linux machines (CentOS 7/Ubuntu Server 14.04,
Intel Xeon Haswell/Broadwell with 18/8 cores).
Steps to apply the patch:
1. hg clone https://hg.python.org/cpython cpython
2. cd cpython
3. hg update 2.7
4. Copy hash8.patch to the current directory
5. hg import --no-commit hash8.patch
6. ./configure
7. make
In the following, please find our sample performance results measured on a XEON
Haswell machine.
Hardware (HW): Intel XEON (Haswell) 18 Cores
BIOS settings: Intel Turbo Boost Technology: false
Hyper-Threading: false
Operating System: Ubuntu 14.04.3 LTS trusty
OS configuration: CPU freq set at fixed: 2.0GHz by
echo 200 >
/sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
echo 200 >
/sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
Address Space Layout Randomization (ASLR) disabled (to
reduce run to run variation) by
echo 0 > /proc/sys/kernel/randomize_va_space
GCC version:gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
Benchmark: Grand Unified Python Benchmark (GUPB)
GUPB Source: https://hg.python.org/benchmarks/
Python2.7 results:
Python source: hg clone https://hg.python.org/cpython cpython
Python Source: hg update 2.7
Benchmarks Speedup(%)
unpack_sequence 40.32733766
chaos 24.84002537
chameleon 23.01392651
silent_logging 22.27202911
django 20.83842317
etree_process 20.46968294
nqueens 20.34234985
pathlib 19.63445919
pidigits19.34722148
etree_generate 19.25836634
pybench 19.06895825
django_v2 18.06073108
etree_iterparse 17.3797149
fannkuch17.08120879
pickle_list 16.60363602
raytrace16.0316265
slowpickle 15.86611184
pickle_dict 15.30447114
call_simple 14.42909032
richards14.2949594
simple_logging 13.6522626
etree_parse 13.38113097
json_dump_v212.2655
float 11.88164311
mako11.20606516
spectral_norm 11.04356684
hg_startup 10.57686164
mako_v2 10.37912648
slowunpickle10.24030714
go 10.03567319
meteor_contest 9.956231435
normal_startup 9.607401586
formatted_logging 9.601244811
html5lib9.082603748
2to38.741557816
html5lib_warmup 8.268150981
nbody 7.507012306
regex_compile 7.153922724
bzr_startup 7.140244739
telco 6.869411927
slowspitfire5.746323922
tornado_http5.24360121
rietveld3.865704876
regex_v83.777622219
hexiom2 3.586305282
json_dump 3.477551682
spambayes 3.183991854
fastunpickle2.971645347
fastpickle 0.673086656
regex_effbot0.127946837
json_load 0.023727176
Thank you,
Alecsandru
--
components: Interpreter Core
files: hash8-v01.patch
keywords: patch
messages: 250639
nosy: alecsandru.patrascu
priority: normal
severity: normal
status: open
title: Hash computation speedup for {buffer,string,unicode}object
type: enhancement
versions: Python 2.7
Added file: http://bugs.python.org/file40456/hash8-v01.patch
___
Python tracker
<http://bugs.python.org/issue25106>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Yes, sure it can be implemented in smaller space, as Serhiy and Chris well pointed out. I submitted the 01 version like that just to be clear that I don't want to re-implement a new hash computing value and just unroll the loop in the existing one. I will submit a new version based on this observations. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40460/hash8-v02.patch ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: I've submitted a more compact version of the previous code (hash8-v02.patch) -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: This patch is not applicable to Python 3, as it already has a better hash function and has the same unrolling technique applied. As Brett well observed, it is a backport of an optimization technique used in Python 3, applied to the existing hash function in 2.7. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Yes, it doesn't break the compatibility with existing applications. To make sure, we tested it in various scenarios and workloads. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25106] Hash computation speedup for {buffer, string, unicode}object
Alecsandru Patrascu added the comment: Please hold this patch for a while, while I analyze the received feedback. -- ___ Python tracker <http://bugs.python.org/issue25106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Thank you for upstreaming this in both branches of Python! Do you think that a different version of regrtest.py, that will be used only for PGO training, should be better in this case? I mean, by implementing a custom version, I think we can control better the output and errors shown on screen. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I can work on modifying the existing regrtest and adding a distinct flag, --pgo for example, as Antoine suggested. Indeed, it will not be trivial as regrtest has a dual approach (single process and multi process), but I will give it a try and post a patch as soon as possible. I also suggest that I open a new issue for this case as it is somehow a distinct implementation than pure PGO and definitively will be some iterations on regrtest.py for both versions of Python until we reach a common ground. It is ok for everyone? -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
New submission from Alecsandru Patrascu: This issue adds improved support for Profile Guided Optimization builds in the regression test suite. Mainly, we keep the visual progress status in this process and suppress errors or any other unnecessary output that can alarm users. -- components: Library (Lib) messages: 251140 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: regrtest.py improvement for Profile Guided Optimization builds type: performance versions: Python 2.7, Python 3.5, Python 3.6 ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file40532/regrtest_27.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40533/regrtest_36.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: I've added the patches for Python default and 2.7. To apply them please use the command "hg import --no-commit". PGO flag is implemented for both variants (single and multi process). Default in profile-opt will be single process. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: The patch does not display dots, as they are not that intuitive and users cannot make an estimate of the time left or the position in the training phase. Instead, the regrtest.py, will display the progress as [X/Y], like the default version, but without any warnings, failed tests, skipped tests. Also the start and end report are suppressed. I don't understand what do you mean by "keyword-only parameter". I added a new flag (-P/--pgo) and propagated it along all the functions that need it. Another approach would have been to use a global variable, lets say "pgo" and use it directly. This is closer to what you had in mind? -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file40547/regrtest_36_v02.patch ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you for the link. I modified regrtest.py for Python3 according to the specifications. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you for the tweaks. I tested the patch and it is all working fine. Without failure count the PGO run is even more transparent to the user. I tested your final version of the patch on several Linux server and regular desktop distributions and software configurations and I saw that test_eintr does not block and the entire PGO build finishes in good conditions every time. -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25188] regrtest.py improvement for Profile Guided Optimization builds
Alecsandru Patrascu added the comment: Thank you Brett for your effort to merge all the changes made to regrtest despite the changes appeared on the way! -- ___ Python tracker <http://bugs.python.org/issue25188> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
New submission from Alecsandru Patrascu: Title: Link Time Optimizations support for GCC and CLANG Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch that adds support for Link Time Optimization (LTO) when using GCC and CLANG to compile CPython2 and CPython3. LTO is a compiler assisted optimization technique that is performed by the compiler at link time. Combined with Profile Guided Optimization (PGO), enabled when running "make profile-opt", and running the Grand Unified Python Benchmark (GUPB), a speedup up to 11%, with a few regressions, was observed comparing with PGO only. Compared with a default build, a performance gain as high as 26% was observed from PGO+LTO. In addition, we are also seeing 2% boost in throughput rate from our OpenStack Swift setup comparing with PGO only. Our GUPB performance evaluation was conducted on Intel SkyLake/Broadwell systems running CentOS/Ubuntu, with CLANG/LLVM and GCC 4.*/5.*. Our OpenStack Swift performance was done on various systems consisting of XEON and Avoton processors. Steps: == 1. Get the CPython source codes hg clone https://hg.python.org/cpython cpython cd cpython hg update 2.7 (for CPython2) 2. Build the binary a) Default: ./configure make b) PGO: ./configure make profile-opt c) PGO+LTO: Copy the attached patch files hg import --no-commit lto-cpython3-v01.patch (for CPython3) hg import --no-commit lto-cpython2-v01.patch (for CPython2) ./configure make profile-opt Hardware and OS Configuration = Hardware: Intel XEON (Broadwell-DE) 8 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false OS: Ubuntu 14.04.3 LTS Server OS configuration: Address Space Layout Randomization (ASLR) disabled to reduce run to run variation by echo 0 > /proc/sys/kernel/randomize_va_space CPU frequency set fixed at 2.6GHz GCC version:GCC version 4.9.2 Benchmark: Grand Unified Python Benchmark from https://hg.python.org/benchmarks/ Measurements and Results A. Repository: GUPB Benchmark: hg id : 2979f5ce6a0c tip hg --debug id -i : 2979f5ce6a0cee994d5485401945d8457bb0afac CPython3: hg id : 21a28f6de358 hg id -r 'ancestors(.) and tag()': 374f501f4567 (3.5) v3.5.0 hg --debug id -i : 21a28f6de3582833652c958b8fd6ae8448b61c7c CPython2: hg id : a37ea1d56e98 (2.7) hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10 hg --debug id -i : a37ea1d56e98eb158750d3e495a5cf524e8c3980 B. Results: CPython2 and CPython3 sample results, measured on a Broadwell platform, can be viewed in Table 1 and 2. On the first column (Benchmark) you can see the benchmark name, on the second (%D) the speedup compared with the default version and on the third column (%PGO) the speedup compared with just PGO; a higher value is better. Table 1. CPython2 results: Benchmark %D %PGO raytrace18 3 chaos 16 5 django_v2 16 6 mako16 6 pathlib 15 3 simple_logging 15 1 slowpickle 15 5 django 14 4 go 14 4 richards13 -1 float 12 4 slowunpickle12 4 etree_process 11 3 fastunpickle11 6 formatted_logging 11 3 nqueens 11 1 regex_compile 11 3 etree_iterparse 10 4 mako_v2 10 3 telco 10 5 pybench 9 1 hexiom2 9 1 html5lib_warmup 9 3 meteor_contest 9 4 pickle_list 9 5 2to38 2 bzr_startup 8 2 chameleon 8 0 etree_generate 8 2 regex_v88 3 silent_logging 8 1 fannkuch7 1 html5lib7 3 json_load 7 -5 tornado_http7 3 call_method_slots 6 3 json_dump_v26 -4 spambayes 6 2 unpickle_list 6 0 etree_parse 5 3 fastpickle 5 4 rietveld5 1 call_method 4 -1 normal_startup 4 2 startup_nosite 4 2 slowspitfire3 0 ssbench 4 2 call_method_unknown 1 -6 json_dump 1 -4 nbody 1 1 pidigits1 -10 pickle_dict 0 -1 regex_effbot0 -2 spectral_norm 0
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41131/lto-cpython3-v01.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file41130/lto-cpython2-v01.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: LTO exists in GCC since version 4.5, but it is true that only recent versions (>=4.8) perform it in good conditions. It is not enabled by default in this patch, it is only available when building with PGO support. Running just "make" will not activate the LTO flags. Do you see it as an configure option (using, for example, an explicit --with-lto flag) rather than using it automatically? -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Meanwhile I've added the patches (v02) for LTO enabled only if the "./configure --with-lto" command is issued. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41138/lto-cpython3-v02.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41137/lto-cpython2-v02.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17781] optimize compilation options
Alecsandru Patrascu added the comment: Hi, I added a dedicated issue just for LTO only when using GCC and CLANG (http://bugs.python.org/issue25702), that works well with PGO also. -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue17781> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25827] Support ICC in configure
Alecsandru Patrascu added the comment: Thank you David for including me in this issue. On ICC, when executing the instrumented applications, it will generate dynamic information file that has a unique name and .dyn suffix. From my previous experience with GCC and CLANG PGO patch (issue24915), also the output of the autoconf will have to be included in the final diff. Regarding the place for adding the '-fp-model strict' flags, I had the same dillema when working on the LTO patch (issue25702), because I needed the flags to be propagated to _all_ compilation and linking phases, and finally decided to add them to the CONFIGURE_CFLAGS, CONFIGURE_CFLAGS_NODIST, CONFIGURE_CPPFLAGS, CONFIGURE_LDFLAGS variables in the Makefile.pre.in file. You can try there also and keep the BASECFLAGS clean . -- ___ Python tracker <http://bugs.python.org/issue25827> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25925] Coverage support for CPython 2
New submission from Alecsandru Patrascu: Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch that adds extended support for coverage, in the same way that it is used in CPython3. Used in conjunction with the regrtests, it provides a broader view of the hot and cold zones in the interpreter. Thank you, Alecsandru -- components: Build files: coverage-2.7-v01.patch keywords: patch messages: 256839 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: Coverage support for CPython 2 type: enhancement versions: Python 2.7 Added file: http://bugs.python.org/file41385/coverage-2.7-v01.patch ___ Python tracker <http://bugs.python.org/issue25925> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25925] Coverage support for CPython 2
Alecsandru Patrascu added the comment: Thank you for the reviews. I've added a new patch (v02) to correct the issues. -- Added file: http://bugs.python.org/file41386/coverage-2.7-v02.patch ___ Python tracker <http://bugs.python.org/issue25925> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: I'm adding Brett, Gregory, Stefan and Victor as nosy because this issue might be interesting for them also. -- nosy: +brett.cannon, gregory.p.smith, haypo, scoder, skrah ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Hello, I've added an updated set of patches (v03) for the current CPython2 and CPython3 codebase. Also made some small changes to reduce the number of places where the flags are set. -- Added file: http://bugs.python.org/file41493/lto-cpython2-v03.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41494/lto-cpython3-v03.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: Thank you for your feedback, I've updated the patches and now LTO flags are used only when building with PGO (v04). CFLAGS/LDFLAGS remain untouched, as Antoine and Victor suggested is better. -- Added file: http://bugs.python.org/file41665/lto-cpython2-v04.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41666/lto-cpython3-v04.patch ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25702] Link Time Optimizations support for GCC and CLANG
Alecsandru Patrascu added the comment: As Steve mentioned, the Microsoft compiler uses LTO (they call it Link-Time Code Generation) and the flags are used when compiling CPython on Windows systems. Thus our proposal to enable it on GCC and CLANG also. -- ___ Python tracker <http://bugs.python.org/issue25702> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26110] Speedup method calls 1.2x
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26110] Speedup method calls 1.2x
Alecsandru Patrascu added the comment: Yury, thank you for the heads up! Here at Intel, in the Dynamic Scripting Languages Optimization Team, we can help the community with reviewing and measuring this patch in our quiet and stable environment, the same one that we use to provide public CPython daily measurements. We will wait for your update. -- ___ Python tracker <http://bugs.python.org/issue26110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41806/cpython3-deadcode-v01.patch ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
New submission from Alecsandru Patrascu: Hi all, This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I would like to submit a patch that enables garbage collection of unused input sections from the CPython2 and CPython3 binaries, by using the "--gc-sections" linker flag, which decides which input sections are used by examining symbols and relocations. In order for this to work, GCC must place each function or data item into its own section in the output file, thus dedicated flags are used. With this technique, an average of 1% is gained in both interpreters, with a few small regressions. Steps: == 1. Get the CPython source codes hg clone https://hg.python.org/cpython cpython cd cpython hg update 2.7 (for CPython2) 2. Build the binary a) Default: ./configure make b) Unused input sections patch Copy the attached patch files hg import --no-commit cpython2-deadcode-v01.patch.patch (for CPython3) hg import --no-commit cpython2-deadcode-v01.patch (for CPython2) ./configure make Hardware and OS Configuration = Hardware: Intel XEON (Haswell-EP) 18 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false OS: Ubuntu 14.04.3 LTS Server OS configuration: Address Space Layout Randomization (ASLR) disabled to reduce run to run variation by echo 0 > /proc/sys/kernel/randomize_va_space CPU frequency set fixed at 2.6GHz GCC version:GCC version 4.9.2 Benchmark: Grand Unified Python Benchmark from https://hg.python.org/benchmarks/ Measurements and Results CPython2 and CPython3 sample results, measured using GUPB on a Haswell platform, can be viewed in Table 1 and 2. On the first column (Benchmark) you can see the benchmark name and on the second (%S) the speedup compared with the default version; a higher value is better. Table 1. CPython3 results: Benchmark %S -- telco 11 etree_parse 7 call_simple 6 etree_iterparse 5 regex_v84 meteor_contest 3 etree_process 3 call_method_unknown 3 json_dump_v23 formatted_logging 2 hexiom2 2 chaos 2 richards2 django_v3 2 nbody 2 etree_generate 2 pickle_list 1 go 1 nqueens 1 call_method 1 mako_v2 1 raytrace1 chameleon_v21 silent_logging 0 fastunpickle0 2to30 float 0 regex_effbot0 pidigits0 json_load 0 simple_logging 0 normal_startup 0 startup_nosite 0 fastpickle 0 tornado_http0 regex_compile 0 fannkuch0 spectral_norm 0 pickle_dict 0 unpickle_list 0 call_method_slots 0 pathlib -2 unpack_sequence -2 Table 2. CPython2 results: Benchmark %S -- simple_logging 4 formatted_logging 3 slowpickle 2 silent_logging 2 pickle_dict 1 chameleon_v21 hg_startup 1 pickle_list 1 call_method_unknown 1 pidigits1 regex_effbot1 regex_v81 html5lib0 normal_startup 0 regex_compile 0 etree_parse 0 spambayes 0 html5lib_warmup 0 unpack_sequence 0 richards0 rietveld0 startup_nosite 0 raytrace0 etree_iterparse 0 json_dump_v20 fastpickle 0 slowspitfire0 slowunpickle0 call_simple 0 float 0 2to30 bzr_startup 0 json_load 0 hexiom2 0 chaos 0 unpickle_list 0 call_method_slots 0 tornado_http0 fastunpickle0 etree_process 0 spectral_norm 0 meteor_contest 0 pybench 0 go 0 etree_generate 0 mako_v2 0 django_v3 0 fannkuch0 nbody 0 nqueens 0 telco -1 call_method -2 pathlib -3 Thank you, Alecsandru -- components: Build files: cpython2-deadcode-v01.patch keywords: patch messages: 259572 nosy: alecsandru.patrascu priority: normal severity: normal status: open title: Garbage collection of unused input sections from CPython binaries type: performance versions: Python 2.7, Python 3.6 Added file: http://bugs.python.org/file41805/cpython2-deadcode-v01.patch ___ Python tracker <http://bugs.python.org/issue26285> ___ _
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I realize now that I should have explained a bit more the background of this patch. I'll do this now, for everyone to be clear of what is the effect of those flags. This issue was revealed after running the coverage target over various workloads, for both CPython2 and CPython3. After running, it can be observed that there are functions in the interpreter that are not called at all over the lifespan of the interpreter. Even more, these functions occupy space in the resulting binary file, and the CPU is forced to jump to longer offsets than it is required. Furthermore, for production level binaries, it is a good idea to remove these stubs, as they bring no benefit. Now, in order to do this, in the first step, every function or data item must exist in its own section (and the flags -ffunction-sections and -fdata-sections come to help in GCC). In the second step, the linker comes into play and because it has the entire picture of every piece of data or function, it is able to see if there are functions that are never called for the current build (and the flag --gc-sections come to help). This functionality is not unique or new and are used by default in other interpreters, such as V8/Node.JS in their Release target, to achieve exactly the same goal. Another example for behind the scene usage of this functionality is the Microsoft's compiler, which does it automatically in their interprocedural optimization phase. To compress all of the above, the main reason for this speedup is the reduction of the code path length and having the useful function close together, so that the CPU will be able to prefetch them in advance and use them instead of trowing them away because they are not used. -- ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: Sure, I attached them as files because they have a lot of lines for posting here (~90 in total). The linker offers the possibility to show what piece of data/functions was removed, but I intentionally omitted it in order not to clutter the build trace. If you think it will be useful for the user to see it, I can add them to the patch also. -- Added file: http://bugs.python.org/file41808/gc-removed-cpython2.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41809/gc-removed-cpython3.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I've done again the experiments on larger workloads, such as our OpenStack Swift cluster, and it works without any issues. Also, I've attached an archive with a simple external module in CPython3 that uses PyMem_RawMalloc. The output is ok, and it's copied bellow. u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python setup.py build_ext --inplace running build_ext building 'mytest' extension creating build creating build/temp.linux-x86_64-3.6 gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fdata-sections -ffunction-sections -Wl,--gc-sections -fPIC -I/home/u/w/cpython3_deadcode/Include -I/home/u/w/cpython3_deadcode -c mytest.c -o build/temp.linux-x86_64-3.6/mytest.o gcc -pthread -shared build/temp.linux-x86_64-3.6/mytest.o -o /home/u/w/experimente/c_ext3/mytest.cpython-36m-x86_64-linux-gnu.so u@palecsandru:~/w/experimente/c_ext3$ ll total 40 drwxrwxr-x 3 u u 4096 Feb 5 14:29 ./ drwxr-xr-x 12 u u 4096 Feb 5 14:00 ../ drwxrwxr-x 3 u u 4096 Feb 5 14:29 build/ -rw-rw-r-- 1 u u 619 Feb 5 14:16 mytest.c -rwxrwxr-x 1 u u 17856 Feb 5 14:29 mytest.cpython-36m-x86_64-linux-gnu.so* -rw-rw-r-- 1 u u 132 Feb 5 14:15 setup.py u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python Python 3.6.0a0 (default:87dfadd61e0d+, Feb 5 2016, 14:22:57) [GCC 5.2.1 20151010] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import mytest >>> mytest.mytest() 'test' >>> -- Added file: http://bugs.python.org/file41819/c_ext3.zip ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26285] Garbage collection of unused input sections from CPython binaries
Alecsandru Patrascu added the comment: I attached the list for CPython3 (gc-removed-zones-cpython3.txt), but now split in two sections (core and parser), for more clarity as from what and where it is removed. As you can see, the reason why the module works is because the API that can be used by modules remains untouched. What I was trying to say and prove before is that these GCC/LD flags are safe to use in CPython (and for any other software project) and will not break any compatibility with existing or future modules. -- Added file: http://bugs.python.org/file41851/gc-removed-zones-cpython3.txt ___ Python tracker <http://bugs.python.org/issue26285> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: Hello and thank you for your feedback. For CPython this does not apply because due to the structure of the build system, inside the "build" directory there are no PGO profiles saved. You can run find . -name '*.gc??' to see. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: That's interesting. Even on CPython3, I still don't see any gcda's inside the build directory, nor the tree structure you are seeing there. Can you please give me a couple of details regarding your environment (os, distribution, gcc version, 32/64 bit, cross compilation, etc)? If this is a general issue, I can add another patch to fix it. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: For my responses, I modified locally the Makefile so that it will not remove the build directory and any of the gcda files. I will make some more tests tomorrow, but i think that this problem will solve simpler if the removal of the build directory is deleted and all the profile info is kept. -- ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Changes by Alecsandru Patrascu : -- nosy: +alecsandru.patrascu ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26307] no PGO for built-in modules with `make profile-opt`
Alecsandru Patrascu added the comment: Thank you for noticing and signaling this issue. Since I proposed the PGO patches, I will fix it in another patch, as I don't want it to break any builds. -- ___ Python tracker <http://bugs.python.org/issue26307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41917/pgofix-cpython3.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24915] Profile Guided Optimization improvements (better training, llvm support, etc)
Alecsandru Patrascu added the comment: I've added a fix for the PGO builds after a issue pointed out in #26307. Thank you Christos for your observation! -- Added file: http://bugs.python.org/file41916/pgofix-cpython2.patch ___ Python tracker <http://bugs.python.org/issue24915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
