Re: [Python-Dev] Deterministic builds of the interpreter
On Fri, Feb 10, 2017 at 7:58 PM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do you have an idea why in [3], this is Python 2.7, the timestamps are still > incorrect? I think they're all required for `compileall` and somehow it > doesn't seem capable of taking into account DETERMINISTIC_BUILD. Explicitly > removing those pyc and pyo files and recompiling them to bytecode still > results in timestamp issues for these 4 files. Sorry, I have no motivation about Python 2 anymore. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items still seem indeterministic. To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` where $out is the path where I installed Python. Do you have an idea why in [3], this is Python 2.7, the timestamps are still incorrect? I think they're all required for `compileall` and somehow it doesn't seem capable of taking into account DETERMINISTIC_BUILD. Explicitly removing those pyc and pyo files and recompiling them to bytecode still results in timestamp issues for these 4 files. On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki wrote: > On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk > wrote: > > Correct, that was an older version from before I patched > > `_bootstrap_external.py`. A more recent diff can be found at > > > > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa > > > > These all seem to be sets. > > Maybe, PYTHONHASHSEED help you. > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED > > > > > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki > wrote: > >> > >> As reading [4], mtime is not 0. > >> > >> data = bytearray(MAGIC_NUMBER) > >> data.extend(_w_long(mtime)) > >> data.extend(_w_long(source_size)) > >> data.extend(marshal.dumps(code)) > >> > >> First 4 bytes are magic. > >> Next 4 bytes are mtime. > >> > >> │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 > >> k..Xl!.. > >> │ │ │ │ +: 160d 0d0a e631 9c58 6c21 e300 > >> .1.Xl!.. > >> > >> mtime is 6b2e9c58 vs e6319c53 (little endian) > >> > >> maybe, you failed to use customized py_compile when building? > >> > >> > >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < > [email protected]> > >> wrote: > >> > Hi, > >> > > >> > I'm attempting to make the builds of the Python interpreters for > Nixpkgs > >> > [1] > >> > deterministic. > >> > > >> > In the case of Python 2.7 we have a patch [2] that fixes the timestamp > >> > used > >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We > also > >> > remove `wininst*.exe`. This works fine, although there are 4 small > >> > issues > >> > left [3]. Do you have any idea what is going on in these files that > >> > could > >> > make them indeterministic? > >> > > >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and > >> > modified > >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds > >> > are > >> > not yet deterministic [4]. Any suggestions what could be fixed here? > >> > > >> > Kind regards, > >> > > >> > Freddy > >> > > >> > > >> > [1] https://github.com/NixOS/nixpkgs > >> > [2] > >> > > >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/ > development/interpreters/python/cpython/2.7/deterministic-build.patch > >> > [3] https://github.com/NixOS/nixpkgs/issues/22570# > issuecomment-278474082 > >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e > 6b > >> > > >> > ___ > >> > Python-Dev mailing list > >> > [email protected] > >> > https://mail.python.org/mailman/listinfo/python-dev > >> > Unsubscribe: > >> > > >> > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > >> > > > > > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deterministic builds of the interpreter
That should have been `PYTHONHASHSEED=0 $out/bin/python -m compileall -f $out`. On Fri, Feb 10, 2017 at 11:58 AM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do you have an idea why in [3], this is Python 2.7, the timestamps are > still incorrect? I think they're all required for `compileall` and somehow > it doesn't seem capable of taking into account DETERMINISTIC_BUILD. > Explicitly removing those pyc and pyo files and recompiling them to > bytecode still results in timestamp issues for these 4 files. > > On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki > wrote: > >> On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk >> wrote: >> > Correct, that was an older version from before I patched >> > `_bootstrap_external.py`. A more recent diff can be found at >> > >> > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa >> > >> > These all seem to be sets. >> >> Maybe, PYTHONHASHSEED help you. >> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED >> >> > >> > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki >> wrote: >> >> >> >> As reading [4], mtime is not 0. >> >> >> >> data = bytearray(MAGIC_NUMBER) >> >> data.extend(_w_long(mtime)) >> >> data.extend(_w_long(source_size)) >> >> data.extend(marshal.dumps(code)) >> >> >> >> First 4 bytes are magic. >> >> Next 4 bytes are mtime. >> >> >> >> │ │ │ │ -: 160d 0d0a 6b2e 9c58 6c21 e300 >> >> k..Xl!.. >> >> │ │ │ │ +: 160d 0d0a e631 9c58 6c21 e300 >> >> .1.Xl!.. >> >> >> >> mtime is 6b2e9c58 vs e6319c53 (little endian) >> >> >> >> maybe, you failed to use customized py_compile when building? >> >> >> >> >> >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < >> [email protected]> >> >> wrote: >> >> > Hi, >> >> > >> >> > I'm attempting to make the builds of the Python interpreters for >> Nixpkgs >> >> > [1] >> >> > deterministic. >> >> > >> >> > In the case of Python 2.7 we have a patch [2] that fixes the >> timestamp >> >> > used >> >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We >> also >> >> > remove `wininst*.exe`. This works fine, although there are 4 small >> >> > issues >> >> > left [3]. Do you have any idea what is going on in these files that >> >> > could >> >> > make them indeterministic? >> >> > >> >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and >> >> > modified >> >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The >> builds >> >> > are >> >> > not yet deterministic [4]. Any suggestions what could be fixed here? >> >> > >> >> > Kind regards, >> >> > >> >> > Freddy >> >> > >> >> > >> >> > [1] https://github.com/NixOS/nixpkgs >> >> > [2] >> >> > >> >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/developme >> nt/interpreters/python/cpython/2.7/deterministic-build.patch >> >> > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment- >> 278474082 >> >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d11 >> 0f0e6b >> >> > >> >> > ___ >> >> > Python-Dev mailing list >> >> > [email protected] >> >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> >> > >> >> > https://mail.python.org/mailman/options/python-dev/songofaca >> ndy%40gmail.com >> >> > >> > >> > >> > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2017-02-03 - 2017-02-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open5799 (+30) closed 35483 (+60) total 41282 (+90) Open issues with patches: 2489 Issues opened (53) == #29430: zipfile: invalid link http://bugs.python.org/issue29430 reopened by berker.peksag #29438: use after free in key sharing dict http://bugs.python.org/issue29438 opened by audric #29440: _dbm requires -lgdbm if gdbm is built as static libraries http://bugs.python.org/issue29440 opened by Chi Hsuan Yen #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 opened by Chi Hsuan Yen #29446: Improve tkinter 'import *' situation http://bugs.python.org/issue29446 opened by terry.reedy #29447: Add os.PathLike support to the tempfile module http://bugs.python.org/issue29447 opened by brett.cannon #29448: Implement os.Pathlike for importlib.machinery http://bugs.python.org/issue29448 opened by brett.cannon #29450: xattr functions aren't in os.supports_fd, os.supports_follow_s http://bugs.python.org/issue29450 opened by Omar Sandoval #29451: Use _PyArg_Parser for _PyArg_ParseStack(): support positional http://bugs.python.org/issue29451 opened by haypo #29453: Remove reference to undefined dictionary ordering in Tutorial http://bugs.python.org/issue29453 opened by Jim Fasarakis-Hilliard #29454: Shutting down consumer on a remote queue http://bugs.python.org/issue29454 opened by Semi #29455: Mention coverage.py in trace module documentation http://bugs.python.org/issue29455 opened by brett.cannon #29456: bug in unicodedata.normalize: u1176 http://bugs.python.org/issue29456 opened by pusnow #29459: `__contains__` and `get` methods for match objects? http://bugs.python.org/issue29459 opened by irdb #29461: Experiment usage of likely/unlikely in CPython core http://bugs.python.org/issue29461 opened by haypo #29463: Add `docstring` attribute to AST nodes http://bugs.python.org/issue29463 opened by inada.naoki #29464: Specialize FASTCALL for functions with positional-only paramet http://bugs.python.org/issue29464 opened by serhiy.storchaka #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 opened by haypo #29466: pickle does not serialize Exception __cause__ field http://bugs.python.org/issue29466 opened by diekhans #29468: zipfile should catch ValueError as well as OSError to detect b http://bugs.python.org/issue29468 opened by ll #29469: AST-level Constant folding http://bugs.python.org/issue29469 opened by inada.naoki #29470: [python] Error in `/usr/bin/python': free(): invalid size: 0x0 http://bugs.python.org/issue29470 opened by David Ford (FirefighterBlu3) #29471: AST: add an attribute to FunctionDef to distinguish functions http://bugs.python.org/issue29471 opened by haypo #29472: subprocess.communicate with timeout 0 and already terminated p http://bugs.python.org/issue29472 opened by Griffon26 #29473: subprocces check_output http://bugs.python.org/issue29473 opened by Ðадим ÐаÑа #29474: Grammatical errors in weakref.WeakValueDictionary docs http://bugs.python.org/issue29474 opened by Mariatta #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 opened by o11c #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 opened by rhettinger #29477: Lambda with complex arguments is ctx STORE http://bugs.python.org/issue29477 opened by malthe #29478: email.policy.Compat32(max_line_length=None) not as documented http://bugs.python.org/issue29478 opened by martin.panter #29479: httplib: could not skip "ACCEPT-ENCODING" header http://bugs.python.org/issue29479 opened by song1st #29480: Mac OSX Installer SSL Roots http://bugs.python.org/issue29480 opened by rahvee #29481: 3.6.0 doc describes 3.6.1 feature - typing.Deque http://bugs.python.org/issue29481 opened by Guy Arad #29502: Should PyObject_Call() call the profiler on C functions, use C http://bugs.python.org/issue29502 opened by haypo #29503: Make embedded-Python detectable http://bugs.python.org/issue29503 opened by Andi Bergmeier #29504: blake2: compile error with -march=bdver2 http://bugs.python.org/issue29504 opened by floppymaster #29505: Submit the re, json, & csv modules to oss-fuzz testing http://bugs.python.org/issue29505 opened by gregory.p.smith #29506: Incorrect documentation for the copy module http://bugs.python.org/issue29506 opened by pgacv2 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 opened by haypo #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 opened by inada.naoki #29511: Add 'find' as build-in method for lists http://bugs.python.org/issue29511 opened by george-shuklin #29512:
[Python-Dev] bugs.python.org access problems
Today I have been getting the following for about 1/3 of access attempts (with Firefox). ''' Secure Connection Failed An error occurred during a connection to bugs.python.org. A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. [Try Again] ''' A couple of retries usually work, but I don't remember such repeated errors before. Anyone else having a similar problem? -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] CPython is now on GitHub
[this is a duplicate of what I just sent python-committers; rendered version at https://paper.dropbox.com/doc/CPython-workflow-changes-mx1k8G6M0rg5JLy80F1r6 ] # CPython workflow changes After more than two years, our new GitHub workflow is ready to accept changes (you can look back to my first “[vision statement]( https://mail.python.org/pipermail/python-dev/2014-December/137472.html)” on changing our workflow to see how things have changed since I started working on this)! I hope you are all excited to see this finished; I know my wife is very excited as she’s tired of listening to me talk about it for a third of our marriage. ;) # Thanks First and foremost, I want to thank everyone who helped with this. Thanks to Donald and Barry for writing the initial PEPs proposing [GitHub]( https://www.python.org/dev/peps/pep-0481/) and [GitLab]( https://www.python.org/dev/peps/pep-0507/) and Nick [pre-proposing RhodeCode](https://www.python.org/dev/peps/pep-0474/). Thanks to everyone on [core-workflow](https://mail.python.org/mailman/listinfo/core-workflow) for helping out with various discussion (and which will continue to host discussions on future improvements on our workflow). Thanks to Ezio, Maciej, and Anish Shah for helping with the changes required to bugs.python.org in order to keep the issue tracker around. Thanks to the infrastructure team for helping deal with the migration of the [peps]( https://github.com/python/peps) and [devguide]( https://github.com/python/devguide) repos (especially Donald and Ernest). Thanks to Senthil for doing the conversion of the repo itself. Thanks to Benjamin for helping with hg.python.org stuff. Thanks to Zach for helping with the buildbots (and the devguide). Thanks to Mariatta, Carol Willing, Berker, Oleg, and Stéphane Wirtel for helping with the devguide. There are also plenty of other people who have provided feedback over the past 2 years on mailing lists and in-person. # What has changed The documentation in the [devguide](https://cpython-devguide.readthedocs.io) should be up-to-date, so don’t worry about keeping this around as a reference. Consider this more of an announcement letter to get people quickly up-to-speed and excited about the new workflow. ## The location of the code repository CPython’s code now lives at https://github.com/python/cpython . hg.python.org/cpython is and will stay read-only (no determination has been made as to how long the Mercurial repository will be kept running). It should also be mentioned that we are doing squash commits and not rebased commits or merge commits as some projects on GitHub do. This basically means we will continue to have a single contribution lead to a single commit, keeping our history linear and compact. To up the bus factor on the new repository, I have set up a team for [Python release managers]( https://github.com/orgs/python/teams/python-release-managers) and made them administrators on the repository. I don’t necessarily expect RMs to use this power, but it’s there in case any of them need to change a setting in order to get a release out (to be upfront about it, I’m also in the team as its creator, but I have administrative privileges for the [Python team]( https://github.com/python) on GitHub so it doesn’t change what I’m able to do). ## Specifying issue numbers Traditionally we have specified issues as “Issue #”. The problem with this format is that [GitHub automatically links]( https://help.github.com/articles/autolinked-references-and-urls/#issues-and-pull-requests) text in this format to GitHub issues and pull requests. While our repository will initially have no issues or PRs to automatically link to, this might not be true long-term (GitHub does the automatic linking eagerly at push time, so there’s no worry for older commit messages; we actually almost mutated the history to match the new format but in the end I made the decision not to as I didn’t consult with python-committers prior to the migration to make sure such a change was acceptable to everyone). To avoid this issue we are going to start specifying issues as “bpo-”. This clearly delineates that issue numbers directly relate to bugs.python.org since “[namespaces are one honking great idea]( https://www.python.org/dev/peps/pep-0020/)”. This is also a format that GitHub supports — “GH-” — as well as JIRA who [lets you specify the prefix]( https://confluence.atlassian.com/adminjiraserver072/changing-the-project-key-format-828787722.html), so there’s precedent for choosing it. This change applies both to `[Misc/NEWS]( https://cpython-devguide.readthedocs.io/committing.html#what-s-new-and-news-entries)` and in [commit messages]( https://cpython-devguide.readthedocs.io/committing.html#commit-messages). Mentioning an issue this way in a pull request title or comment will connect the PR to the corresponding issue on bugs.python.org. Mentioning an issue this way in a commit message will cause a comment to show up in the issue rel
Re: [Python-Dev] bugs.python.org access problems
On Fri, 10 Feb 2017 16:15:46 -0500 Terry Reedy wrote: > Today I have been getting the following for about 1/3 of access > attempts (with Firefox). > ''' > Secure Connection Failed > > An error occurred during a connection to bugs.python.org. A PKCS #11 > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > The page you are trying to view cannot be shown because the > authenticity of the received data could not be verified. > Please contact the website owners to inform them of this problem. > > [Try Again] > ''' > A couple of retries usually work, but I don't remember such repeated > errors before. Anyone else having a similar problem? Yes, I do. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
