[Python-Dev] Compiling 2.7.2 on OS/2
Hi All,
I'm working on updating my port of Python 2.6.5 to v2.7.2 for the OS/2
platform.
I have python.exe and python27.dll compiling find, but when starting to
build sharedmods I'm getting the following error:
running build
running build_ext
Traceback (most recent call last):
File "./setup.py", line 2092, in
main()
File "./setup.py", line 2087, in main
'Lib/smtpd.py']
File "U:/DEV/python-2.7.2/Lib/distutils/core.py", line 152, in setup
dist.run_commands()
File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 953, in
run_commands
self.run_command(cmd)
File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 972, in
run_command
cmd_obj.run()
File "U:/DEV/python-2.7.2/Lib/distutils/command/build.py", line 127,
in run
self.run_command(cmd_name)
File "U:/DEV/python-2.7.2/Lib/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 972, in
run_command
cmd_obj.run()
File "U:/DEV/python-2.7.2/Lib/distutils/command/build_ext.py", line
340, in run
self.build_extensions()
File "./setup.py", line 152, in build_extensions
missing = self.detect_modules()
File "./setup.py", line 1154, in detect_modules
for arg in sysconfig.get_config_var("CONFIG_ARGS").split()]
AttributeError: 'NoneType' object has no attribute 'split'
make: *** [sharedmods] Error 1
Any suggestions? A google showed a similar error on AIX with no clear
resolution.
Thanks in advance,
Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, 5 Jan 2012 15:26:27 +1100 Andrew Bennetts wrote: > > I don't think that's news either. > http://mail.python.org/pipermail/python-dev/2003-May/035907.html and > http://twistedmatrix.com/pipermail/twisted-python/2003-June/004339.html for > instance show that in 2003 it was clearly known to at least be likely to be an > exploitable DoS in common code (a dict of HTTP headers or HTTP form keys). > > There was debate about whether it's the language's responsibility to mitigate > the problem or if apps should use safer designs for handling untrusted input > (e.g. limit the number of keys input is allowed to create, or use something > other than dicts), and debate about just how practical an effective exploit > would be. But I think it was understood to be a real concern 8 years ago, so > not exactly sudden. That's not news indeed, but that doesn't make it less of a problem, especially now that the issue has been widely publicized through a conference and announcements on several widely-read Web sites. That said, only doing the security fix in 3.3 would have the nice side effect of pushing people towards Python 3, so perhaps I'm for it after all. Half-jokingly, Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Testing the tests by modifying the ordering of dict items.
Hi, Python code should not depend upon the ordering of items in a dict. Unfortunately it seems that a number of tests in the standard library do just that. Changing PyDict_MINSIZE from 8 to either 4 or 16 causes the following tests to fail: test_dis test_email test_inspect test_nntplib test_packaging test_plistlib test_pprint test_symtable test_trace test_sys also fails, but this is a legitimate failure in sys.getsizeof() Changing the collision resolution function from f(n) = 5n + 1 to f(n) = n + 1 results in the same failures, except for test_packaging and test_symtable which pass. Finally, changing the seed in unicode_hash() from (implicit) 0 to an arbitrary value (12345678) causes the above tests to fail plus: test_json test_set test_ttk_textonly test_urllib test_urlparse I think this is a real issue as the unicode_hash() function is likely to change soon due to http://bugs.python.org/issue13703. Should I: 1. Submit one big bug report? 2. Submit a bug report for each "failing" test separately? 3. Ignore it, since the tests only fail when I start messing about? Cheers, Mark. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing the tests by modifying the ordering of dict items.
On Thu, 05 Jan 2012 13:46:52 + Mark Shannon wrote: > > Should I: > > 1. Submit one big bug report? > > 2. Submit a bug report for each "failing" test separately? I would say a separate bug report for each failing test file, i.e. one report for test_dis, one for test_email etc. Hope this doesn't eat too much of your time :) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling 2.7.2 on OS/2
2012/1/5 Paul Smedley
> Hi All,
>
> I'm working on updating my port of Python 2.6.5 to v2.7.2 for the OS/2
> platform.
>
> I have python.exe and python27.dll compiling find, but when starting to
> build sharedmods I'm getting the following error:
> running build
> running build_ext
> Traceback (most recent call last):
> File "./setup.py", line 2092, in
>main()
> File "./setup.py", line 2087, in main
>'Lib/smtpd.py']
> File "U:/DEV/python-2.7.2/Lib/**distutils/core.py", line 152, in setup
>dist.run_commands()
> File "U:/DEV/python-2.7.2/Lib/**distutils/dist.py", line 953, in
> run_commands
>self.run_command(cmd)
> File "U:/DEV/python-2.7.2/Lib/**distutils/dist.py", line 972, in
> run_command
>cmd_obj.run()
> File "U:/DEV/python-2.7.2/Lib/**distutils/command/build.py", line 127,
> in run
>self.run_command(cmd_name)
> File "U:/DEV/python-2.7.2/Lib/**distutils/cmd.py", line 326, in
> run_command
>self.distribution.run_command(**command)
> File "U:/DEV/python-2.7.2/Lib/**distutils/dist.py", line 972, in
> run_command
>cmd_obj.run()
> File "U:/DEV/python-2.7.2/Lib/**distutils/command/build_ext.**py", line
> 340, in run
>self.build_extensions()
> File "./setup.py", line 152, in build_extensions
>missing = self.detect_modules()
> File "./setup.py", line 1154, in detect_modules
>for arg in sysconfig.get_config_var("**CONFIG_ARGS").split()]
> AttributeError: 'NoneType' object has no attribute 'split'
> make: *** [sharedmods] Error 1
>
>
> Any suggestions? A google showed a similar error on AIX with no clear
> resolution.
>
Is it in the part that configures the "dbm" module?
This paragraph is already protected by a "if platform not in ['cygwin']:",
I suggest to exclude 'os2emx' as well.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing the tests by modifying the ordering of dict items.
On Jan 05, 2012, at 01:46 PM, Mark Shannon wrote: >2. Submit a bug report for each "failing" test separately? I'm sure it will be a pain, but this is really the best thing to do. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, Jan 5, 2012 at 3:39 PM, Antoine Pitrou wrote: > On Thu, 5 Jan 2012 15:26:27 +1100 > Andrew Bennetts wrote: >> >> I don't think that's news either. >> http://mail.python.org/pipermail/python-dev/2003-May/035907.html and >> http://twistedmatrix.com/pipermail/twisted-python/2003-June/004339.html for >> instance show that in 2003 it was clearly known to at least be likely to be >> an >> exploitable DoS in common code (a dict of HTTP headers or HTTP form keys). >> >> There was debate about whether it's the language's responsibility to mitigate >> the problem or if apps should use safer designs for handling untrusted input >> (e.g. limit the number of keys input is allowed to create, or use something >> other than dicts), and debate about just how practical an effective exploit >> would be. But I think it was understood to be a real concern 8 years ago, so >> not exactly sudden. > > That's not news indeed, but that doesn't make it less of a problem, > especially now that the issue has been widely publicized through a > conference and announcements on several widely-read Web sites. > > That said, only doing the security fix in 3.3 would have the nice side > effect of pushing people towards Python 3, so perhaps I'm for it after > all. > > Half-jokingly, > > Antoine. > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com Just to make things clear - stdlib itself has 1/64 of tests relying on dict order. Changing dict order in *older* pythons will break everyone's tests and some peoples code. Making this new 2.6.x release would mean that people using new python 2.6 would have to upgrade an unspecified amount of their python packages, that does not sound very cool. Also consider that new 2.6.x would go as a security fix to old ubuntu, but all other packages won't, because they'll not contain security fixes. Just so you know Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 1/5/2012 9:34 AM, Maciej Fijalkowski wrote: Also consider that new 2.6.x would go as a security fix to old ubuntu, but all other packages won't, because they'll not contain security fixes. Just so you know Why should CPython by constrained by broken policies of Ubuntu? If the other packages must be fixed so they work correctly with a security fix in Python, then they should be considered as containing a security fix. If they aren't, then that is a broken policy. On the other hand, it is very true that the seductive convenience of dict (readily available, good performance) in normal cases have created the vulnerability because its characteristics are a function of the data inserted, and when used for data that is from unknown, possibly malicious sources, that is a bug in the program that uses dict, not in dict itself. So it seems to me that: 1) the security problem is not in CPython, but rather in web servers that use dict inappropriately. 2) changing CPython in a way that breaks code is not a security fix to CPython, but rather a gratuitous breakage of compatibility promises, wrapped in a security-fix lie. The problem for CPython here can be summarized as follows: a) it is being blamed for problems in web servers that are not problems in CPython b) perhaps dict documentation is a bit too seductive, in not declaring that data from malicious sources could cause its performance to degrade significantly (but then, any programmer who has actually taken a decent set of programming classes should understand that, but on the other hand, there are programmers who have not taken such classes). c) CPython provides no other mapping data structures that rival the performance and capabilities of dict as an alternative, nor can such data structures be written in CPython, as the performance of dict comes not only from hashing, but also from being written in C. The solutions could be: A) push back on the blame: it is not a CPython problem B) perhaps add a warning to the documentation for the naïve, untrained programmers C) consider adding an additional data structure to the language, and mention it in the B warning for versions 3.3+. On the other hand, the web server vulnerability could be blamed on CPython in another way: identify vulnerable packages in the stdlib that are likely the be used during the parsing of user-supplied data. Ones that come to mind (Python 3.2) are: urllib.parse (various parse* functions) (package names different in Python 2.x) cgi (parse_multipart, FieldStorage) So, fixing the vulnerable packages could be a sufficient response, rather than changing the hash function. How to fix? Each of those above allocates and returns a dict. Simply have each of those allocate and return and wrapped dict, which has the following behaviors: i) during __init__, create a local, random, string. ii) for all key values, prepend the string, before passing it to the internal dict. Changing these vulnerable packages rather than the hash function is a much more constrained change, and wouldn't create bugs in programs that erroneously depend on the current hash function directly or indirectly. This would not fix web servers that use their own parsing and storage mechanism for fields, if they have also inappropriately used a dict as their storage mechanism for user supplied data. However, a similar solution could be similarly applied by the authors of those web servers, and would be a security fix to such packages, so should be applied to Ubuntu, if available there, or other systems with security-only fix acceptance. This solution does not require changes to the hash, does not require a cryptographicly secure hash, and does not require code to be added to the initialization of Python before normal objects and mappings can be created. If a port doesn't contain a good random number generator, a weak one can be subsitituted, but such decisions can be made in Python code after the interpreter is initialized, and use of stdlib packages is available. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, 5 Jan 2012 19:34:13 +0200 Maciej Fijalkowski wrote: > > Just to make things clear - stdlib itself has 1/64 of tests relying on > dict order. Changing dict order in *older* pythons will break > everyone's tests and some peoples code. Breaking tests is not a problem: they are typically not run by production code and so people can take the time to fix them. Breaking other code is a problem if it is legitimate. Relying on dict ordering is totally wrong and I don't think we should care about such cases. The only issue is when relying on hash() being stable accross runs. But hashing already varies from build to build (32-bit vs. 64-bit) and I think that anyone seriously relying on it should already have been bitten. > Making this new 2.6.x release > would mean that people using new python 2.6 would have to upgrade an > unspecified amount of their python packages, that does not sound very > cool. How about 2.7? Do you think it should also remain untouched? I am ok for leaving 2.6 alone (that's Barry's call anyway) but 2.7 is another matter - should people migrate to 3.x to get the security fix? As for 3.2, it should certainly get the fix IMO. There are not many Python 3 legacy applications relying on hash() stability, I think. > Also consider that new 2.6.x would go as a security fix to old > ubuntu, but all other packages won't, because they'll not contain > security fixes. Ubuntu can decide *not* to ship the fix if they prefer it like that. Their policies and decisions, though, should not taint ours. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, 2012-01-05 at 19:34 +0200, Maciej Fijalkowski wrote: > On Thu, Jan 5, 2012 at 3:39 PM, Antoine Pitrou wrote: > > On Thu, 5 Jan 2012 15:26:27 +1100 > > Andrew Bennetts wrote: > >> > >> I don't think that's news either. > >> http://mail.python.org/pipermail/python-dev/2003-May/035907.html and > >> http://twistedmatrix.com/pipermail/twisted-python/2003-June/004339.html for > >> instance show that in 2003 it was clearly known to at least be likely to > >> be an > >> exploitable DoS in common code (a dict of HTTP headers or HTTP form keys). > >> > >> There was debate about whether it's the language's responsibility to > >> mitigate > >> the problem or if apps should use safer designs for handling untrusted > >> input > >> (e.g. limit the number of keys input is allowed to create, or use something > >> other than dicts), and debate about just how practical an effective exploit > >> would be. But I think it was understood to be a real concern 8 years ago, > >> so > >> not exactly sudden. > > > > That's not news indeed, but that doesn't make it less of a problem, > > especially now that the issue has been widely publicized through a > > conference and announcements on several widely-read Web sites. > > > > That said, only doing the security fix in 3.3 would have the nice side > > effect of pushing people towards Python 3, so perhaps I'm for it after > > all. > > > > Half-jokingly, > > > > Antoine. > > Just to make things clear - stdlib itself has 1/64 of tests relying on > dict order. Changing dict order in *older* pythons will break > everyone's tests and some peoples code. Making this new 2.6.x release > would mean that people using new python 2.6 would have to upgrade an > unspecified amount of their python packages, that does not sound very > cool. Also consider that new 2.6.x would go as a security fix to old > ubuntu, but all other packages won't, because they'll not contain > security fixes. Just so you know We have similar issues in RHEL, with the Python versions going much further back (e.g. 2.3) When backporting the fix to ancient python versions, I'm inclined to turn the change *off* by default, requiring the change to be enabled via an environment variable: I want to avoid breaking existing code, even if such code is technically relying on non-guaranteed behavior. But we could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. That way /usr/bin/python would default to the old behavior, but web apps would have some protection. Any such logic here also suggests the need for an attribute in the sys module so that you can verify the behavior. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/05/2012 02:14 PM, Glenn Linderman wrote: > 1) the security problem is not in CPython, but rather in web servers > that use dict inappropriately. Most webapp vulnerabilities are due to their use of Python's cgi module, which it uses a dict to hold the form / query string data being supplied by untrusted external users. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8F/uEACgkQ+gerLs4ltQ679QCgqKPYYwEetKR3bEMVh5eukLin cA8An3XJMYWhK5MutjbOCxCfYzKXmDzc =V3lh -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling 2.7.2 on OS/2
Hi Amaury, On 06/01/12 00:32, Amaury Forgeot d'Arc wrote: 2012/1/5 Paul Smedley mailto:[email protected]>> Hi All, I'm working on updating my port of Python 2.6.5 to v2.7.2 for the OS/2 platform. I have python.exe and python27.dll compiling find, but when starting to build sharedmods I'm getting the following error: running build running build_ext Traceback (most recent call last): File "./setup.py", line 2092, in main() File "./setup.py", line 2087, in main 'Lib/smtpd.py'] File "U:/DEV/python-2.7.2/Lib/__distutils/core.py", line 152, in setup dist.run_commands() File "U:/DEV/python-2.7.2/Lib/__distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "U:/DEV/python-2.7.2/Lib/__distutils/dist.py", line 972, in run_command cmd_obj.run() File "U:/DEV/python-2.7.2/Lib/__distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "U:/DEV/python-2.7.2/Lib/__distutils/cmd.py", line 326, in run_command self.distribution.run_command(__command) File "U:/DEV/python-2.7.2/Lib/__distutils/dist.py", line 972, in run_command cmd_obj.run() File "U:/DEV/python-2.7.2/Lib/__distutils/command/build_ext.__py", line 340, in run self.build_extensions() File "./setup.py", line 152, in build_extensions missing = self.detect_modules() File "./setup.py", line 1154, in detect_modules for arg in sysconfig.get_config_var("__CONFIG_ARGS").split()] AttributeError: 'NoneType' object has no attribute 'split' make: *** [sharedmods] Error 1 Any suggestions? A google showed a similar error on AIX with no clear resolution. Is it in the part that configures the "dbm" module? This paragraph is already protected by a "if platform not in ['cygwin']:", I suggest to exclude 'os2emx' as well. It is - however adding os2 the the list of platforms to the ones to exclude gets me only a little further: It then bombs with: running build running build_ext Traceback (most recent call last): File "./setup.py", line 2092, in main() File "./setup.py", line 2087, in main 'Lib/smtpd.py'] File "U:/DEV/python-2.7.2/Lib/distutils/core.py", line 152, in setup dist.run_commands() File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 972, in run_command cmd_obj.run() File "U:/DEV/python-2.7.2/Lib/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "U:/DEV/python-2.7.2/Lib/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "U:/DEV/python-2.7.2/Lib/distutils/dist.py", line 972, in run_command cmd_obj.run() File "U:/DEV/python-2.7.2/Lib/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "./setup.py", line 152, in build_extensions missing = self.detect_modules() File "./setup.py", line 1368, in detect_modules if '--with-system-expat' in sysconfig.get_config_var("CONFIG_ARGS"): TypeError: argument of type 'NoneType' is not iterable make: *** [sharedmods] Error 1 Which again points to problems with sysconfig.get_config_var("CONFIG_ARGS"): Thanks, Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 1/5/2012 11:49 AM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/05/2012 02:14 PM, Glenn Linderman wrote: 1) the security problem is not in CPython, but rather in web servers that use dict inappropriately. Most webapp vulnerabilities are due to their use of Python's cgi module, which it uses a dict to hold the form / query string data being supplied by untrusted external users. Yes, I understand that (and have some such web apps in production). In fact, I pointed out urllib.parse and cgi as specific modules for which a proposed fix could be made without impacting the Python hash function. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/05/2012 02:14 PM, Glenn Linderman wrote: 1) the security problem is not in CPython, but rather in web servers that use dict inappropriately. Most webapp vulnerabilities are due to their use of Python's cgi module, which it uses a dict to hold the form / query string data being supplied by untrusted external users. And Glenn suggested further down that an appropriate course of action would be to fix the cgi module (and others) instead of messing with dict. ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 5 January 2012 19:33, David Malcolm wrote: > We have similar issues in RHEL, with the Python versions going much > further back (e.g. 2.3) > > When backporting the fix to ancient python versions, I'm inclined to > turn the change *off* by default, requiring the change to be enabled via > an environment variable: I want to avoid breaking existing code, even if > such code is technically relying on non-guaranteed behavior. But we > could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. > That way /usr/bin/python would default to the old behavior, but web apps > would have some protection. Any such logic here also suggests the need > for an attribute in the sys module so that you can verify the behavior. Uh, surely no-one is suggesting backporting to "ancient" versions? I couldn't find the statement quickly on the python.org website (so this is via google), but isn't it true that 2.6 is in security-only mode and 2.5 and earlier will never get the fix? Having a source-only release for 2.6 means the fix is "off by default" in the sense that you can choose not to build it. Or add a #ifdef to the source if it really matters. Personally, I find it hard to see this as a Python security hole, but I can sympathise with the idea that it would be nice to make dict "safer by default". (Although the benefit for me personally would be zero, so I'm reluctant for the change to have a detectable cost...) My feeling is that it should go into 2.7, 3.2, and 3.3+, but with no bells and whistles to switch it off or the like. If it's not suitable to go in on that basis, restrict it to 3.3+ (where it's certainly OK) and advise users of earlier versions to either upgrade or code defensively to avoid hitting the pathological case. Surely that sort of defensive code should be second nature to the people who might be affected by the issue? Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Jan 05, 2012, at 02:33 PM, David Malcolm wrote: >We have similar issues in RHEL, with the Python versions going much >further back (e.g. 2.3) > >When backporting the fix to ancient python versions, I'm inclined to >turn the change *off* by default, requiring the change to be enabled via >an environment variable: I want to avoid breaking existing code, even if >such code is technically relying on non-guaranteed behavior. But we >could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. >That way /usr/bin/python would default to the old behavior, but web apps >would have some protection. This sounds like a reasonable compromise for all stable Python releases. It can be turned on by default for Python 3.3. If you also make the default setting easy to change (i.e. parameterized in one place), then distros can make their own decision about the default, although I'd argue for the above default approach for Debian/Ubuntu. >Any such logic here also suggests the need for an attribute in the sys module >so that you can verify the behavior. That would be read-only though, right? -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Jan 05, 2012, at 08:35 PM, Paul Moore wrote: >Uh, surely no-one is suggesting backporting to "ancient" versions? I >couldn't find the statement quickly on the python.org website (so this >is via google), but isn't it true that 2.6 is in security-only mode >and 2.5 and earlier will never get the fix? Having a source-only >release for 2.6 means the fix is "off by default" in the sense that >you can choose not to build it. Or add a #ifdef to the source if it >really matters. Correct, although there's no reason why a patch for versions older than 2.6 couldn't be included on a python.org security page for reference in CVE or other security notifications. Distros that care about versions older than Python 2.6 will basically be back-porting the patch anyway. >My feeling is that it should go into 2.7, 3.2, and 3.3+, but with no >bells and whistles to switch it off or the like. I like David Malcolm's suggestion, but I have no problem applying it to 3.3, enabled by default with no way to turn it off. The off-by-default on-switch policy for stable releases would be justified by maximum backward compatibility conservativeness. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, Jan 05, 2012 at 08:35:57PM +, Paul Moore wrote: > On 5 January 2012 19:33, David Malcolm wrote: > > We have similar issues in RHEL, with the Python versions going much > > further back (e.g. 2.3) > > > > When backporting the fix to ancient python versions, I'm inclined to > > turn the change *off* by default, requiring the change to be enabled via > > an environment variable: I want to avoid breaking existing code, even if > > such code is technically relying on non-guaranteed behavior. But we > > could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. > > That way /usr/bin/python would default to the old behavior, but web apps > > would have some protection. Any such logic here also suggests the need > > for an attribute in the sys module so that you can verify the behavior. > > Uh, surely no-one is suggesting backporting to "ancient" versions? I > couldn't find the statement quickly on the python.org website (so this > is via google), but isn't it true that 2.6 is in security-only mode > and 2.5 and earlier will never get the fix? > I think when dmalcolm says "backporting" he means that he'll have to backport the fix from modern, supported-by-python.org python to the ancient python's that he's supporting as part of the Linux distributions where he's the python package maintainer. I'm thinking he's mentioning it here mainly to see if someone thinks that his approach for those distributions causes anyone to point out a reason not to diverge from upstream in that manner. > Having a source-only > release for 2.6 means the fix is "off by default" in the sense that > you can choose not to build it. Or add a #ifdef to the source if it > really matters. > I don't think that this would satisfy dmalcolm's needs. What he's talking about sounds more like a runtime switch (possibly only when initializing, though, not on-the-fly). -Toshio pgp7qk95cGJ9b.pgp Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, 2012-01-05 at 20:35 +, Paul Moore wrote: > On 5 January 2012 19:33, David Malcolm wrote: > > We have similar issues in RHEL, with the Python versions going much > > further back (e.g. 2.3) > > > > When backporting the fix to ancient python versions, I'm inclined to > > turn the change *off* by default, requiring the change to be enabled via > > an environment variable: I want to avoid breaking existing code, even if > > such code is technically relying on non-guaranteed behavior. But we > > could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. > > That way /usr/bin/python would default to the old behavior, but web apps > > would have some protection. Any such logic here also suggests the need > > for an attribute in the sys module so that you can verify the behavior. > > Uh, surely no-one is suggesting backporting to "ancient" versions? I > couldn't find the statement quickly on the python.org website (so this > is via google), but isn't it true that 2.6 is in security-only mode > and 2.5 and earlier will never get the fix? Having a source-only > release for 2.6 means the fix is "off by default" in the sense that > you can choose not to build it. Or add a #ifdef to the source if it > really matters. Sorry, if I was unclear. I don't expect python-dev to do this backporting, but those of us who do maintain such ancient pythons via Linux distributions may want to do the backport for our users. My email was to note that it may make sense to pick more conservative defaults for such a scenario, as compared to 2.6 onwards. [snip] Hope this is helpful Dave ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 01/05/2012 09:45 PM, Barry Warsaw wrote: > On Jan 05, 2012, at 02:33 PM, David Malcolm wrote: > >>We have similar issues in RHEL, with the Python versions going much >>further back (e.g. 2.3) >> >>When backporting the fix to ancient python versions, I'm inclined to >>turn the change *off* by default, requiring the change to be enabled via >>an environment variable: I want to avoid breaking existing code, even if >>such code is technically relying on non-guaranteed behavior. But we >>could potentially tweak mod_python/mod_wsgi so that it defaults to *on*. >>That way /usr/bin/python would default to the old behavior, but web apps >>would have some protection. > > This sounds like a reasonable compromise for all stable Python releases. It > can be turned on by default for Python 3.3. If you also make the default > setting easy to change (i.e. parameterized in one place), then distros can > make their own decision about the default, although I'd argue for the above > default approach for Debian/Ubuntu. Agreed. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Am 05.01.2012 21:45, schrieb Barry Warsaw: > This sounds like a reasonable compromise for all stable Python releases. It > can be turned on by default for Python 3.3. If you also make the default > setting easy to change (i.e. parameterized in one place), then distros can > make their own decision about the default, although I'd argue for the above > default approach for Debian/Ubuntu. Hey Barry, stop stealing my ideas! :) I've argued for these default settings for days. ver deliveryrandomized hashing == 2.3 patch disabled by default 2.4 patch disabled 2.5 patch disabled 2.6 release disabled 2.7 release disabled 3.0 ignore? disabled 3.1 release disabled 3.2 release disabled 3.3 n/a yet enabled by default 2.3 to 2.5 are still used in production (RHEL, Ubuntu LTS). Guido has stated that he needs a patch for 2.4, too. I think we may safely ignore Python 3.0. Nobody should use Python 3.0 on a production system. I've suggested the env var PYRANDOMHASH. It's easy to set env vars in Apache. For example Debian/Ubuntu has /etc/apache2/envvars. Settings for PYRANDOMHASH: PYRANDOMHASH=1 enable randomized hashing function PYRANDOMHASH=/path/to/seed enable randomized hashing function and read seed from 'seed' PYRANDOMHASH=0 disable randomed hashing function Since there isn't an easy way to set env vars in a shebang line since something like #!/usr/bin/env PYRANDOMHASH=1 python2.7 doesn't work, we could come up with a solution the shebang. IMHO the setting for the default setting should be a compile time option. It's reasonable easy to extend the configure script to support --enable-randomhash / --disable-randomhash. The MS VC build scripts can grow a flag, too. I still think that the topic needs a PEP. A couple of days ago I started with a PEP. But Guido told me that he doesn't see a point in a PEP because he prefers a small and quick solution, so I stopped working on it. However the arguments, worries and ideas in this enormous topic have repeated over and over. We know from experience that a PEP is a great way to explain the how, what and why of the change as well as the paths we didn't take. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] usefulness of Python version of threading.RLock
Hi, Issue #13697 (http://bugs.python.org/issue13697) deals with a problem with the Python version of threading.RLock (a signal handler which tries to acquire the same RLock is called right at the wrong time) which doesn't affect the C version. Whether such a use case can be considered good practise or the best way to fix this is not settled yet, but the question that arose to me is: "why do we have both a C and Python version?". Here's Antoine answer (he suggested to me to bring this up on python-dev": """ The C version is quite recent, and there's a school of thought that we should always provide fallback Python implementations. (also, arguably a Python implementation makes things easier to prototype, although I don't think it's the case for an RLock) """ So, what do you guys think? Would it be okay to nuke the Python version? Do you have more details on this "school of thought"? Also, while we're at it, Victor created #13550 to try to rewrite the "logging hack" of the threading module: there again, I think we could just remove this logging altogether. What do you think? Cheers, cf ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Am 05.01.2012 21:10, schrieb Ethan Furman: > Tres Seaver wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 01/05/2012 02:14 PM, Glenn Linderman wrote: >>> 1) the security problem is not in CPython, but rather in web servers >>> that use dict inappropriately. >> >> Most webapp vulnerabilities are due to their use of Python's cgi module, >> which it uses a dict to hold the form / query string data being supplied >> by untrusted external users. > > And Glenn suggested further down that an appropriate course of action > would be to fix the cgi module (and others) instead of messing with dict. You'd have to fix any Python core module that may handle data from untrusted sources. The issue isn't limited to web apps and POST requests. It's possible to trigger the DoS from JSON, a malicious PDF, JPEG's EXIF metadata or any other data. Oh, and somebody has to fix all 3rd party modules, too. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Thu, 05 Jan 2012 22:40:58 +0100 Christian Heimes wrote: > Am 05.01.2012 21:45, schrieb Barry Warsaw: > > This sounds like a reasonable compromise for all stable Python releases. It > > can be turned on by default for Python 3.3. If you also make the default > > setting easy to change (i.e. parameterized in one place), then distros can > > make their own decision about the default, although I'd argue for the above > > default approach for Debian/Ubuntu. > > Hey Barry, stop stealing my ideas! :) I've argued for these default > settings for days. > > ver deliveryrandomized hashing > == > 2.3 patch disabled by default > 2.4 patch disabled > 2.5 patch disabled > 2.6 release disabled > 2.7 release disabled > 3.0 ignore? disabled > 3.1 release disabled > 3.2 release disabled > 3.3 n/a yet enabled by default I don't think we (python-dev) are really concerned with 2.3, 2.4, 2.5 and 3.0. They're all unsupported, and people do what they want with their local source trees. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] usefulness of Python version of threading.RLock
2012/1/5 Charles-François Natali : > Hi, > > Issue #13697 (http://bugs.python.org/issue13697) deals with a problem > with the Python version of threading.RLock (a signal handler which > tries to acquire the same RLock is called right at the wrong time) > which doesn't affect the C version. > Whether such a use case can be considered good practise or the best > way to fix this is not settled yet, but the question that arose to me > is: "why do we have both a C and Python version?". > Here's Antoine answer (he suggested to me to bring this up on python-dev": > """ > The C version is quite recent, and there's a school of thought that we > should always provide fallback Python implementations. > (also, arguably a Python implementation makes things easier to > prototype, although I don't think it's the case for an RLock) > """ > > So, what do you guys think? > Would it be okay to nuke the Python version? > Do you have more details on this "school of thought"? >From what I understand, the biggest motivation for pure Python versions is cooperation with the other Python implementations. See http://www.python.org/dev/peps/pep-0399/ -eric > > Also, while we're at it, Victor created #13550 to try to rewrite the > "logging hack" of the threading module: there again, I think we could > just remove this logging altogether. What do you think? > > Cheers, > > cf > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Am 05.01.2012 22:59, schrieb Antoine Pitrou: > I don't think we (python-dev) are really concerned with 2.3, 2.4, > 2.5 and 3.0. They're all unsupported, and people do what they want > with their local source trees. Let me reply with a quote from Barry: > Correct, although there's no reason why a patch for versions > older than 2.6 couldn't be included on a python.org security > page for reference in CVE or other security notifications. > Distros that care about versions older than Python 2.6 will > basically be back-porting the patch anyway. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
05.01.12 21:14, Glenn Linderman написав(ла):
So, fixing the vulnerable packages could be a sufficient response,
rather than changing the hash function. How to fix? Each of those
above allocates and returns a dict. Simply have each of those allocate
and return and wrapped dict, which has the following behaviors:
i) during __init__, create a local, random, string.
ii) for all key values, prepend the string, before passing it to the
internal dict.
Good idea.
# -*- coding: utf-8 -*-
from collections import MutableMapping
import random
class SafeDict(dict, MutableMapping):
def __init__(self, *args, **kwds):
dict.__init__(self)
self._prefix = str(random.getrandbits(64))
self.update(*args, **kwds)
def clear(self):
dict.clear(self)
self._prefix = str(random.getrandbits(64))
def _safe_key(self, key):
return self._prefix + repr(key), key
def __getitem__(self, key):
try:
return dict.__getitem__(self, self._safe_key(key))
except KeyError as e:
e.args = (key,)
raise e
def __setitem__(self, key, value):
dict.__setitem__(self, self._safe_key(key), value)
def __delitem__(self, key):
try:
dict.__delitem__(self, self._safe_key(key))
except KeyError as e:
e.args = (key,)
raise e
def __iter__(self):
for skey, key in dict.__iter__(self):
yield key
def __contains__(self, key):
return dict.__contains__(self, self._safe_key(key))
setdefault = MutableMapping.setdefault
update = MutableMapping.update
pop = MutableMapping.pop
popitem = MutableMapping.popitem
keys = MutableMapping.keys
values = MutableMapping.values
items = MutableMapping.items
def __repr__(self):
return '{%s}' % ', '.join('%s: %s' % (repr(k), repr(v))
for k, v in self.items())
def copy(self):
return self.__class__(self)
@classmethod
def fromkeys(cls, iterable, value=None):
d = cls()
for key in iterable:
d[key] = value
return d
def __eq__(self, other):
return all(k in other and other[k] == v for k, v in self.items()) and \
all(k in self and self[k] == v for k, v in other.items())
def __ne__(self, other):
return not self == other
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] usefulness of Python version of threading.RLock
On Thu, 5 Jan 2012 15:02:42 -0700 Eric Snow wrote: > 2012/1/5 Charles-François Natali : > > Hi, > > > > Issue #13697 (http://bugs.python.org/issue13697) deals with a problem > > with the Python version of threading.RLock (a signal handler which > > tries to acquire the same RLock is called right at the wrong time) > > which doesn't affect the C version. > > Whether such a use case can be considered good practise or the best > > way to fix this is not settled yet, but the question that arose to me > > is: "why do we have both a C and Python version?". > > Here's Antoine answer (he suggested to me to bring this up on python-dev": > > """ > > The C version is quite recent, and there's a school of thought that we > > should always provide fallback Python implementations. > > (also, arguably a Python implementation makes things easier to > > prototype, although I don't think it's the case for an RLock) > > """ > > > > So, what do you guys think? > > Would it be okay to nuke the Python version? > > Do you have more details on this "school of thought"? > > >From what I understand, the biggest motivation for pure Python > versions is cooperation with the other Python implementations. See > http://www.python.org/dev/peps/pep-0399/ Apologies, I didn't remember it was written down in PEP. A bit more than a school of thought, then :-) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling 2.7.2 on OS/2
On 1/5/2012 3:01 PM, Paul Smedley wrote:
File "./setup.py", line 1154, in detect_modules
for arg in sysconfig.get_config_var("__CONFIG_ARGS").split()]
AttributeError: 'NoneType' object has no attribute 'split'
make: *** [sharedmods] Error 1
File "./setup.py", line 1368, in detect_modules
if '--with-system-expat' in sysconfig.get_config_var("CONFIG_ARGS"):
TypeError: argument of type 'NoneType' is not iterable
make: *** [sharedmods] Error 1
Which again points to problems with
sysconfig.get_config_var("CONFIG_ARGS"):
[The earlier call was with "__CONFIG_ARGS", for whatever difference that
makes.] It appears to be returning None instead of [] (or a populated list).
In 3.2.2, at line 579 of sysconfig.py is
def get_config_var(name):
return get_config_vars().get(name)
That defaults to None if name is not a key in the dict returned by
get_config_vars(). My guess is that it always is and and the the value
is always a list for tested win/*nix/mac systems. So either setup.py has
the bug of assuming that there is always a list value for "CONFIG_ARGS"
or sysconfig.py has the bug of not setting it for os2, perhaps because
of a bug elsewhere.
At line 440 of sysconfig.py is
def get_config_var(*args):
global _CONFIG_VARS
if _CONFIG_VARS is None:
_CONFIG_VARS = {}
if os.name in ('nt', 'os2'):
_init_non_posix(_CONFIG_VARS)
if args:
vals = []
for name in args:
vals.append(_CONFIG_VARS.get(name))
return vals
else:
return _CONFIG_VARS
At 456 is
def _init_non_posix(vars):
"""Initialize the module as appropriate for NT"""
# set basic install directories
...
"CONFIG_ARGS" is not set explicitly for any system anywhere in the file,
so I do not know how the call ever works.
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Fri, Jan 6, 2012 at 8:15 AM, Serhiy Storchaka wrote: > 05.01.12 21:14, Glenn Linderman написав(ла): >> >> So, fixing the vulnerable packages could be a sufficient response, >> rather than changing the hash function. How to fix? Each of those >> above allocates and returns a dict. Simply have each of those allocate >> and return and wrapped dict, which has the following behaviors: >> >> i) during __init__, create a local, random, string. >> ii) for all key values, prepend the string, before passing it to the >> internal dict. > > > Good idea. Not a good idea - a lot of the 3rd party tests that depend on dict ordering are going to be using those modules anyway, so scattering our solution across half the standard library is needlessly creating additional work without really reducing the incompatibility problem. If we're going to change anything, it may as well be the string hashing algorithm itself. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 1/5/2012 3:10 PM, Ethan Furman wrote: Tres Seaver wrote: 1) the security problem is not in CPython, but rather in web servers that use dict inappropriately. Most webapp vulnerabilities are due to their use of Python's cgi module, which it uses a dict to hold the form / query string data being supplied by untrusted external users. And Glenn suggested further down that an appropriate course of action would be to fix the cgi module (and others) instead of messing with dict. I think both should be done. For web applications, it would be best to reject DOS attempts with 'random' keys in O(1) time rather than in O(n) time even with improved hash. But some other apps, like the Python interpreter itself, 'random' names may be quite normal. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
David Malcolm wrote:
When backporting the fix to ancient python versions, I'm inclined to
turn the change *off* by default, requiring the change to be enabled via
an environment variable: I want to avoid breaking existing code, even if
such code is technically relying on non-guaranteed behavior. But we
could potentially tweak mod_python/mod_wsgi so that it defaults to *on*.
That way /usr/bin/python would default to the old behavior, but web apps
would have some protection. Any such logic here also suggests the need
for an attribute in the sys module so that you can verify the behavior.
Surely the way to verify the behaviour is to run this from the shell:
python -c print(hash("abcde"))
twice, and see that the calls return different values. (Or have I
misunderstood the way the fix is going to work?)
In any case, I wouldn't want to rely on the presence of a flag in the sys
module to verify the behaviour, I'd want to see for myself that hash
collisions are no longer predictable.
--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Jan 05, 2012, at 10:40 PM, Christian Heimes wrote: >Hey Barry, stop stealing my ideas! :) I've argued for these default >settings for days. :) >I've suggested the env var PYRANDOMHASH. It's easy to set env vars in >Apache. For example Debian/Ubuntu has /etc/apache2/envvars. For consistency, it really should be PYTHONSOMETHING. I personally don't care how long it is (e.g. PYTHONIOENCODING). >Settings for PYRANDOMHASH: > > PYRANDOMHASH=1 > enable randomized hashing function > > PYRANDOMHASH=/path/to/seed > enable randomized hashing function and read seed from 'seed' > > PYRANDOMHASH=0 > disable randomed hashing function Why not PYTHONHASHSEED then? >Since there isn't an easy way to set env vars in a shebang line since >something like > > #!/usr/bin/env PYRANDOMHASH=1 python2.7 > >doesn't work, we could come up with a solution the shebang. We have precedence for mirroring startup options and envars, so it doesn't bother me to add such a switch to Python 3.3. It *does* bother me to add a switch to any stable release. >IMHO the setting for the default setting should be a compile time >option. It's reasonable easy to extend the configure script to support >--enable-randomhash / --disable-randomhash. The MS VC build scripts can >grow a flag, too. > >I still think that the topic needs a PEP. A couple of days ago I started >with a PEP. But Guido told me that he doesn't see a point in a PEP >because he prefers a small and quick solution, so I stopped working on >it. However the arguments, worries and ideas in this enormous topic have >repeated over and over. We know from experience that a PEP is a great >way to explain the how, what and why of the change as well as the paths >we didn't take. One way to look at it is to have a quick-and-dirty solution for stable releases. It could be suboptimal from a ui point of view because of backward compatibility issues. The PEP could then outline the boffo perfect solution for Python 3.3, which a section on how it will be backported to stable releases. Cheers, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano wrote:
> Surely the way to verify the behaviour is to run this from the shell:
>
> python -c print(hash("abcde"))
>
> twice, and see that the calls return different values. (Or have I
> misunderstood the way the fix is going to work?)
>
> In any case, I wouldn't want to rely on the presence of a flag in the sys
> module to verify the behaviour, I'd want to see for myself that hash
> collisions are no longer predictable.
More directly, you can just check that the hash of the empty string is non-zero.
So -1 for a flag in the sys module - "hash('') != 0" should serve as a
sufficient check whether or not process-level string hash
randomisation is in effect.
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
2012/1/6 Barry Warsaw : >>Settings for PYRANDOMHASH: >> >> PYRANDOMHASH=1 >> enable randomized hashing function >> >> PYRANDOMHASH=/path/to/seed >> enable randomized hashing function and read seed from 'seed' >> >> PYRANDOMHASH=0 >> disable randomed hashing function > > Why not PYTHONHASHSEED then? See my patch attached to the issue #13703? I prepared the code to be able to set easily the hash seed (it has a LCG, it's seed can be provided by the user directly). I agree that the value 0 should give the same behaviour than the actual hash (disable the randomized hash). I will add the variable in the next version of my patch. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Am 06.01.2012 01:34, schrieb Nick Coghlan:
> On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano wrote:
>> Surely the way to verify the behaviour is to run this from the shell:
>>
>> python -c print(hash("abcde"))
>>
>> twice, and see that the calls return different values. (Or have I
>> misunderstood the way the fix is going to work?)
>>
>> In any case, I wouldn't want to rely on the presence of a flag in the sys
>> module to verify the behaviour, I'd want to see for myself that hash
>> collisions are no longer predictable.
>
> More directly, you can just check that the hash of the empty string is
> non-zero.
>
> So -1 for a flag in the sys module - "hash('') != 0" should serve as a
> sufficient check whether or not process-level string hash
> randomisation is in effect.
This might not work as we have to special case empty strings and perhaps
\0 strings, too. Otherwise we would give away the random seed to an
attacker if an attacker can somehow get hold of hash('') or hash(n * '\0').
Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
2012/1/5 Nick Coghlan :
> On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano wrote:
>> Surely the way to verify the behaviour is to run this from the shell:
>>
>> python -c print(hash("abcde"))
>>
>> twice, and see that the calls return different values. (Or have I
>> misunderstood the way the fix is going to work?)
>>
>> In any case, I wouldn't want to rely on the presence of a flag in the sys
>> module to verify the behaviour, I'd want to see for myself that hash
>> collisions are no longer predictable.
>
> More directly, you can just check that the hash of the empty string is
> non-zero.
>
> So -1 for a flag in the sys module - "hash('') != 0" should serve as a
> sufficient check whether or not process-level string hash
> randomisation is in effect.
What exactly is the disadvantage of a sys attribute? That would seem
preferable to an obscure incarnation like that.
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Fri, 06 Jan 2012 01:50:00 +0100
Christian Heimes wrote:
> Am 06.01.2012 01:34, schrieb Nick Coghlan:
> > On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano
> > wrote:
> >> Surely the way to verify the behaviour is to run this from the shell:
> >>
> >> python -c print(hash("abcde"))
> >>
> >> twice, and see that the calls return different values. (Or have I
> >> misunderstood the way the fix is going to work?)
> >>
> >> In any case, I wouldn't want to rely on the presence of a flag in the sys
> >> module to verify the behaviour, I'd want to see for myself that hash
> >> collisions are no longer predictable.
> >
> > More directly, you can just check that the hash of the empty string is
> > non-zero.
> >
> > So -1 for a flag in the sys module - "hash('') != 0" should serve as a
> > sufficient check whether or not process-level string hash
> > randomisation is in effect.
>
> This might not work as we have to special case empty strings and perhaps
> \0 strings, too.
The special case value doesn't have to be zero. Make it age(Barry) for
example (which, I think, is still representable in a 32-bit integer!).
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Fri, Jan 6, 2012 at 10:59 AM, Benjamin Peterson wrote: > What exactly is the disadvantage of a sys attribute? That would seem > preferable to an obscure incarnation like that. Adding sys attributes in maintenance (or security) releases makes me nervous. However, Victor and Christian are right about the need for a special case to avoid leaking information, so my particular suggested check won't work. The most robust check would be to run sys.executable in a subprocess and check if it gives the same hash for a non-empty string as the current process. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Benjamin Peterson wrote:
2012/1/5 Nick Coghlan :
On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano wrote:
Surely the way to verify the behaviour is to run this from the shell:
python -c print(hash("abcde"))
twice, and see that the calls return different values. (Or have I
misunderstood the way the fix is going to work?)
In any case, I wouldn't want to rely on the presence of a flag in the sys
module to verify the behaviour, I'd want to see for myself that hash
collisions are no longer predictable.
More directly, you can just check that the hash of the empty string is non-zero.
So -1 for a flag in the sys module - "hash('') != 0" should serve as a
sufficient check whether or not process-level string hash
randomisation is in effect.
What exactly is the disadvantage of a sys attribute? That would seem
preferable to an obscure incarnation like that.
There's nothing obscure about directly testing the hash. That's about as far
from obscure as it is possible to get: you are directly testing the presence
of a feature by testing the feature.
Relying on a flag to tell you whether hashes are randomised adds additional
complexity: now you need to care about whether hashes are randomised AND know
that there is a flag you can look up and what it is called.
And since the flag won't exist in all versions of Python, or even in all
builds of a particular Python version, it isn't a matter of just testing the
flag, but of doing the try...except or hasattr() dance to check whether it
exists first.
At some point, presuming that there is no speed penalty, the behaviour will
surely become not just enabled by default but mandatory. Python has never
promised that hashes must be predictable or consistent, so apart from
backwards compatibility concerns for old versions, future versions of Python
should make it mandatory. Presuming that there is no speed penalty, I'd argue
in favour of making it mandatory for 3.3. Why do we need a flag for something
that is going to be always on?
--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
2012/1/5 Steven D'Aprano :
> Benjamin Peterson wrote:
>>
>> 2012/1/5 Nick Coghlan :
>>>
>>> On Fri, Jan 6, 2012 at 10:07 AM, Steven D'Aprano
>>> wrote:
Surely the way to verify the behaviour is to run this from the shell:
python -c print(hash("abcde"))
twice, and see that the calls return different values. (Or have I
misunderstood the way the fix is going to work?)
In any case, I wouldn't want to rely on the presence of a flag in the
sys
module to verify the behaviour, I'd want to see for myself that hash
collisions are no longer predictable.
>>>
>>> More directly, you can just check that the hash of the empty string is
>>> non-zero.
>>>
>>> So -1 for a flag in the sys module - "hash('') != 0" should serve as a
>>> sufficient check whether or not process-level string hash
>>> randomisation is in effect.
>>
>>
>> What exactly is the disadvantage of a sys attribute? That would seem
>> preferable to an obscure incarnation like that.
>
>
> There's nothing obscure about directly testing the hash. That's about as far
> from obscure as it is possible to get: you are directly testing the presence
> of a feature by testing the feature.
It's obscure because hash('') != 0 doesn't necessarily mean the hashes
are randomized. A different hashing algorithm could be in effect.
--
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Am 06.01.2012 03:04, schrieb Benjamin Peterson:
> It's obscure because hash('') != 0 doesn't necessarily mean the hashes
> are randomized. A different hashing algorithm could be in effect.
Also in 1 of 2**32 or 2**64 tries hash('') is 0 although randomizing is
active.
Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] usefulness of Python version of threading.RLock
On Fri, Jan 6, 2012 at 11:17 AM, Antoine Pitrou wrote: >> >From what I understand, the biggest motivation for pure Python >> versions is cooperation with the other Python implementations. See >> http://www.python.org/dev/peps/pep-0399/ > > Apologies, I didn't remember it was written down in PEP. > A bit more than a school of thought, then :-) It needs to be correct to aid other implementation though, doesn't it? Copying/reusing something buggy won't help... -Rob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Benjamin Peterson wrote:
2012/1/5 Steven D'Aprano :
[...]
There's nothing obscure about directly testing the hash. That's about as far
from obscure as it is possible to get: you are directly testing the presence
of a feature by testing the feature.
It's obscure because hash('') != 0 doesn't necessarily mean the hashes
are randomized. A different hashing algorithm could be in effect.
Fair point, but I didn't actually suggest testing hash('') != 0, that was
Nick's suggestion, which he's since withdrawn.
--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On 1/5/2012 5:52 PM, Steven D'Aprano wrote: At some point, presuming that there is no speed penalty, the behaviour will surely become not just enabled by default but mandatory. Python has never promised that hashes must be predictable or consistent, so apart from backwards compatibility concerns for old versions, future versions of Python should make it mandatory. Presuming that there is no speed penalty, I'd argue in favour of making it mandatory for 3.3. Why do we need a flag for something that is going to be always on? I think the whole paragraph is invalid, because it presumes there is no speed penalty. I presume there will be a speed penalty, until benchmarking shows otherwise. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] usefulness of Python version of threading.RLock
I'm pretty sure the Python version of RLock is in use in several alternative implementations that provide an alternative _thread.lock. I think gevent would fall into this camp, as well as a personal project of mine in a similar vein that operates on python3. 2012/1/6 Charles-François Natali > Hi, > > Issue #13697 (http://bugs.python.org/issue13697) deals with a problem > with the Python version of threading.RLock (a signal handler which > tries to acquire the same RLock is called right at the wrong time) > which doesn't affect the C version. > Whether such a use case can be considered good practise or the best > way to fix this is not settled yet, but the question that arose to me > is: "why do we have both a C and Python version?". > Here's Antoine answer (he suggested to me to bring this up on python-dev": > """ > The C version is quite recent, and there's a school of thought that we > should always provide fallback Python implementations. > (also, arguably a Python implementation makes things easier to > prototype, although I don't think it's the case for an RLock) > """ > > So, what do you guys think? > Would it be okay to nuke the Python version? > Do you have more details on this "school of thought"? > > Also, while we're at it, Victor created #13550 to try to rewrite the > "logging hack" of the threading module: there again, I think we could > just remove this logging altogether. What do you think? > > Cheers, > > cf > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com > -- ಠ_ಠ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
06.01.12 02:10, Nick Coghlan написав(ла): Not a good idea - a lot of the 3rd party tests that depend on dict ordering are going to be using those modules anyway, so scattering our solution across half the standard library is needlessly creating additional work without really reducing the incompatibility problem. If we're going to change anything, it may as well be the string hashing algorithm itself. Changing the string hashing algorithm will hit the general performance and also will break down any code that depend on dict ordering. Specialized dict slow down only needed parts of some applications. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling 2.7.2 on OS/2
Hi Terry,
On 06/01/12 10:25, Terry Reedy wrote:
On 1/5/2012 3:01 PM, Paul Smedley wrote:
File "./setup.py", line 1154, in detect_modules
for arg in sysconfig.get_config_var("__CONFIG_ARGS").split()]
AttributeError: 'NoneType' object has no attribute 'split'
make: *** [sharedmods] Error 1
File "./setup.py", line 1368, in detect_modules
if '--with-system-expat' in sysconfig.get_config_var("CONFIG_ARGS"):
TypeError: argument of type 'NoneType' is not iterable
make: *** [sharedmods] Error 1
Which again points to problems with
sysconfig.get_config_var("CONFIG_ARGS"):
[The earlier call was with "__CONFIG_ARGS", for whatever difference that
makes.] It appears to be returning None instead of [] (or a populated
list).
In 3.2.2, at line 579 of sysconfig.py is
def get_config_var(name):
return get_config_vars().get(name)
That defaults to None if name is not a key in the dict returned by
get_config_vars(). My guess is that it always is and and the the value
is always a list for tested win/*nix/mac systems. So either setup.py has
the bug of assuming that there is always a list value for "CONFIG_ARGS"
or sysconfig.py has the bug of not setting it for os2, perhaps because
of a bug elsewhere.
At line 440 of sysconfig.py is
def get_config_var(*args):
global _CONFIG_VARS
if _CONFIG_VARS is None:
_CONFIG_VARS = {}
if os.name in ('nt', 'os2'):
_init_non_posix(_CONFIG_VARS)
if args:
vals = []
for name in args:
vals.append(_CONFIG_VARS.get(name))
return vals
else:
return _CONFIG_VARS
At 456 is
def _init_non_posix(vars):
"""Initialize the module as appropriate for NT"""
# set basic install directories
...
"CONFIG_ARGS" is not set explicitly for any system anywhere in the file,
so I do not know how the call ever works.
This looks pretty much the same as the code in 2.7.2 - I don't
understand Python code well enough to debug the script :(
Thanks for the response,
Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
