Re: [Python-Dev] SSL issues in Python stdlib and 3rd party code

2013-08-13 Thread Christian Heimes 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

CVE-2013-4238 has been signed to NULL bytes in subjectAltName issue.

  http://bugs.python.org/issue18709
  http://www.openwall.com/lists/oss-security/2013/08/13/2

Should we assign a CVE to issue in ssl.match_hostname(), too? Even
more projects have copied our code (bzr, tornado, pip, setuptools):

  http://bugs.python.org/issue17997
  https://bugs.mageia.org/show_bug.cgi?id=10391
  https://bugzilla.redhat.com/show_bug.cgi?id=963260#c11

Christian
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBCgAGBQJSCfcLAAoJEMeIxMHUVQ1FOC0P/0bPHK67qHbLf6HkHiVGNoAe
NUX5oT28bm00RyfmjU9ZPA3RWnPjFL9yiVXqP0mWzs4OzdPjGrHkw+uH285d/rFv
Di/Bcckq1lz/wzzsBeF/vviPVaSdV3tjlABgl/M6b902XhqEhZGg3RtiWmOvn+tc
1uKnXM4kWr/nUDbKYC2mBqbZD0IvN+XBQcy2cikjEtYcZc4QO80Dq9pL6g+3c4jH
7PpcMDyffsqD+Cd/PKK+Aq2tJOSHdHnK7V3/kTpRd+jheKSnq6idZYwQDU9sOkHT
NcVjqJtFkhGTzSD7u1/kNtD0UEleXn8sOxJwBLjcAqg+dV0BUEJk8uwuUn4Mi9Di
MaZbCs7NU/gPFdrS9pVxujaKaANbM4BJJwravA1/YYgPOGt1MhWlREbTg6W69w2+
57/PXs2Vt1nHISEyvCJLkIDVHeZx8ccm57YJ+zEMI2MKIBP7+21zY3Yq+86RwHs0
/h2mkzj8EQVcwvaVT4XfjezMp0A6Tbh/iwIQEbY6zUQ8OSBlbQ7FhF8VNXOqb5fh
pSVv0B6j1nNB8IaAAlMC56wRX2cmT8LvejUfGUq0duP+yiDYScknuqnhPePM1PZz
oPHSDbbfLI5s0Ab9d0encKKWatNmeoml/V7td5PUEAicDHJ1WnTB+FM9Qxv3qNQn
5J+eNhg2Bjj2en8PnbFo
=NiC2
-END PGP SIGNATURE-

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating the formatter module

2013-08-13 Thread Phil Elson 
On 12 August 2013 22:01, Ryan  wrote:

> Keep it, but put better documentation. It's needed.



There are many a useful package outside of the standard library. If this is
genuinely useful in some specialist use cases then I'm sure the code will
find its way to a github repo and be maintained as a standalone package in
itself. Who knows, being outside of the stdlib might breathe a new lease of
life into the concept if the release cycles are not bound to Python
releases.

Personally, I'd say delete it, unless there is a good reason not to.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating the formatter module

2013-08-13 Thread Serhiy Storchaka 

12.08.13 22:22, Brett Cannon написав(ла):

I have created http://bugs.python.org/issue18716 to deprecate the
formatter module for removal in Python 3.6 unless someone convinces me
otherwise that deprecation and removal is the wrong move.


The formatter module doesn't look such buggy as the audioop module. If 
the formatter module was not removed in Python 3.0 I don't see why it 
can't wait to 4.0.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks

2013-08-13 Thread Brett Cannon 
On Mon, Aug 12, 2013 at 10:10 PM, Arnaud Fontaine <
[email protected]> wrote:

> Brett Cannon  writes:
> > On Mon, Aug 12, 2013 at 5:12 AM, Arnaud Fontaine <
> [email protected] wrote:
> >> Yes, I saw the bug report and its patch implementing the import lock per
> >> module (mentioned in my initial email) and watched the presentation by
> >> Brett Cannon (BTW, I could not find the diagram explained during the
> >> presentation, anyone knows if it's available somewhere?).
> >
> > http://prezi.com/mqptpza9xbic/?utm_campaign=share&utm_medium=copy
>
> Thanks. Is the full diagram only available somewhere? (I mean as an
> image or PDF file, not within the presentation document itself)
>

Nope. I made the diagrams separately and then joined them directly in the
presentation.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating the formatter module

2013-08-13 Thread Brett Cannon 
On Tue, Aug 13, 2013 at 6:34 AM, Serhiy Storchaka wrote:

> 12.08.13 22:22, Brett Cannon написав(ла):
>
>  I have created 
> http://bugs.python.org/**issue18716to 
> deprecate the
>> formatter module for removal in Python 3.6 unless someone convinces me
>> otherwise that deprecation and removal is the wrong move.
>>
>
> The formatter module doesn't look such buggy as the audioop module. If the
> formatter module was not removed in Python 3.0 I don't see why it can't
> wait to 4.0.


It doesn't help that at PyCon CA we couldn't find a single person who had
heard of the module (which included people like Alex Gaynor and Larry
Hastings).

I'm definitely deprecating it, but we can discuss postponing deletion.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating the formatter module

2013-08-13 Thread Nick Coghlan 
On 13 Aug 2013 09:39, "Brett Cannon"  wrote:
>
>
>
>
> On Tue, Aug 13, 2013 at 6:34 AM, Serhiy Storchaka 
wrote:
>>
>> 12.08.13 22:22, Brett Cannon написав(ла):
>>
>>> I have created http://bugs.python.org/issue18716 to deprecate the
>>> formatter module for removal in Python 3.6 unless someone convinces me
>>> otherwise that deprecation and removal is the wrong move.
>>
>>
>> The formatter module doesn't look such buggy as the audioop module. If
the formatter module was not removed in Python 3.0 I don't see why it can't
wait to 4.0.
>
>
> It doesn't help that at PyCon CA we couldn't find a single person who had
heard of the module (which included people like Alex Gaynor and Larry
Hastings).
>
> I'm definitely deprecating it, but we can discuss postponing deletion.

Unless something is a serious bug magnet or is causing us maintenance
hassles, we shouldn't remove it. Neither applies here, so an indefinite
PendingDeprecation makes the most sense to me.

Cheers,
Nick.

>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks

2013-08-13 Thread Arnaud Fontaine 
Antoine Pitrou  writes:
> On Tue, 13 Aug 2013 11:06:51 +0900 Arnaud Fontaine 
>  wrote:
>> I suggested the same in my initial email, but I was wondering if there
>> could be any issue by releasing the lock in find_module()/load_module()
>> until the module is actually added to sys.modules.
>
> Well, you are obviously on your own with such hacks. There is a reason
> the lock exists.

Yes. Actually, I was thinking about implementing something similar to
what has been done in Python 3.3 but for Python 2.7 with a corser-grain
lock. From my understanding of import.c, it should work but I was hoping
that someone with more experience in import code would confirm:

  Currently, I have a package, foo, registered into sys.meta_path which
  loads modules through its find_module() and load_module() methods
  (PEP302).

  Access to load_module() of this package is protected by a RLock I
  defined, so that modules within foo cannot be imported in
  parallel. Until the module is added to sys.modules and then the code
  loaded, release the import lock.

  For find_module(), only filter the full module name and if this is a
  module from foo package, then acquired the same RLock defined for
  load_module() to access variables shared with find_module().


Also, if a patch backporting the features from Python 3.3 to import.c
for Python 2.7 would be written, is there any chance it could be
accepted?

Regards,
-- 
Arnaud Fontaine
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks

2013-08-13 Thread Antoine Pitrou 
Le Tue, 13 Aug 2013 17:28:42 +0900,
Arnaud Fontaine  a écrit :
> Antoine Pitrou  writes:
> > On Tue, 13 Aug 2013 11:06:51 +0900 Arnaud Fontaine
> >  wrote:
> >> I suggested the same in my initial email, but I was wondering if
> >> there could be any issue by releasing the lock in
> >> find_module()/load_module() until the module is actually added to
> >> sys.modules.
> >
> > Well, you are obviously on your own with such hacks. There is a
> > reason the lock exists.
> 
> Yes. Actually, I was thinking about implementing something similar to
> what has been done in Python 3.3 but for Python 2.7 with a
> corser-grain lock. From my understanding of import.c, it should work
> but I was hoping that someone with more experience in import code
> would confirm:

It's probably possible, but it will be non-trivial and delicate to get
right.

> Also, if a patch backporting the features from Python 3.3 to import.c
> for Python 2.7 would be written, is there any chance it could be
> accepted?

Definitely not. We generally don't backport any features, especially
when the risk is high due to a feature's implementation complexity.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL issues in Python stdlib and 3rd party code

2013-08-13 Thread Terry Reedy 

On 8/13/2013 5:06 AM, Christian Heimes wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

CVE-2013-4238 has been signed to NULL bytes in subjectAltName issue.


assigned...


   http://bugs.python.org/issue18709
   http://www.openwall.com/lists/oss-security/2013/08/13/2

Should we assign a CVE to issue in ssl.match_hostname(), too? Even
more projects have copied our code (bzr, tornado, pip, setuptools):

   http://bugs.python.org/issue17997
   https://bugs.mageia.org/show_bug.cgi?id=10391
   https://bugzilla.redhat.com/show_bug.cgi?id=963260#c11


I personlly thought that the CVE people did the assigning, or are you 
talking about asking them? What are the implications of 'yes' versus 
'no'? If a number would get more attention, and you think that needed, 
do it.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Steven D'Aprano 

Hi,

I have raise a tracker item and PEP for adding a statistics module to the 
standard library:

http://bugs.python.org/issue18606

http://www.python.org/dev/peps/pep-0450/


and I'm about to submit a patch containing my updated code and tests, but I've 
run into a problem with testing. My existing tests use unittest, and follow the 
basic boilerplate documented here:

http://docs.python.org/3/library/test.html

and when run directly they pass, but when I try to run them using Python 3.4a 
-m test -j3 they break. My question is, is it acceptable to post the code and 
tests to the tracker as-is, and ask for a pronouncement on the PEP first, and 
then fix the test breakage later? If not, can somebody mentor me in 
understanding what I need to do here?

To avoid all doubt, the tests pass if I call them like this:


./python Lib/test/test_statistics.py


but raise errors when I call them like this:

./python -m test -j3


Thanks in advance,




--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Ethan Furman 

On 08/13/2013 04:51 PM, Steven D'Aprano wrote:


I have raise a tracker item and PEP for adding a statistics module to the 
standard library:

http://bugs.python.org/issue18606

http://www.python.org/dev/peps/pep-0450/


The bug-tracker doesn't think you've submitted a CLA yet.  If you haven't 
you'll need to get that done.

--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Ethan Furman 

On 08/13/2013 04:51 PM, Steven D'Aprano wrote:


My question is, is it acceptable to post the code and tests to
 the tracker as-is, and ask for a pronouncement on the PEP first,
and then fix the test breakage later?


Certainly.

--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Victor Stinner 
Send the patch somewhere (ex: attach it to an email, or to the bug
tracker, as you want), or give the error message, if you want some
help.

> Ask for a pronouncement on the PEP first, and then fix the test breakage 
> later?

Sometimes, it's possible to pronounce on a PEP without a working
implementation. But it's easier to pronounce with a working
implementation :-)

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Nick Coghlan 
On 13 Aug 2013 19:40, "Victor Stinner"  wrote:
>
> Send the patch somewhere (ex: attach it to an email, or to the bug
> tracker, as you want), or give the error message, if you want some
> help.
>
> > Ask for a pronouncement on the PEP first, and then fix the test
breakage later?
>
> Sometimes, it's possible to pronounce on a PEP without a working
> implementation. But it's easier to pronounce with a working
> implementation :-)

It's more typical for reference implementations to be "proof of concept"
quality code, though. It's very rare to have a full, ready to apply patch
(although we certainly don't complain if that happens!)

(To directly answer the original question: a test integration glitch isn't
a blocker for PEP acceptance, just for actually committing the
implementation)

Cheers,
Nick.

>
> Victor
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating the formatter module

2013-08-13 Thread Steven D'Aprano 

On 13/08/13 23:36, Brett Cannon wrote:

On Tue, Aug 13, 2013 at 6:34 AM, Serhiy Storchaka wrote:


12.08.13 22:22, Brett Cannon написав(ла):

  I have created 
http://bugs.python.org/**issue18716to 
deprecate the

formatter module for removal in Python 3.6 unless someone convinces me
otherwise that deprecation and removal is the wrong move.



The formatter module doesn't look such buggy as the audioop module. If the
formatter module was not removed in Python 3.0 I don't see why it can't
wait to 4.0.



It doesn't help that at PyCon CA we couldn't find a single person who had
heard of the module (which included people like Alex Gaynor and Larry
Hastings).



You know, there may be one or two Python programmers who didn't go to PyCon 
CA... :-)

I knew of formatter before this thread. I've looked at it for use in my own code, but the 
lack of useful documentation or clear examples put me off, so I put it in the pile of 
"things to investigate later", but it is definitely a module that interests me.

The documentation for 2.7 claims that it is used by HTMLParser, but that 
doesn't appear to be true. The claim is removed from the 3.3 documentation.

Over on Python-Ideas mailing list, there are periodic frenzies of requests to 
delete all sorts of things, e.g. a recent thread debating deleting various 
string methods in favour of using {} string formatting. My answer there is the 
same as my answer here: unless the formatter module is actively harmful, then 
deprecation risks causing more pain than benefit. All those thousands of coders 
who don't use formatter? The benefit to them of deprecating it is next to zero. 
That one guy in Uberwald you've never heard of who does use it? It's going to 
cause him a lot of pain.


--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Terry Reedy 

On 8/13/2013 7:51 PM, Steven D'Aprano wrote:


http://bugs.python.org/issue18606


Tests at end of statistics.patch.


and I'm about to submit a patch containing my updated code and tests,
but I've run into a problem with testing. My existing tests use
unittest, and follow the basic boilerplate documented here:

http://docs.python.org/3/library/test.html


+def test_main():
+# run_unittest()
+unittest.main()
+
+
+if __name__ == "__main__":
+test_main()

This is faulty, as explained below. It is not the boilerplate in
http://docs.python.org/3/library/test.html#writing-unit-tests-for-the-test-package

which is correct. Compress to just

if __name__ == "__main__":
   unittest_main()


To avoid all doubt, the tests pass if I call them like this:
./python Lib/test/test_statistics.py


That is a faulty test as explained below.

Patch applies cleanly on Win7, 32bit build. And indeed,
F:\Python\dev\py34\PCbuild>python_d ../Lib/test/test_statistics.py
works. (the progess dots will have to go before being applied ;-).

The above should be equivalent to
...> python -m test.test_statistics.
but it is not.

F:\Python\dev\py34\PCbuild>python_d -m test.test_statistics
Traceback (most recent call last):
  File "F:\Python\dev\py34\lib\runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "F:\Python\dev\py34\lib\runpy.py", line 73, in _run_code
exec(code, run_globals)
  File "F:\Python\dev\py34\lib\test\test_statistics.py", line 16, in 


from test_statistics_approx import NumericTestCase
ImportError: No module named 'test_statistics_approx'

Your test *depends* on Lib/test being the current directory, at the 
beginning of the path. It should not. Your import must be

   from test.test_statistics_appox import NumericTestCase

With this fixed, the above command works.

Once the test runs from anywhere with unittest, run with regrtest,

...> python -m test test_statistics

Initially, this gives a traceback with the same last three lines. With 
the revision, I get


  File "F:\Python\dev\py34\lib\unittest\loader.py", line 113, in 
loadTestsFromName

parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_statistics'

This is related to the first fault I mentioned above. If regrtest finds 
the old style 'test_main', it uses the regrtest loader. If it does not, 
it uses the unittest loader. This is documented in the code ;-). With 
the proper 2-line boilerplate as a second change, the second line now works.



but raise errors when I call them like this:
./python -m test -j3


LOL. In one jump, you changed the current directory, the test runner, 
the number of test files run, and the mode of testing. Try just one 
change at a time.


When you run the suite, you run test_statistics_approx.py, including the 
test case imported into test_statistics. (Is it run twice?).


Tested as part of the suite gives a bizarre message I do not pretend to 
understand.

[290/379/3] test_statistics_approx
Usage: regrtest.py [options]

regrtest.py: error: no such option: --slaveargs
test test_statistics_approx crashed -- Traceback (most recent call last):
  File "F:\Python\dev\py34\lib\optparse.py", line 1391, in parse_args
stop = self._process_args(largs, rargs, values)
  File "F:\Python\dev\py34\lib\optparse.py", line 1431, in _process_args
self._process_long_opt(rargs, values)
  File "F:\Python\dev\py34\lib\optparse.py", line 1484, in 
_process_long_opt

opt = self._match_long_opt(opt)
  File "F:\Python\dev\py34\lib\optparse.py", line 1469, in _match_long_opt
return _match_abbrev(opt, self._long_opt)
  File "F:\Python\dev\py34\lib\optparse.py", line 1674, in _match_abbrev
raise BadOptionError(s)
optparse.BadOptionError: no such option: --slaveargs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\Python\dev\py34\lib\test\regrtest.py", line 1305, in 
runtest_inner

test_runner()
  File "F:\Python\dev\py34\lib\test\test_statistics_approx.py", line 
597, in test_main

unittest.main()
  File "F:\Python\dev\py34\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
  File "F:\Python\dev\py34\lib\unittest\main.py", line 148, in parseArgs
options, args = parser.parse_args(argv[1:])
  File "F:\Python\dev\py34\lib\optparse.py", line 1393, in parse_args
self.error(str(err))
  File "F:\Python\dev\py34\lib\optparse.py", line 1573, in error
self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
  File "F:\Python\dev\py34\lib\optparse.py", line 1563, in exit
sys.exit(status)
SystemExit: 2

So back up and test it by itself first. It passes under unittest, but 
needs the revised boilerplate for regrtest. The two together work fine. 
When I rerun the suite, poof!, the mysterious tracback is gone and both 
pass.


The following appears after the 2nd. '''
..
--
Ran 2 t