Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!

2013-10-26 Thread Kristján Valur Jónsson 
In that case, how about adding a client/server feature?
If you standardize the format, a minimal tracing client could write a log, or 
send it to a socket, in a way that can be turned into a snapshot by a 
corresponsing utility reading from a file or listenting to a socket.
Just a though.  Could be a future addition…

From: Python-Dev [mailto:[email protected]] 
On Behalf Of Victor Stinner
Sent: 24. október 2013 16:45
To: python-dev
Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics!


> When I was looking for memory leaks in the regex module I simply wrote all of 
> the allocations, reallocations and deallocations to a log file and then 
> parsed it afterwards using a Python script. Simple, but effective.

He he, it's funny because you described exactly my first implementation of 
tracemalloc! I wrote output using fprintf() into a text file. The parser was 
written in Python but too slow to be used it in practice. When running on the 
slow set top box, it took minutes (5 maybe 10) to analyze the file. Transfering 
the file to a PC took also minutes: the file was very large (maybe 1 GB, I 
don't remember) and SimpleHTTPServer too slow for the transfer.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -)

2013-10-26 Thread A.M. Kuchling 
On Sat, Oct 26, 2013 at 05:34:05AM +0200, tim.peters wrote:
>   Fiddled Thread.join() to be a little simpler.  Kinda ;-)
> -# else it's a negative timeout - precise behavior isn't documented
> -# then, but historically .join() returned in this case
> +else:
> +# the behavior of a negative timeout isn't documented, but
> +# historically .join() has acted as if timeout=0 then

I think the final 'then' on the new comment is an editing error.
Unless perhaps you meant to write a further sentence of comment?

--amk
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Stefan Krah 
nick.coghlan  wrote:
> http://hg.python.org/cpython/rev/a9bbc2d0c1dc
> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
> -   sys.platform == 'win32' or
> -   sysconfig.get_config_var('WITH_DOC_STRINGS'))
> +# Rather than trying to enumerate all the cases where docstrings may be
> +# disabled, we just check for that directly
> +
> +def _check_docstrings():
> +"""Just used to check if docstrings are enabled"""
> +
> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)
>  
>  requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,

I think that does not detect --without-doc-strings (i.e. the C docstrings are
empty).


Stefan Krah



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pip SSL

2013-10-26 Thread Donald Stufft 
This pull request should solve this https://github.com/pypa/pip/pull/1256

On Oct 20, 2013, at 12:32 AM, Nick Coghlan  wrote:

> On 20 October 2013 05:46, Ian Cordasco  wrote:
>> Also the three of us maintaining requests and the author of urllib3
>> are all very conscious that the packaged pem file is outdated. We have
>> an open issue about how to rebuild it accurately while taking into
>> consideration (and not including) the ones that have been revoked. Any
>> suggestions you have can be sent to me off list or reported on the
>> issue tracker.
> 
> The requests issue Ian is referring to:
> https://github.com/kennethreitz/requests/issues/1659
> 
> The next version of PEP 453 will include getting this resolved as part
> of the integration timeline:
> 
> 
> * by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2)
> 
>  ``requests`` certificate management issue resolved
>  ``ensurepip`` updated to the final release of pip 1.5, or a subsequent
>  maintenance release (including a suitably updated vendored copy of
>  ``requests``)
> 
> 
> And also mentions it under the "security considerations" section for
> the bootstrapping mechanism:
> 
> 
> Only users that choose to use ``pip`` to communicate with PyPI will
> need to pay attention to the additional security considerations that come
> with doing so.
> 
> However, the core CPython team will also assist with reviewing and
> resolving the `certificate update management issue
> `__ currently
> affecting the ``requests`` project (and hence ``pip``).
> 
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Serhiy Storchaka 

26.10.13 15:50, Stefan Krah написав(ла):

nick.coghlan  wrote:

http://hg.python.org/cpython/rev/a9bbc2d0c1dc
-HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
-   sys.platform == 'win32' or
-   sysconfig.get_config_var('WITH_DOC_STRINGS'))
+# Rather than trying to enumerate all the cases where docstrings may be
+# disabled, we just check for that directly
+
+def _check_docstrings():
+"""Just used to check if docstrings are enabled"""
+
+HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)

  requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,


I think that does not detect --without-doc-strings (i.e. the C docstrings are
empty).


Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C 
docstrings. Python docstrings tests are skipped if sys.flags.optimize >= 2.



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -)

2013-10-26 Thread Tim Peters 
On Sat, Oct 26, 2013 at 6:29 AM, A.M. Kuchling  wrote:
> On Sat, Oct 26, 2013 at 05:34:05AM +0200, tim.peters wrote:
>>   Fiddled Thread.join() to be a little simpler.  Kinda ;-)
>> -# else it's a negative timeout - precise behavior isn't documented
>> -# then, but historically .join() returned in this case
>> +else:
>> +# the behavior of a negative timeout isn't documented, but
>> +# historically .join() has acted as if timeout=0 then
>
> I think the final 'then' on the new comment is an editing error.
> Unless perhaps you meant to write a further sentence of comment?

Reads OK to me.  Read "then" as an abbreviation of "in that case"?
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -)

2013-10-26 Thread Glenn Linderman 

On 10/26/2013 8:42 AM, Tim Peters wrote:

Read "then" as an abbreviation of "in that case"?
In that case, then, "then" is a very poor abbreviation of "in that 
case", and is unlikely to be interpreted with that expansion by many 
readers.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Nick Coghlan 
On 26 October 2013 08:51, PJ Eby  wrote:
> Mostly, this just seems like an ugly wart -- Python should be dynamic
> by default, and that includes reloading.  While the import machinery
> has lots of ugly caching under the hood, a user-level function like
> reload() should not require you to do the equivalent of saying, "no,
> really...  I want you to *really* reload, not just pull in whatever
> exists where you found it last time, while ignoring whether I switched
> from module to package or vice versa, or just fixed my sys.path so I
> can load the right version of the module."
>
> It is a really tiny thing in the overall scheme of things, because
> reload() is not used all that often, but it's still a thing.  If this
> isn't treated as a bug, then the docs for reload() at least need to
> include a forward-supported workaround so you can say "no, really...
> *really* reload" in an approved fashion.
>
> (ISTM that any production code out there that currently uses reload()
> would want to perform the "really reload" incantation in order to
> avoid the edge cases, even if they haven't actually run into any of
> them yet.)

Having imp.reload() kick off the full reload cycle makes sense to me.
The only reason it doesn't currently do that in importlib is because
there was no test for it in the regression test suite, and PEP 302 and
the library reference are entirely vague on how module reloading works
(aside from the requirement that it reuse the same module namespace).

>> And in a PEP 451 world it should be dead-simple to make this work the way
>> you want in your own code even if this doesn't go the way you want::
>>
>>   spec = importlib.find_spec(name)
>>   module.__spec__ = spec
>>   importlib.reload(module)  # Which in itself is essentially
>> init_module_attrs(spec, module); spec.loader.exec_module(module)
>>
>> Heck, you can do this in Python 3.3 right now::
>>
>>   loader = importlib.find_loader(name)
>>   module = sys.modules[name]
>>   module.__loader__ = loader
>>   importlib.reload(module)
>
> And will that later version still work correctly in a PEP 451 world,
> or will you have to detect which world you live in before waving this
> particular dead chicken?  ;-)

Yep, we already tweaked PEP 451 to say that it has to respect
post-import modifications made to the module attributes.

>> Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance
>> in order to support reloading by the loader, not an explicit finder + loader
>> use-case.
>
> Yeah, it actually was to ensure that you could reload a module using a
> different loader than the one that originally loaded it, e.g. due to a
> change in path hooks, etc.

Yeah, the rationale makes sense, we only missed it due to the lack of
a regression test for the behaviour.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Nick Coghlan 
On 27 October 2013 01:10, Serhiy Storchaka  wrote:
> 26.10.13 15:50, Stefan Krah написав(ла):
>
>> nick.coghlan  wrote:
>>>
>>> http://hg.python.org/cpython/rev/a9bbc2d0c1dc
>>> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
>>> -   sys.platform == 'win32' or
>>> -   sysconfig.get_config_var('WITH_DOC_STRINGS'))
>>> +# Rather than trying to enumerate all the cases where docstrings may be
>>> +# disabled, we just check for that directly
>>> +
>>> +def _check_docstrings():
>>> +"""Just used to check if docstrings are enabled"""
>>> +
>>> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)
>>>
>>>   requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,
>>
>>
>> I think that does not detect --without-doc-strings (i.e. the C docstrings
>> are
>> empty).
>
>
> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings.
> Python docstrings tests are skipped if sys.flags.optimize >= 2.

That's *extraordinarily* confusing, especially when Serhiy suggested I
use the flag when testing a pure Python module.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Nick Coghlan 
On 27 October 2013 03:32, Nick Coghlan  wrote:
> On 27 October 2013 01:10, Serhiy Storchaka  wrote:
>> 26.10.13 15:50, Stefan Krah написав(ла):
>>
>>> nick.coghlan  wrote:

 http://hg.python.org/cpython/rev/a9bbc2d0c1dc
 -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
 -   sys.platform == 'win32' or
 -   sysconfig.get_config_var('WITH_DOC_STRINGS'))
 +# Rather than trying to enumerate all the cases where docstrings may be
 +# disabled, we just check for that directly
 +
 +def _check_docstrings():
 +"""Just used to check if docstrings are enabled"""
 +
 +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)

   requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,
>>>
>>>
>>> I think that does not detect --without-doc-strings (i.e. the C docstrings
>>> are
>>> empty).
>>
>>
>> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings.
>> Python docstrings tests are skipped if sys.flags.optimize >= 2.
>
> That's *extraordinarily* confusing, especially when Serhiy suggested I
> use the flag when testing a pure Python module.

http://bugs.python.org/issue19412

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Serhiy Storchaka 

26.10.13 20:32, Nick Coghlan написав(ла):

On 27 October 2013 01:10, Serhiy Storchaka  wrote:

26.10.13 15:50, Stefan Krah написав(ла):


nick.coghlan  wrote:


http://hg.python.org/cpython/rev/a9bbc2d0c1dc
-HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
-   sys.platform == 'win32' or
-   sysconfig.get_config_var('WITH_DOC_STRINGS'))
+# Rather than trying to enumerate all the cases where docstrings may be
+# disabled, we just check for that directly
+
+def _check_docstrings():
+"""Just used to check if docstrings are enabled"""
+
+HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)

   requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,



I think that does not detect --without-doc-strings (i.e. the C docstrings
are
empty).



Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings.
Python docstrings tests are skipped if sys.flags.optimize >= 2.


That's *extraordinarily* confusing, especially when Serhiy suggested I
use the flag when testing a pure Python module.


I'm sorry for misleading you.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): 2.7.6rc1

2013-10-26 Thread Christian Heimes 
Am 26.10.2013 20:58, schrieb benjamin.peterson:
> http://hg.python.org/cpython/rev/4913d0e9be30
> changeset:   86665:4913d0e9be30
> branch:  2.7
> tag: v2.7.6rc1
> user:Benjamin Peterson 
> date:Sat Oct 26 14:57:21 2013 -0400
> summary:
>   2.7.6rc1

We need to solve http://bugs.python.org/issue19227 before you release
2.7.6. I have suggested two approaches
http://bugs.python.org/issue19227#msg201401

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Eric Snow 
On Fri, Oct 25, 2013 at 9:44 AM, Eric Snow  wrote:
> On Fri, Oct 25, 2013 at 9:40 AM, Brett Cannon  wrote:
>> After reading Eric's doc patch, I realized there is one change I want to
>> make to the current semantics and that's not to backfill __package__ when
>> set to None. Since import is now going to take over the job of setting
>> __package__ (along with other attributes), this seems like a slight waste of
>> effort. It also kills (or at least complicates) having a loader which does
>> lazy loading since reading the attribute to see if it is None would trigger
>> the load before leaving the import code, thus killing any postponed loading.
>
> Fine with me.  I may have time today to make the outstanding updates to the 
> PEP.

The changes are in now, as is an update to the doc patch on
issue18864.  If I missed anything, let me know.  Thanks for the
feedback!

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests

2013-10-26 Thread Nick Coghlan 
On 27 Oct 2013 04:16, "Serhiy Storchaka"  wrote:
>
> 26.10.13 20:32, Nick Coghlan написав(ла):
>
>> On 27 October 2013 01:10, Serhiy Storchaka  wrote:
>>>
>>> 26.10.13 15:50, Stefan Krah написав(ла):
>>>
 nick.coghlan  wrote:
>
>
> http://hg.python.org/cpython/rev/a9bbc2d0c1dc
> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
> -   sys.platform == 'win32' or
> -   sysconfig.get_config_var('WITH_DOC_STRINGS'))
> +# Rather than trying to enumerate all the cases where docstrings may
be
> +# disabled, we just check for that directly
> +
> +def _check_docstrings():
> +"""Just used to check if docstrings are enabled"""
> +
> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)
>
>requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,



 I think that does not detect --without-doc-strings (i.e. the C
docstrings
 are
 empty).
>>>
>>>
>>>
>>> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C
docstrings.
>>> Python docstrings tests are skipped if sys.flags.optimize >= 2.
>>
>>
>> That's *extraordinarily* confusing, especially when Serhiy suggested I
>> use the flag when testing a pure Python module.
>
>
> I'm sorry for misleading you.

And my apologies for going ahead and assuming I understood what the flag
was for rather than seeking clarification.

I'll put together a patch to split it into two flags for the different
meanings.

Cheers,
Nick.
>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OrderedDict.values() behavior for modified instance

2013-10-26 Thread Ethan Furman 

Nikolaus,

Good write-up.  Please submit it to the bug tracker: http://bugs.python.org

--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OrderedDict.values() behavior for modified instance

2013-10-26 Thread Nikolaus Rath 
Ethan Furman  writes:
> Nikolaus,
>
> Good write-up.  Please submit it to the bug tracker:
> http://bugs.python.org

Submitted as http://bugs.python.org/issue19414.

If someone gives me the go-ahead for one of the proposed solutions, I'd
be happy to create a full patch.

Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Eric Snow 
On Sat, Oct 26, 2013 at 11:27 AM, Nick Coghlan  wrote:
> On 26 October 2013 08:51, PJ Eby  wrote:
>> Yeah, it actually was to ensure that you could reload a module using a
>> different loader than the one that originally loaded it, e.g. due to a
>> change in path hooks, etc.
>
> Yeah, the rationale makes sense, we only missed it due to the lack of
> a regression test for the behaviour.

I put up a patch that should fix this without a lot of work, including
a test that would have caught this.

http://bugs.python.org/issue19413

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Nick Coghlan 
On 27 October 2013 03:27, Nick Coghlan  wrote:
> On 26 October 2013 08:51, PJ Eby  wrote:
>> Mostly, this just seems like an ugly wart -- Python should be dynamic
>> by default, and that includes reloading.  While the import machinery
>> has lots of ugly caching under the hood, a user-level function like
>> reload() should not require you to do the equivalent of saying, "no,
>> really...  I want you to *really* reload, not just pull in whatever
>> exists where you found it last time, while ignoring whether I switched
>> from module to package or vice versa, or just fixed my sys.path so I
>> can load the right version of the module."
>>
>> It is a really tiny thing in the overall scheme of things, because
>> reload() is not used all that often, but it's still a thing.  If this
>> isn't treated as a bug, then the docs for reload() at least need to
>> include a forward-supported workaround so you can say "no, really...
>> *really* reload" in an approved fashion.
>>
>> (ISTM that any production code out there that currently uses reload()
>> would want to perform the "really reload" incantation in order to
>> avoid the edge cases, even if they haven't actually run into any of
>> them yet.)
>
> Having imp.reload() kick off the full reload cycle makes sense to me.
> The only reason it doesn't currently do that in importlib is because
> there was no test for it in the regression test suite, and PEP 302 and
> the library reference are entirely vague on how module reloading works
> (aside from the requirement that it reuse the same module namespace).

I've been thinking about this, and I've come to the conclusion that:

1. The old __name__ based behaviour was broken (since __name__ may not
be the actual module name, as in "__main__" or when a pseudo-module is
lying about its identity)
2. The 3.3 behaviour is still broken for similar reasons *and* added
the dependency on __loader__
3. Even PEP 451 still underspecifies the semantics of reloading (e.g.
it's implied create_module isn't called for reload(), but not stated
explicitly)

Accordingly, I think we should add a "How Reloading Will Work"
section, akin to the existing "How Loading Will Work"
(http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We
may also want to spit out some import warnings for edge cases that are
likely to do the wrong thing.

Side note: in "How Loading Will Work", _init_module_attrs needs to be
passed both the module *and* the spec (current text only passes it the
module). It also needs to be updated to handle the namespace package
case (as shown below)

And a sketch of some possible fully specified reload semantics:

def reload(module):
# Get the name to reload from the spec if it is available,
otherwise __name__
current_spec = getattr(module, "__spec__")
current_name = module.__name__
name_to_reload = current_name if current_spec is None else
current_spec.name
# Check this module is properly imported before trying to reload it
loaded_module = sys.modules.get(name_to_reload)
if loaded_module is not module:
msg = "module {} not in sys.modules"
raise ImportError(msg.format(name_to_reload), name=name_to_reload)
parent_name = name_to_reload.rpartition('.')[0]
while parent_name:
if parent_name not in sys.modules:
msg = "parent {!r} not in sys.modules"
raise ImportError(msg.format(parent_name), name=parent_name)
parent_name = parent_name.rpartition('.')[0]
# Check for a modified spec (e.g. if the import hooks have changed)
updated_spec = importlib.find_spec(name_to_reload)
# The import-related module attributes get updated here
# We leave __name__ alone, so reloading __main__ has some hope
of working correctly
_init_module_attrs(loaded_module, updated_spec, set_name=False)

# Now re-execute the module
loader = updated_spec.loader
if loader is None:
# Namespace package, already reinitialised above
return loaded_module
elif hasattr(loader, 'exec_module'):
loader.exec_module(module)
else:
loader.load_module(name_to_reload)

# Reloading modules that replace themselves in sys.modules doesn't work
reloaded_module = sys.modules[name_to_reload]
if reloaded_module is not loaded_module:
# Perhaps this should initially be a DeprecationWarning?
# We've never actively *prevented* this case before, even
though it is
# almost guaranteed to break things if you do it.
Alternatively, we could
# leave "imp.reload()" as the more permissive version, and
include this
# check in the new importlib.reload() (with imp.reload
calling an internal
# importlib._unchecked_reload() instead)
msg = "Module {!r} was replaced in sys.modules during reload"
raise ImportError(

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Eric Snow 
On Sat, Oct 26, 2013 at 9:44 PM, Nick Coghlan  wrote:
> I've been thinking about this, and I've come to the conclusion that:
>
> 1. The old __name__ based behaviour was broken (since __name__ may not
> be the actual module name, as in "__main__" or when a pseudo-module is
> lying about its identity)
> 2. The 3.3 behaviour is still broken for similar reasons *and* added
> the dependency on __loader__
> 3. Even PEP 451 still underspecifies the semantics of reloading (e.g.
> it's implied create_module isn't called for reload(), but not stated
> explicitly)
>
> Accordingly, I think we should add a "How Reloading Will Work"
> section, akin to the existing "How Loading Will Work"
> (http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We
> may also want to spit out some import warnings for edge cases that are
> likely to do the wrong thing.

I'm tempted to just say reload should not be a blocker for PEP 451.
The premise is that the PEP mostly maintains the status quo, just
shifting reload-based-on-loader to reload-based-on-spec (it's still
"*load* the same way you did the first time").

I agree that it would be worth getting reload working right, but I'm
not convinced it's worth it to delay PEP 451 more for the sake of
reload.

>
> Side note: in "How Loading Will Work", _init_module_attrs needs to be
> passed both the module *and* the spec (current text only passes it the
> module).

Good catch.  Fixed!

> It also needs to be updated to handle the namespace package
> case (as shown below)

I knew I missed something.  I'll fix this, but it may be a couple days
before I get a minute. :(

-eric
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451 update

2013-10-26 Thread Nick Coghlan 
On 27 October 2013 14:29, Eric Snow  wrote:
> On Sat, Oct 26, 2013 at 9:44 PM, Nick Coghlan  wrote:
>> Accordingly, I think we should add a "How Reloading Will Work"
>> section, akin to the existing "How Loading Will Work"
>> (http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We
>> may also want to spit out some import warnings for edge cases that are
>> likely to do the wrong thing.
>
> I'm tempted to just say reload should not be a blocker for PEP 451.
> The premise is that the PEP mostly maintains the status quo, just
> shifting reload-based-on-loader to reload-based-on-spec (it's still
> "*load* the same way you did the first time").
>
> I agree that it would be worth getting reload working right, but I'm
> not convinced it's worth it to delay PEP 451 more for the sake of
> reload.

I don't think we can postpone it to later, as we need to be clear on:

1. Does reload use __name__ or __spec__.name when both are available?
2. Does __name__ get restored to its original value if reloading via
__spec__.name?
3. Do other import related attributes get restored to their original values?
4. Does create_module get called if the loader has an exec_module method?
5. Does the loader have access to the previous spec when reloading a module?

My answers to these questions (note that my answer to 2 differs from
what I had in my initial sketch):

1. __spec__.name
2. Yes, __name__ is updated to match __spec__.name, *expect* if it is
currently "__main__"
3. Yes, other import related attributes are restored to their baseline values
4. No, create_module is never called when reloading
5. Currently no, but I'm thinking that may be worth changing (more on
that below)

The reload() implementation in my message is actually based on the
current importlib.reload implementation. The only PEP 451 related
changes were:

- using __spec__.name (if available) instead of __name__
- checking all parent modules exist rather than just the immediate parent module
- calling import.find_spec() rather than using the __loader__ attribute directly
- being explicit that __name__ is left at the value it had prior to the reload
- handling the namespace package, exec_module and no exec_module cases

I also added an explicit check that the module was re-used properly,
but I agree *that* change is out of scope for the PEP and should be
considered as a separate question.

Now, regarding the signature of exec_module(): I'm back to believing
that loaders should receive a clear indication that a reload is taking
place. Legacy loaders have to figure that out for themselves (by
seeing that the module already exists in sys.modules), but we can do
better for the new API by making the exec_module signature look like:

def exec_module(self, module, previous_spec=None):
# module is as per the current PEP 451 text
# previous_spec would be set *only* in the reload() case
# loaders that don't care still need to accept it, but can
just ignore it
...

So, with those revisions, the reload() semantics would be defined as:

def reload(module):
# Get the name to reload from the spec if it is available,
# otherwise use __name__ directly
current_spec = getattr(module, "__spec__")
current_name = module.__name__
name_to_reload = current_name if current_spec is None else
current_spec.name

# Check this module is properly imported before trying to reload it
loaded_module = sys.modules.get(name_to_reload)
if loaded_module is not module:
msg = "module {} not in sys.modules"
raise ImportError(msg.format(name_to_reload), name=name_to_reload)
parent_name = name_to_reload.rpartition('.')[0]
while parent_name:
if parent_name not in sys.modules:
msg = "parent {!r} not in sys.modules"
raise ImportError(msg.format(parent_name), name=parent_name)
parent_name = parent_name.rpartition('.')[0]

# Check for a modified spec (e.g. if the import hooks have changed)
updated_spec = importlib.find_spec(name_to_reload)

# The import-related module attributes get updated here
_init_module_attrs(loaded_module, updated_spec)
if current_name == "__main__":
loaded_module.__name__ = "__main__"

# Now re-execute the module
loader = updated_spec.loader
if loader is None:
# Namespace package, already reinitialised above
return loaded_module
elif hasattr(loader, 'exec_module'):
loader.exec_module(module, current_spec)
else:
loader.load_module(name_to_reload)

# Allow for the module to be replaced in sys.modules
# (even though that likely won't work very well)
return sys.modules[name_to_reload]

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev ma