Re: [Python-Dev] IO module improvements

2010-02-06 Thread Pascal Chambon 

Antoine Pitrou a écrit :
  
What is the difference between "file handle" and a regular C file descriptor?

Is it some Windows-specific thing?
If so, then perhaps it deserves some Windows-specific attribute ("handle"?).
  
At the moment it's windows-specific, but it's not impossible that some 
other OSes also rely on specific file handles (only emulating C file 
descriptors for compatibility).
I've indeed mirrored the fileno concept, with a "handle" argument for 
constructors, and a handle() getter.




On Fri, Feb 5, 2010 at 5:28 AM, Antoine Pitrou  wrote:
  

Pascal Chambon  gmail.com> writes:


By the way, I'm having trouble with the "name" attribute of raw files,
which can be string or integer (confusing), ambiguous if containing a
relative path,
  


Why is it ambiguous? It sounds like you're using str() of the name and
then can't tell whether the file is named e.g. '1' or whether it
refers to file descriptor 1 (i.e. sys.stdout).

  
As Jean-Paul mentioned, I find confusing the fact that it can be a 
relative path, and sometimes not a path at all. I'm pretty sure many 
programmers haven't even cared in their library code that it could be a 
non-string, using concatenation etc. on it...
However I guess that the history is so high on it, that I'll have to 
conform to this semantic, putting all paths/fileno/handle in the same 
"name" property, and adding an "origin" property telling how to 
interpret the "name"...


Methods too would deserve some auto-forwarding. If you want to bufferize 
a raw stream which also offers size(), times(), lock_file() and other 
methods, how can these be accessed from a top-level buffering/text 
stream ?



I think it's a bad idea. If you forget to implement one of the standard IO
methods (e.g. seek()), it will get forwarded to the raw stream, but with the
wrong semantics (because it won't take buffering into account).

It's better to require the implementor to do the forwarding explicitly if
desired, IMO.
  
The problem is, doing that forwarding is quite complicated. IO is a 
collection of "core tools for working with streams", but it's currently 
not flexible enough to let people customize them too...
For example, if I want to add a new series of methods to all standard 
streams, which simply forward calls to new raw stream features, what do 
I do ? Monkey-patching base classes (RawFileIO, BufferedIOBase...) ? Not 
a good pattern. Subclassing 
FileIO+BufferedWriter+BufferredReader+BufferedRandom+TextIOWrapper ? 
That's really redundant...


And there are sepecially flaws around BufferedRandom. This stream 
inherits BufferedWriter and BufferedRandom, and overrides some methods. 
How do I do to extend it ? I'd want to reuse its methods, but then have 
it forward calls to MY buffered classes, not original BufferedWriter or 
BufferredReader classes. Should I modify its __bases__ to edit the 
inheritance tree ? Handy but not a good pattern... I'm currently getting 
what I want with a triple inheritance (praying for the MRO to be as I 
expect), but it's really not straightforward.
Having BufferedRandom as an additional layer would slow down the system, 
but allow its reuse with custom buffered writers and readers...


- I feel thread-safety locking and stream stream status checking are 
currently overly complicated. All methods are filled with locking calls 
and CheckClosed() calls, which is both a performance loss (most io 
streams will have 3 such levels of locking, when 1 would suffice)



FileIO objects don't have a lock, so there are 2 levels of locking at worse, not
3 (and, actually, TextIOWrapper doesn't have a lock either, although perhaps it
should).
As for the checkClosed() calls, they are probably cheap, especially if they
bypass regular attribute lookup.
  
CheckClosed calls are cheap, but they can easily be forgotten in one of 
the dozens of methods involved...
My own FileIO class alas needs locking, because for example, on windows 
truncating a file means seeking + setting end of file + restoring pointer.
And I TextIOWrapper seems to deserve locks. Maybe excerpts like this one 
really are thread-safe, but a long study would be required to ensure it.


  if whence == 2: # seek relative to end of file
   if cookie != 0:
   raise IOError("can't do nonzero end-relative seeks")
   self.flush()
   position = self.buffer.seek(0, 2)
   self._set_decoded_chars('')
   self._snapshot = None
   if self._decoder:
   self._decoder.reset()
   return position

  
Since we're anyway in a mood of imbricating streams, why not simply 
adding a "safety stream" on top of each stream chain returned by open() 
? That layer could gracefully handle mutex locking, CheckClosed() calls, 
and even, maybe, the attribute/method forwarding I evocated above.



It's an interesting idea, but it could also end up slower than the current
situation.
First because you are adding a level of indirection

Re: [Python-Dev] IO module improvements

2010-02-06 Thread David Cournapeau 
On Fri, Feb 5, 2010 at 10:28 PM, Antoine Pitrou  wrote:
> Pascal Chambon  gmail.com> writes:
>>
>> By the way, I'm having trouble with the "name" attribute of raw files,
>> which can be string or integer (confusing), ambiguous if containing a
>> relative path, and which isn't able to handle the new case of my
>> library, i.e opening a file from an existing file handle (which is ALSO
>> an integer, like C file descriptors...)
>
> What is the difference between "file handle" and a regular C file descriptor?
> Is it some Windows-specific thing?
> If so, then perhaps it deserves some Windows-specific attribute ("handle"?).

When wondering about the same issue, I found the following useful:

http://www.codeproject.com/KB/files/handles.aspx

The C library file descriptor as returned by C open is emulated by
win32. Only HANDLE is considered "native" (can be passed freely
however you want within one process).

cheers,

David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASED] Python 2.7 alpha 3

2010-02-06 Thread Benjamin Peterson 
On behalf of the Python development team, I'm cheerful to announce the third
alpha release of Python 2.7.

Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version
in the 2.x series.  Though more major releases have not been absolutely ruled
out, it's likely that the 2.7 release will an extended period of maintenance for
the 2.x series.

2.7 includes many features that were first released in Python 3.1.  The faster
io module, the new nested with statement syntax, improved float repr, set
literals, dictionary views, and the memoryview object have been backported from
3.1. Other features include an ordered dictionary implementation, unittests
improvements, a new sysconfig module, and support for ttk Tile in Tkinter.  For
a more extensive list of changes in 2.7, see
http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python
distribution.

To download Python 2.7 visit:

 http://www.python.org/download/releases/2.7/

Please note that this is a development release, intended as a preview of new
features for the community, and is thus not suitable for production use.

The 2.7 documentation can be found at:

 http://docs.python.org/2.7

Please consider trying Python 2.7 with your code and reporting any bugs you may
notice to:

 http://bugs.python.org


Enjoy!

--
Benjamin Peterson
2.7 Release Manager
benjamin at python.org
(on behalf of the entire python-dev team and 2.7's contributors)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote:

>Can you clarify? In Python 3, __file__ always points to the source.
>Clearly that is the way of the future. For 99.99% of uses of __file__,
>if it suddenly never pointed to a .pyc file any more (even if one
>existed) that would be just fine. So what's this talk of switching to
>__source__?

Upon further reflection, I agree.  __file__ also points to the source in
Python 2.7.  Do we need an attribute to point to the compiled bytecode file?

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 05, 2010, at 07:37 PM, Nick Coghlan wrote:

>Brett Cannon wrote:
>> Does code exist out there where people are constructing bytecode from
>>  multiple files for a single module?
>
>I'm quite prepared to call YAGNI on that idea and just return a 2-tuple
>of source filename and compiled filename.

Me too.  I think a 2-tuple of (source-path, compiled-path) is probably going
to be fine for all practical purposes.  I'd assign the former to a module's
__file__ (as is done today in Python >= 2.7) and the latter to a module's
__cached__.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote:

>How about using an optionally relative cache dir setting to let
>the user decide ?

Why do we need that level of flexibility?

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote:

>It's also the case that having to run Python to manage my own filesystem
>would very annoying. If a dev has a broken .pyc that prevents the
>affected Python build from even starting how are they meant to use the
>nonfunctioning interpreter to find and delete the offending file? How is
>someone meant to find and delete the .pyc files if they prefer to use a
>graphical file manager over (or in conjunction with) the command line?

I agree.  I'd prefer to have a predictable place for the cached files,
independent of having to run Python to tell you where that is.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 03, 2010, at 09:26 AM, Floris Bruynooghe wrote:

>On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote:
>> I don't understand the distinction you're making between those two
>> options. Can you explain what you mean by each of “siblings” and
>> “folder-per-folder”?
>
>sibilings: the original proposal, i.e.:
>
>foo.py
>foo.pyr/
>MAGIC1.pyc
>MAGIC1.pyo
>...
>bar.py
>bar.pyr/
>MAGIC1.pyc
>MAGIC1.pyo
>...
>
>folder-per-folder:
>
>foo.py
>bar.py
>__pyr__/
>foo.MAGIC1.pyc
>foo.MAGIC1.pyo
>foo.MAGIC2.pyc
>bar.MAGIC1.pyc
>...
>
>IIUC

Correct.  If necessary, I'll define those two terms in the PEP.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Ezio Melotti 
In #7712 I was trying to change regrtest to always run the tests in a 
temporary CWD (e.g. /tmp/@test_1234_cwd/).
The patches attached to the issue add a context manager that changes the 
CWD, and it works fine when I run ./python -m test.regrtest from trunk/. 
However, when I try from trunk/Lib/ it fails with ImportErrors (note 
that the latest patch by Florent Xicluna already tries to workaround the 
problem). The traceback points to "the_package = __import__(abstest, 
globals(), locals(), [])" in runtest_inner (in regrtest.py), and a 
"print __import__('test').__file__" there returns 'test/__init__.pyc'.

This can be reproduced quite easily:

trunk$ ./python
Python 2.7a2+ (trunk:77941M, Feb  3 2010, 06:40:49)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> os.getcwd()
'/home/wolf/dev/trunk'
>>> import test
>>> test.__file__  # absolute
'/home/wolf/dev/trunk/Lib/test/__init__.pyc'
>>> os.chdir('/tmp')
>>> test.__file__
'/home/wolf/dev/trunk/Lib/test/__init__.pyc'
>>> from test import test_unicode  # works
>>> test_unicode.__file__
'/home/wolf/dev/trunk/Lib/test/test_unicode.pyc'
>>>
[21]+  Stopped ./python

trunk$ cd Lib/
trunk/Lib$ ../python
Python 2.7a2+ (trunk:77941M, Feb  3 2010, 06:40:49)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> os.getcwd()
'/home/wolf/dev/trunk/Lib'
>>> import test
>>> test.__file__  # relative
'test/__init__.pyc'
>>> os.chdir('/tmp')
>>> from test import test_unicode  # fails
Traceback (most recent call last):
  File "", line 1, in 
ImportError: cannot import name test_unicode

Is there a reason why in the second case test.__file__ is relative?

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread exarkun 

On 08:21 pm, [email protected] wrote:

On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote:

Can you clarify? In Python 3, __file__ always points to the source.
Clearly that is the way of the future. For 99.99% of uses of __file__,
if it suddenly never pointed to a .pyc file any more (even if one
existed) that would be just fine. So what's this talk of switching to
__source__?


Upon further reflection, I agree.  __file__ also points to the source 
in
Python 2.7.  Do we need an attribute to point to the compiled bytecode 
file?


What if, instead of trying to annotate the module object with this 
assortment of metadata - metadata which depends on lots of things, and 
can vary from interpreter to interpreter, and even from module to module 
(depending on how it was loaded) - we just stuck with the __loader__ 
annotation, and encouraged/allowed/facilitated the use of the loader 
object to learn all of this extra information?


Jean-Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] IO module improvements

2010-02-06 Thread Antoine Pitrou 
Pascal Chambon  gmail.com> writes:
> 
> The problem is, doing that forwarding is quite complicated.

Hmm, why is it complicated? I agree it can be tedious (especially in C), but it
doesn't seem complicated in itself.

> My own FileIO class alas needs locking, because for example, on windows
> truncating a file means seeking + setting end of file + restoring
> pointer. 

That's assuming you need FileIO to be thread-safe at all. If you always wrap it
in a Buffered object, the Buffered object will ensure thread-safety using its
own lock.
(I suppose use cases for unbuffered file IO *in Python* must be pretty rare, so
most of the time you shouln't use an unwrapped FileIO anyway)

> And I TextIOWrapper seems to deserve locks. Maybe excerpts like this
> one really are thread-safe, but a long study would be required to
> ensure it.
> [snip]

Actually, TextIOWrapper is simply not thread-safe for most of its operations. I
think we did the work for simple writing, though, since it's better for
multi-threaded use of print().

> There are chances that my approach is slower, but the gains are so high
> in terms of maintainability and use of use, that I would definitely
> advocate it.

I agree that optimizations must always be balanced with maintainability and
simplicity. In this case, though, the IO system is really a critical part and
I'm not sure users would like us to pessimize the implementation.

> Typically, the micro-optimizations you speak about can please heavy
> programs, but they make code a mined land (maybe that's why they
> haven't been put into _pyio :p). 

Well, there's no point in micro-optimizing _pyio since it's dramatically slower
than the C version :) It's there more as a reference implementation.

> Maybe I should take the latest _pyio version, and make a fork offering
> high level flexibility and security, for those who don't care about so
> high performances ?

You can, but be aware that _pyio is *really* slow... I'm not sure it would be a
service to many users.

cheers

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Absolute imports in Python 2.x?

2010-02-06 Thread Brett Cannon 
On Wed, Feb 3, 2010 at 05:57, Nick Coghlan  wrote:
> Mark Dickinson wrote:
>> Agreed on all points.  Would it be terrible to simply add all relevant
>> tags the moment a PEP is accepted?  E.g., if a PEP pronounces some
>> particular behaviour deprecated in Python 3.3 and removed in Python
>> 3.4, then corresponding release blockers for 3.3 and 3.4 could be
>> opened as part of implementing the PEP.
>
> That strikes me as a really good idea, since the tracker already serves
> as a global "to do" list for the project (heck, I've used it as a
> semi-private to do list at times, when I've thought of things I need to
> fix when I don't have the roundtuits free to actually fix them).

Martin filled me in on how to add new versions, so when we need a new
one people can let me know.

-Brett


>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
> ---
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making loggerClass an attribute of the logger manager?

2010-02-06 Thread Georg Brandl 
Am 25.11.2009 11:32, schrieb Vinay Sajip:
> Georg Brandl  gmx.net> writes:
> 
>> Making the loggerClass configurable per manager would solve the
>> problem for me, and AFAICS since most applications don't use
>> different managers anyway, there should not be any detrimental
>> effects.  What do you think?
> 
> Seems reasonable. Apart from the API to set/get, _loggerClass is only used by
> the manager when instantiating a new logger.

I've created a patch for this, see .  I'd like
to get this into 2.7 before beta1 :)

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 12:21 PM, Barry Warsaw  wrote:
> On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote:
>>Can you clarify? In Python 3, __file__ always points to the source.
>>Clearly that is the way of the future. For 99.99% of uses of __file__,
>>if it suddenly never pointed to a .pyc file any more (even if one
>>existed) that would be just fine. So what's this talk of switching to
>>__source__?
>
> Upon further reflection, I agree.  __file__ also points to the source in
> Python 2.7.

Not in the 2.7 svn repo I have access to. It still points to the .pyc
file if it was used.

And I propose not to disturb this in 2.7, at least not by default. I'm
fine though with a flag or distro-overridable config setting to change
this behavior.

> Do we need an attribute to point to the compiled bytecode file?

I think we do. Quite unrelated to this discussion I have a use case
for knowing easily whether a module was actually loaded from bytecode
or not -- but I also have a need for __file__ to point to the source.
So having both __file__ and __compiled__ makes sense to me.

When there is no source code but only bytecode I am file with both
pointing to the bytecode; in that case I presume that the bytecode is
not in a __pyr__ subdirectory. For dynamically loaded extension
modules I think both should be left unset, and some other __xxx__
variable could point to the .so or .dll file. FWIW the most common use
case for __file__ is probably to find data files relative to it. Since
the data won't be in the __pyr__ directory we couldn't make __file__
point to the __pyr__/pyc file without much code breakage.

(Yes, I am still in favor of the folder-per-folder model.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 12:49 PM, Ezio Melotti  wrote:
> In #7712 I was trying to change regrtest to always run the tests in a
> temporary CWD (e.g. /tmp/@test_1234_cwd/).
> The patches attached to the issue add a context manager that changes the
> CWD, and it works fine when I run ./python -m test.regrtest from trunk/.
> However, when I try from trunk/Lib/ it fails with ImportErrors (note that
> the latest patch by Florent Xicluna already tries to workaround the
> problem). The traceback points to "the_package = __import__(abstest,
> globals(), locals(), [])" in runtest_inner (in regrtest.py), and a "print
> __import__('test').__file__" there returns 'test/__init__.pyc'.
> This can be reproduced quite easily:
>
> trunk$ ./python
> Python 2.7a2+ (trunk:77941M, Feb  3 2010, 06:40:49)
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import os, sys
 os.getcwd()
> '/home/wolf/dev/trunk'
 import test
 test.__file__  # absolute
> '/home/wolf/dev/trunk/Lib/test/__init__.pyc'
 os.chdir('/tmp')
 test.__file__
> '/home/wolf/dev/trunk/Lib/test/__init__.pyc'
 from test import test_unicode  # works
 test_unicode.__file__
> '/home/wolf/dev/trunk/Lib/test/test_unicode.pyc'

> [21]+  Stopped                 ./python
>
> trunk$ cd Lib/
> trunk/Lib$ ../python
> Python 2.7a2+ (trunk:77941M, Feb  3 2010, 06:40:49)
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 import os, sys
 os.getcwd()
> '/home/wolf/dev/trunk/Lib'
 import test
 test.__file__  # relative
> 'test/__init__.pyc'
 os.chdir('/tmp')
 from test import test_unicode  # fails
> Traceback (most recent call last):
>  File "", line 1, in 
> ImportError: cannot import name test_unicode
>
> Is there a reason why in the second case test.__file__ is relative?

I haven't tried to repro this particular example, but the reason is
that we don't want to have to call getpwd() on every import nor do we
want to have some kind of in-process variable to cache the current
directory. (getpwd() is relatively slow and can sometimes fail
outright, and trying to cache it has a certain risk of being wrong.)

What we do instead, is code in site.py that walks over the elements of
sys.path and turns them into absolute paths. However this code runs
before '' is inserted in the front of sys.path, so that the initial
value of sys.path is ''.

You may want to print the value of sys.path at various points to see
for yourself.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote:

>>  It does this by
>> allowing many different byte compilation files (.pyc files) to be
>> co-located with the Python source file (.py file).  
>
>It would be nice if all the compilation files could be tucked
>into one single zipfile per directory to reduce directory clutter.
>
>It has several benefits besides tidiness. It hides the implementation
>details of when magic numbers get shifted.  And it may allow faster
>start-up times when the zipfile is in the disk cache.

This is closer in spirit to the original (uncirculated) PEP which called for
fat pyc files, but without the complicated implementation details.  It's still
an interesting approach to explore.

Writer concurrency can be handled with dot-lock files, but that does incur
some extra overhead, such as the remove() of the lock file.

-Barry



signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread exarkun 

On 10:29 pm, [email protected] wrote:
On Sat, Feb 6, 2010 at 12:49 PM, Ezio Melotti  
wrote:

In #7712 I was trying to change regrtest to always run the tests in a
temporary CWD (e.g. /tmp/@test_1234_cwd/).
The patches attached to the issue add a context manager that changes 
the
CWD, and it works fine when I run ./python -m test.regrtest from 
trunk/.
However, when I try from trunk/Lib/ it fails with ImportErrors (note 
that

the latest patch by Florent Xicluna already tries to workaround the
problem). The traceback points to "the_package = __import__(abstest,
globals(), locals(), [])" in runtest_inner (in regrtest.py), and a 
"print

__import__('test').__file__" there returns 'test/__init__.pyc'.
This can be reproduced quite easily:

[snip]

I haven't tried to repro this particular example, but the reason is
that we don't want to have to call getpwd() on every import nor do we
want to have some kind of in-process variable to cache the current
directory. (getpwd() is relatively slow and can sometimes fail
outright, and trying to cache it has a certain risk of being wrong.)


Assuming you mean os.getcwd():

exar...@boson:~$ python -m timeit -s 'def f(): pass' 'f()'
1000 loops, best of 3: 0.132 usec per loop
exar...@boson:~$ python -m timeit -s 'from os import getcwd' 'getcwd()'
100 loops, best of 3: 1.02 usec per loop
exar...@boson:~$
So it's about 7x more expensive than a no-op function call.  I'd call 
this pretty quick.  Compared to everything else that happens during an 
import, I'm not convinced this wouldn't be lost in the noise.  I think 
it's at least worth implementing and measuring.


Jean-Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote:

>It's an interesting challenge to write the file in such a way that
>it's safe for a reader and writer to co-exist. Like Brett, I
>considered an append-only scheme, but one needs to handle the case
>where the bytecode for a particular magic number changes. At some
>point you'd need to sweep garbage from the file. All solutions seem
>unnecessarily complex, and unnecessary since in practice the case
>should not come up.

I don't think that part's difficult.  The byte code's only going to change if
the source file has changed, and in that case, /all/ the byte code in the "fat
pyc" file will be invalidated, so the whole thing can be deleted by the first
writer.  I'd worked that out in the original fat pyc version of the PEP.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw 
On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote:

>So what would you do for concurrent writers, then? The current
>implementation relies on creat(O_EXCL) to be atomic, so a second
>writer would just fail. This is but the only IO operation that is
>guaranteed to be atomic (along with mkdir(2)), so reusing the current
>approach doesn't work.

I believe rename(2) is atomic also, at least on POSIX.  I'm not sure if that
helps us though.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 3:22 PM,   wrote:
> On 10:29 pm, [email protected] wrote:
>>
>> On Sat, Feb 6, 2010 at 12:49 PM, Ezio Melotti 
>> wrote:
>>>
>>> In #7712 I was trying to change regrtest to always run the tests in a
>>> temporary CWD (e.g. /tmp/@test_1234_cwd/).
>>> The patches attached to the issue add a context manager that changes the
>>> CWD, and it works fine when I run ./python -m test.regrtest from trunk/.
>>> However, when I try from trunk/Lib/ it fails with ImportErrors (note that
>>> the latest patch by Florent Xicluna already tries to workaround the
>>> problem). The traceback points to "the_package = __import__(abstest,
>>> globals(), locals(), [])" in runtest_inner (in regrtest.py), and a "print
>>> __import__('test').__file__" there returns 'test/__init__.pyc'.
>>> This can be reproduced quite easily:
>>
>> [snip]
>>
>> I haven't tried to repro this particular example, but the reason is
>> that we don't want to have to call getpwd() on every import nor do we
>> want to have some kind of in-process variable to cache the current
>> directory. (getpwd() is relatively slow and can sometimes fail
>> outright, and trying to cache it has a certain risk of being wrong.)
>
> Assuming you mean os.getcwd():

Yes.

> exar...@boson:~$ python -m timeit -s 'def f(): pass' 'f()'
> 1000 loops, best of 3: 0.132 usec per loop
> exar...@boson:~$ python -m timeit -s 'from os import getcwd' 'getcwd()'
> 100 loops, best of 3: 1.02 usec per loop
> exar...@boson:~$
> So it's about 7x more expensive than a no-op function call.  I'd call this
> pretty quick.  Compared to everything else that happens during an import,
> I'm not convinced this wouldn't be lost in the noise.  I think it's at least
> worth implementing and measuring.

But it's a system call, and its speed depends on a lot more than the
speed of a simple function call. It depends on the OS kernel, possibly
on the filesystem, and so on. Also "os.getcwd()" abstracts away
various platform details that the C import code would have to
replicate. Really, the approach of preprocessing sys.path makes much
more sense. If an app wants sys.path[0] to be an absolute path too
they can modify it themselves.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw  wrote:
> On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote:
>
>>It's an interesting challenge to write the file in such a way that
>>it's safe for a reader and writer to co-exist. Like Brett, I
>>considered an append-only scheme, but one needs to handle the case
>>where the bytecode for a particular magic number changes. At some
>>point you'd need to sweep garbage from the file. All solutions seem
>>unnecessarily complex, and unnecessary since in practice the case
>>should not come up.
>
> I don't think that part's difficult.  The byte code's only going to change if
> the source file has changed, and in that case, /all/ the byte code in the "fat
> pyc" file will be invalidated, so the whole thing can be deleted by the first
> writer.  I'd worked that out in the original fat pyc version of the PEP.

I'm sorry, but I'm totally against fat bytecode files. They make
things harder for all tools. The beauty of the existing bytecode
format is that it's totally trivial: magic number, source mtime,
unmarshalled code object. You can't beat the beauty of that.

For the traditional "skinny" bytecode files, I believe that the
existing algorithm which writes zeros in the place of the magic number
first, writes the rest of the file, and then goes back to write the
correct magic number, is correct with a single writer and multiple
readers (assuming the readers ignore the file if its magic number is
invalid). The creat(O_EXCL) option ensures that there won't be
multiple writers. No rename() is necessary; POSIX rename() may be
atomic, but it's a directory modification which makes it potentially
slow.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Christian Heimes 
Guido van Rossum wrote:
> What we do instead, is code in site.py that walks over the elements of
> sys.path and turns them into absolute paths. However this code runs
> before '' is inserted in the front of sys.path, so that the initial
> value of sys.path is ''.
> 
> You may want to print the value of sys.path at various points to see
> for yourself.

I ran into the issue on Debian or Ubuntu (can't remember) several years
ago. The post-install script of the Python package did something like
"cd /usr/lib/pythonX.Y && ./compileall.py", so all pyc files were
created relative to the library root of Python. The __file__ attribute
of all pre-compiled Python files were relative, too.

Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposal for the getpass module

2010-02-06 Thread Tarek Ziadé 
Hello,

I would like to propose a small change in the getpass module so it's
able to get passwords from keyrings (like KWallet, Keychain, etc)

The idea is to provide a getpass.cfg configuration file where people
can provide the name of a function to use when getpass is called.
Then third-party projects can implement this function. For example the
Python Keyring library.[1] could be installed and configured to be
used
by people that wants getpass calls to be handled by this tool.

That's a backward compatible change, and it avoids adding any new
module in the stdlib. Plus, it offers a greatly improved getpass
module
with no risks for the stdlib stability : it becomes a reference
implementation with an interface for third-party implementers.

A prototype is here : http://bitbucket.org/tarek/getpass/ (work in
progress but you can get the idea)

[1] http://pypi.python.org/pypi/keyring

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 4:04 PM, Christian Heimes  wrote:
> Guido van Rossum wrote:
>> What we do instead, is code in site.py that walks over the elements of
>> sys.path and turns them into absolute paths. However this code runs
>> before '' is inserted in the front of sys.path, so that the initial
>> value of sys.path is ''.
>>
>> You may want to print the value of sys.path at various points to see
>> for yourself.
>
> I ran into the issue on Debian or Ubuntu (can't remember) several years
> ago. The post-install script of the Python package did something like
> "cd /usr/lib/pythonX.Y && ./compileall.py", so all pyc files were
> created relative to the library root of Python. The __file__ attribute
> of all pre-compiled Python files were relative, too.

Are you sure you remember this right? The .co_filename
attributes will be unmarshalled straight from the bytecode file which
indeed will have the relative path in this case (hopefully we'll
finally fix this in 3.2 and 2.7). But if I read the code in import.c
correctly, __file__ is set on the basis of the path of the file read,
which in turn comes from sys.path which will have been "absolufied" by
site.py. Or maybe this was so long ago that site.py didn't yet do
that?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney 
Barry Warsaw  writes:

> On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote:
>
> >It's also the case that having to run Python to manage my own
> >filesystem would very annoying.
[…]

Files that are problematic wouldn't need Python to manage any more than
currently. The suggestion was just that, a suggestion for Python to
expose information to assist; it wouldn't be required.

> I agree. I'd prefer to have a predictable place for the cached files,
> independent of having to run Python to tell you where that is.

Right; I don't see who would disagree with that. I don't see any
conflict between “decouple compiled bytecode file locations from source
file locations” versus “predictable location for the compiled bytecode
files”.

-- 
 \ “All television is educational television. The question is: |
  `\   what is it teaching?” —Nicholas Johnson |
_o__)  |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Christian Heimes 
Guido van Rossum schrieb:
> Are you sure you remember this right? The .co_filename
> attributes will be unmarshalled straight from the bytecode file which
> indeed will have the relative path in this case (hopefully we'll
> finally fix this in 3.2 and 2.7). But if I read the code in import.c
> correctly, __file__ is set on the basis of the path of the file read,
> which in turn comes from sys.path which will have been "absolufied" by
> site.py. Or maybe this was so long ago that site.py didn't yet do
> that?

I ran into the problem years ago. I can recall the Python version but it
must have been 2.2 or 2.3, maybe 2.1. I'm not entirely sure how it
happened, too. All I can remember that I traced the cause down to the
way compileall was called. I've tried to reproduce the issue with Python
2.6 but failed. It looks like the code does the right thing.

Christian
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal for the getpass module

2010-02-06 Thread Guido van Rossum 
[redirecting to python-ideas]

On Sat, Feb 6, 2010 at 4:08 PM, Tarek Ziadé  wrote:
> Hello,
>
> I would like to propose a small change in the getpass module so it's
> able to get passwords from keyrings (like KWallet, Keychain, etc)
>
> The idea is to provide a getpass.cfg configuration file where people
> can provide the name of a function to use when getpass is called.
> Then third-party projects can implement this function. For example the
> Python Keyring library.[1] could be installed and configured to be
> used by people that wants getpass calls to be handled by this tool.
>
> That's a backward compatible change, and it avoids adding any new
> module in the stdlib. Plus, it offers a greatly improved getpass
> module with no risks for the stdlib stability : it becomes a reference
> implementation with an interface for third-party implementers.
>
> A prototype is here : http://bitbucket.org/tarek/getpass/ (work in
> progress but you can get the idea)
>
> [1] http://pypi.python.org/pypi/keyring

Don't you usually have to tell the keyring which password to retrieve?
The signature for getpass() doesn't have enough info for that.

I'm not sure that not adding a new module to the stdlib is a
sufficient reason to foist the new semantics on an old function.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 4:27 PM, Ben Finney  wrote:
> Barry Warsaw  writes:
>> I agree. I'd prefer to have a predictable place for the cached files,
>> independent of having to run Python to tell you where that is.
>
> Right; I don't see who would disagree with that. I don't see any
> conflict between “decouple compiled bytecode file locations from source
> file locations” versus “predictable location for the compiled bytecode
> files”.

The conflict is purely that PEP 3147 proposes the new behavior to be
optional, and adds a flag (-R) and an environment variable
($PYTHONPYR) to change it. I presume Barry is proposing this out of
fear that the new behavior might upset somebody; personally I think it
would be better if the behavior weren't optional. At least not in new
Python releases -- in backports such as a distribution that wants this
feature might make, it may make sense to be more conservative, or at
least to have a way to turn it off.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 4:36 PM, Christian Heimes  wrote:
> Guido van Rossum schrieb:
>> Are you sure you remember this right? The .co_filename
>> attributes will be unmarshalled straight from the bytecode file which
>> indeed will have the relative path in this case (hopefully we'll
>> finally fix this in 3.2 and 2.7). But if I read the code in import.c
>> correctly, __file__ is set on the basis of the path of the file read,
>> which in turn comes from sys.path which will have been "absolufied" by
>> site.py. Or maybe this was so long ago that site.py didn't yet do
>> that?
>
> I ran into the problem years ago. I can recall the Python version but it
> must have been 2.2 or 2.3, maybe 2.1. I'm not entirely sure how it
> happened, too. All I can remember that I traced the cause down to the
> way compileall was called. I've tried to reproduce the issue with Python
> 2.6 but failed. It looks like the code does the right thing.

Hm. The timing doesn't match. From the svn logs for site.py looks like
this was introduced in r17768 on 2000-09-28, which puts it before 2.0
was released.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan 
Ben Finney wrote:
> Right; I don't see who would disagree with that. I don't see any
> conflict between “decouple compiled bytecode file locations from source
> file locations” versus “predictable location for the compiled bytecode
> files”.

The more decoupled they are, the harder it is to manually find the
bytecode file.

With the current .pyc scheme, .pyr folders or an SVN style Python cache
directory, finding the bytecode file is pretty easy, since the cached
file is either in the same directory as the source file or in a
subdirectory.

With any form of shadow hierarchy though, it gets trickier because you
have to:
1. Find the root of the shadow hierarchy
2. Navigate within the shadow hierarchy down to the point that matches
where your source file was

It's a fairly significant increase in mental overhead. It gets much
worse if the location of the shadow hierarchy root is configurable in
any way (e.g. based on sys.path contents or an environment variable).

Restricting the caching mechanism to the folder containing the source
file keeps things a lot simpler.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
---
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum 
On Sat, Feb 6, 2010 at 5:10 PM, Nick Coghlan  wrote:
> Ben Finney wrote:
>> Right; I don't see who would disagree with that. I don't see any
>> conflict between “decouple compiled bytecode file locations from source
>> file locations” versus “predictable location for the compiled bytecode
>> files”.
>
> The more decoupled they are, the harder it is to manually find the
> bytecode file.
>
> With the current .pyc scheme, .pyr folders or an SVN style Python cache
> directory, finding the bytecode file is pretty easy, since the cached
> file is either in the same directory as the source file or in a
> subdirectory.
>
> With any form of shadow hierarchy though, it gets trickier because you
> have to:
> 1. Find the root of the shadow hierarchy
> 2. Navigate within the shadow hierarchy down to the point that matches
> where your source file was
>
> It's a fairly significant increase in mental overhead. It gets much
> worse if the location of the shadow hierarchy root is configurable in
> any way (e.g. based on sys.path contents or an environment variable).
>
> Restricting the caching mechanism to the folder containing the source
> file keeps things a lot simpler.

Great way of explaining why the basic folder-per-folder model wins
over the folder-per-sys.path-entry model! The basic folder-per-folder
model doesn't need to know what sys.path is. (And I hadn't followed
previous messages in the thread with enough care to understand the
subtlen implications of Ben's point. Sorry!)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney 
Nick Coghlan  writes:

> The more decoupled they are, the harder it is to manually find the
> bytecode file.

Okay. So it's not so much about “predictable”, but rather about
“predictable by a human without too much cognitive effort”.

I can see value in that, though it's best to be explicit that this is a
goal (to be clear that “a program can tell you where they live” isn't a
solution).

> It's a fairly significant increase in mental overhead. It gets much
> worse if the location of the shadow hierarchy root is configurable in
> any way (e.g. based on sys.path contents or an environment variable).
>
> Restricting the caching mechanism to the folder containing the source
> file keeps things a lot simpler.

Simpler for the human working on the source code; not for the human
trying to fit this scheme in with an OS package management system.
(Again, I'm just clarifying and making the contrast explicit, not
judging relative values.)

This makes it clearer to me that there is a glaring incompatibility
between this desire for “keep the compiled bytecode files close to the
source files” versus “decouple the locations so the OS package manager
can do its job of managing installed files”.

I recognise after earlier discussion in this thread that's not an issue
being addressed by PEP 3147.

-- 
 \ “Those are my principles. If you don't like them I have |
  `\others.” —Groucho Marx |
_o__)  |
Ben Finney

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __file__ is not always an absolute path

2010-02-06 Thread exarkun 

On 6 Feb, 11:53 pm, [email protected] wrote:

On Sat, Feb 6, 2010 at 3:22 PM,   wrote:

On 10:29 pm, [email protected] wrote:


[snip]

I haven't tried to repro this particular example, but the reason is
that we don't want to have to call getpwd() on every import nor do we
want to have some kind of in-process variable to cache the current
directory. (getpwd() is relatively slow and can sometimes fail
outright, and trying to cache it has a certain risk of being wrong.)


Assuming you mean os.getcwd():


Yes.

exar...@boson:~$ python -m timeit -s 'def f(): pass' 'f()'
1000 loops, best of 3: 0.132 usec per loop
exar...@boson:~$ python -m timeit -s 'from os import getcwd' 
'getcwd()'

100 loops, best of 3: 1.02 usec per loop
exar...@boson:~$
So it's about 7x more expensive than a no-op function call. I'd call 
this
pretty quick. Compared to everything else that happens during an 
import,
I'm not convinced this wouldn't be lost in the noise. I think it's at 
least

worth implementing and measuring.


But it's a system call, and its speed depends on a lot more than the
speed of a simple function call. It depends on the OS kernel, possibly
on the filesystem, and so on.


Do you know of a case where it's actually slow?  If not, how convincing 
should this argument really be?  Perhaps we can measure it on a few 
platforms before passing judgement.


For reference, my numbers are from Linux 2.6.31 and my filesystem 
(though I don't think it really matters) is ext3.  I have eglibc 2.10.1 
compiled by gcc version 4.4.1.

Also "os.getcwd()" abstracts away
various platform details that the C import code would have to
replicate.


That logic can all be hidden behind a C API which os.getcwd() can then 
be implemented in terms of.  There's no reason for it to be any harder 
to invoke from C than it is from Python.

Really, the approach of preprocessing sys.path makes much
more sense. If an app wants sys.path[0] to be an absolute path too
they can modify it themselves.


That may turn out to be the less expensive approach.  I'm not sure in 
what other ways it is the approach that makes much more sense.


Quite the opposite: centralizing the responsibility for normalizing this 
value makes a lot of sense if you consider things like reducing code 
duplication and, in turn, removing the possibility for bugs.


Adding better documentation for __file__ is another task which I think 
is worth undertaking, regardless of whether any change is made to how 
its value is computed.  At the moment, the two or three sentences about 
it in PEP 302 are all I've been able to find, and they don't really get 
the job done.


Jean-Paul
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan 
[email protected] wrote:
> On 08:21 pm, [email protected] wrote:
>> On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote:
>>> Can you clarify? In Python 3, __file__ always points to the source.
>>> Clearly that is the way of the future. For 99.99% of uses of __file__,
>>> if it suddenly never pointed to a .pyc file any more (even if one
>>> existed) that would be just fine. So what's this talk of switching to
>>> __source__?
>>
>> Upon further reflection, I agree.  __file__ also points to the source in
>> Python 2.7.  Do we need an attribute to point to the compiled bytecode
>> file?
> 
> What if, instead of trying to annotate the module object with this
> assortment of metadata - metadata which depends on lots of things, and
> can vary from interpreter to interpreter, and even from module to module
> (depending on how it was loaded) - we just stuck with the __loader__
> annotation, and encouraged/allowed/facilitated the use of the loader
> object to learn all of this extra information?

Trickier than it sounds. In the case of answering the question "was this
module loaded from bytecode or not?", the loader will need somewhere to
store the answer for each file.

The easiest per-module store is the module's own global namespace - the
loader's own attribute namespace isn't appropriate, since one loader may
handle multiple modules.

The filesystem can't be used as a reference because even when the file
is loaded from source, the bytecode file will usually be created as a
side effect.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
---
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com