Re: [Python-Dev] 2.6 object.__init__ & deling __new__

[Greg Ewing]

P.J. Eby wrote:

In effect, 2.6 forces you to have a common known base class *other* than 
'object' in order to write co-operative classes.  :-(


You have to do that anyway if you want to make cooperative
calls to any method *other* that __init__.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
Ratnakumar wrote:
> Here are my comments regarding PEP 376 with respect to PyPM (the Python
> package manager being developd at ActiveState)
>
>
> Multiple versions: I understand that the PEP does not support
> installation (thus uninstallation) of multiple versions of the same
> package. Should this be explicitly mentioned in the PEP -- as
> `get_distribution` API accepts only `name` argument, and not a `version`
> argument?

That's another can of worms ;)

Before I answer here's a bit of background, i's a bit long but required, sorry

For people that don't want to read the rest, here's the idea :
multiple version support imho should
be introduced later, if they are to be introduced, by extending PEP
302 protocol.

The long explanantion now:

given a "foo" package, containing a "bar" module, multiple versions
support implies to do one of these:

1 - a custom PEP 302-like loader/importer that picks a version of
"foo" when the code imports the "bar" module.
  this works if the "foo" package is not directly available in
sys.path, and if the custom loader/importer is put in sys.meta_path
  for example.  If "foo 0.9" is located in /var/packages/foo/0.9 and
if "foo 1.0" is in  /var/packages/foo/1.0,
  The loader will select the right foo package to load and return with
through a loader that scans   /var/packages/foo/*

  To make it work it requires 2 things :
a/ a version comparison system (see PEP 386) that will make
the loader pick the "latest" version by default
b/ an API that will force the loader to pick one particular version

2 - changing the paths in sys.path to include the path containing the
right version, and let the existing importer/loader do the work.
That's what setuptools does with its multiple version system: an API
called "require" will let you change sys.path on the fly

   >>> from pkg_resources import require
   >>> require('docutils==0.4') <--- looks for a docutils
egg distribution and adds it in the path

So if we support multiple versions in Python, (I'd love too). PEP 376
would need to be able to find the various versions
of each distribution, not by scanning sys.path but rather by scanning
a arbitrary collection of directories, then publishing the
right ones in sys.path (with a PEP 302 loader, or ala setuptools by
inserting them in sys.path)

In other words this would require changing the way the distributions
are stored. e.g. in self-contained eggs or in a brand-new
storage tree. (I am currently experimenting this with "virtual
site-packages" see http://bitbucket.org/tarek/vsp/src/tip/README.txt)

But as we said earlier, people might want to store their modules
anywhere (on a sql database, on Mars, etc.) and provide
a PEP 302-like loader for them. PJE has "eggs" but John Doe might want
to store its packages differently and provide an importer/loader for
them.

So each one of them provides a "package manager", which should composed of :

A- a loader/importer system
B- an installation system (that is easy_install -m for setuptools)
C- query APIs
D- a version comparison system
E- an uninstaller

So the real solution is to work with PEP 302 importers/loaders (A)
(e.g. "package managers")

Which means that PEP 302 need to be changed to become
'distribution-aware' as Paul said.
Because we would then be able to browse distributions (C) that are not
already loaded in sys.path, so work on two versions
of the same distribution.

but some open questions remains:

It also implies that each package manager provides its installer (B)
and a version comparison system (D)

I'm not sure about the way package installers could be declared. Plus,
how people would deal with several installers ?
For the version comparison system I am not sure either, but it would
require to have one global version comparison
system to rule them all otherwise some conflicts may occur.

So there's no plan to support multiple versions yet, because that
requires another PEP imho.

Since distutils is a package manager in some ways (it provides an
installer, and stores distributions that are made
available in sys.path) My feeling is that we need first to finish
what's missing to make it fully usable (eg query apis + uninstaller)

Then maybe think about extending PEP 302

>
>> get_distribution(name) -> Distribution or None.
>> Scans all elements in sys.path and looks for all directories ending
>> with .egg-info. Returns a Distribution corresponding to the .egg-info
>> directory that contains a PKG-INFO that matches name for the name
>> metadata.
>> Notice that there should be at most one result. The first result
>> founded is returned. If the directory is not found, returns None.
>
> Some packages have package names with mixed case. Example: ConfigObj
> .. as registered in setup.py. However, other packages such as turbogears
> specifies "configobj" (lowercase) in their install_requires.
>
> Is `get_distribution(name)` supposed to handle mixed cases? Will 

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Paul Moore]

2009/7/15 Tarek Ziadé :
> On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
> Ratnakumar wrote:
>> Here are my comments regarding PEP 376 with respect to PyPM (the Python
>> package manager being developd at ActiveState)
>>
>>
>> Multiple versions: I understand that the PEP does not support
>> installation (thus uninstallation) of multiple versions of the same
>> package. Should this be explicitly mentioned in the PEP -- as
>> `get_distribution` API accepts only `name` argument, and not a `version`
>> argument?
>
> That's another can of worms ;)

:-)

> Before I answer here's a bit of background, i's a bit long but required, sorry
>
> For people that don't want to read the rest, here's the idea :
> multiple version support imho should
> be introduced later, if they are to be introduced, by extending PEP
> 302 protocol.

Disclaimer: I've only read the short version, so if some of this is
covered in the longer explanation, I apologise now.

-1.

In my view, multiple version support is not at all related to PEP 302
- or to core Python in general. The import statement has no concept of
versions, any version handling is done by explicit user manipulation
of sys.path.

PEP 302 is currently purely an import protocol. As such, it only cares
about locating the correct code to run to populate sys.modules['foo'].
Once the code has been located, there are a number of other details
that might be useful, hence the extensions like get_data,
get_filename, etc. But note that these are all *loader* extensions -
their starting point is an imported module.

The PEP 376 support I've just added is a *finder* extension, which is
working alongside the scanning of the container - but rather than
looking for modules, it's looking for distributions. Disappointingly
(for me) it turned out not to give much opportunity to share code -
the finder extensions could just as easily have been a completely
independent protocol.

PEP 376 support has added a requirement for 3 additional methods to
the existing 1 finder method in PEP 302. That's already a 300%
increase in complexity. I'm against adding any further complexity to
PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
*own* locator protocol and avoid any extra burden on PEP 302. I'm not
sure implementers of PEP 302 importers will even provide the current
PEP 376 extensions.

I propose that before the current prototype is turned into a final
(spec and) implementation, the PEP 302 extensions are extracted and
documented as an independent protocol, purely part of PEP 376. (This
*helps* implementers, as they can write support for, for example,
eggs, without needing to modify the existing egg importer). I'll
volunteer to do that work - but I won't start until the latest
iteration of questions and discussions has settled down and PEP 376
has achieved a stable form with the known issues addressed.

Of course, this is moving more and more towards saying that the design
of setuptools, with its generic means for locating distributions, etc
etc, is the right approach. We're reinventing the wheel here. But the
problem is that too many people dislike setuptools as it stands for it
to gain support. My understanding is that the current set of PEPs were
intended to be a stripped down, more generally acceptable subset of
setuptools. Let's keep them that way (and omit the complexities of
multi-version support).

If you want setuptools, you know where to get it...

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Michael Foord]

Paul Moore wrote:

2009/7/15 Tarek Ziadé :
  

On Tue, Jul 14, 2009 at 2:12 AM, Sridhar
Ratnakumar wrote:


Here are my comments regarding PEP 376 with respect to PyPM (the Python
package manager being developd at ActiveState)


Multiple versions: I understand that the PEP does not support
installation (thus uninstallation) of multiple versions of the same
package. Should this be explicitly mentioned in the PEP -- as
`get_distribution` API accepts only `name` argument, and not a `version`
argument?
  

That's another can of worms ;)



:-)

  

Before I answer here's a bit of background, i's a bit long but required, sorry

For people that don't want to read the rest, here's the idea :
multiple version support imho should
be introduced later, if they are to be introduced, by extending PEP
302 protocol.



Disclaimer: I've only read the short version, so if some of this is
covered in the longer explanation, I apologise now.

-1.
  


I agree. People with versioning issues *should* be using virtualenv.

Michael Foord


In my view, multiple version support is not at all related to PEP 302
- or to core Python in general. The import statement has no concept of
versions, any version handling is done by explicit user manipulation
of sys.path.

PEP 302 is currently purely an import protocol. As such, it only cares
about locating the correct code to run to populate sys.modules['foo'].
Once the code has been located, there are a number of other details
that might be useful, hence the extensions like get_data,
get_filename, etc. But note that these are all *loader* extensions -
their starting point is an imported module.

The PEP 376 support I've just added is a *finder* extension, which is
working alongside the scanning of the container - but rather than
looking for modules, it's looking for distributions. Disappointingly
(for me) it turned out not to give much opportunity to share code -
the finder extensions could just as easily have been a completely
independent protocol.

PEP 376 support has added a requirement for 3 additional methods to
the existing 1 finder method in PEP 302. That's already a 300%
increase in complexity. I'm against adding any further complexity to
PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
*own* locator protocol and avoid any extra burden on PEP 302. I'm not
sure implementers of PEP 302 importers will even provide the current
PEP 376 extensions.

I propose that before the current prototype is turned into a final
(spec and) implementation, the PEP 302 extensions are extracted and
documented as an independent protocol, purely part of PEP 376. (This
*helps* implementers, as they can write support for, for example,
eggs, without needing to modify the existing egg importer). I'll
volunteer to do that work - but I won't start until the latest
iteration of questions and discussions has settled down and PEP 376
has achieved a stable form with the known issues addressed.

Of course, this is moving more and more towards saying that the design
of setuptools, with its generic means for locating distributions, etc
etc, is the right approach. We're reinventing the wheel here. But the
problem is that too many people dislike setuptools as it stands for it
to gain support. My understanding is that the current set of PEPs were
intended to be a stripped down, more generally acceptable subset of
setuptools. Let's keep them that way (and omit the complexities of
multi-version support).

If you want setuptools, you know where to get it...

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
  



--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Antoine Pitrou]

Brett Cannon  python.org> writes:
> 
> 
> I implemented get_filename() as specified in PEP 302 for importlib's source
and bytecode loaders and I was starting to create the ABC for importlib.abc,
but then I realized that perhaps the loader should live in runpy instead of
importlib.
> 
> Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot. But
this addition to the PEP 302 protocols is very specific to runpy and is not
directly required to make imports work (the implementation was just a
reshuffling of pre-existing code from one method to another). I am tempted to
leave the ABC out of importlib and stop at implementing get_filename() for
importlib.abc.PyLoader and PyPycLoader.

I am not sure when this discussion started. Are you replying to a 3 month-old
message of yours? :)

In any case, keeping all import-related ABCs in a single place sounds like a
good idea.
Alternatively, if the method is runpy specific, it doesn't have its place in an
ABC, does it?


Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Paul Moore]

2009/7/15 Brett Cannon :
> I implemented get_filename() as specified in PEP 302 for importlib's source
> and bytecode loaders and I was starting to create the ABC for importlib.abc,
> but then I realized that perhaps the loader should live in runpy instead of
> importlib.
> Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot.
> But this addition to the PEP 302 protocols is very specific to runpy and is
> not directly required to make imports work (the implementation was just a
> reshuffling of pre-existing code from one method to another). I am tempted
> to leave the ABC out of importlib and stop at implementing get_filename()
> for importlib.abc.PyLoader and PyPycLoader.
> Any opinions?

importlib.

In my view, all of the interfaces defined by PEP 302 should be
available from one place. And importlib is that place.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.6 object.__init__ & deling __new__

[Nick Coghlan]

Dino Viehland wrote:
> Based upon the behavior I'm seeing it seems to me that the
> presence of __new__ / __init__ must be getting cached somewhere
> and the deletion isn't updating the cache and that's specifically
> what struck me as odd here.

Digging through typeobject.c, it isn't clear to me why this wouldn't be
getting picked up.

If all is working correctly:

1. When the class is created, tp_new and tp_init are copied from the
base class (see inherit_special and inherit_slots)

2. When object_new and object_init are called they compare the values
stored in the tp_new and tp_init slots with their own function addresses
to decide whether or not to raise the warning

3. When either of those is deleted, the update_slots call in
type_setattro should fix up the slot inheritance, and hence affect
future calls to __new__ and __init__

That said, the following comments in update_one_slot() are leading me to
wonder if Guido might owe Dino a beer:

{
/* The __new__ wrapper is not a wrapper descriptor,
so must be special-cased differently.
If we don't do this, creating an instance will
always use slot_tp_new which will look up
__new__ in the MRO which will call tp_new_wrapper
which will look through the base classes looking
for a static base and call its tp_new (usually
PyType_GenericNew), after performing various
sanity checks and constructing a new argument
list.  Cut all that nonsense short -- this speeds
up instance creation tremendously. */
specific = (void *)type->tp_new;
/* XXX I'm not 100% sure that there isn't a hole
in this reasoning that requires additional
sanity checks.  I'll buy the first person to
point out a bug in this reasoning a beer. */
}

I *think* this logic may be getting confused when update_one_slot() is
called immediately after the "del x.__new__" call has stuffed a NULL
into the tp_new slot. In the normal class creation case the tp_new
method will have been copied down from the parent class so the
conditions when update_one_slot() gets called are different (ditto for
when you actually *set* x.__new__ rather than delete it).

If nobody else gets anywhere with this I should have a chance to
actually play with it (rather than just reading code) tomorrow evening.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Nick Coghlan]

Antoine Pitrou wrote:
> I am not sure when this discussion started. Are you replying to a 3 month-old
> message of yours? :)

That depends on how you define the beginning of the discussion...

However, the fact that importlib doesn't implement the comparatively
recent get_filename() optional extension documented in PEP 302 came up
in one of the PEP 376 threads within the last week or so.

> In any case, keeping all import-related ABCs in a single place sounds like a
> good idea.
> Alternatively, if the method is runpy specific, it doesn't have its place in 
> an
> ABC, does it?

While runpy is the only client in the standard library for the
get_filename() method, the method is still a PEP 302 extension. I
documented the extension in the PEP as loaders are the only things
reliably in a position to provide the filename details that runpy needs
to set __file__ and sys.argv[0] correctly and until importlib came along
PEP 302 itself was the only real documentation of that API.

Since importlib is now the "go-to" location for people that want to
write their own PEP 302 importers and loaders, I would say that it is
also the right place for the new ExecutionLoader ABC.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore wrote:
>
> Disclaimer: I've only read the short version, so if some of this is
> covered in the longer explanation, I apologise now.

Next time I won't put a short version ;)


>
> PEP 376 support has added a requirement for 3 additional methods to
> the existing 1 finder method in PEP 302. That's already a 300%
> increase in complexity. I'm against adding any further complexity to
> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
> *own* locator protocol and avoid any extra burden on PEP 302. I'm not
> sure implementers of PEP 302 importers will even provide the current
> PEP 376 extensions.
>
> I propose that before the current prototype is turned into a final
> (spec and) implementation, the PEP 302 extensions are extracted and
> documented as an independent protocol, purely part of PEP 376. (This
> *helps* implementers, as they can write support for, for example,
> eggs, without needing to modify the existing egg importer). I'll
> volunteer to do that work - but I won't start until the latest
> iteration of questions and discussions has settled down and PEP 376
> has achieved a stable form with the known issues addressed.

Sure that makes sense. I am all for having these 302 extensions
flipped on PEP 376
side, then think about the "locator" protocol.

I am lagging a bit in the discussions, I have 10 messages left to read or so,
but the known issues I've listed so far are about the RECORD file and
absolute paths,
I am waiting for PJE example on the syntax he proposed for prefixes,
on the docutils example.

> Of course, this is moving more and more towards saying that the design
> of setuptools, with its generic means for locating distributions, etc
> etc, is the right approach.
> We're reinventing the wheel here. But the
> problem is that too many people dislike setuptools as it stands for it
> to gain support.

I don't think it's about setuptools design. I think it's more likely
to be about the fact
that there's no way in Python to install two different versions of the
same distribution
without "hiding" one from each other, using setuptools, virtualenv or
zc.buildout.

"installing" a distribution in Python means that its activated
globally, whereas people
need it locally at the application level.

> My understanding is that the current set of PEPs were
> intended to be a stripped down, more generally acceptable subset of
> setuptools. Let's keep them that way (and omit the complexities of
> multi-version support).
>
> If you want setuptools, you know where to get it...

Sure, but let's not forget that the multiple-version issue is a global
issue OS packagers
also meet. (setuptools is not the problem) :

- application Foo uses docutils 0.4 and doesn't work with docutils 0.5
- application Bar uses docutils 0.5

if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
shipped with it.

So right now application developers must use tools to isolate their
dependencies from
the rest of the system, using virtualenv or zc.buildout, and ship the
dependencies with them.
So they are sure that their applications are not broken by the system.

This is not optimal of course, because you end up with several
occurences of the same
versions sometimes. And can be a nightmare for os packagers and maintainers.

And virtualenv and such tools are now required when you develop
applications (mid-size and large ones)
and the "good practice" is to avoid any installation of any
distributions in Python itself.

So basically "site-packages" is a distribution location that is
avoided by everyone because it doesn't
know how to handle multiple versions. If we had a multi-versions
support protocol, that would
help os packagers and application developers to be friends again imho ;)

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

On Wed, Jul 15, 2009 at 12:17 PM, Michael
Foord wrote:
>> Disclaimer: I've only read the short version, so if some of this is
>> covered in the longer explanation, I apologise now.
>>
>> -1.
>>
>
> I agree. People with versioning issues *should* be using virtualenv.
>

Let's remove site-packages from Python then.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[Peter Hanecak]

Hello,

when I'm trying to use 64-bit integer values with SimpleXMLRPCServer, 
I'm getting "OverflowError: int exceeds XML-RPC limits" error each time 
I use an integer with value greater or equal to 2^31.


I googled this:

http://bugs.python.org/issue2985


So, my question is: In which Python release has been this fix distributed?

Thank you in advance for information.

Btw, I've made some dummy scripts to demonstrate the issue:

- server.py:
from SimpleXMLRPCServer import SimpleXMLRPCServer

def dummy(number):
  return number

server = SimpleXMLRPCServer(("localhost", 8000))
server.register_function(dummy, 'dummy')
server.serve_forever()

- client.py
import xmlrpclib

proxy = xmlrpclib.ServerProxy("http://localhost:8000/";)
print proxy.dummy(0x7FFF)
print proxy.dummy(0x8000)


The output from client is:
localhost.localdomain - - [15/Jul/2009 15:24:12] "POST / HTTP/1.0" 200 -
2147483647
Traceback (most recent call last):
  File "./client.py", line 7, in 
print proxy.dummy(0x8000)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1483, in __request
allow_none=self.__allow_none)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1132, in dumps
data = m.dumps(params)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 677, in dumps
dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 710, in dump_int
raise OverflowError, "int exceeds XML-RPC limits"
OverflowError: int exceeds XML-RPC limits


Have a nice day.

Peter

p.s.: I'm not subscribed to the list, se please keep me in CC when 
replying. Thank you.


--
Peter Hanecak

ePC Developer | Alcatel-Lucent
Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic
email: peter.hane...@alcatel-lucent.com
phone: +421 (0)2 49264 857
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[Benjamin Peterson]

2009/7/15 Peter Hanecak :
> So, my question is: In which Python release has been this fix distributed?

Python 2.6 and above.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[R. David Murray]

On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote:

2009/7/15 Peter Hanecak :

So, my question is: In which Python release has been this fix distributed?


Python 2.6 and above.


But it doesn't solve your problem, since the ticket says it only fixes
reading long ints, not writing them.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[Peter Hanecak]

Hello,

thank you David and Benjamin for quick response.

So my subsequent question is: What can help me solve the "writing" part?

Sincerely

Peter

On 07/15/2009 04:39 PM, R. David Murray wrote:

On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote:

2009/7/15 Peter Hanecak :

So, my question is: In which Python release has been this fix
distributed?


Python 2.6 and above.


But it doesn't solve your problem, since the ticket says it only fixes
reading long ints, not writing them.

--David


--
Peter Hanecak

ePC Developer | Alcatel-Lucent
Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic
email: peter.hane...@alcatel-lucent.com
phone: +421 (0)2 49264 857
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[Tarek Ziadé]

On Wed, Jul 15, 2009 at 4:42 PM, Peter
Hanecak wrote:
> Hello,
>
> thank you David and Benjamin for quick response.
>
> So my subsequent question is: What can help me solve the "writing" part?

Use strings.

Send str(0x7FFF) from the client for example , and get back your
number on server side with a long(number)  call.

That's what I do.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits

[Dirkjan Ochtman]

On Wed, Jul 15, 2009 at 16:42, Peter
Hanecak wrote:
> So my subsequent question is: What can help me solve the "writing" part?

The XML-RPC protocol, as specified at [1], doesn't support integers
with more than 32 bits (in fact, the i4 alias can be used to make the
use of 4 bytes explicit). So, either don't use XML-RPC, or encode it
in some other type (e.g. a string or base64).

Cheers,

Dirkjan

[1] http://www.xmlrpc.com/spec
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Paul Moore]

2009/7/15 Tarek Ziadé :
> On Wed, Jul 15, 2009 at 12:17 PM, Michael
> Foord wrote:
>>> Disclaimer: I've only read the short version, so if some of this is
>>> covered in the longer explanation, I apologise now.
>>>
>>> -1.
>>>
>>
>> I agree. People with versioning issues *should* be using virtualenv.
>>
>
> Let's remove site-packages from Python then.

If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then
maybe you're right.

For me, py2exe (and a clean virtual machine if I require an absolutely
pristine environment, I guess virtualenv would help here too) does
what I need for application packaging. But I'll freely admit that my
needs are minimal.

Bluntly, as Python stands, import and sys.path do not offer any core
support for multiple versions. Custom solutions can be built on top of
that - that's what setuptools does. But they are precisely that -
custom solutions, and should be supported as such, outside the core
(and stdlib).

If standard Python support for multi-version imports is required (it's
not by me, but I accept that some people want it), then it should be
designed in thoughout:

- how do I import version 0.7.1 of package foo, rather than 0.7.2?
- how do I use foo 0.8 at my interactive prompt, and import bar 0.2
which relies on foo 0.7.1?
- what happens if I import foo 2.0 (which requires baz 0.1) and bar
1.5 (which requires baz 0.2)?
- what does "import foo" (without a version number) mean? Is it
different if it's at the command line or in bar 0.5 (which explicitly
declares a dependency on foo 0.7 in its setup.py)? Does the answer to
that mean that imports always need to read dependency information?
- does your head hurt yet? I could go on...

Any PEP dealing with multi versions should address these issues. It's
a big area, and I have no interest in it myself, but I do have an
interest in avoiding partial solutions which only look at some of the
questions that might arise.

Look - I really, really don't mind if people use setuptools. Honest.
But I do mind if core python gets changed to support little bits of
what setuptools does, adding complexity to deal with issues that
setuptools handles, but without making it possible to avoid using
setuptools. Where's the benefit to anyone then?

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Paul Moore]

2009/7/15 Nick Coghlan :
> While runpy is the only client in the standard library for the
> get_filename() method, the method is still a PEP 302 extension. I
> documented the extension in the PEP as loaders are the only things
> reliably in a position to provide the filename details that runpy needs
> to set __file__ and sys.argv[0] correctly and until importlib came along
> PEP 302 itself was the only real documentation of that API.

Yes, sorry about that :-)

> Since importlib is now the "go-to" location for people that want to
> write their own PEP 302 importers and loaders, I would say that it is
> also the right place for the new ExecutionLoader ABC.

Agreed. (But I said that before...)

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Joachim König]

Tarek Ziadé wrote:

So basically "site-packages" is a distribution location that is
avoided by everyone because it doesn't
know how to handle multiple versions. 
I think you overrate the importance of having multiple versions of a 
package available
for the same python interpreter. If you have m different versions of n 
packages then

you could have n**m different combinations for an application so you need a
possiblilty to select one combination from n**m possible ones at application
startup time. Is this really worth it?


If we had a multi-versions
support protocol, that would
help os packagers and application developers to be friends again imho ;)
  
Let's remove site-packages from Python then.
The _one_ site-packages folder stands for _one_ python interpreter. All 
the clever
efforts to provide a set of package versions at runtime to an 
application (that uses
the singleton python interpreter) do logically create a new python 
interpreter with
a site-packages folder that contains just the versions of the packages 
the application
needs, unfortunately by mucking with PYTHONPATH and .pth, 
site.py etc
making it very difficult to understand what is happening for the joe 
average python developer.




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

On Wed, Jul 15, 2009 at 5:14 PM, Paul Moore wrote:
> 2009/7/15 Tarek Ziadé :
>> On Wed, Jul 15, 2009 at 12:17 PM, Michael
>> Foord wrote:
 Disclaimer: I've only read the short version, so if some of this is
 covered in the longer explanation, I apologise now.

 -1.

>>>
>>> I agree. People with versioning issues *should* be using virtualenv.
>>>
>>
>> Let's remove site-packages from Python then.
>
> If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then
> maybe you're right.

They do offer a solution, but these solution are hard to maintain from
a OS packager point of view.

In any case I don't see any use case to have a "site-packages"
remaining in Python itself.

> If standard Python support for multi-version imports is required (it's
> not by me, but I accept that some people want it), then it should be
> designed in thoughout:
> [..]
>
> Any PEP dealing with multi versions should address these issues. It's
> a big area, and I have no interest in it myself, but I do have an
> interest in avoiding partial solutions which only look at some of the
> questions that might arise.
>
> Look - I really, really don't mind if people use setuptools. Honest.
> But I do mind if core python gets changed to support little bits of
> what setuptools does, adding complexity to deal with issues that
> setuptools handles, but without making it possible to avoid using
> setuptools. Where's the benefit to anyone then?

I totally agree. But I think that the boundary between what Python+stdlib
should provide feature-wise and what third party packages provides is
still fuzzy and should
be made clearer.

At some point, third party projects are trying hard to isolate dependencies
because they can't do it with the execution environment Python sets by
default when
you launch a script or an interpreter.

site.py loads site-packages and user site-packages at startup
basically, and you can
use PYTHONPATH / sys.path to add more, but that's partially shared by all apps.

To address this issued, a project like zc.buildout will create a local
repository of distributions, and will
change sys.path on the fly so the local repository is used.

virtualenv on its side creates an isolated Python installation for
your application,

So If the stdlib doesn't provide a standard protocol that goes further
than that,
people that have this need will continue to use zc.buildout or
virtualenv and install many times
the same libs on a system.

It's perfectly fine from an application developer PoV, but it make
site-packages obsoletes and
every application installed that way looks like a black box for os
packagers. They don't want that.

At the end, application developers have to care about the way their applications
dependencies are installed, were they should just give the list of
those dependencies,
and let any package manager project install them.

If these matters are not adressed by Python sdtlib, then we should
remove the loading of
site-packages at Python startup completely, and make it cristal clear
that it's not the core/stdlib
problem.

Otherwise we should think hard about how to let os packagers manage a
single repository
of Python distributions (and multiple versions) and how application
developers could use them from within
their applications. This protocol imho could be in the sdtlib even if
the implementation
is outside, but that's just me.

In any case, these issues can be postponed after PEP 376, because
that's a another (bigger) part of the puzzle.

Regards
Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[David Cournapeau]

On Wed, Jul 15, 2009 at 11:00 PM, Tarek Ziadé wrote:
> On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore wrote:
>>
>> Disclaimer: I've only read the short version, so if some of this is
>> covered in the longer explanation, I apologise now.
>
> Next time I won't put a short version ;)
>
>
>>
>> PEP 376 support has added a requirement for 3 additional methods to
>> the existing 1 finder method in PEP 302. That's already a 300%
>> increase in complexity. I'm against adding any further complexity to
>> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its
>> *own* locator protocol and avoid any extra burden on PEP 302. I'm not
>> sure implementers of PEP 302 importers will even provide the current
>> PEP 376 extensions.
>>
>> I propose that before the current prototype is turned into a final
>> (spec and) implementation, the PEP 302 extensions are extracted and
>> documented as an independent protocol, purely part of PEP 376. (This
>> *helps* implementers, as they can write support for, for example,
>> eggs, without needing to modify the existing egg importer). I'll
>> volunteer to do that work - but I won't start until the latest
>> iteration of questions and discussions has settled down and PEP 376
>> has achieved a stable form with the known issues addressed.
>
> Sure that makes sense. I am all for having these 302 extensions
> flipped on PEP 376
> side, then think about the "locator" protocol.
>
> I am lagging a bit in the discussions, I have 10 messages left to read or so,
> but the known issues I've listed so far are about the RECORD file and
> absolute paths,
> I am waiting for PJE example on the syntax he proposed for prefixes,
> on the docutils example.
>
>> Of course, this is moving more and more towards saying that the design
>> of setuptools, with its generic means for locating distributions, etc
>> etc, is the right approach.
>> We're reinventing the wheel here. But the
>> problem is that too many people dislike setuptools as it stands for it
>> to gain support.
>
> I don't think it's about setuptools design. I think it's more likely
> to be about the fact
> that there's no way in Python to install two different versions of the
> same distribution
> without "hiding" one from each other, using setuptools, virtualenv or
> zc.buildout.
>
> "installing" a distribution in Python means that its activated
> globally, whereas people
> need it locally at the application level.
>
>> My understanding is that the current set of PEPs were
>> intended to be a stripped down, more generally acceptable subset of
>> setuptools. Let's keep them that way (and omit the complexities of
>> multi-version support).
>>
>> If you want setuptools, you know where to get it...
>
> Sure, but let's not forget that the multiple-version issue is a global
> issue OS packagers
> also meet. (setuptools is not the problem) :
>
> - application Foo uses docutils 0.4 and doesn't work with docutils 0.5
> - application Bar uses docutils 0.5
>
> if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
> shipped with it.

As was stated by Debian packagers on the distutils ML, the problem is
that docutils 0.5 breaks packages which work with docutils 0.4 in the
first place.

http://www.mail-archive.com/distutils-...@python.org/msg05775.html

And current hacks to work around lack of explicit version handling for
module import is a maintenance burden:

http://www.mail-archive.com/distutils-...@python.org/msg05742.html

setuptools has given the incentive to use versioning as a workaround
for API/ABI compatibility. That's the core problem, and most problems
brought by setuptools (sys.path and .pth hacks with the unreliability
which ensued) are consequences of this. I don't see how virtualenv
solves anything in that regard for deployment issues. I doubt using
things like virtualenv will make OS packagers happy.

David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.6 object.__init__ & deling __new__

[P.J. Eby]

At 07:29 PM 7/15/2009 +1200, Greg Ewing wrote:

P.J. Eby wrote:

In effect, 2.6 forces you to have a common known base class *other* 
than 'object' in order to write co-operative classes.  :-(


You have to do that anyway if you want to make cooperative
calls to any method *other* that __init__.


I haven't found that to be an issue, actually, since there's usually 
a base where the "bottom" version of those methods live, and anything 
implementing those methods is going to inherit it via the mixin.  The 
problem is that for at least __init__ and __new__, 'object' *is* the 
"bottom" -- and there may be more than one family of mixins that need 
to call it.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Paul Moore]

2009/7/15 David Cournapeau :
> As was stated by Debian packagers on the distutils ML, the problem is
> that docutils 0.5 breaks packages which work with docutils 0.4 in the
> first place.
>
> http://www.mail-archive.com/distutils-...@python.org/msg05775.html
>
> And current hacks to work around lack of explicit version handling for
> module import is a maintenance burden:
>
> http://www.mail-archive.com/distutils-...@python.org/msg05742.html
>
> setuptools has given the incentive to use versioning as a workaround
> for API/ABI compatibility. That's the core problem, and most problems
> brought by setuptools (sys.path and .pth hacks with the unreliability
> which ensued) are consequences of this. I don't see how virtualenv
> solves anything in that regard for deployment issues. I doubt using
> things like virtualenv will make OS packagers happy.

So, I think what you're saying is:

- The real issues is packages not maintaining backward compatibility (I agree)
- Setuptools is a workaround (I agree, at least it's *one* workaround)
- Virtualenv isn't a workaround (I don't know virtualenv, I'll take
your word for it)

Three points:

- When building *applications*, bundling everything (py2exe-style) is
an alternative workaround - universal on Windows, apparently uncommon
on Unix/Linux.
- When managing multiple packages in a "toolkit" style interactive
Python installation, I'm not aware of a good solution (other than
avoiding core that hits this issue in the first place).
- I do not believe that it's clear that sanctioning the setuptools
workaround as the "right" approach by building it into the Python
core/stdlib is the right thing to do.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

On Wed, Jul 15, 2009 at 5:16 PM, Joachim König wrote:
> Tarek Ziadé wrote:
>>
>> So basically "site-packages" is a distribution location that is
>> avoided by everyone because it doesn't
>> know how to handle multiple versions.
>
> I think you overrate the importance of having multiple versions of a package
> available
> for the same python interpreter. If you have m different versions of n
> packages then
> you could have n**m different combinations for an application so you need a
> possiblilty to select one combination from n**m possible ones at application
> startup time. Is this really worth it?

When you create an application that uses several libs, and when you
create your distribution,
you end up pinning a version for each one of your dependency to avoid
any problems.

While it's workable in a small application to list the dependencies in
a text file, and let your users
install them manually, it's impossible in bigger applications.

Applications based on Zope for example have **hundreds** of dependencies
you need to have installed, in very specific versions.

If you install a Zope 2 app on your system, and a Zope 3 one, you have
good chances of
breaking them because the "zope.schema" distribution is incompatible.

So either each application ships its own collection of dependencies
and ignore site-packages (that's what zope-based
applications does with zc.buildout), either they have a way to pick
the right version of each package.


>
>> If we had a multi-versions
>> support protocol, that would
>> help os packagers and application developers to be friends again imho ;)
>>  Let's remove site-packages from Python then.
>
> The _one_ site-packages folder stands for _one_ python interpreter. All the
> clever
> efforts to provide a set of package versions at runtime to an application
> (that uses
> the singleton python interpreter) do logically create a new python
> interpreter with
> a site-packages folder that contains just the versions of the packages the
> application
> needs, unfortunately by mucking with PYTHONPATH and .pth, site.py
> etc
> making it very difficult to understand what is happening for the joe average
> python developer.

That's what we have nowadays. python packages installed in different places,
and scripts tweaking the path to launch an application with them.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 11:10 AM 7/15/2009 +0100, Paul Moore wrote:

I propose that before the current prototype is turned into a final
(spec and) implementation, the PEP 302 extensions are extracted and
documented as an independent protocol, purely part of PEP 376. (This
*helps* implementers, as they can write support for, for example,
eggs, without needing to modify the existing egg importer).


Btw, this is why setuptools chose to implement these things as 
adapters or generic functions in pkg_resources (and in the vestigial 
bits that were added to python 2.5's pkgutil).


So as you can see, trying to solve these particular problems tends to 
lead to reinventing setuptools or at least portions thereof.  ;-)





But the
problem is that too many people dislike setuptools as it stands for it
to gain support. My understanding is that the current set of PEPs were
intended to be a stripped down, more generally acceptable subset of
setuptools. Let's keep them that way (and omit the complexities of
multi-version support).


Even without multi-version support, the parts of PEP 376 that aren't 
about uninstallation are still reinventing chunks of 
pkg_resources.  Had pkg_resources been in the stdlib a couple years 
back (note that bugs and changes in it are still quite rare), the PEP 
376 bits for pkgutil could have been focused strictly on 
uninstallation, and just reused pkg_resources API for finding 
distributions, reading metadata, getting version info, etc. etc.


All that stuff is extremely stable code, very widely used for a very 
long time.  If politics is the only thing keeping it from being used, 
then that's a pretty sad statement about the community, given that 
pkg_resources is not to blame for 99% of what people complain about 
in relation to setuptools.  All pkg_resources does is find stuff (on 
request) and maybe add it to sys.path (on request), pull out data and 
metadata from distributions (on request), or set up namespace 
packages (on request).


So if politics demands that it be rejected by association with 
"setuptools", then just search-and-replace the API, PEP 8-ify it, 
call it something different, and lie to everyone about where it came 
from.  I won't tell if you won't.  ;-)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 04:14 PM 7/15/2009 +0100, Paul Moore wrote:

Look - I really, really don't mind if people use setuptools. Honest.
But I do mind if core python gets changed to support little bits of
what setuptools does, adding complexity to deal with issues that
setuptools handles, but without making it possible to avoid using
setuptools.


I don't see how that's being proposed.  Nothing in PEP 376 requires 
setuptools or any portion thereof, AFAICT.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 05:16 PM 7/15/2009 +0200, Joachim König wrote:

f you have m different versions of n packages then
you could have n**m different combinations for an application so you need a
possiblilty to select one combination from n**m possible ones at application
startup time. Is this really worth it?


Obviously yes, as neither buildout nor setuptools would exist otherwise.  ;-)
Nor would Fedora be packaging certain library versions as eggs 
specifically to get certain multi-version scenarios to work.


The specific solutions for handling n*m problems aren't fantastic, 
but they are clearly needed.


(Buildout, btw, actually hardwires the n*m choice at install time, 
which is probably better for production situations than setuptools' 
dynamic approach.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 04:59 PM 7/15/2009 +0100, Paul Moore wrote:

- Virtualenv isn't a workaround (I don't know virtualenv, I'll take
your word for it)


It's not one for system package maintainers because it would 
effectively be managing multiple instances of 'python'.  Really not a 
suitable solution.




- I do not believe that it's clear that sanctioning the setuptools
workaround as the "right" approach by building it into the Python
core/stdlib is the right thing to do.


I still don't understand how we're doing that.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[R. David Murray]

On Wed, 15 Jul 2009 at 16:14, Paul Moore wrote:

Bluntly, as Python stands, import and sys.path do not offer any core
support for multiple versions. Custom solutions can be built on top of
that - that's what setuptools does. But they are precisely that -
custom solutions, and should be supported as such, outside the core
(and stdlib).

If standard Python support for multi-version imports is required (it's
not by me, but I accept that some people want it), then it should be
designed in thoughout:


Isn't this problem directly analogous to the '.so version' (*) problem?
Can we learn anything from the solution to that problem?  Does not the
fact that a (standard) solution to that problem was required indicate
that a similar solution needs to be provided by core Python?  (And yes,
it's out of scope for PEP 376).

--David

(*) or, for our Windows users, DLL Hell and the Side By Side Component
Sharing solution...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

2009/7/15 P.J. Eby :
[...]
> So if politics demands that it be rejected by association with "setuptools",
> then just search-and-replace the API, PEP 8-ify it, call it something
> different, and lie to everyone about where it came from.  I won't tell if
> you won't.  ;-)

While setuptools did address some issues, we are now adressing in PEP 376,
it does a *lot* of other things and it's impossible to extract the
features we would
want to re-use. e.g. what I would call the good bits.

The prototype code we are building on bitbucket is clear, easy to
understand, read and adopt and already PEP8-fied (which is required
for sdtlib inclusion)

Reading and understanding pkg_resources requires a lot of time.

A side note on setuptools, I don't mean to be rude here or to tell you
what you should do,
really, but:

  While I am very glad that you are part of PEP 376 discussions, and
that you helped
  on almost every section of it, I really don't understand why you
keep on pushing setuptools
  *code* here and at distutils-SIG since you don't maintain it anymore
(the last commit is > 9 months)

  Part of the rejection of setuptools is because of that and because you don't
  bless anyone to maintain the project code base, or do releases,
neither to communicate clearly
  on what's its roadmap.

Regards
Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - from PyPM's point of view

[Sridhar Ratnakumar]

On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziadé   
wrote:



get_installed_files(local=False) -> iterator of (path, md5, size)


Will this also return the directories /created/ during the installation?
For example, will it also contain the entry "docutils" .. along with
"docutils/__init__.py"?


I don't think it's necessary to add "docutils" if
"docutils/__init__.py" is present

But for empty directories added during installation we should add the I  
guess.


So, I'll add a note.


It seems that you overlooked the below paragraph.


If not, how is the installer (pip, pypm, etc..) supposed to know which
directories to remove (docutils/) and which directories not to remove
(site-packages/, bin/, etc..)?


Quoting from the PEP:

[quote]'(...)uninstall uses the APIs described earlier and remove all  
unique files, as long as their hash didn't change. Then it removes empty  
directories left behind.'[endquote]


Let's assume that site-packages/ contained only one package 'Foo'. Will  
uninstall('Foo') remove the site-packages/ directory just because it  
turned out to be empty after removing 'Foo'? To explain, let's assume the  
RECORD of 'Foo' contains:


$ cat RECORD
Foo/__init__.py
Foo/bar/__init__.py
Foo/bar/test.py

and according to what you wrote in the PEP ("it removes empty directories  
left behind"):


$ python -m distutils.util.uninstall Foo
rm /.../site-packages/Foo/__init__.py
rm /.../site-packages/Foo/bar/__init__.py
rm /.../site-packages/Foo/bar/test.py
rm empty dir /.../site-packages/Foo/bar
rm empty dir /.../site-packages/Foo/
rm empty dir /.../site-packages/# !

it also remove the site-packages directory!

Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do  
you see my point?



btw: is PyPM a public project ?


If by 'public', you meant open source, then no.

-srid
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Paul Moore]

2009/7/15 P.J. Eby :
> At 11:10 AM 7/15/2009 +0100, Paul Moore wrote:
>>
>> I propose that before the current prototype is turned into a final
>> (spec and) implementation, the PEP 302 extensions are extracted and
>> documented as an independent protocol, purely part of PEP 376. (This
>> *helps* implementers, as they can write support for, for example,
>> eggs, without needing to modify the existing egg importer).
>
> Btw, this is why setuptools chose to implement these things as adapters or
> generic functions in pkg_resources (and in the vestigial bits that were
> added to python 2.5's pkgutil).
>
> So as you can see, trying to solve these particular problems tends to lead
> to reinventing setuptools or at least portions thereof.  ;-)

Yes, that's the impression I'm getting :-)

And yet you seem resistant to reading any message into the fact that
people find it easier to solve the problems from scratch than to
extract the relevant parts of the existing code. To be explicit - it's
too flipping hard to identify, extract and tidy up the appropriate
bits of the setuptools code! (Disclaimer: I haven't looked at the code
for some time - if the style has changed utterly in the last 2 years
or so, so that it no longer resembles the style of the pkgutil code
you contributed to the stdlib, I'll retract that criticism of the
setuptools codebase).

>> But the
>> problem is that too many people dislike setuptools as it stands for it
>> to gain support. My understanding is that the current set of PEPs were
>> intended to be a stripped down, more generally acceptable subset of
>> setuptools. Let's keep them that way (and omit the complexities of
>> multi-version support).
>
> Even without multi-version support, the parts of PEP 376 that aren't about
> uninstallation are still reinventing chunks of pkg_resources.  Had
> pkg_resources been in the stdlib a couple years back (note that bugs and
> changes in it are still quite rare), the PEP 376 bits for pkgutil could have
> been focused strictly on uninstallation, and just reused pkg_resources API
> for finding distributions, reading metadata, getting version info, etc. etc.
>
> All that stuff is extremely stable code, very widely used for a very long
> time.  If politics is the only thing keeping it from being used, then that's
> a pretty sad statement about the community, given that pkg_resources is not
> to blame for 99% of what people complain about in relation to setuptools.
>  All pkg_resources does is find stuff (on request) and maybe add it to
> sys.path (on request), pull out data and metadata from distributions (on
> request), or set up namespace packages (on request).

I don't think it's quite "politics" as you describe it. Maybe
"marketing", if you want a simple tag for it.

The problem is that the setuptools "brand name" is associated with a
lot of baggage, that maybe it doesn't deserve, but equally no-one
seems interested in addressing. It's certainly not a technical
argument at this point. And yet you counter criticisms with technical
responses. But if I recall the history correctly, one of the initial
goals of setuptools was to become a de-facto standard precisely by
marketing and filling the ecological niche - a goal it has achieved
fantastically well. The fact that in doing so it alienated certain
groups is a more or less inevitable consequence.

My argument - which you may well disagree with - is that the
"setuptools ecosystem" (by which I mean
setuptools/easy_install/pkg_resources/eggs and the various subsystems
around these) is documented in a way which is:

- confusing to the casual user
- not at all modular (ie, it's impossible to easily pick out individual "bits")
- full of new concepts, so it's hard to find an easy place to start from

There's not much technically at issue here, it's all about presentation.

> So if politics demands that it be rejected by association with "setuptools",
> then just search-and-replace the API, PEP 8-ify it, call it something
> different, and lie to everyone about where it came from.  I won't tell if
> you won't.  ;-)

Unless the documentation and code is cleaned up, it'll still be
obvious from the style where it came from - that's the problem. Who's
going to maintain the code within the standard library, and are they
happy with the style of the existing code - even with the serial
numbers filed off?

And of course, someone has to do the clean-up. It seems to me that the
fact that people are more inclined to reinvent the code than to try to
understand the existing codebase and pick out the relevant bits, says
something important about how easy it would be to maintain the
existing code within the Python core...

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - from PyPM's point of view

[Terry Reedy]

Tarek Ziadé wrote:


In any case I don't see any use case to have a "site-packages"
remaining in Python itself.


I have and am using it.  Where else would you have me put library 
packages meant to be accessible by any Python program?


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Sridhar Ratnakumar]

On Wed, 15 Jul 2009 08:22:03 -0700, David Cournapeau   
wrote:



if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is
shipped with it.

As was stated by Debian packagers on the distutils ML, the problem is
that docutils 0.5 breaks packages which work with docutils 0.4 in the
first place.


Thus I am -1 on multi-version support .. and would rather have the python  
developers make their packages backward compatible just like what Armin  
did with Jinja and Jinja2. Debian at the moment has only one package  
"python-docutils" which is 0.5. How is a debian application supposed to  
depend upon 0.4? With Jinja, there is no problem .. there are  
'python-jinja' and 'python-jinja2'. Similarly, CherryPy should have gone  
with renaming their package names to CherryPy2 and CherryPy3.


-srid


PS: Quoting from a fellow developer:


> [...] it sounds like CherryPy 3.x is really an incompatible
> module that just kept the old name. That is rather unfortunate, but can
> sometimes happen.
 I have never seen a Python package changing its name (when significantly
changing API, design, etc..). The only exception that comes to my mind
is Jinja2 (renamed from 'Jinja'):
 [Armin] (...) Because we love backwards compatibility these changes will
go into a package called "jinja2"



Well, congrats to the Jinja team then!  The others will eventually learn...
Mixing incompatible APIs in a single namespace and using a non-standardized
version numbering system to keep things apart creates a world of pain!


The challenge however is in compartmentalizing versions according to
their major release numbers. Versioning in the Python world is already a
mess which we are beginning to sort out:
http://wiki.python.org/moin/Distutils/VersionComparison (PyPM relies on
this standard and the ongoing implementation - verlib.py)


How embarrassing for a cult that prides itself on having only one way for
everything they do...   CPAN versions numbers are just as much if not
more of a mess, but at least you can argue that it is the price for there
being "more than one way to do it".
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 07:07 PM 7/15/2009 +0200, Tarek Ziadé wrote:
  Part of the rejection of setuptools is because of that and 
because you don't

  bless anyone to maintain the project code base, or do releases,
neither to communicate clearly
  on what's its roadmap.


Jim Fulton and Ian Bicking have been officially blessed for *years*, 
and I've posted plenty about what's on its roadmap.




I really don't understand why you keep on pushing setuptools *code*


pkg_resources is not setuptools; its stability and revision history 
are very different, as it addresses different problems.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[P.J. Eby]

At 06:40 PM 7/15/2009 +0100, Paul Moore wrote:

And of course, someone has to do the clean-up. It seems to me that the
fact that people are more inclined to reinvent the code than to try to
understand the existing codebase and pick out the relevant bits, says
something important about how easy it would be to maintain the
existing code within the Python core...


That's normal for any code that contains "legacy" issues, which is 
why people always prefer rewriting code they don't understand: it's 
more fun to write than to read.  However, as Joel Spolsky has well 
explained, rewriting such code inevitably means that you must 
re-learn the lessons that were learned by the original author.


It seems like a waste, but then, I suppose those lessons must be 
relearned *some* way.  I just think it'd be better if, having 
re-learned most of the lessons by trying to rewrite, one could then 
go back and learn the rest from the code.  ;-)


That having been said, it's obviously a dead parrot... one which I 
will now cease attempting to revive.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Chris McDonough]

I've been trying to follow this discussion now for weeks. The signal to noise 
ratio is pretty low.


I'd love to have an stdlib solution for distribution packaging and installation. 
 But I think we might as well pack it up and go home if the folks whom are 
contributing to the discussion "recreationally" (whom are not themselves 
implementers and potential implementers or spec writers or potential spec 
writers of packaging systems) continue to chime in on *every single issue*, 
contributing only stop energy.  It's just completely pointless.


- C


On 7/15/09 12:34 PM, P.J. Eby wrote:

At 04:59 PM 7/15/2009 +0100, Paul Moore wrote:

- Virtualenv isn't a workaround (I don't know virtualenv, I'll take
your word for it)


It's not one for system package maintainers because it would effectively
be managing multiple instances of 'python'. Really not a suitable solution.



- I do not believe that it's clear that sanctioning the setuptools
workaround as the "right" approach by building it into the Python
core/stdlib is the right thing to do.


I still don't understand how we're doing that.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/lists%40plope.com



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Brett Cannon]

On Wed, Jul 15, 2009 at 06:51, Nick Coghlan  wrote:

> Antoine Pitrou wrote:
> > I am not sure when this discussion started. Are you replying to a 3
> month-old
> > message of yours? :)
>
> That depends on how you define the beginning of the discussion...
>

Especially since I was offlist three months ago. =)


>
> However, the fact that importlib doesn't implement the comparatively
> recent get_filename() optional extension documented in PEP 302 came up
> in one of the PEP 376 threads within the last week or so.
>

I have been ignoring the PEP 372 discussions since I came into it late and I
don't deal with packaging enough to want to influence the discussion, so I
missed that point being made.


>
> > In any case, keeping all import-related ABCs in a single place sounds
> like a
> > good idea.
> > Alternatively, if the method is runpy specific, it doesn't have its place
> in an
> > ABC, does it?
>
> While runpy is the only client in the standard library for the
> get_filename() method, the method is still a PEP 302 extension. I
> documented the extension in the PEP as loaders are the only things
> reliably in a position to provide the filename details that runpy needs
> to set __file__ and sys.argv[0] correctly and until importlib came along
> PEP 302 itself was the only real documentation of that API.
>
> Since importlib is now the "go-to" location for people that want to
> write their own PEP 302 importers and loaders, I would say that it is
> also the right place for the new ExecutionLoader ABC.


OK. I will add an ExecutionLoader ABC to importlib that simply subclasses
InspectLoader and adds the get_filename() abstract method. Thanks to
everyone for their feedback.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mercurial: tag generation incorrect

[Martin v. Löwis]

Hagen Fürstenau wrote:
>> be32850b093f is listed
>> as having a child revision, 52b0a279fec6, and ISTM that *this*
>> should be the revision that got tagged.
> 
> I think the tag is correct. Note that the concept of tagging is
> different in Mercurial, where a tag can only refer to a revision
> previous to the one where it is inserted in .hgtags. If I understand
> correctly, all relevant tagging revisions from SVN are replaced by
> Mercurial revisions setting tags, which then refer to their immediate
> predecessors.

Ah, ok. Thanks for the explanation.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy?

[Nick Coghlan]

Brett Cannon wrote:
> On Wed, Jul 15, 2009 at 06:51, Nick Coghlan  > wrote:
> However, the fact that importlib doesn't implement the comparatively
> recent get_filename() optional extension documented in PEP 302 came up
> in one of the PEP 376 threads within the last week or so.
> 
> I have been ignoring the PEP 372 discussions since I came into it late
> and I don't deal with packaging enough to want to influence the
> discussion, so I missed that point being made.

Since I'm pretty sure you were the one that mentioned it, it's possible
I just misremembered which thread it came up in. The various discussions
can blur together sometimes!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view

[Paul Moore]

2009/7/15 Chris McDonough :
> I've been trying to follow this discussion now for weeks. The signal to
> noise ratio is pretty low.

I agree :-(

> I'd love to have an stdlib solution for distribution packaging and
> installation.  But I think we might as well pack it up and go home if the
> folks whom are contributing to the discussion "recreationally" (whom are not
> themselves implementers and potential implementers or spec writers or
> potential spec writers of packaging systems) continue to chime in on *every
> single issue*, contributing only stop energy.  It's just completely
> pointless.

I thought that as someone who is writing code for Tarek's PEP 376
prototype implementation, I counted as offering useful input. Thinking
further on what I've said, I'm no longer sure that's true.

I do think that at *some* point, the feedback from the non-packagers,
negative or irrelevant though it might be, needs to be requested and
considered. Maybe now isn't the right time, but it needs to be done at
some point.

Until then, I'll try to avoid adding to the noise.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 376 - from pythonpkgmgr point of view

[David Lyon]

On Wed, 15 Jul 2009 13:47:35 -0400, Chris McDonough 
wrote:
> I've been trying to follow this discussion now for weeks. The signal to
> noise ratio is pretty low.

I'm -1 on that..

As a relative newcomer to python packaging I'm finding all these
discussions very informative. :-)

> I'd love to have an stdlib solution for distribution packaging and
> installation.

That's right.

I'd love to find out what the process is for submitting my pythonpkgmgr
project for consideration into the stdlib.

As it fills a huge gap that you've just identified that currently exists 
within the python that is being shipped today. That is, allowing users to 
easily install packages from pypi.

>   But I think we might as well pack it up and go home if the folks whom
are
> contributing to the discussion "recreationally" (whom are not themselves 
> implementers and potential implementers or spec writers or potential spec

> writers of packaging systems) continue to chime in on *every single
issue*,
> 
> contributing only stop energy.  It's just completely pointless.

We can all work in different ways..

I'm not writing PEPS and don't care too much about how the internals of
a package work. As long as there are APIs for such stuff I'm happy.

What I'm working on is to try to improve the user experience which
isn't so handled so easily by a design by commitee process. 

Packaging and Distutils, need to be looked at wholistically as
well. It's not just about the api's, but how well they work.

It's not noise it's just the process

ps:

(http://peak.telecommunity.com/DevCenter/EasyInstall)

"Easy Install
...
Please share your experiences with us! If you encounter difficulty
installing a package, please contact us via the distutils mailing list.
(Note: please DO NOT send private email directly to the author of
setuptools; it will be discarded. "

So what are we to do if we find issues with setuptools other than to post
to distutils?

David


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 376 - from PyPM's point of view

[Tarek Ziadé]

2009/7/15 Sridhar Ratnakumar :
> On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziadé 
> wrote:
>
 get_installed_files(local=False) -> iterator of (path, md5, size)
>>>
>>> Will this also return the directories /created/ during the installation?
>>> For example, will it also contain the entry "docutils" .. along with
>>> "docutils/__init__.py"?
>>
>> I don't think it's necessary to add "docutils" if
>> "docutils/__init__.py" is present
>>
>> But for empty directories added during installation we should add the I
>> guess.
>>
>> So, I'll add a note.
>
> It seems that you overlooked the below paragraph.
>
>>> If not, how is the installer (pip, pypm, etc..) supposed to know which
>>> directories to remove (docutils/) and which directories not to remove
>>> (site-packages/, bin/, etc..)?
>
> Quoting from the PEP:
>
> [quote]'(...)uninstall uses the APIs described earlier and remove all unique
> files, as long as their hash didn't change. Then it removes empty
> directories left behind.'[endquote]
>
> Let's assume that site-packages/ contained only one package 'Foo'. Will
> uninstall('Foo') remove the site-packages/ directory just because it turned
> out to be empty after removing 'Foo'? To explain, let's assume the RECORD of
> 'Foo' contains:
>
> $ cat RECORD
> Foo/__init__.py
> Foo/bar/__init__.py
> Foo/bar/test.py
>
> and according to what you wrote in the PEP ("it removes empty directories
> left behind"):
>
> $ python -m distutils.util.uninstall Foo
> rm /.../site-packages/Foo/__init__.py
> rm /.../site-packages/Foo/bar/__init__.py
> rm /.../site-packages/Foo/bar/test.py
> rm empty dir /.../site-packages/Foo/bar
> rm empty dir /.../site-packages/Foo/
> rm empty dir /.../site-packages/        # !
>
> it also remove the site-packages directory!
>
> Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do
> you see my point?

I didn't mean that of course. While we can avoid your example for the
code, by removing only
packages that are fully emptied, and are alongside the egg-info directory,
we might not be able to do it properly for the data.

So let's add the directories as well
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com