Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Greg Ewing wrote:
> Nick Coghlan wrote:
>> I'd say the answer to where we put it will be dependent on what happens to 
>> the 
>> idea of adding a NumArray style fixed dimension array type to the standard 
>> library. If that gets exposed through the array module as array.dimarray, 
>> then 
>> it would make sense to expose the associated data layout descriptors as 
>> array.datatype.
> 
> Seem to me that arrays are a sub-concept of binary data,
> not the other way around. So maybe both arrays and data
> types should be in a module called 'binary' or some such.

Yes, very good point.

That's probably one reason I'm proposing the data-type first before the 
array interface in the extended buffer protocol.

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Greg Ewing wrote:
> Travis E. Oliphant wrote:
> 
>> The 'kind' does not specify how "big" the data-type (data-format) is.
> 
> What exactly does "bit" mean in that context?   

Do you mean "big" ?  It's how many bytes the kind is using.

So, 'u4' is a 4-byte unsigned integer and 'u2' is a 2-byte unsigned 
integer.


-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Greg Ewing wrote:
> Nick Coghlan wrote:
> 
>> Greg Ewing wrote:
> 
>>> Also, what if I want to refer to fields by name
>>> but don't want to have to work out all the offsets
> 
>> Use the list definition form. With the changes I've 
>> suggested above, you wouldn't even have to name the fields you don't 
>> care about - just describe them.
> 
> That would be okay.
> 
> I still don't see a strong justification for having a
> one-big-string form as well as a list/tuple/dict form,
> though.

Compaction of representation is all. It's used quite a bit in numarray, 
   which is where most of the 'kind' names came from as well.   When you 
don't want to name fields it is a really nice feature (but it doesn't 
nest well).

-Travis


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Travis E. Oliphant schrieb:
> What is needed is a definitive way to describe data and then have
> 
> array
> struct
> ctypes
> 
> all be compatible with that same method.That's why I'm proposing the 
> PEP.  It's a unification effort not yet-another-method.

As I unification mechanism, I think it is insufficient. I doubt it
can express all the concepts that ctypes supports.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Greg Ewing wrote:
> Travis E. Oliphant wrote:
> 
>> How to handle unicode data-formats could definitely be improved. 
>> Suggestions are welcome.
> 
> 'U4*10'  string of 10 4-byte Unicode chars
> 

I like that.  Thanks.

-Travis


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Robert Kern]

Martin v. Löwis wrote:
> Travis E. Oliphant schrieb:
>> What is needed is a definitive way to describe data and then have
>>
>> array
>> struct
>> ctypes
>>
>> all be compatible with that same method.That's why I'm proposing the 
>> PEP.  It's a unification effort not yet-another-method.
> 
> As I unification mechanism, I think it is insufficient. I doubt it
> can express all the concepts that ctypes supports.

What do you think is missing that can't be added?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Travis E. Oliphant schrieb:
> How to handle unicode data-formats could definitely be improved. 

As before, I'm doubtful what the actual needs are. For example, is
it desired to support generation of ID3v2 tags with such a data
format? The tag is specified here:

http://www.id3.org/id3v2.4.0-structure.txt

In ID3v1, text fields have a specified width, and are supposed
to be encoded in Latin-1, and padded with zero bytes.

In ID3v2, text fields start with an encoding declaration
(say, \x03 for UTF-8), then followed with a null-terminated
sequence of UTF-8 bytes.

Is it the intent of this PEP to support such data structures,
and allow the user to fill in a Unicode object, and then the
processing is automatic? (i.e. in ID3v1, the string gets
automatically Latin-1-encoded and zero-padded, in ID3v2, it
gets automatically UTF-8 encoded, and null-terminated)

If that is not to be supported, what are the use cases?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Martin v. Löwis wrote:
> Travis E. Oliphant schrieb:
>> What is needed is a definitive way to describe data and then have
>>
>> array
>> struct
>> ctypes
>>
>> all be compatible with that same method.That's why I'm proposing the 
>> PEP.  It's a unification effort not yet-another-method.
> 
> As I unification mechanism, I think it is insufficient. I doubt it
> can express all the concepts that ctypes supports.
> 

Please clarify what you mean.

Are you saying that a single object can't carry all the information 
about binary data that ctypes allows with it's multi-object approach?

I don't agree with you, if that is the case.  Sure, perhaps I've not 
included certain cases, so give an example.

Besides, I don't think this is the right view of "unification".  I'm not 
saying that ctypes should get rid of it's many objects used for 
interfacing with C-functions.

I'm saying we should introduce a single-object mechanism for describing 
binary data so that the many-object approach of c-types does not become 
some kind of de-facto standard.  C-types can "translate" this 
object-instance to its internals if and when it needs to.

In the mean-time, how are other packages supposed to communicate binary 
information about data with each other?

Remember the context that the data-format object is presented in.  Two 
packages need to share a chunk of memory (the package authors do not 
know each other and only have and Python as a common reference).  They 
both want to describe that the memory they are sharing has some 
underlying binary structure.

How do they do that? Please explain to me how the buffer protocol can be 
extended so that information about "what is in the memory" can be shared 
without a data-format object?

-Travis


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Travis E. Oliphant]

Martin v. Löwis wrote:
> Travis E. Oliphant schrieb:
>> How to handle unicode data-formats could definitely be improved. 
> 
> As before, I'm doubtful what the actual needs are. For example, is
> it desired to support generation of ID3v2 tags with such a data
> format? The tag is specified here:
> 

Perhaps I was not clear enough about what I'm try to do.   For a long 
time a lot of people have wanted something like Numeric in Python 
itself.  There have been many hurdles to that goal.

After discussions at SciPy 2006 with Guido, we decided that the best way 
to proceed at this point was to extend the buffer protocol to allow 
packages to share array-like information with each-other.

There are several things missing from the buffer protocol that NumPy 
needs in order to be able to really understand the (fixed-size) memory 
another package has allocated and is sharing.

The most important of these is

1) Shape information
2) Striding information
3) Data-format information  (how is each element perceived).

Shape and striding information can be shared with a C-array of integers.

How is data-format information supposed to be shared?

We've come up with a very flexible way to do this in NumPy using a 
single Python object.  This Python object supports describing the layout 
of any fixed-size chunk of memory (right now in units of bytes --- bit 
fields could be added, though).

I'm proposing to add this object to Python so that the buffer protcol 
has a fast and efficient way to share #3.   That's really all I'm after.

It also bothers me that so many ways to describe binary data are being 
used out there.  This is a problem that deserves being solved.  And, no, 
ctypes hasn't solved it (we can't directly use the ctypes solution). 
Perhaps this PEP doesn't hit all the corners, but a data-format object 
*is* a useful thing to consider.

The array object in Python already has a PyArray_Descr * structure that 
is a watered-down version of what I'm talking about.   In fact, this is 
what Numeric built from (or vice-versa actually).  And NumPy has greatly 
enhanced this object for any conceivable structure.

Guido seemed to think the data-type objects were nice when he saw them 
at SciPy 2006, and so I'm presenting a PEP.

Without the data-format object, I'm don't know how to extend the buffer 
protocol to communicate data-format information.  Do you have a better 
idea?

I have no trouble limiting the data-type object to the buffer protocol 
extension PEP, but I do think it could gain wider use.

> 
> Is it the intent of this PEP to support such data structures,
> and allow the user to fill in a Unicode object, and then the
> processing is automatic? (i.e. in ID3v1, the string gets
> automatically Latin-1-encoded and zero-padded, in ID3v2, it
> gets automatically UTF-8 encoded, and null-terminated)
>

No, the point of the data-format object is to communicate information 
about data-formats not to encode or decode anything.   Users of the 
data-format object could decide what they wanted to do with that 
information.   We just need a standard way to communicate it through the 
buffer protocol.

-Travis

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Travis E. Oliphant schrieb:
> I'm proposing to add this object to Python so that the buffer protcol 
> has a fast and efficient way to share #3.   That's really all I'm after.

I admit that I don't understand this objective. Why is it desirable to
support such an extended buffer protocol? What specific application
would be made possible if it was available and implemented in the
relevant modules and data types? What are the relevant modules and data
types that should implement it?

> It also bothers me that so many ways to describe binary data are being 
> used out there.  This is a problem that deserves being solved.  And, no, 
> ctypes hasn't solved it (we can't directly use the ctypes solution). 
> Perhaps this PEP doesn't hit all the corners, but a data-format object 
> *is* a useful thing to consider.

IMO, it is only useful if it realistically can support all the use cases
that it intends to support. If this PEP is about defining the elements
of arrays, I doubt it can realistically support everything you can
express in ctypes. There is no support for pointers (except for
PyObject*), no support for incomplete (recursive) types, no support
for function pointers, etc.

Vice versa: why exactly can't you use the data type system of ctypes?
If I want to say "int[10]", I do

py> ctypes.c_long * 10


To rewrite the examples from the PEP:

datatype(float) => ctypes.c_double
datatype(int)   => ctypes.c_long
datatype((int, 5)) => ctypes.c_long * 5
datatype((float, (3,2)) => (ctypes.c_double * 3) * 2

struct {
  int  simple;
  struct nested {
   char name[30];
   char addr[45];
   int  amount;
  }
=>
py> from ctypes import *
py> class nested(Structure):
...  _fields_ = [("name", c_char*30), ("addr", c_char*45), ("amount",
c_long)]
...
py> class struct(Structure):
...   _fields_ = [("simple", c_int), ("nested", nested)]
...

> Guido seemed to think the data-type objects were nice when he saw them 
> at SciPy 2006, and so I'm presenting a PEP.

I have no objection to including NumArray as-is into Python. I just
wonder were the rationale for this PEP comes from, i.e. why do you
need to exchange this information across different modules?

> Without the data-format object, I'm don't know how to extend the buffer 
> protocol to communicate data-format information.  Do you have a better 
> idea?

See above: I can't understand where the need for an extended buffer
protocol comes from. I can see why NumArray needs reflection, and
needs to keep information to interpret the bytes in the array.
But why is it important that the same information is exposed by
other data types?

>> Is it the intent of this PEP to support such data structures,
>> and allow the user to fill in a Unicode object, and then the
>> processing is automatic? (i.e. in ID3v1, the string gets
>> automatically Latin-1-encoded and zero-padded, in ID3v2, it
>> gets automatically UTF-8 encoded, and null-terminated)
>>
> 
> No, the point of the data-format object is to communicate information 
> about data-formats not to encode or decode anything.   Users of the 
> data-format object could decide what they wanted to do with that 
> information.   We just need a standard way to communicate it through the 
> buffer protocol.

This was actually a different sub-thread: why do you need to support
the 'U' code (or the 'S' code, for that matter)? In what application
do you have fixed size Unicode arrays, as opposed to Unicode strings?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Travis E. Oliphant schrieb:
>> As I unification mechanism, I think it is insufficient. I doubt it
>> can express all the concepts that ctypes supports.
>>
> 
> Please clarify what you mean.
> 
> Are you saying that a single object can't carry all the information 
> about binary data that ctypes allows with it's multi-object approach?

I'm not sure what you mean by "single object". If I use the tuple
syntax, e.g.

datatype((float, (3,2))

There are also multiple objects (the float, the 3, and the 2). You
get a single "root" object back, but so do you in ctypes.

But this isn't really what I meant. Instead, I think the PEP lacks
various concepts from C data types, such as pointers, unions,
function pointers, alignment/packing.

> In the mean-time, how are other packages supposed to communicate binary 
> information about data with each other?

This is my other question. Why should they?

> Remember the context that the data-format object is presented in.  Two 
> packages need to share a chunk of memory (the package authors do not 
> know each other and only have and Python as a common reference).  They 
> both want to describe that the memory they are sharing has some 
> underlying binary structure.

Can you please give an example of such two packages, and an application
that needs them share data?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Robert Kern schrieb:
>> As I unification mechanism, I think it is insufficient. I doubt it
>> can express all the concepts that ctypes supports.
> 
> What do you think is missing that can't be added?

I can factually only report what is missing. Whether it can be added,
I don't know. As I just wrote in a few other messages: pointers,
unions, functions pointers, packed structs, incomplete/recursive
types. Also "flexible array members" (i.e. open-ended arrays).

While it may be possible to come up with a string syntax to describe
all these things (*), I wonder whether it should be done, and whether
NumArray can then support this extended data model.

Regards,
Martin

(*) perhaps with the exception of incomplete types: C needs forward
references in its own syntax.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] build bots, log output

[Anthony Baxter]

On Saturday 28 October 2006 23:39, Georg Brandl wrote:
> Hi,
>
> I wonder if it's possible that the build bot notification mails that go
> to python-checkins include the last 10-15 lines from the log. This would
> make it much easier to decide whether a buildbot failure is an old,
> esoteric one (e.g.

A better solution (awaiting sufficient round-tuits) would be to add an option 
to regrtest that's used by the buildslaves that uses particularly markup 
around success/fail indications. The buildmaster can pick those up, and keep 
track of existing pass/fails. Then it could send an email only when one 
changes. We might also add a daily or every couple of days reminder 
saying "The following tests are failing on the following platforms, and have 
been for X days now". 

Buildmaster code is on dinsdale, in (I think) ~buildbot. It's also in SVN.

This solution doesn't require changes to the buildslave code at all - only to 
the buildmaster and to regrtest.


-- 
Anthony Baxter <[EMAIL PROTECTED]>
It's never too late to have a happy childhood.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyCon: proposals due by Tuesday 10/31

[A.M. Kuchling]

Final reminder: if you want to submit a proposal to PyCon, you should
do it by end of Tuesday, October 31st.

 for more info

The deadline for tutorials is November 15th:

http://us.pycon.org/TX2007/CallForTutorials

PyCon is the Python community conference, held next February 23-25
near Dallas; a tutorial day will be on February 22.  See
http://us.pycon.org/ for more info.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Edward C. Jones]

Travis E. Oliphant wrote:
 > It also bothers me that so many ways to describe binary data are
 > being used out there.  This is a problem that deserves being solved.

Is there a survey paper somewhere about binary formats? What formats are 
used in particle physics, bio-informatics, astronomy, etc? What software 
is used to read and write binary data? What descriptive languages are 
used for data (SQL, XML, etc)?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Neal Becker]

I have watched numpy with interest for a long time.  My own interest is to
possibly use the c-api to wrap c++ algorithms to use from python.

One thing that has concerned me, and continues to concern me with this
proposal, is that it seems to suffer from a very fat interface.  I
certainly have not studied the options in any depth, but my gut feeling is
that the interface is too fat and too complex.  I wonder if it's possible
to avoid this.  I wonder if this is an example of all the methods sinking
to the base class.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Paul Moore]

On 10/29/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Travis E. Oliphant schrieb:
> > Remember the context that the data-format object is presented in.  Two
> > packages need to share a chunk of memory (the package authors do not
> > know each other and only have and Python as a common reference).  They
> > both want to describe that the memory they are sharing has some
> > underlying binary structure.
>
> Can you please give an example of such two packages, and an application
> that needs them share data?

Here's an example. PIL handles images (in various formats) in memory,
as blocks of binary image data. NumPy provides methods for
manipulating in-memory blocks of data. Now, if I want to use NumPy to
manipulate that data in place (for example, to cap the red component
at 128, and equalise the range of the green component) my code needs
to know the format of the memory block that PIL exposes. I am assuming
that in-place manipulation is better, because there is no need for
repeated copies of the data to be made (this would be true for large
images).

If PIL could expose a descriptor for its data structure, NumPy code
could manipulate it in place without fear of corrupting it. Of course,
this can be done by the end user reading the PIL documentation and
transcribing the documented format into the NumPy code. But I would
argue that it's better if the PIL block is self-describing in a way
that avoids the need for a manual transcription of the format.

To do this *without* needing the PIL and NumPy developers to
co-operate needs an independent standard, which is what I assume this
PEP is intended to provide.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Josiah Carlson]

"Paul Moore" <[EMAIL PROTECTED]> wrote:
> On 10/29/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > Travis E. Oliphant schrieb:
> > > Remember the context that the data-format object is presented in.  Two
> > > packages need to share a chunk of memory (the package authors do not
> > > know each other and only have and Python as a common reference).  They
> > > both want to describe that the memory they are sharing has some
> > > underlying binary structure.
> >
> > Can you please give an example of such two packages, and an application
> > that needs them share data?
> 
> To do this *without* needing the PIL and NumPy developers to
> co-operate needs an independent standard, which is what I assume this
> PEP is intended to provide.

One could also toss wxPython, VTK, or any one of the other GUI libraries
into the mix for visualizing those images, of which wxPython just
acquired no-copy display of PIL images, and being able to manipulate
them with numpy (of which some wxPython built in classes use numpy to
speed up manipulation) would be very useful.

Of all of the intended uses, I'd say that zero-copy sharing of
information on the graphics/visualization front is the most immediate
'people will be using it tomorrow' feature.


I personally don't have my pulse on the Scientific Python community, so
I don't know about other uses, but in regards to Martin's list of
missing features: "pointers, unions, function pointers,
alignment/packing [, etc.]" I'm going to go out on a limb and say for
the majority of those YAGNI, or really, NOHAFIAFACT (no one has asked
for it, as far as I can tell).  Someone who knows the scipy community,
feel free to correct me.


 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] build bots, log output

[Martin v. Löwis]

Anthony Baxter schrieb:
> A better solution (awaiting sufficient round-tuits) would be to add an option 
> to regrtest that's used by the buildslaves that uses particularly markup 
> around success/fail indications. The buildmaster can pick those up, and keep 
> track of existing pass/fails. Then it could send an email only when one 
> changes. We might also add a daily or every couple of days reminder 
> saying "The following tests are failing on the following platforms, and have 
> been for X days now". 

As yet another alternative, we could put the names of the builders on
which builds are expected to fail (or the system names of these systems)
into the test cases, and then report "expected failures"; regrtest would
give a "success" status if all failures are expected.

The consequence would be that these systems would appear "green" on
the buildbot page, and you'd have to look into the log file to find
out which of the expected failures actually happened. This all could
work without changes to buildbot at all.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Paul Moore schrieb:
> Here's an example. PIL handles images (in various formats) in memory,
> as blocks of binary image data. NumPy provides methods for
> manipulating in-memory blocks of data. Now, if I want to use NumPy to
> manipulate that data in place (for example, to cap the red component
> at 128, and equalise the range of the green component) my code needs
> to know the format of the memory block that PIL exposes. I am assuming
> that in-place manipulation is better, because there is no need for
> repeated copies of the data to be made (this would be true for large
> images).

Thanks, that looks like a good example. Is it possible to elaborate
that? E.g. what specific image format would I use (could that work
for jpeg, even though this format has compression in it), and
what specific NumPy routines would I use to implement the capping
and equalising? What would the datatype description look like that
those tools need to exchange?

Looking at this in more detail, PIL in-memory images (ImagingCore
objects) either have the image8 UINT8**, or the image32 INT32**;
they have separate fields for pixelsize and linesize. In the image8
case, there are three options:
- each value is an 8-bit integer (IMAGING_TYPE_UINT8) (1)
- each value is a 16-bit integer, either little (2) or big endian (3)
  (IMAGING_TYPE_SPECIAL, mode either I;16 or I;16B)
In the image32 case, there are five options:
- two 8-bit values per four bytes, namely byte 0 and byte 3 (4)
- three 8-bit values (bytes 0, 1, 2) (5)
- four 8-bit values (6)
- a single 32-bit int (7)
- a single 32-bit float (8)

Now, what would be the algorithm in NumPy that I could use to
implement capping and equalising?

> If PIL could expose a descriptor for its data structure, NumPy code
> could manipulate it in place without fear of corrupting it. Of course,
> this can be done by the end user reading the PIL documentation and
> transcribing the documented format into the NumPy code. But I would
> argue that it's better if the PIL block is self-describing in a way
> that avoids the need for a manual transcription of the format.

Without digging further, I think some of the formats simply don't allow
for the kind of manipulation you suggest, namely all palette formats
(which are the single-valued ones, plus the two-band version with
 a palette number and an alpha value), and greyscale images. So
in any case, the application has to look at the mode of the image to
find out whether the operation is even meaningful. And then, the
application has to tell NumPy somehow what fields to operate on.

> To do this *without* needing the PIL and NumPy developers to
> co-operate needs an independent standard, which is what I assume this
> PEP is intended to provide.

Ok, I now understand the goal, although I still like to understand
this usecase better.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 355 status

[BJörn Lindqvist]

On 10/28/06, Talin <[EMAIL PROTECTED]> wrote:
> BJörn Lindqvist wrote:
> > I'd like to write a post mortem for PEP 355. But one important
> > question that haven't been answered is if there is a possibility for a
> > path-like PEP to succeed in the future? If so, does the path-object
> > implementation have to prove itself in the wild before it can be
> > included in Python? From earlier posts it seems like you don't like
> > the concept of path objects, which others have found very interesting.
> > If that is the case, then it would be nice to hear it explicitly. :)
>
> So...how's that post mortem coming along? Did you get a sufficient
> answer to your questions?

Yes and no. All posts have very exhaustively explained why the
implementation in PEP 355 is far from optimal. And I can see why it
is. However, what I am uncertain of is Guido's opinion on the
background and motivation of the PEP:

"Many have felt that the API for manipulating file paths as offered in
the os.path module is inadequate."

"Currently, Python has a large number of different functions scattered
over half a dozen modules for handling paths.  This makes it hard for
newbies and experienced developers to to choose the right method."

IMHO, the current API is very messy. But when it comes to PEPs, it is
mostly Guido's opinion that counts. :) Unless he sees a problem with
the current situation, then there is no point in writing more PEPs.

> And the more interesting question is, will the effort to reform Python's
> path functionality continue?

I certainly hope so. But maybe it is better to target Python 3000, or
maybe the Python devs already have ideas for how they want the path
APIs to look like?

> So what happens next?

I really hope that Guido will give his input when he has more time.

Mvh Björn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Martin v. Löwis]

Josiah Carlson schrieb:
> One could also toss wxPython, VTK, or any one of the other GUI libraries
> into the mix for visualizing those images, of which wxPython just
> acquired no-copy display of PIL images, and being able to manipulate
> them with numpy (of which some wxPython built in classes use numpy to
> speed up manipulation) would be very useful.

I'm doubtful that this PEP alone would allow zero-copy sharing of images
for display. Often, the libraries need the data in a different format.
So they need to copy, even if they could understand the other format.
However, the PEP won't allow "understanding" the format. If I know I
have an array of 4-byte values: which of them is R, G, B, and A?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 355 status

[Talin]

BJörn Lindqvist wrote:
> On 10/28/06, Talin <[EMAIL PROTECTED]> wrote:
>> BJörn Lindqvist wrote:
>> > I'd like to write a post mortem for PEP 355. But one important
>> > question that haven't been answered is if there is a possibility for a
>> > path-like PEP to succeed in the future? If so, does the path-object
>> > implementation have to prove itself in the wild before it can be
>> > included in Python? From earlier posts it seems like you don't like
>> > the concept of path objects, which others have found very interesting.
>> > If that is the case, then it would be nice to hear it explicitly. :)
>>
>> So...how's that post mortem coming along? Did you get a sufficient
>> answer to your questions?
> 
> Yes and no. All posts have very exhaustively explained why the
> implementation in PEP 355 is far from optimal. And I can see why it
> is. However, what I am uncertain of is Guido's opinion on the
> background and motivation of the PEP:
> 
> "Many have felt that the API for manipulating file paths as offered in
> the os.path module is inadequate."
> 
> "Currently, Python has a large number of different functions scattered
> over half a dozen modules for handling paths.  This makes it hard for
> newbies and experienced developers to to choose the right method."
> 
> IMHO, the current API is very messy. But when it comes to PEPs, it is
> mostly Guido's opinion that counts. :) Unless he sees a problem with
> the current situation, then there is no point in writing more PEPs.
> 
>> And the more interesting question is, will the effort to reform Python's
>> path functionality continue?
> 
> I certainly hope so. But maybe it is better to target Python 3000, or
> maybe the Python devs already have ideas for how they want the path
> APIs to look like?

I think targeting Py3K is a good idea. The whole purpose of Py3K is to 
"clean up the messes" of past decisions, and to that end, a certain 
amount of backwards-compatibility breakage will be allowed (although if 
that can be avoided, so much the better.)

And to the second point, having been following the Py3K list, I don't 
anyone has expressed any preconceived notions of how they want things to 
look (well, except I know I do, but I'm not a core dev :) :).

>> So what happens next?
> 
> I really hope that Guido will give his input when he has more time.

First bit of advice is, don't hold your breath.

Second bit of advice is, if you really do want Guido's feedback (or the 
core python devs), start my creating a (short) list of the outstanding 
points of controversy to be resolved. Once those issues have been 
decided, then proceed to the next stage, building consensus by increments.

Basically, anything that requires Guido to read more than a page of 
material isn't going to get done quickly. At least, in my experience :)

> Mvh Björn

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Josiah Carlson]

"Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Josiah Carlson schrieb:
> > One could also toss wxPython, VTK, or any one of the other GUI libraries
> > into the mix for visualizing those images, of which wxPython just
> > acquired no-copy display of PIL images, and being able to manipulate
> > them with numpy (of which some wxPython built in classes use numpy to
> > speed up manipulation) would be very useful.
> 
> I'm doubtful that this PEP alone would allow zero-copy sharing of images
> for display. Often, the libraries need the data in a different format.
> So they need to copy, even if they could understand the other format.
> However, the PEP won't allow "understanding" the format. If I know I
> have an array of 4-byte values: which of them is R, G, B, and A?

...in the cases I have seen, which includes BMP, TGA, uncompressed TIFF,
a handful of platform-specific bitmap formats, etc.,  you _always_ get
them in RGBA order.  If the alpha channel is to be left out, then you
get them as RGB.

The trick with allowing zero-copy sharing is 1) to understand the format,
and 2) to manipulate/display in-place.  The former is necessary for the
latter, which is what Travis is shooting for.  Also, because wxPython
has figured out how PIL images are structured, they can do #2, and so
far no one has mentioned any examples where the standard RGB/RGBA format
hasn't worked for them.

In the case of jpegs (as you mentioned in another message), PIL
uncompresses all images it understands into some kind of 'natural'
format (from what I understand). For 24/32 bit images, that is RGB or
RGBA. For palletized images (gif, 8-bit png, 8-bit bmp, etc.) maybe it
is a palletized format, or maybe it is RGB/RGBA?  I don't know, all of
my images are 24/32 bit, but I can just about guarantee it's not an
issue for the case that Paul mentioned.


 - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Status of new issue tracker

[Brett Cannon]

The initial admins for the Roundup installation have been chosen: Paul DuBois, Michael Twomey, Stefan Seefeld, and Erik Forsberg.  The offer from Upfront Systems (http://www.upfrontsystems.co.za/
) has been accepted for professional Roundup hosting.Discussion of how to handle the new tracker (including the design of it, handling the transition, etc.) will take place on the tracker-discuss mailing list (
http://mail.python.org/mailman/listinfo/tracker-discuss).  If you want to provide input on what you want the new tracker to do, please join the list.  Input from members of python-dev will take precedence so please participate if you have any interest.
I don't have a timeline on when all of this will happen (talks amongst the four admins has already started on the mailing list and Upfront has started the process of getting us our account).  The first step is to get the admins situated with their new server.  Then we start worrying about what info we want the tracker to store and how to transition off of SF.
-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Greg Ewing]

Travis E. Oliphant wrote:

> Greg Ewing wrote:

>>What exactly does "bit" mean in that context?   
> 
> Do you mean "big" ?

No, you've got a data type there called "bit",
which seems to imply a size, in contradiction
to the size-independent nature of the other
types. I'm asking what size-independent
information it's meant to convey.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Greg Ewing]

Travis E. Oliphant wrote:
> Martin v. Löwis wrote:
> 
>>Travis E. Oliphant schrieb:

>>Is it the intent of this PEP to support such data structures,
>>and allow the user to fill in a Unicode object, and then the
>>processing is automatic?

> No, the point of the data-format object is to communicate information 
> about data-formats not to encode or decode anything.

Well, there's still the issue of how much detail you
want to be able to convey, so I think the question
is valid. Is the encoding of a Unicode string something
we want to be able to communicate via this mechanism,
or is that outside its scope?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Adding data-type objects to Python

[Greg Ewing]

Josiah Carlson wrote:

> ...in the cases I have seen ... you _always_ get
> them in RGBA order.

Except when you don't. I've had cases where I've had to
convert between RGBA and BGRA (for stuffing directly into
a frame buffer on Linux, as far as I remember).

So it may be worth including some features in the standard
for describing pixel formats.

Pygame seems to have a very detailed and flexible system
for doing this, so it might be a good idea to have a
look at that.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] test_codecs failures

[skip]

I recently began running a Pybots buildslave for SQLAlchemy.  I am still
struggling to get that working correctly.  Today, Python's test_codecs test
began failing:

test test_codecs failed -- Traceback (most recent call last):
  File 
"/Library/Buildbot/pybot/trunk.montanaro-g5/build/Lib/test/test_codecs.py", 
line 1165, in test_basics
encoder = codecs.getincrementalencoder(encoding)("ignore")
  File 
"/Library/Buildbot/pybot/trunk.montanaro-g5/build/Lib/encodings/bz2_codec.py", 
line 56, in __init__
assert errors == 'strict'
AssertionError

This failure seems to coincide with some checkins by Georg.  Full output
here:


http://www.python.org/dev/buildbot/community/all/?show=g5%20OSX%20trunk&show=g5%20OSX%202.5

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_codecs failures

[Neal Norwitz]

On 10/29/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I recently began running a Pybots buildslave for SQLAlchemy.  I am still
> struggling to get that working correctly.  Today, Python's test_codecs test
> began failing:

I checked in a fix for this that hasn't quite completed yet.  (Only
finished successfully on one box so far.)  So this should be taken
care of.  I *think* the fix was correct, but I'm not entirely
positive.

Also the refleak problem is fixed AFAIK.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_codecs failures

[Walter Dörwald]

Neal Norwitz wrote:

> On 10/29/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>> I recently began running a Pybots buildslave for SQLAlchemy.  I am still
>> struggling to get that working correctly.  Today, Python's test_codecs test
>> began failing:
> 
> I checked in a fix for this that hasn't quite completed yet.  (Only
> finished successfully on one box so far.)  So this should be taken
> care of.  I *think* the fix was correct, but I'm not entirely
> positive.

The fix *is* indeed correct. bz2 didn't get built on my box, so I didn't
see the failure.

Servus,
   Walter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com