Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Mark Dickinson
On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: > [...] struct already supports > variable-width formats. Unfortunately, that's not really true: the Pascal strings it supports are in some sense variable length, but are stored in a fixed-width field. The internals of the struct module rely

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Rhodri James
On 19/01/17 08:31, Mark Dickinson wrote: On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: [...] struct already supports variable-width formats. Unfortunately, that's not really true: the Pascal strings it supports are in some sense variable length, but are stored in a fixed-width field

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Elizabeth Myers
On 19/01/17 05:58, Rhodri James wrote: > On 19/01/17 08:31, Mark Dickinson wrote: >> On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano >> wrote: >>> [...] struct already supports >>> variable-width formats. >> >> Unfortunately, that's not really true: the Pascal strings it supports >> are in some s

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Elizabeth Myers
On 19/01/17 06:47, Elizabeth Myers wrote: > On 19/01/17 05:58, Rhodri James wrote: >> On 19/01/17 08:31, Mark Dickinson wrote: >>> On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano >>> wrote: [...] struct already supports variable-width formats. >>> >>> Unfortunately, that's not really tr

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Chris Angelico
On Fri, Jan 20, 2017 at 5:08 AM, Elizabeth Myers wrote: > I do understand it might require a possible major rewrite or major > changes the struct module, but in the long run, I think it's worth it > (especially because the struct module is not all that big in scope). As > it stands, the struct mod

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Joao S. O. Bueno
I am for upgrading struct to these, if possible. But besides my +1, I am writting in to remember folks thatthere is another "struct" model in the stdlib: ctypes.Structure - For reading a lot of records with the same structure it is much more handy than struct, since it gives one a suitable Pyt

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Nick Timkovich
ctypes.Structure is *literally* the interface to the C struct that as Chris mentions has fixed offsets for all members. I don't think that should (can?) be altered. In file formats (beyond net protocols) the string size + variable length string motif comes up often and I am frequently re-implement

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Nathaniel Smith
I haven't had a chance to use it myself yet, but I've heard good things about https://construct.readthedocs.io/en/latest/ It's certainly far more comprehensive than struct for this and other problems. As usual, there's some tension between adding stuff to the stdlib versus using more specialized

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Nick Timkovich
Construct has radical API changes and should remain apart. It feels to me like a straw-man to introduce a large library to the discussion as justification for it being too-specialized. This proposal to me seems much more modest: add another format character (or two) to the existing set of a dozen

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread MRAB
On 2017-01-19 12:47, Elizabeth Myers wrote: On 19/01/17 05:58, Rhodri James wrote: On 19/01/17 08:31, Mark Dickinson wrote: On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: [...] struct already supports variable-width formats. Unfortunately, that's not really true: the Pascal strings

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Yury Selivanov
This is a neat idea, but this will only work for parsing framed binary protocols. For example, if you protocol prefixes all packets with a length field, you can write an efficient read buffer and use your proposal to decode all of message's fields in one shot. Which is good. Not all protocols us

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Ethan Furman
There is now an issue for this: http://bugs.python.org/issue29328 -- ~Ethan~ ___ Python-ideas mailing list [email protected] https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Steven D'Aprano
On Thu, Jan 19, 2017 at 08:31:03AM +, Mark Dickinson wrote: > On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: > > [...] struct already supports > > variable-width formats. > > Unfortunately, that's not really true: the Pascal strings it supports > are in some sense variable length, bu

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Steven D'Aprano
On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris Angelico wrote: > To be fair, the name "struct" implies a C-style structure, which > _does_ have a fixed size, or at least fixed offsets for its members Ah, the old "everyone thinks in C terms" fallacy raises its ugly head agan :-) The name doesn

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Chris Angelico
On Fri, Jan 20, 2017 at 11:38 AM, Steven D'Aprano wrote: > On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris Angelico wrote: > >> To be fair, the name "struct" implies a C-style structure, which >> _does_ have a fixed size, or at least fixed offsets for its members > > > Ah, the old "everyone thinks

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Guido van Rossum
Nevertheless the C meaning *is* the etymology of the module name. :-) --Guido (mobile) On Jan 19, 2017 16:54, "Chris Angelico" wrote: > On Fri, Jan 20, 2017 at 11:38 AM, Steven D'Aprano > wrote: > > On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris Angelico wrote: > > > >> To be fair, the name "

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Cameron Simpson
On 19Jan2017 12:08, Elizabeth Myers wrote: I also didn't mention that when you are unpacking iteratively (e.g., you have multiple strings), the code becomes a bit more hairy: test_bytes = b'\x00\x05hello\x00\x07goodbye\x00\x04test' offset = 0 while offset < len(test_bytes): ... length = s

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Cameron Simpson
On 19Jan2017 16:04, Yury Selivanov wrote: This is a neat idea, but this will only work for parsing framed binary protocols. For example, if you protocol prefixes all packets with a length field, you can write an efficient read buffer and use your proposal to decode all of message's fields in on

Re: [Python-ideas] Ideas for improving the struct module

2017-01-19 Thread Mark Dickinson
On Fri, Jan 20, 2017 at 12:30 AM, Steven D'Aprano wrote: > Does it require a PEP just to add one more > format code? (Maybe it will, if the format code requires a complete > re-write of the entire module.) Yes, I think a PEP would be useful in this case. The proposed change *would* entail some fa