[Python-Dev] PEP 467: Minor bytes and bytearray improvements
The final PEP with SC feedback incorporated and one last addition: `bytes.ascii` as a replacement for the Python 2 idiom
of `str(some_var)` to get the bytes version, and the Python 3 workaround of either the correct
`str(some_var).encode('astii') or the potentially wrong `ascii(some_var).encode('ascii').
The rendered version is at https://www.python.org/dev/peps/pep-0467/
Happy reading!
PEP: 467
Title: Minor API improvements for binary sequences
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan , Ethan Furman
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Mar-2014
Python-Version: 3.11
Post-History: 2014-03-30 2014-08-15 2014-08-16 2016-06-07 2016-09-01 2021-04-13
2021-11-03
Abstract
This PEP proposes five small adjustments to the APIs of the ``bytes`` and
``bytearray`` types to make it easier to operate entirely in the binary domain:
* Add ``fromsize`` alternative constructor
* Add ``fromint`` alternative constructor
* Add ``ascii`` alternative constructor
* Add ``getbyte`` byte retrieval method
* Add ``iterbytes`` alternative iterator
Rationale
=
During the initial development of the Python 3 language specification, the
core ``bytes`` type for arbitrary binary data started as the mutable type
that is now referred to as ``bytearray``. Other aspects of operating in
the binary domain in Python have also evolved over the course of the Python
3 series, for example with PEP 461.
Motivation
==
With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols.
This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text). The addition of
the new constructors, methods, and iterators will aid both in writing new
wire format code, and in porting any remaining Python 2 wire format code.
Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.
Proposals
=
Addition of explicit "count and byte initialised sequence" constructors
---
To replace the now discouraged behavior, this PEP proposes the addition of an
explicit ``fromsize`` alternative constructor as a class method on both
``bytes`` and ``bytearray`` whose first argument is the count, and whose
second argument is the fill byte to use (defaults to ``\x00``)::
>>> bytes.fromsize(3)
b'\x00\x00\x00'
>>> bytearray.fromsize(3)
bytearray(b'\x00\x00\x00')
>>> bytes.fromsize(5, b'\x0a')
b'\x0a\x0a\x0a\x0a\x0a'
>>> bytearray.fromsize(5, fill=b'\x0a')
bytearray(b'\x0a\x0a\x0a\x0a\x0a')
``fromsize`` will behave just as the current constructors behave when passed a
single integer, while allowing for non-zero fill values when needed.
Addition of explicit "single byte" constructors
---
As binary counterparts to the text ``chr`` function, this PEP proposes
the addition of an explicit ``fromint`` alternative constructor as a class
method on both ``bytes`` and ``bytearray``::
>>> bytes.fromint(65)
b'A'
>>> bytearray.fromint(65)
bytearray(b'A')
These methods will only accept integers in the range 0 to 255 (inclusive)::
>>> bytes.fromint(512)
Traceback (most recent call last):
File "", line 1, in
ValueError: integer must be in range(0, 256)
>>> bytes.fromint(1.0)
Traceback (most recent call last):
File "", line 1, in
TypeError: 'float' object cannot be interpreted as an integer
The documentation of the ``ord`` builtin will be updated to explicitly note
that ``bytes.fromint`` is the primary inverse operation for binary data, while
``chr`` is the inverse operation for text data, and that ``bytearray.fromint``
also exists.
Behaviorally, ``bytes.fromint(x)`` will be equivalent to the current
``bytes([x])`` (and similarly for ``bytearray``). The new spelling is
expected to be easier to discover and easier to read (especially when used
in conjunction with indexing operations on binary sequence types).
As a separate method, the new spelling will also work better with higher
order functions like ``map``.
These new methods intentionally do NOT offer the same level of general integer
support as the existing ``int.to_bytes`` conversion method, which allows
arbitrarily large integers to be converted to arbitrarily long bytes objects.
The
restriction to only accept positive integers that fit in a single byte means
that no byte order information is needed, and there is no need to handle
negative numbers. The documentation of the new methods will refer readers to
``int.to_bytes`` for use cases where handling of arbitrary integers is needed.
Addition of "ascii" constructors
In Python 2 converting a
[Python-Dev] Oh look, I've been subscribed to python/issues-test-2 notifications again
I guess this is part of the migration from bpo to GitHub issues? Maybe the initial work could be done in a private repo, to cut down on the spurious email notifications to literally everybody subscribed to cpython? Which is a lot of people. //arry/ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/H5KW6GRHIF2VWOGNRH5367WB3K2GPARO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467: Minor bytes and bytearray improvements
On Thu, Nov 4, 2021 at 12:01 AM Ethan Furman wrote:
> >>> bytearray.fromsize(5, fill=b'\x0a')
> bytearray(b'\x0a\x0a\x0a\x0a\x0a')
>
What happens if you supply more than one byte for the fill argument?
Silent truncation, raise ValueError('too long') or ???
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/GHXKHPNJSEK6SRNG2O3ZLZMKAEZTMVLS/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467: Minor bytes and bytearray improvements
On Thu, Nov 4, 2021 at 10:37 AM Eric Fahlgren
wrote:
> On Thu, Nov 4, 2021 at 12:01 AM Ethan Furman wrote:
>
>> >>> bytearray.fromsize(5, fill=b'\x0a')
>> bytearray(b'\x0a\x0a\x0a\x0a\x0a')
>>
>
> What happens if you supply more than one byte for the fill argument?
> Silent truncation, raise ValueError('too long') or ???
>
It would seem reasonable to me for a multi-byte sequence to be filled as-is
in a repeating pattern, perhaps truncating the last repetition if len(fill)
is not an even multiple of the size. At least that's the intuitive behavior
for me.
That said, I don't know if such behavior would be useful in practice (i.e.
whether there's a use case for it).
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/I4OE6VPBOBYBUQD45PJJKNR4BHA5EQF6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467: Minor bytes and bytearray improvements
On Fri, Nov 5, 2021 at 2:59 AM Jonathan Goble wrote:
>
> On Thu, Nov 4, 2021 at 10:37 AM Eric Fahlgren wrote:
>>
>> On Thu, Nov 4, 2021 at 12:01 AM Ethan Furman wrote:
>>>
>>> >>> bytearray.fromsize(5, fill=b'\x0a')
>>> bytearray(b'\x0a\x0a\x0a\x0a\x0a')
>>
>>
>> What happens if you supply more than one byte for the fill argument? Silent
>> truncation, raise ValueError('too long') or ???
>
>
> It would seem reasonable to me for a multi-byte sequence to be filled as-is
> in a repeating pattern, perhaps truncating the last repetition if len(fill)
> is not an even multiple of the size. At least that's the intuitive behavior
> for me.
>
> That said, I don't know if such behavior would be useful in practice (i.e.
> whether there's a use case for it).
>
It's definitely useful behaviour, but aligns better with sequence
multiplication than a fill= constructor parameter. My expectation (or
if you prefer: my preferred shed colour) would be ValueError.
ChrisA
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/R35YPY4TRHMGDBELZTO25Z2UZBAEB7IC/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Oh look, I've been subscribed to python/issues-test-2 notifications again
What notification? (I fully admit I may not have gotten one due to some team I'm in, but I have no such notification if it happened recently.) On Thu, Nov 4, 2021 at 12:16 AM Larry Hastings wrote: > > I guess this is part of the migration from bpo to GitHub issues? Maybe > the initial work could be done in a private repo, to cut down on the > spurious email notifications to literally everybody subscribed to cpython? > Which is a lot of people. > > > */arry* > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/H5KW6GRHIF2VWOGNRH5367WB3K2GPARO/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/OJYGOHGS27CUY7KWHE5PEC2CKYH6M5PI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Oh look, I've been subscribed to python/issues-test-2 notifications again
On 11/4/21 12:21 PM, Brett Cannon wrote: > What notification? (I fully admit I may not have gotten one due to some team I'm in, but I have > no such notification if it happened recently.) I've received 20-30 in the last three or four days. I'm not concerned about it, just providing a data point. -- ~Ethan~ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/LFF2AQPBXJPVNFDBTW6ZVSIO55AYOHVM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Oh look, I've been subscribed to python/issues-test-2 notifications again
I was looking into migrating twisted trac to github, and contacted GitHub support who told me there's a secret API that doesn't notify subscribers https://gist.github.com/jonmagic/5282384165e0f86ef105 On Thu, 4 Nov 2021, 19:28 Brett Cannon, wrote: > What notification? (I fully admit I may not have gotten one due to some > team I'm in, but I have no such notification if it happened recently.) > > On Thu, Nov 4, 2021 at 12:16 AM Larry Hastings wrote: > >> >> I guess this is part of the migration from bpo to GitHub issues? Maybe >> the initial work could be done in a private repo, to cut down on the >> spurious email notifications to literally everybody subscribed to cpython? >> Which is a lot of people. >> >> >> */arry* >> ___ >> Python-Dev mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/[email protected]/message/H5KW6GRHIF2VWOGNRH5367WB3K2GPARO/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/OJYGOHGS27CUY7KWHE5PEC2CKYH6M5PI/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/X5OWAZZRM6ZL2O6U4V7Q6EXBLG4NTJUT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Oh look, I've been subscribed to python/issues-test-2 notifications again
I've had about 5 or 6 of them. Paul On Thu, 4 Nov 2021 at 19:22, Brett Cannon wrote: > > What notification? (I fully admit I may not have gotten one due to some team > I'm in, but I have no such notification if it happened recently.) > > On Thu, Nov 4, 2021 at 12:16 AM Larry Hastings wrote: >> >> >> I guess this is part of the migration from bpo to GitHub issues? Maybe the >> initial work could be done in a private repo, to cut down on the spurious >> email notifications to literally everybody subscribed to cpython? Which is >> a lot of people. >> >> >> /arry >> >> ___ >> Python-Dev mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/[email protected]/message/H5KW6GRHIF2VWOGNRH5367WB3K2GPARO/ >> Code of Conduct: http://python.org/psf/codeofconduct/ > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/OJYGOHGS27CUY7KWHE5PEC2CKYH6M5PI/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JGPFO6R4AUDTF7LG6SSFDC6Y6GRTA66S/ Code of Conduct: http://python.org/psf/codeofconduct/
