ValueError vs IndexError, unpacking arguments with string.split
Hi there. I was adding a port specification feature to my surveil project as shown here: https://github.com/morphex/surveil/commit/703318f87c4c450a37944b565a10718ef27b57b4 A bit later I was surprised when the script raised an exception, and that I had to catch a ValueError instead of an IndexError: https://github.com/morphex/surveil/commit/d5c49c54d4037557aaca1f78178b76441853c7af I've been reading up on a bit of C++, Assembler etc. lately, so maybe my mind expected an IndexError because of that, but isn't it logical that the string is parsed and split, and then later the unpacking operation fails with an IndexError? Regards, Morten -- Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
Morten W. Petersen wrote:
> Hi there.
>
> I was adding a port specification feature to my surveil project as shown
> here:
>
>
https://github.com/morphex/surveil/commit/703318f87c4c450a37944b565a10718ef27b57b4
>
> A bit later I was surprised when the script raised an exception, and that
> I had to catch a ValueError instead of an IndexError:
>
>
https://github.com/morphex/surveil/commit/d5c49c54d4037557aaca1f78178b76441853c7af
>
> I've been reading up on a bit of C++, Assembler etc. lately, so maybe my
> mind expected an IndexError because of that, but isn't it logical that the
> string is parsed and split, and then later the unpacking operation fails
> with an IndexError?
You might think that
a, b = c
is equivalent to
a = c[0]
b = c[1]
but the c above can be an arbitrary iterable:
>>> a, b = iter("ab")
>>> a, b
('a', 'b')
>>> a, b = iter("abc")
Traceback (most recent call last):
File "", line 1, in
ValueError: too many values to unpack (expected 2)
So there is not necessarily a lookup by index.
--
https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Fri, Nov 30, 2018 at 4:02 PM Peter Otten <__pete...@web.de> wrote:
> Morten W. Petersen wrote:
>
> > I've been reading up on a bit of C++, Assembler etc. lately, so maybe my
> > mind expected an IndexError because of that, but isn't it logical that
> the
> > string is parsed and split, and then later the unpacking operation fails
> > with an IndexError?
>
> You might think that
>
> a, b = c
>
> is equivalent to
>
> a = c[0]
> b = c[1]
>
> but the c above can be an arbitrary iterable:
>
> >>> a, b = iter("ab")
> >>> a, b
> ('a', 'b')
> >>> a, b = iter("abc")
> Traceback (most recent call last):
> File "", line 1, in
> ValueError: too many values to unpack (expected 2)
>
> So there is not necessarily a lookup by index.
>
Ah, of course. OK, right you are. :)
-Morten
--
Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/
--
https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On 11/30/18 7:35 AM, Morten W. Petersen wrote: ... but isn't it logical that the string is parsed and split, and then later the unpacking operation fails with an IndexError? With slightly simpler code and the full text of the exception, the details becomes more apparent: >>> x, y = [4] Traceback (most recent call last): File "", line 1, in ValueError: not enough values to unpack (expected 2, got 1) Python validates that the right hand side contains exactly the right number of elements before beginning to unpack, presumably to ensure a successful operation before failing part way through. Effectively, Python is saying that you can't unpack a one-element array into two pieces *before* it gets to the indexing logic. HTH, Dan -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Fri, Nov 30, 2018 at 4:25 PM Dan Sommers < 2qdxy4rzwzuui...@potatochowder.com> wrote: > On 11/30/18 7:35 AM, Morten W. Petersen wrote: > > > ... but isn't it logical that the > > string is parsed and split, and then later the unpacking operation fails > > with an IndexError? > > With slightly simpler code and the full text of the exception, > the details becomes more apparent: > > >>> x, y = [4] > Traceback (most recent call last): >File "", line 1, in > ValueError: not enough values to unpack (expected 2, got 1) > > Python validates that the right hand side contains exactly the > right number of elements before beginning to unpack, presumably > to ensure a successful operation before failing part way through. > > Effectively, Python is saying that you can't unpack a one-element > array into two pieces *before* it gets to the indexing logic. > Hm. Well I like the simplicity and abstraction of Python, it makes it a very productive language. And since iterable object can be used, I guess it makes sense to raise a ValueError. And trying >>> a,b,c = None Traceback (most recent call last): File "", line 1, in TypeError: 'NoneType' object is not iterable >>> raises a TypeError, and this is all in-hand with the Python docs. But since you mention it, why is it necessary to ensure a successful operation? Is it so that when a,b,c = [1,2] fails, none of the variables a,b,c have been assigned to, and because of that, one avoids "rolling back" any assignment that would have been done without checking the right-hand argument first ? Regards, Morten -- Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On 11/30/18 10:57 AM, Morten W. Petersen wrote: > On Fri, Nov 30, 2018 at 4:25 PM Dan Sommers <2qdxy4rzwzuui...@potatochowder.com> wrote: >> Python validates that the right hand side contains exactly the right >> number of elements before beginning to unpack, presumably to ensure a >> successful operation before failing part way through. > Hm. Well I like the simplicity and abstraction of Python, it makes it > a very productive language. Agreed, and hold that thought. > But since you mention it, why is it necessary to ensure a successful > operation? > > Is it so that when > > a,b,c = [1,2] > > fails, none of the variables a,b,c have been assigned to, and because > of that, one avoids "rolling back" any assignment that would have been > done without checking the right-hand argument first ? That's what I was getting at, but apparently failed to express clearly. Yes, there are use cases for a short iterator just not assigning the rest of the variables, and for a long iterator using only what it needs, but also explicit is better than implicit, and errors should never pass silently, and other bits of wisdom learned from horrible debugging experiences. -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Fri, Nov 30, 2018 at 6:21 PM Dan Sommers < 2qdxy4rzwzuui...@potatochowder.com> wrote: > On 11/30/18 10:57 AM, Morten W. Petersen wrote: > > On Fri, Nov 30, 2018 at 4:25 PM Dan Sommers > <2qdxy4rzwzuui...@potatochowder.com> wrote: > [...] > > But since you mention it, why is it necessary to ensure a successful > > operation? > > > > Is it so that when > > > > a,b,c = [1,2] > > > > fails, none of the variables a,b,c have been assigned to, and because > > of that, one avoids "rolling back" any assignment that would have been > > done without checking the right-hand argument first ? > > That's what I was getting at, but apparently failed to express clearly. > > Yes, there are use cases for a short iterator just not assigning the > rest of the variables, and for a long iterator using only what it needs, > but also explicit is better than implicit, and errors should never pass > silently, and other bits of wisdom learned from horrible debugging > experiences. > Aha, yes well you have head, tail, etc. I guess syntax could be added, so that a, b, @c = some sequence would initialize a and b, and leave anything remaining in c. We could then call this @ syntax "teh snek". 😂 ..I guess I haven't punished myself with enough Perl yet. Regards, Morten -- Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On 11/30/18 12:00 PM, Morten W. Petersen wrote: > I guess syntax could be added, so that > > a, b, @c = some sequence > > would initialize a and b, and leave anything remaining in c. We could > then call this @ syntax "teh snek". Close. ;-) Try this: a, b, *c = [4, 5, 6, 7] -- https://mail.python.org/mailman/listinfo/python-list
RAR functionality
Folks, Can anyone explain to me why RAR support isn’t available natively in python? We have the zipfile and tarfile modules… Yes, there is rarfile and PaTools but they both rely on RAR being available to shell out to… What is it about RAR files that makes it unreasonable to handle in natively in python? Licensing? Bad rar file design? - Benjamin -- https://mail.python.org/mailman/listinfo/python-list
Re: RAR functionality
On Fri, 30 Nov 2018 16:42:06 -0500, Benjamin Schollnick wrote: > Folks, > > Can anyone explain to me why RAR support isn’t available natively in python? > > We have the zipfile and tarfile modules… > > Yes, there is rarfile and PaTools but they both rely on RAR being available > to shell out to… > What is it about RAR files that makes it unreasonable to handle in natively > in python? > Licensing? Bad rar file design? > > - Benjamin RAR is proprietary. -- GNU/Linux user #557453 The cow died so I don't need your bull! -- https://mail.python.org/mailman/listinfo/python-list
Re: RAR functionality
On Sat, Dec 1, 2018 at 8:43 AM Benjamin Schollnick wrote: > > Folks, > > Can anyone explain to me why RAR support isn’t available natively in python? > > We have the zipfile and tarfile modules… > > Yes, there is rarfile and PaTools but they both rely on RAR being available > to shell out to… > What is it about RAR files that makes it unreasonable to handle in natively > in python? > Licensing? Bad rar file design? > Licensing, yes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: multiple JSON documents in one file, change proposal
Paul Rubin : > Maybe someone can convince me I'm misusing JSON but I often want to > write out a file containing multiple records, and it's convenient to > use JSON to represent the record data. > > The obvious way to read a JSON doc from a file is with "json.load(f)" > where f is a file handle. Unfortunately, this throws an exception I have this "multi-JSON" need quite often. In particular, I exchange JSON-encoded messages over byte stream connections. There are many ways of doing it. Having rejected different options (https://en.wikipedia.org/wiki/JSON_streaming>), I settled with terminating each JSON value with an ASCII NUL character, which is illegal in JSON proper. > I also recommend the following article to those not aware of how badly > designed JSON is: http://seriot.ch/parsing_json.php JSON is not ideal, but compared with XML, it's a godsend. What would be ideal? I think S-expressions would come close, but people can mess up even them: https://www.ietf.org/rfc/rfc2693.txt>. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: multiple JSON documents in one file, change proposal
On 01Dec2018 00:40, Marko Rauhamaa wrote: Paul Rubin : Maybe someone can convince me I'm misusing JSON but I often want to write out a file containing multiple records, and it's convenient to use JSON to represent the record data. The obvious way to read a JSON doc from a file is with "json.load(f)" where f is a file handle. Unfortunately, this throws an exception I have this "multi-JSON" need quite often. In particular, I exchange JSON-encoded messages over byte stream connections. There are many ways of doing it. Having rejected different options (https://en.wikipedia.org/wiki/JSON_streaming>), I settled with terminating each JSON value with an ASCII NUL character, which is illegal in JSON proper. There's a common format called Newline Delimited JSON (NDJSON) for just this need. Just format the outbound records as JSON with no newlines (i.e. make the separator a space or the empty string), put a newline at the end of each. On ingest, read lines of text, and JSON parse each line separately. This is very easy, very partable, and makes for very human readable data files. I strongly recommend it. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: multiple JSON documents in one file, change proposal
On Sat, Dec 1, 2018 at 9:46 AM Marko Rauhamaa wrote:
>
> Paul Rubin :
> > Maybe someone can convince me I'm misusing JSON but I often want to
> > write out a file containing multiple records, and it's convenient to
> > use JSON to represent the record data.
> >
> > The obvious way to read a JSON doc from a file is with "json.load(f)"
> > where f is a file handle. Unfortunately, this throws an exception
>
> I have this "multi-JSON" need quite often. In particular, I exchange
> JSON-encoded messages over byte stream connections. There are many ways
> of doing it. Having rejected different options ( https://en.wikipedia.org/wiki/JSON_streaming>), I settled with
> terminating each JSON value with an ASCII NUL character, which is
> illegal in JSON proper.
There actually is a way to parse concatenated JSON (where you start
the next object straight after the previous one, possibly with
whitespace). It's not hugely well proclaimed, but it's based on this
function:
https://docs.python.org/3/library/json.html#json.JSONDecoder.raw_decode
Example usage:
data = """\
{"foo": 1, "bar": 2}
{"foo": 3, "bar": 4}
{"foo": 5, "bar": 6}
"""
from json import JSONDecoder
def parse(s, *, decoder=JSONDecoder()):
obj, offset = decoder.raw_decode(s)
return obj, s[offset:].strip()
while data:
obj, data = parse(data)
print(obj)
> > I also recommend the following article to those not aware of how badly
> > designed JSON is: http://seriot.ch/parsing_json.php
(There's some irony in using a PHP web site to discuss bad design.)
> JSON is not ideal, but compared with XML, it's a godsend.
>
> What would be ideal? I think S-expressions would come close, but people
> can mess up even them: https://www.ietf.org/rfc/rfc2693.txt>.
Crockford is opinionated and egotistical. And like all humans, often
wrong. JSON is far from a perfect interchange format, and there have
been numerous attempts to improve on it (for instance, SUPPORTING
TRAILING COMMAS!! ARGH!). See, for instance, JSON5 (https://json5.org/
and https://pypi.org/project/json5/), which is I believe a strict
superset of JSON and also a strict subset of JavaScript, keeping it
fully compatible both ways.
JSON isn't perfect, but for many purposes, it is good enough. Where it
isn't, there are often reasonable extensions (such as the use of
raw_decode to read multiple objects from a string) that can help.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Fri, Nov 30, 2018 at 7:25 PM Dan Sommers < 2qdxy4rzwzuui...@potatochowder.com> wrote: > On 11/30/18 12:00 PM, Morten W. Petersen wrote: > > > I guess syntax could be added, so that > > > > a, b, @c = some sequence > > > > would initialize a and b, and leave anything remaining in c. We could > > then call this @ syntax "teh snek". > > Close. ;-) Try this: > > a, b, *c = [4, 5, 6, 7] > ☺ I did not know that. Or maybe I read it somewhere and it was in the subconscious. -Morten -- Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Sat, Dec 1, 2018 at 10:18 AM Morten W. Petersen wrote: > > On Fri, Nov 30, 2018 at 7:25 PM Dan Sommers < > 2qdxy4rzwzuui...@potatochowder.com> wrote: > > > On 11/30/18 12:00 PM, Morten W. Petersen wrote: > > > > > I guess syntax could be added, so that > > > > > > a, b, @c = some sequence > > > > > > would initialize a and b, and leave anything remaining in c. We could > > > then call this @ syntax "teh snek". > > > > Close. ;-) Try this: > > > > a, b, *c = [4, 5, 6, 7] > > > > > > I did not know that. Or maybe I read it somewhere and it was in the > subconscious. > Be aware that this isn't the same as "leaving" everything in c. It will specifically gather everything in the rest of the iterable and put it all into a list in c. For this example, that wouldn't be a problem, but consider: >>> x = range(10) >>> a, b, *c = x >>> print(x) range(0, 10) >>> print(a, b, c) 0 1 [2, 3, 4, 5, 6, 7, 8, 9] But as long as you never dive into infinite iterators, it's going to be fine. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: multiple JSON documents in one file, change proposal
On 30/11/2018 23:40, Marko Rauhamaa wrote: > Paul Rubin : >> Maybe someone can convince me I'm misusing JSON but I often want to >> write out a file containing multiple records, and it's convenient to >> use JSON to represent the record data. >> >> The obvious way to read a JSON doc from a file is with "json.load(f)" >> where f is a file handle. Unfortunately, this throws an exception > > I have this "multi-JSON" need quite often. In particular, I exchange > JSON-encoded messages over byte stream connections. There are many ways > of doing it. Having rejected different options ( https://en.wikipedia.org/wiki/JSON_streaming>), I settled with > terminating each JSON value with an ASCII NUL character, which is > illegal in JSON proper. FWIW, YAML supports multiple documents in one stream (separated by ---) If you just want multiple JSON objects in a file (without any streaming needs), you can just use a JSON array... > >> I also recommend the following article to those not aware of how badly >> designed JSON is: http://seriot.ch/parsing_json.php > > JSON is not ideal, but compared with XML, it's a godsend. > > What would be ideal? I think S-expressions would come close, but people > can mess up even them: https://www.ietf.org/rfc/rfc2693.txt>. > > > Marko > -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Sat, Dec 1, 2018 at 12:25 AM Chris Angelico wrote: > On Sat, Dec 1, 2018 at 10:18 AM Morten W. Petersen > wrote: > > > > On Fri, Nov 30, 2018 at 7:25 PM Dan Sommers < > > 2qdxy4rzwzuui...@potatochowder.com> wrote: > > > > > On 11/30/18 12:00 PM, Morten W. Petersen wrote: > > > > > > > I guess syntax could be added, so that > > > > > > > > a, b, @c = some sequence > > > > > > > > would initialize a and b, and leave anything remaining in c. We > could > > > > then call this @ syntax "teh snek". > > > > > > Close. ;-) Try this: > > > > > > a, b, *c = [4, 5, 6, 7] > > > > > > > > > > > I did not know that. Or maybe I read it somewhere and it was in the > > subconscious. > > > > Be aware that this isn't the same as "leaving" everything in c. It > will specifically gather everything in the rest of the iterable and > put it all into a list in c. For this example, that wouldn't be a > problem, but consider: > > >>> x = range(10) > >>> a, b, *c = x > >>> print(x) > range(0, 10) > >>> print(a, b, c) > 0 1 [2, 3, 4, 5, 6, 7, 8, 9] > > But as long as you never dive into infinite iterators, it's going to be > fine. > Mm, yes. This was the intuitive understanding I had of it. The range in Python 3 is what xrange was in Python 2, and if I do x=list(range(1)) it takes quite a bit of time, as opposed to x = range(1) But this raises the question of how to write Python code, short and sweet, that could handle infinite iterators in such an unpack with multiple variables to assign to. Which I guess is mostly theoretical, as there are other ways of designing code to avoid needing to unpack an infinite iterator using the * prefix. -Morten -- Videos at https://www.youtube.com/user/TheBlogologue Twittering at http://twitter.com/blogologue Blogging at http://blogologue.com Playing music at https://soundcloud.com/morten-w-petersen Also playing music and podcasting here: http://www.mixcloud.com/morten-w-petersen/ On Google+ here https://plus.google.com/107781930037068750156 On Instagram at https://instagram.com/morphexx/ -- https://mail.python.org/mailman/listinfo/python-list
Re: ValueError vs IndexError, unpacking arguments with string.split
On Sat, Dec 1, 2018 at 10:55 AM Morten W. Petersen wrote: > But this raises the question of how to write Python code, > short and sweet, that could handle infinite iterators in > such an unpack with multiple variables to assign to. > > Which I guess is mostly theoretical, as there are other > ways of designing code to avoid needing to unpack > an infinite iterator using the * prefix. It could only be done with the cooperation of the iterable in question. For instance, a range object could implement a "remove first" operation that does this, and several itertools types wouldn't need to change at all. But it can't be done generically other than the way it now is (pump the iterator the rest of the way). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
