Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Tue, Dec 19, 2017 at 04:56:16PM -0800, Steve Dower wrote:
> On 19Dec2017 1004, Chris Barker wrote:
> >(though I assume order is still ignored when comparing dicts, so:
> >eval(pprint(a_dict)) == a_dict will still hold.
>
> Order had better be ignored when comparing dicts, or plenty of code will
> break. For example:
>
> >>> {'a': 1, 'b': 2} == {'b': 2, 'a': 1}
> True
>
> Saying that "iter(dict)" will produce keys in the same order as they
> were inserted is not the same as saying that "dict" is an ordered
> mapping. As far as I understand, we've only said the first part.
Indeed. Regular dicts preserve insertion order, they don't take
insertion order into account for the purposes of equality. See the
example here:
https://docs.python.org/3.7/library/stdtypes.html#mapping-types-dict
and the description of mapping equality:
https://docs.python.org/3.7/reference/expressions.html#value-comparisons
"Mappings (instances of dict) compare equal if and only if they
have equal (key, value) pairs. Equality comparison of the keys
and values enforces reflexivity."
Changing that would be a *huge* backwards-compatibility breaking change.
Aside: I've just noticed that mapping equality is not transitive:
a == b and b == c does not imply that a == c.
py> from collections import OrderedDict as OD
py> a, b, c = OD.fromkeys('xyz'), dict.fromkeys('xyz'), OD.fromkeys('zyx'))
py> a == b == c
True
py> a == c
False
--
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Tue, 19 Dec 2017 17:32:52 -0800 Nathaniel Smith wrote: > > > In any case, there are so many ways > > to spoil the first point for yourself that it's hardly worth treating as an > > important constraint. > > I guess the underlying issue here is partly the question of what the > pprint module is for. In my understanding, it's primarily a tool for > debugging/introspecting Python programs, and the reason it talks about > "valid input to the interpreter" isn't because we want anyone to > actually feed the data back into the interpreter, [...] Actually, when you want to include a large constant in a Python program, pprint() can be useful to get a nicer formatting for your source code. That said, I do think that pprint() should continue sorting dicts by default. Even though dicts may be ordered *now*, most uses of dict don't expect any particular order. (I also think the pprint() example shows the potential confusion issues with making dict ordered by default, as the user and implementor of an API may not agree whether dict order is significant...) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Mon, Dec 18, 2017 at 08:49:54PM -0800, Nathaniel Smith wrote:
> On Mon, Dec 18, 2017 at 7:58 PM, Steven D'Aprano wrote:
> > I have a script which today prints data like so:
[...]
> To make sure I understand, do you actually have a script like this, or
> is this hypothetical?
The details are much simplified, but basically, and my users probably
won't literally yell at me, but yes I do.
But does it matter? The thing about backwards-compatibility guarantees
is that we have to proceed as if somebody does have such a script. We
don't know who, we don't know why, but we have to assume that they are
relying on whatever guarantees we've given and will be greatly
inconvenienced by any change without sufficient notice.
> > Now, maybe that's my own damn fault for using
> > pprint
[...]
> > so I think I can be excused having relied on that feature.
>
> No need to get aggro -- I asked a question, it wasn't a personal attack.
I didn't interpret it as an attack. Sorry for any confusion, I was
trying to be funny -- at least, it sounded funny in my own head.
> At a high-level, pprint's job is to "pretty-print arbitray Python data
> structures in a form which can be used as input to the interpreter"
> (quoting the first sentence of its documentation), i.e., like repr()
The *high* level purpose of pprint is to *pretty-print* values, like the
name says. If all we wanted was something that outputs an eval()'able
representation, we already had that: repr().
But even that requirement that output can be used as input to the
interpreter is a non-core promise. There are plenty of exceptions:
recursive data structures, functions, any object with the default repr,
etc. Even when it works, the guarantee is quite weak. For instance, even
the object type is not preserved:
py> class MyDict(dict):
... pass
...
py> d = MyDict()
py> x = eval(repr(d))
py> assert d == x
py> assert type(d) == type(x)
Traceback (most recent call last):
File "", line 1, in
AssertionError
So the "promise" that eval(repr(obj)) will round-trip needs to be
understood as being one of those Nice To Have non-core promises, not an
actual guaranteed feature. (The bold print giveth, and the fine print
taketh away.)
So the fact that the output of pprint doesn't preserve the order of the
dict won't be breaking any documented language guarantees. (It is
probably worth documenting explicitly though, rather than just letting
it be implied by the sorted keys guarantee.)
> it's fundamentally intended as a debugging tool that's supposed to
> match how Python works, not any particular externally imposed output
> format.
The point of pprint is not merely to duplicate what repr() already does,
but to output an aesthetically pleasing view of the data structure.
There is no reason to think that is only for the purposes of debugging.
pprint is listed in the docs under Data Types, not Debugging:
https://docs.python.org/3/library/datatypes.html
https://docs.python.org/3/library/debug.html
> Now, how Python works has changed. Previously dict order was
> arbitrary, so picking the arbitrary order that happened to be sorted
> was a nice convenience.
Beware of promising a feature for convenience, because people will come
to rely on it.
In any case, lexicographic (the default sorting) order is in some ways
the very opposite of "arbitrary order".
> Now, dict order isn't arbitrary,
No, we can't say that. Dicts *preserve insertion order*, that is all.
There is no requirement that the insertion order be meaningful or
significant in any way: it may be completely arbitrary. If I build
a mapping of (say) product to price:
d = {'hammer': 5, 'screwdriver': 3, 'ladder': 116}
the order the items are inserted is arbitrary, probably representing the
historical accident of when they were added to the database/catalog or
when I thought of them while typing in the dict.
The most we can say is that for *some* cases, dict order *may* be meaningful.
We're under no obligation to break backwards-compatibility guarantees in
order for pretty printing to reflect a feature of dicts which may or may
not be of any significance to the user.
> and sorting dicts both obscures the actual structure of the
> Python objects,
You can't see the actual structure of Python objects via pprint. For
example, you can't see whether the dict is a split table (shared keys)
or combined table. You can only see the parts of the public interface
which the repr, or pprint, chooses to show.
That's always been the case so nothing changes here.
If pprint were new to 3.7, I daresay there would be a good argument
to have it display keys in insertion order, but given backwards
compatibility, that's not tenable without either an opt-in switch, or a
period of deprecation.
> and also breaks round-tripping through pprint.
Round-tripping need not promise to preserve order, since dicts
don't care about order for the purposes of equality.
Round-tripping already i
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
Following this discussion, I opened two issues: * https://bugs.python.org/issue32387: "Disallow untagged C extension import on major platforms" * https://bugs.python.org/issue32388: "Remove cross-version binary compatibility" Regards Antoine. On Sat, 16 Dec 2017 14:22:57 +0100 Antoine Pitrou wrote: > Hello, > > Nowadays we have an official mechanism for third-party C extensions to > be binary-compatible accross feature releases of Python: the stable ABI. > > But, for non-stable ABI-using C extensions, there are also mechanisms > in place to *try* and ensure binary compatibility. One of them is the > way in which we add tp_ slots to the PyTypeObject structure. > > Typically, when adding a tp_XXX slot, you also need to add a > Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures > that have been compiled against a recent enough PyTypeObject > definition. This way, extensions compiled against Python N-1 are > supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, > the core Python runtime won't try to access the (non-existing) tp_XXX > member. > > However, beside internal code complication, it means you need to add a > new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 > such bits available (many of them already taken), it is a very limited > resource. Is it worth it? (*) Can an extension compiled against Python > N-1 really claim to be compatible with Python N, despite other possible > differences? > > (*) we can't extend the tp_flags field to 64 bits, precisely because of > the binary compatibility problem... > > Regards > > Antoine. > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usefulness of binary compatibility accross Python versions?
On Mon, Dec 18, 2017, at 05:23, Antoine Pitrou wrote: > On Sun, 17 Dec 2017 21:07:02 -0500 > Random832 wrote: > > > > Is there any practical for of having the flag off for one slot and on > > for another slot that's been added later? > > > > Could this be replaced (that is, a slot for such a thing added before > > it's too late) with a simple counter that goes up with each version, and > > any "unused" slot should have NULL or some other sentinel value? > > Any replacement here would break binary compatibility, which is what > those flags are precisely meant to avoid. I meant replacing the mechanism for new fields, rather than existing ones. > > If it > > really is important to have the flags themselves, just add another set > > of flags - Py_TPFLAGS_HAVE_MORE_FLAGS. > > Yes, we could... but it's more complication again. Hmm, maybe that could be eased with macros... /* Doing this preprocessor trick because if we used a ternary * operator, dummy macros needed to prevent compile errors may * become an attractive nuisance */ #define Py_TPFLAG_CHK(tp, flagname) \ Py__TPFCHK_##flagname(tp, flagname) #define Py__TPFCHK_OLD(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_##flagname) #define Py__TPFCHK_NEW(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_TPFLAGVER \ && (tp).tp_flagver >= Py_TPFLAGVER_##flagname \ && Py_TPFLAGCHK_##flagname(tp)) #define Py__TPFCHK_HEAPTYPE Py__TPFCHK_OLD #define Py_TPFLAGS_HEAPTYPE (1UL<<9) #define Py__TPFCHK_TPFLAGVER Py__TPFCHK_OLD #define Py_TPFLAGS_TPFLAGVER (1UL<<31) #define Py__TPFCHK_NEWFLD Py__TPFCHK_NEW #define Py_TPFLAGVER_NEWFLD 32 #define Py_TPFLAGCHK_NEWFLD(tp) ((tp).tp_newfld != NULL) So to check heap type you get Py_TPFLAG_CHK(tp, HEAPTYPE) ((tp).tp_flags & (1UL<<9)) And to check newfld1 you get Py_TPFLAG_CHK(tp, NEWFLD) ((tp).tp_flags & (1UL<<31) && (tp).tp_flagver >= 32 && ((tp).tp_newfld != ((void *)0))) Or in a "more flags" scenario: #define Py__TPFCHK_NEW(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_TPMOREFLAGS \ && (tp).tp_moreflag & PY_TPMOREFLAGS_##flagname) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Wed, Dec 20, 2017 at 03:23:16PM -0800, Chris Barker wrote: > On Wed, Dec 20, 2017 at 2:31 AM, Steven D'Aprano > wrote: > > Even when it works, the guarantee is quite weak. For instance, even > > the object type is not preserved: > > > > py> class MyDict(dict): > > ... pass > > ... > > py> d = MyDict() > > py> x = eval(repr(d)) > > py> assert d == x > > py> assert type(d) == type(x) > > Traceback (most recent call last): > > File "", line 1, in > > AssertionError > > > > Oh come on! If you subclass, and don't override __repr__ -- you're written > a (very slightly) broken class (OK, a class with a broken __repr__). Why is it broken? Is it documented somewhere that every subclass MUST override __repr__? If there's a bug here, and I'm not sure that there is, the bug is in dict itself, for having a repr which isn't friendly to subclasses. But in practice, why would I care? Obviously sometimes I do care, and for debugging it is often good to have a custom repr for subclasses, but it isn't mandatory or even always useful. Especially since in practice, it isn't that common to round-trip repr though eval (apart from the REPL itself, of course). -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is static typing still optional?
On 2017-12-19 02:53, Paul Moore wrote: Also, the fact that no-one raised this issue during the whole time the PEP was being discussed (at least as far as I recollect) and that Guido (who of all of us should be most aware of what is and isn't acceptable use of annotations in the stdlib) approved the PEP, suggests to me that this isn't that big a deal. Hi, I asked about this in the first posting of the PEP and agree with Chris. https://mail.python.org/pipermail/python-dev/2017-September/149406.html There is definitely a passive bias towards using types with dataclasses in that the Eric (the author) doesn't appear to want an example without them in the pep/docs. It seems that typing proponents are sufficiently enamored with them that they can't imagine anyone else feeling differently, haha. Personally, I wouldn't use types with Python unless I was leading a large project with a large team of folks with different levels of experience. That's where types shine, and those folks might be better served by Java or Kotlin. So we hearing that "types are optional" while the docs may imply the opposite. Liked the ellipsis since None is often used as a sentinel value and an extra import is a drag. -Mike ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is static typing still optional?
On 12/20/2017 6:57 PM, Mike Miller wrote: On 2017-12-19 02:53, Paul Moore wrote: Also, the fact that no-one raised this issue during the whole time the PEP was being discussed (at least as far as I recollect) and that Guido (who of all of us should be most aware of what is and isn't acceptable use of annotations in the stdlib) approved the PEP, suggests to me that this isn't that big a deal. Hi, I asked about this in the first posting of the PEP and agree with Chris. https://mail.python.org/pipermail/python-dev/2017-September/149406.html There is definitely a passive bias towards using types with dataclasses in that the Eric (the author) doesn't appear to want an example without them in the pep/docs. I'm not sure what such an example would look like. Do you mean without annotations? Or do you mean without specifying the "correct" type, like: @dataclass class C: x: int = 'hello world' ? Or something else? Can you provide an example of what you'd like to see? It seems that typing proponents are sufficiently enamored with them that they can't imagine anyone else feeling differently, haha. I've never used typing or mypy, so you're not talking about me. I do like the conciseness that annotations bring to dataclasses, though. If you buy that (and you might not), then I don't see the point of not using a correct type annotation. Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is static typing still optional?
On 12/20/2017 8:13 PM, Eric V. Smith wrote: There is definitely a passive bias towards using types with dataclasses in that the Eric (the author) doesn't appear to want an example without them in the pep/docs. I'm not sure what such an example would look like. Do you mean without annotations? Or do you mean without specifying the "correct" type, like: @dataclass class C: x: int = 'hello world' ? Or something else? Can you provide an example of what you'd like to see? Re-reading my post you referenced, is it just an example using typing.Any? I'm okay with that in the docs, I just didn't want to focus on it in the PEP. I want the PEP to only have the one reference to typing, for typing.ClassVar. I figure the people reading the PEP can extrapolate to all of the possible uses for annotations that they don't need to see a typing.Any example. Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is static typing still optional?
On Wed, Dec 20, 2017 at 5:29 PM, Eric V. Smith wrote: > There is definitely a passive bias towards using types with dataclasses in > that the Eric (the author) doesn't appear to want an example without them > in the pep/docs. > >> >> I'm not sure what such an example would look like. Do you mean without >> annotations? > > IIUC, there is not way to make a dataclass without annotations, yes? That it, using annotations to determine the fields is the one and only way the decorator works. So it's impossible to give an example without annotations, yes? > Or do you mean without specifying the "correct" type, like: >> >> @dataclass >> class C: >> x: int = 'hello world' >> > It may be a good idea to have an example like that in the docs (but probably not the PEP) to make it clear that the type is not used in any way at run time. But I don't think that anyone is suggesting that would be a recommended practice. I suggest that it be clear in the docs, and ideally in the PEP, that the dataclass decorator is using the *annotation" syntax, and that the the only relevant part it uses is that an annotation exists, but the value of the annotation is essentially (completely?) ignored. So we should have examples like: @dataclass class C: a: ... # field with no default b: ... = 0 # filed with a default value Then maybe: @dataclass class C: a: "the a parameter" # field with no default b: "another, different parameter" = 0.0 # field with a default Then the docs can go to say that if the user wants to specify a type for use with a static type checking pre-processor, they can do it like so: @dataclass class C: a: int # integer field with no default b: float = 0.0 # float field with a default And the types will be recognized by type checkers such as mypy. And I think the non-typed examples should go first in the docs. This is completely analogous to how all the other parts of python are taught. Would anyone suggest that the very first example of a function definition that a newbie sees would be: def func(a: int, b:float = 0.0): body_of_function Then, _maybe_ way down on the page, you mention that oh, by the way, those types are completely ignored by Python. And not even give any examples without types? > Re-reading my post you referenced, is it just an example using typing.Any? I actually think that is exactly the wrong point -- typing.Any is still using type hinting -- it's an explicit way to say, "any type will do", but it's only relevant if you are using a type checker. We really need examples for folks that don't know or care about type hinting at all. typing.Any is for use by people that are explicitly adding type hinting, and should be discussed in type hinting documentation. > I'm okay with that in the docs, I just didn't want to focus on it in the PEP. I want the PEP to only > have the one reference to typing, for typing.ClassVar. I figure the people reading the PEP can > extrapolate to all of the possible uses for annotations that they don't need to see a typing.Any > example. no they don't, but they DO need to see examples without type hints at all. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
