[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[André Malo]

Steve Dower wrote:

> On a policy level, we don't make changes that would break users of the C 
> API. Because we can't track everyone who's using it, we have to assume 
> that everything is used and any change will cause breakage.
> 
> To make sure it's possible to keep developing CPython, we declare parts 
> of the API off limits (typically by prepending them with an underscore). 
> If you use these, and you break, we're sorry but we aren't going to fix it.
> 
> This line of discussion is basically saying that we would designate a 
> broader section of the API that is off limits, most likely the parts 
> that are only useful for increased performance (rather than increased 
> functionality). We would then specifically include the Cython 
> team/volunteers in discussions about how to manage changes to these 
> parts of the API to avoid breaking them, and possibly do simultaneous 
> releases to account for changes so that their users have more time to 
> rebuild.
> 
> Effectively, when we change our APIs, we would break everyone except 
> Cython because we've worked with them to avoid the breakage. Anyone else 
> using it has to make their own effort to follow CPython development and 
> detect any breakage themselves (just like today).
> 
> So probably the part you're missing is where we would give ourselves 
> permission to break more APIs in a release, while simultaneously 
> encouraging people to use Cython as an isolation layer from those breaks.

The encouraging part is not working for me :-) And seriously, my gut tells me, 
we're split at 50/50 here. People usually write C for a reason and Cython is 
not. For, let's say, half of the cases that's fine, speeding up inner loops 
and all that, which not touching the C level at all. The other half wants to 
solve different issues.

I think, it does not serve well as a policy for CPython. Since we're talking 
hypotheticals right now, if Cython vanishes tomorrow, we're kind of left empty 
handed. Such kind of a runtime, if considered part of the compatibility 
"promise", should be provided by the core itself, no?
A good way to test that promise (or other implications like performance) might 
also be to rewrite the standard library extensions in Cython and see where it 
leads.

I personally see myself using the python-provided runtime (types, methods, 
GC), out of convenience (it's there, so why not use it). The vision of the 
future outlined here can easily lead to backing off from that and rebuilding 
all those things and really only keep touchpoints with python when it comes to 
interfacing with python itself. It's probably even desirable that way. But 
definitely more work (for an extension author).

As a closing word, I don't mind either way. IOW I'm not complaining. I'm just 
putting more opinion from the "outside" into the ring. Thanks for listening 
:-)

Cheers,
nd

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZDPNR3PO4RVQ3EHQQLEFEKUVBF72A2MX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Stefan Behnel]

Steve Dower schrieb am 14.04.20 um 00:27:
> On 13Apr2020 2308, André Malo wrote:
>> For one thing, if you open up APIs for Cython, they're open for everybody
>> (Cython being "just" another C extension).
>> More to the point: The ABIs have the same problem as they have now,
>> regardless
>> how responsive the Cython developers are. Once you compiled the extension,
>> you're using the ABI and are supposedly not required to recompile to stay
>> compatible.
>>
>> So, where I'm getting at is: Either you open up to everybody or nobody. In C
>> there's not really an in-between.
> 
> On a technical level, you are correct.
> 
> On a policy level, we don't make changes that would break users of the C
> API. Because we can't track everyone who's using it, we have to assume that
> everything is used and any change will cause breakage.
> 
> To make sure it's possible to keep developing CPython, we declare parts of
> the API off limits (typically by prepending them with an underscore). If
> you use these, and you break, we're sorry but we aren't going to fix it.
> 
> This line of discussion is basically saying that we would designate a
> broader section of the API that is off limits, most likely the parts that
> are only useful for increased performance (rather than increased
> functionality). We would then specifically include the Cython
> team/volunteers in discussions about how to manage changes to these parts
> of the API to avoid breaking them, and possibly do simultaneous releases to
> account for changes so that their users have more time to rebuild.
> 
> Effectively, when we change our APIs, we would break everyone except Cython
> because we've worked with them to avoid the breakage. Anyone else using it
> has to make their own effort to follow CPython development and detect any
> breakage themselves (just like today).
> 
> So probably the part you're missing is where we would give ourselves
> permission to break more APIs in a release, while simultaneously
> encouraging people to use Cython as an isolation layer from those breaks.

To add to that, the main difference for users here is a choice:

1) I want to use whatever is in the C-API and will fix my broken code
myself whenever there's a new CPython release.

2) I write my code against the stable ABI, accept the performance
limitations, and hope that it'll "never" break and my code just keeps
working (even through future compatibility layers, if necessary).

3) I use Cython and rerun it on my code at least once for each new CPython
release series, because I want to get the best performance for each target
version.

4) I use Cython and activate its (yet to be completed) stable ABI mode, so
that I don't have to target separate (C)Python releases but can release a
single wheel, at the cost of reduced performance.

And then there are a couple of grey areas, e.g. people using Cython plus a
bit of the C-API directly, for which they are then responsible themselves
again. But it's still way easier to adapt 3% of your code every couple of
CPython releases than all of your modules for each new release. That's just
the normal price that you pay for manual optimisations.

A nice feature of Cython here is that 3) and 4) are actually not mutually
exclusive, at least as it looks so far. You should eventually be able to
generate both from your same sources (we are trying hard to keep them in
the same C file), and even mix them on PyPI, e.g. distribute a generic
stable ABI wheel for all Pythons that support it, plus accelerated wheels
for CPython 3.9 and 3.10. You may even be able to release a pure Python
wheel as well, as we currently do for Cython itself to better support PyPy.

And to drive the point home, if CPython starts changing its C-API more
radically, or comes up with a new one, we can add the support for it to
Cython and then, in the best case, users will still only have to rerun it
on their code to target that new API. Compare that to case 1).


> (Cython is still just a placeholder name here, btw. There are 1-2 other
> projects that could be considered instead, though I think Cython is the
> only one that also provides a usability improvement as well as API
> stability.)

pybind11 and mypyc could probably make a similar offer to users. The
important point is just that we centralise the abstraction and adaptation work.

Stefan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BHH3XKTCKZ73WQNHPVHYNBMJPYBELZFV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Stefan Behnel]

André Malo schrieb am 14.04.20 um 13:39:
> I think, it does not serve well as a policy for CPython. Since we're talking 
> hypotheticals right now, if Cython vanishes tomorrow, we're kind of left 
> empty 
> handed. Such kind of a runtime, if considered part of the compatibility 
> "promise", should be provided by the core itself, no?

There was some discussion a while ago about integrating a stripped-down
variant of Cython into CPython's stdlib. I was arguing against that because
the selling point of Cython is really what it is, and stripping that down
wouldn't lead to something equally helpful for users.

I think it's good to have separate projects (and, in fact, it's more than
one) deal with this need.

In the end, it's an external tool, like your editor, your C compiler, your
debugger and whatever else you need for developing Python extensions. It
spits out C code and lets you do with it what you want. There's no reason
it should be part of the CPython project, core or stdlib. It's even written
in Python. If it doesn't work for you, you can fix it.


> A good way to test that promise (or other implications like performance) 
> might 
> also be to rewrite the standard library extensions in Cython and see where it 
> leads.

Not sure I understand what you're saying here. stdlib extension modules are
currently written in C, with a bit of code generation. How is that different?


> I personally see myself using the python-provided runtime (types, methods, 
> GC), out of convenience (it's there, so why not use it). The vision of the 
> future outlined here can easily lead to backing off from that and rebuilding 
> all those things and really only keep touchpoints with python when it comes 
> to 
> interfacing with python itself. It's probably even desirable that way

That's actually not an uncommon thing to do. Some packages really only use
Cython or pybind11 to wrap their otherwise native C or C++ code. It's a
choice given specific organisational/project/developer constraints, and
choices are good.

Stefan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QZSX36TPAKLXAA3O6KLUNCPKVJ2SKASN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Stefan Behnel]

Paul Moore schrieb am 13.04.20 um 14:25:
> On a related but different note, what is the recommended policy
> (assuming it's not to use the C API) for embedding Python, and for
> exposing the embedding app to Python as a C extension? My standard
> example of this is the Vim interface to Python - see
> https://github.com/vim/vim/blob/master/src/if_python3.c. I originally
> wrote this back in the Python 1.5 days, so it's *very* old, and quite
> likely not how I'd write it now, even using the C API. But what's the
> recommendation for code like that in the face of these changes, and
> the suggestion that using 3rd party tools is the normal way to write C
> extensions?

Embedding is not very well documented overall. I recently looked through
the docs to collect what a user would need to know in this case, and ended
up creating at least a little link collection, because I failed to find a
good place to refer users to. The things people need to know from the
CPython docs are scattered across different places, and lack a complete
real-world-like example that "most people" could start from. (I don't think
many users will pass strings into Python to execute code there.)

https://cython.readthedocs.io/en/latest/src/tutorial/embedding.html

From Cython's PoV, the main thing that future embedders need to understand
is that it's not really different from extending – you just have to start
the Python runtime before doing anything else. I think there should be some
help for getting that done, and then it's just executing your Python code
in some module. Cython then has its ways to go back and forth from there,
e.g. by writing cdef (C) functions as entry points for your application.

Cython currently doesn't really have "direct" support for embedding. You
can let it generate a C main function for you to start your program, but
that's not what you want in the case of vim. There's a "cython_freeze"
script that generates an inittab list in addition, but it's a bit
simplistic and not integrated. We have a beginners ticket for integrating
it better:

https://github.com/cython/cython/issues/2849

What I would like to see eventually is to let users pass a list of modules
into Cython's frontend (maybe cythonize(), maybe not), and then it would
just generate a single distutils Extension from them that links everything
together and registers all modules on import, optionally with a generated
exported C function that starts up the whole thing. That seems simple
enough to do and use, and you end up with a shared library that your
application can load. PRs welcome. :)

Stefan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y6VRSVWYSV63AFQNAQEIJZBDZZG7QOTM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[André Malo]

Stefan Behnel wrote:
> André Malo schrieb am 14.04.20 um 13:39:
> 
> > I think, it does not serve well as a policy for CPython. Since we're
> > talking 
 hypotheticals right now, if Cython vanishes tomorrow, we're
> > kind of left empty handed. Such kind of a runtime, if considered part of
> > the compatibility "promise", should be provided by the core itself, no?
> 
> 
> There was some discussion a while ago about integrating a stripped-down
> variant of Cython into CPython's stdlib. I was arguing against that because
> the selling point of Cython is really what it is, and stripping that down
> wouldn't lead to something equally helpful for users.
> 
> I think it's good to have separate projects (and, in fact, it's more than
> one) deal with this need.
> 
> In the end, it's an external tool, [...]

Thank you, that is my point exactly. It's the same "external" as everything 
else. I'm still trying to understand where to separate the different sets of 
"external".

> 
> > A good way to test that promise (or other implications like performance)
> > might 
 also be to rewrite the standard library extensions in Cython and
> > see where it leads.
> 
> 
> Not sure I understand what you're saying here. stdlib extension modules are
> currently written in C, with a bit of code generation. How is that
> different? 

They are C extensions like the ones everybody could write. They should use the 
same APIs. What I'm saying is, that it would be a good test if the APIs are 
good enough (for everybody else). If, say, Cython is recommended, some attempt 
should be made to achieve the same results with Cython. Or some other sets of 
APIs which are considered for "the public".

I don't think, the current stdlib modules restrict themselves to a limited 
API. The distinction between "inside" and "outside" bothers me.


> > I personally see myself using the python-provided runtime (types, methods,
> > 
 GC), out of convenience (it's there, so why not use it). The vision of
> > the future outlined here can easily lead to backing off from that and
> > rebuilding all those things and really only keep touchpoints with python
> > when it comes to interfacing with python itself. It's probably even
> > desirable that way
> 
> That's actually not an uncommon thing to do. Some packages really only use
> Cython or pybind11 to wrap their otherwise native C or C++ code. It's a
> choice given specific organisational/project/developer constraints, and
> choices are good.

Agreed. Nevertheless, the choices are going to be limited by extra 
constraints.

Cheers,
nd

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U7KHC7KV5GOOQ4ST5HI3MZKAW4CMRJ6S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Ronald Oussoren via Python-Dev]

> On 10 Apr 2020, at 19:20, Victor Stinner  wrote:
> 
[…]

> 
> 
> PEP xxx: Modify the C API to hide implementation details
> 
> 
> Abstract
> 
> 
> * Hide implementation details from the C API to be able to `optimize
>  CPython`_ and make PyPy more efficient.
> * The expectation is that `most C extensions don't rely directly on
>  CPython internals`_ and so will remain compatible.
> * Continue to support old unmodified C extensions by continuing to
>  provide the fully compatible "regular" CPython runtime.
> * Provide a `new optimized CPython runtime`_ using the same CPython code
>  base: faster but can only import C extensions which don't use
>  implementation details. Since both CPython runtimes share the same
>  code base, features implemented in CPython will be available in both
>  runtimes.
> * `Stable ABI`_: Only build a C extension once and use it on multiple
>  Python runtimes and different versions of the same runtime.
> * Better advertise alternative Python runtimes and better communicate on
>  the differences between the Python language and the Python
>  implementation (especially CPython).
> 
> Note: Cython and cffi should be preferred to write new C extensions.

I’m too old… I still prefer the CPython ABI over the other two mostly because 
that’s what I know best but also the reduce dependencies. 

> This PEP is about existing C extensions which cannot be rewritten with
> Cython.

I’m not sure what this PEP  proposes beyond “lets make the stable ABI the 
default API” and provide a mechanism to get access to the current API.  I guess 
the proposal also expands the scope for the stable ABI, some internals that are 
currently exposed in the stable ABI would no longer be so. 

I’m not  opposed to this as long as it is still possible to use the current 
API, possibly with clean-ups and correctness fixes, As you write the CPython 
API has some features that make writing correct code harder, in particular the 
concept of borrowed references. There’s still good reasons to want be as close 
to the metal as possible, both to get maximal performance and to accomplish 
things that aren’t possible using the stable ABI.

[…]
> 
> API and ABI incompatible changes
> 
> 
> * Make structures opaque: move them to the internal C API.
> * Remove functions from the public C API which are tied to CPython
>  internals. Maybe begin by marking these functions as private (rename
>  ``PyXXX`` to ``_PyXXX``) or move them to the internal C API.
> * Ban statically allocated types (by making ``PyTypeObject`` opaque):
>  enforce usage of ``PyType_FromSpec()``.
> 
> Examples of issues to make structures opaque:
> 
> * ``PyGC_Head``: https://bugs.python.org/issue40241
> * ``PyObject``: https://bugs.python.org/issue39573
> * ``PyTypeObject``: https://bugs.python.org/issue40170
> * ``PyThreadState``: https://bugs.python.org/issue39573
> 
> Another example are ``Py_REFCNT()`` and ``Py_TYPE()`` macros which can
> currently be used l-value to modify an object reference count or type.
> Python 3.9 has new ``Py_SET_REFCNT()`` and ``Py_SET_TYPE()`` macros
> which should be used instead. ``Py_REFCNT()`` and ``Py_TYPE()`` macros
> should be converted to static inline functions to prevent their usage as
> l-value.
> 
> **Backward compatibility:** backward incompatible on purpose. Break the
> limited C API and the stable ABI, with the assumption that `Most C
> extensions don't rely directly on CPython internals`_ and so will remain
> compatible.

This is definitely backward incompatible in a way that affects all extensions 
defining types without using  PyTypeSpec due to having PyObject ad PyTypeObject 
in the list. I wonder how large a percentage of existing extensions is affected 
by this.  

Making “PyObject” opaque will also affect the stable ABI because even types 
defined using the PyTypeSpec API embed a “PyObject” value in the structure 
defining the instance layout. It is easy enough to change this in a way that 
preserves source-code compatibility, but I’m  not sure it is possible to avoid 
breaking the stable ABI. 

BTW. This will require growing the PyTypeSpec ABI a little, there are features 
you cannot implement using that API for example the buffer protocol. 

[…]
> 
> 
> CPython specific behavior
> =
> 
> Some C functions and some Python functions have a behavior which is
> closely tied to the current CPython implementation.
> 
> is operator
> ---
> 
> The "x is y" operator is closed tied to how CPython allocates objects
> and to ``PyObject*``.
> 
> For example, CPython uses singletons for numbers in [-5; 256] range::
> 
 x=1; (x + 1) is 2
>True
 x=1000; (x + 1) is 1001
>False
> 
> Python 3.8 compiler now emits a ``SyntaxWarning`` when the right operand
> of the ``is`` and ``is not`` operators is a literal (ex: integer or
> s

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Matěj Cepl]

On 2020-04-13, 17:39 GMT, Eric Fahlgren wrote:
> Ok, so put that in a Pros/Cons list that provides guidance as to what
> interface and tools to choose when writing a new extension module.
> Personally, I'd put Cython (and other "big" packages, numpy, requests and
> such) on par with CPython itself with respect to "likely to implode and
> become unusable."

Time for the unplesant questions: what is the bus factor of Cython?

Best,

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
To love another person
Is to see the face of God.
  -- yes, incredibly cheesy verse from the screenplay of the
 movie Les Miserables (2012)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RT4HSRZF37KYED2ZREDMF37EAYYFNR4R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Matěj Cepl]

On 2020-04-14, 12:35 GMT, Stefan Behnel wrote:
>> A good way to test that promise (or other implications like performance) 
>> might 
>> also be to rewrite the standard library extensions in Cython and see where 
>> it 
>> leads.
>
> Not sure I understand what you're saying here. stdlib extension modules are
> currently written in C, with a bit of code generation. How is that different?

When you are saying that writing C extensions is unnecessary,
because everything can be easily written in Cython, start
persuading me by rewriting all C extensions included in CPython
into Cython. If you are not willing to do it, why I should
I start rewriting my 7k lines of SWIG code to Cython, just
because you hope that somebody finally finally (please!) notices
existence of PyPy and hopefully starts to care about it. No,
they won’t.

Matěj
-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
Never, never, never believe any war will be smooth and easy, or
that anyone who embarks on the strange voyage can measure the
tides and hurricanes he will encounter. The statesman who yields
to war fever must realise that once the signal is given, he is no
longer the master of policy but the slave of unforeseeable and
uncontrollable events.
-- Winston Churchill, 1930

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FXUBENCEONTRFTAKR2SNIJ5OOEE22FBY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 14Apr2020 1557, André Malo wrote:

Stefan Behnel wrote:

André Malo schrieb am 14.04.20 um 13:39:

A good way to test that promise (or other implications like performance)
might

  also be to rewrite the standard library extensions in Cython and

see where it leads.



Not sure I understand what you're saying here. stdlib extension modules are
currently written in C, with a bit of code generation. How is that
different?


They are C extensions like the ones everybody could write. They should use the
same APIs. What I'm saying is, that it would be a good test if the APIs are
good enough (for everybody else). If, say, Cython is recommended, some attempt
should be made to achieve the same results with Cython. Or some other sets of
APIs which are considered for "the public".

I don't think, the current stdlib modules restrict themselves to a limited
API. The distinction between "inside" and "outside" bothers me.


It should not bother you. The standard library is not a testing ground 
for the public API - it's a layer to make those APIs available to users 
in a reliable, compatible format. Think of it like your C runtime, which 
uses a lot of system calls that have changed far more often than libc.


We can change the interface between the runtime and the included modules 
as frequently as we like, because it's private. And we do change them, 
and the changes go unnoticed because we adapt both sides of the contract 
at once. For example, we recently changed the calling conventions for 
certain functions, which didn't break anyone because we updated the 
callers as well. And we completely reimplemented stat() emulation on 
Windows recently, which wasn't incompatible because the public part of 
the API didn't change (except to have fewer false errors).


Modules that are part of the core runtime deliberately use private APIs 
so that other extension modules don't have to. It's not any sort of 
unfair advantage - it's a deliberate aspect of the software's design.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RFYAK5NDBY4DYHKBYOQK5SUKMIT4VZZX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[André Malo]

Steve Dower wrote:
> On 14Apr2020 1557, André Malo wrote:
> 
> > Stefan Behnel wrote:
> > 
> >> André Malo schrieb am 14.04.20 um 13:39:
> >> 
> >>> A good way to test that promise (or other implications like
> >>> performance)
> >>> might
> >>> 
> >   also be to rewrite the standard library extensions in Cython and
> >   
> >>> see where it leads.
> >>
> >>
> >>
> >>
> >> Not sure I understand what you're saying here. stdlib extension modules
> >> are
 currently written in C, with a bit of code generation. How is that
> >> different?
> > 
> > 
> > They are C extensions like the ones everybody could write. They should use
> > the
 same APIs. What I'm saying is, that it would be a good test if the
> > APIs are good enough (for everybody else). If, say, Cython is
> > recommended, some attempt should be made to achieve the same results with
> > Cython. Or some other sets of APIs which are considered for "the public".
> > 
> > I don't think, the current stdlib modules restrict themselves to a
> > limited
> > API. The distinction between "inside" and "outside" bothers me.
> 
> 
> It should not bother you. The standard library is not a testing ground 
> for the public API - it's a layer to make those APIs available to users 
> in a reliable, compatible format. Think of it like your C runtime, which 
> uses a lot of system calls that have changed far more often than libc.

I can agree up to a certain level. There are extensions and there are 
extensions, see below.

> 
> We can change the interface between the runtime and the included modules 
> as frequently as we like, because it's private. And we do change them, 
> and the changes go unnoticed because we adapt both sides of the contract 
> at once. For example, we recently changed the calling conventions for 
> certain functions, which didn't break anyone because we updated the 
> callers as well. And we completely reimplemented stat() emulation on 
> Windows recently, which wasn't incompatible because the public part of 
> the API didn't change (except to have fewer false errors).
> 
> Modules that are part of the core runtime deliberately use private APIs 
> so that other extension modules don't have to. It's not any sort of 
> unfair advantage - it's a deliberate aspect of the software's design.

Ah, hmm, maybe I was not clear enough. I was talking about extensions like 
itertools or datetime. Not core builtins like sys or the type system. I think, 
there's a difference. People do use especially the former ones also as a 
template how things are done "correctly".

I agree, it's easy enough to change everything at once, assuming a good test 
suite :-)

Cheers,
nd

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZEYZX7PRCACRZEZIHU35CLWIPHQBALDV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Victor Stinner]

Hi Ronald,

Le mar. 14 avr. 2020 à 18:25, Ronald Oussoren  a écrit :
> Making “PyObject” opaque will also affect the stable ABI because even types 
> defined using the PyTypeSpec API embed a “PyObject” value in the structure 
> defining the instance layout. It is easy enough to change this in a way that 
> preserves source-code compatibility, but I’m  not sure it is possible to 
> avoid breaking the stable ABI.

Oh, that's a good point. I tracked this issue at:
https://bugs.python.org/issue39573#msg366473

> BTW. This will require growing the PyTypeSpec ABI a little, there are 
> features you cannot implement using that API for example the buffer protocol.

I tracked this feature request at:
https://bugs.python.org/issue40170#msg366474

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IG3SU3RTOSO24OHWT6PQIZJP4WMGKADA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Improvement to SimpleNamespace

[Raymond Hettinger]

SimpleNamespace() is really good at giving attribute style-access. I would like 
to make that functionality available to the JSON module (or just about anything 
else that accepts a custom dict) by adding the magic methods for mappings so 
that this works:

 catalog = json.load(f, object_hook=SimpleNamespace)
 print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])  
# currently possible with dict()
 print(catalog.clothing.mens.shoes.extra_wide.quantity]) # 
proposed with SimpleNamespace()
 print(catalog.clothing.boys['3t'].tops.quantity
   # would also be supported

I've already seen something like this in production; however, people are having 
to write custom subclasses to do it.  This is kind of bummer because the custom 
subclasses are a pain to write, are non-standard, and are generally somewhat 
slow.  I would like to see a high-quality version this made more broadly 
available.

The core idea is keep the simple attribute access but make it easier to load 
data programmatically:

>>> ns = SimpleNamespace(roses='red', violets='blue')
>>> thing = input()
sugar
>>> quality = input()
sweet
>>> setattr(ns, thing, quality)# current
>>> ns['sugar'] = 'sweet'   # proposed

If the PEP 584 __ior__ method were supported, updating a SimpleNamespace would 
be much cleaner:

  ns |= some_dict

I posted an issue on the tracker: https://bugs.python.org/issue40284 .  There 
was a suggestion to create a different type for this, but I don't see the point 
in substantially duplicating everything SimpleNamespace already does just so we 
can add some supporting dunder methods.   Please add more commentary so we can 
figure-out the best way to offer this powerful functionality.


Raymond


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Guido van Rossum]

I've seen this pattern a lot at a past employer, and despite the obvious
convenience I've come to see it as an anti-pattern: for people expecting
Python semantics it's quite surprising to read code that writes foo.bar and
then reads back foo['bar']. We should not try to import JavaScript's object
model into Python.

On Tue, Apr 14, 2020 at 8:06 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> SimpleNamespace() is really good at giving attribute style-access. I would
> like to make that functionality available to the JSON module (or just about
> anything else that accepts a custom dict) by adding the magic methods for
> mappings so that this works:
>
>  catalog = json.load(f, object_hook=SimpleNamespace)
>
>  print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])
> # currently possible with dict()
>  print(catalog.clothing.mens.shoes.extra_wide.quantity])
># proposed with SimpleNamespace()
>  print(catalog.clothing.boys['3t'].tops.quantity
>  # would also be supported
>
> I've already seen something like this in production; however, people are
> having to write custom subclasses to do it.  This is kind of bummer because
> the custom subclasses are a pain to write, are non-standard, and are
> generally somewhat slow.  I would like to see a high-quality version this
> made more broadly available.
>
> The core idea is keep the simple attribute access but make it easier to
> load data programmatically:
>
> >>> ns = SimpleNamespace(roses='red', violets='blue')
> >>> thing = input()
> sugar
> >>> quality = input()
> sweet
> >>> setattr(ns, thing, quality)# current
> >>> ns['sugar'] = 'sweet'   # proposed
>
> If the PEP 584 __ior__ method were supported, updating a SimpleNamespace
> would be much cleaner:
>
>   ns |= some_dict
>
> I posted an issue on the tracker: https://bugs.python.org/issue40284 .
> There was a suggestion to create a different type for this, but I don't see
> the point in substantially duplicating everything SimpleNamespace already
> does just so we can add some supporting dunder methods.   Please add more
> commentary so we can figure-out the best way to offer this powerful
> functionality.
>
>
> Raymond
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EJ6XJW6SZ23V2RJTSPPYT4Z3FB7BMQGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Kyle Stanley]

Guido van Rossum wrote:
> I've seen this pattern a lot at a past employer, and despite the obvious
convenience I've come to see it as an anti-pattern: for people expecting
Python semantics it's quite surprising to read code that writes foo.bar and
then reads back foo['bar'].

Would it be significantly less surprising if it were to be exclusively
allowed for the JSON module rather than anything that accepts custom dicts?
To me, that seems like the most useful and intuitive location for the dot
notation, and it would keep the Python semantics intact everywhere else. I
think it's fairly unlikely that most users of JSON would be surprised by
it, but I can see that it could be surprising elsewhere.

On Tue, Apr 14, 2020 at 11:33 PM Guido van Rossum  wrote:

> I've seen this pattern a lot at a past employer, and despite the obvious
> convenience I've come to see it as an anti-pattern: for people expecting
> Python semantics it's quite surprising to read code that writes foo.bar and
> then reads back foo['bar']. We should not try to import JavaScript's object
> model into Python.
>
> On Tue, Apr 14, 2020 at 8:06 PM Raymond Hettinger <
> raymond.hettin...@gmail.com> wrote:
>
>> SimpleNamespace() is really good at giving attribute style-access. I
>> would like to make that functionality available to the JSON module (or just
>> about anything else that accepts a custom dict) by adding the magic methods
>> for mappings so that this works:
>>
>>  catalog = json.load(f, object_hook=SimpleNamespace)
>>
>>  print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])
>> # currently possible with dict()
>>  print(catalog.clothing.mens.shoes.extra_wide.quantity])
>># proposed with SimpleNamespace()
>>  print(catalog.clothing.boys['3t'].tops.quantity
>>  # would also be supported
>>
>> I've already seen something like this in production; however, people are
>> having to write custom subclasses to do it.  This is kind of bummer because
>> the custom subclasses are a pain to write, are non-standard, and are
>> generally somewhat slow.  I would like to see a high-quality version this
>> made more broadly available.
>>
>> The core idea is keep the simple attribute access but make it easier to
>> load data programmatically:
>>
>> >>> ns = SimpleNamespace(roses='red', violets='blue')
>> >>> thing = input()
>> sugar
>> >>> quality = input()
>> sweet
>> >>> setattr(ns, thing, quality)# current
>> >>> ns['sugar'] = 'sweet'   # proposed
>>
>> If the PEP 584 __ior__ method were supported, updating a SimpleNamespace
>> would be much cleaner:
>>
>>   ns |= some_dict
>>
>> I posted an issue on the tracker: https://bugs.python.org/issue40284 .
>> There was a suggestion to create a different type for this, but I don't see
>> the point in substantially duplicating everything SimpleNamespace already
>> does just so we can add some supporting dunder methods.   Please add more
>> commentary so we can figure-out the best way to offer this powerful
>> functionality.
>>
>>
>> Raymond
>>
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EJ6XJW6SZ23V2RJTSPPYT4Z3FB7BMQGO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5B6ACDDG5HR2NY32SZY2GPFLNKOVBMDJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Raymond Hettinger]

[GvR]
> We should not try to import JavaScript's object model into Python.

Yes, I get that.  Just want to point-out that working with heavily nested 
dictionaries (typical for JSON) is no fun with square brackets and quotation 
marks.


Raymond
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G5SJKRQ7S5VY3JKLAVOTCCA7RSDUNWXS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Guido van Rossum]

Well, as a user of JSON in Python I *would* be surprised by it, since the
actual JSON notation uses dicts, and most Python code I've seen that access
raw JSON data directly uses dict notation. Where you see dot notation is if
the raw JSON dict is verified and converted to a regular object (usually
with the help of some schema library), but there dict notation is
questionable.

Accepted if you replace "users of JSON" with "users of JavaScript" -- but
if you want to go there, try CoffeeScript. :-)

On Tue, Apr 14, 2020 at 8:50 PM Kyle Stanley  wrote:

> Guido van Rossum wrote:
> > I've seen this pattern a lot at a past employer, and despite the obvious
> convenience I've come to see it as an anti-pattern: for people expecting
> Python semantics it's quite surprising to read code that writes foo.bar and
> then reads back foo['bar'].
>
> Would it be significantly less surprising if it were to be exclusively
> allowed for the JSON module rather than anything that accepts custom dicts?
> To me, that seems like the most useful and intuitive location for the dot
> notation, and it would keep the Python semantics intact everywhere else. I
> think it's fairly unlikely that most users of JSON would be surprised by
> it, but I can see that it could be surprising elsewhere.
>
> On Tue, Apr 14, 2020 at 11:33 PM Guido van Rossum 
> wrote:
>
>> I've seen this pattern a lot at a past employer, and despite the obvious
>> convenience I've come to see it as an anti-pattern: for people expecting
>> Python semantics it's quite surprising to read code that writes foo.bar and
>> then reads back foo['bar']. We should not try to import JavaScript's object
>> model into Python.
>>
>> On Tue, Apr 14, 2020 at 8:06 PM Raymond Hettinger <
>> raymond.hettin...@gmail.com> wrote:
>>
>>> SimpleNamespace() is really good at giving attribute style-access. I
>>> would like to make that functionality available to the JSON module (or just
>>> about anything else that accepts a custom dict) by adding the magic methods
>>> for mappings so that this works:
>>>
>>>  catalog = json.load(f, object_hook=SimpleNamespace)
>>>
>>>  print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity'])
>>> # currently possible with dict()
>>>  print(catalog.clothing.mens.shoes.extra_wide.quantity])
>>>  # proposed with SimpleNamespace()
>>>  print(catalog.clothing.boys['3t'].tops.quantity
>>># would also be supported
>>>
>>> I've already seen something like this in production; however, people are
>>> having to write custom subclasses to do it.  This is kind of bummer because
>>> the custom subclasses are a pain to write, are non-standard, and are
>>> generally somewhat slow.  I would like to see a high-quality version this
>>> made more broadly available.
>>>
>>> The core idea is keep the simple attribute access but make it easier to
>>> load data programmatically:
>>>
>>> >>> ns = SimpleNamespace(roses='red', violets='blue')
>>> >>> thing = input()
>>> sugar
>>> >>> quality = input()
>>> sweet
>>> >>> setattr(ns, thing, quality)# current
>>> >>> ns['sugar'] = 'sweet'   # proposed
>>>
>>> If the PEP 584 __ior__ method were supported, updating a SimpleNamespace
>>> would be much cleaner:
>>>
>>>   ns |= some_dict
>>>
>>> I posted an issue on the tracker: https://bugs.python.org/issue40284 .
>>> There was a suggestion to create a different type for this, but I don't see
>>> the point in substantially duplicating everything SimpleNamespace already
>>> does just so we can add some supporting dunder methods.   Please add more
>>> commentary so we can figure-out the best way to offer this powerful
>>> functionality.
>>>
>>>
>>> Raymond
>>>
>>>
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56PJGRN7FQQLLCWONE5Z7R2EKXW/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/EJ6XJW6SZ23V2RJTSPPYT4Z3FB7BMQGO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

_

[Python-Dev] Re: Improvement to SimpleNamespace

[Chris Angelico]

On Wed, Apr 15, 2020 at 2:09 PM Raymond Hettinger
 wrote:
>
> [GvR]
> > We should not try to import JavaScript's object model into Python.
>
> Yes, I get that.  Just want to point-out that working with heavily nested 
> dictionaries (typical for JSON) is no fun with square brackets and quotation 
> marks.
>

My solution to that has usually been something along the lines of:

def get(obj, path):
for step in path.split("-"):
obj = obj[step]
return obj

print(get(catalog, 'clothing-mens-shoes-extra_wide-quantity'))

Will often be custom-tweaked to the situation, but the basic idea is the same.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XNCNGO3PYOTFVWKTYCO6T57JW3YH24D4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[David Mertz]

I've written AttributeDict a fair number of times. Each time I write it
from scratch, which is only a few lines. And I only make a silly wore about
50% of the time when I do so.

I wonder if a separate type in collections might be a more natural way to
get the desired effect. I do recognize that such a type is very similar to
SimpleNamespace, so get Raymond's concern with making a new thing for
trivial change. But the *name* of the class should help with it's purpose
IMO.

On Wed, Apr 15, 2020, 12:04 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> [GvR]
> > We should not try to import JavaScript's object model into Python.
>
> Yes, I get that.  Just want to point-out that working with heavily nested
> dictionaries (typical for JSON) is no fun with square brackets and
> quotation marks.
>
>
> Raymond
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/G5SJKRQ7S5VY3JKLAVOTCCA7RSDUNWXS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C44IWFALVZ6RUIKLECSGAWZB3KXX3FAO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Guido van Rossum]

On Tue, Apr 14, 2020 at 9:08 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> [GvR]
> > We should not try to import JavaScript's object model into Python.
>
> Yes, I get that.  Just want to point-out that working with heavily nested
> dictionaries (typical for JSON) is no fun with square brackets and
> quotation marks.
>

Yeah, I get that too. So maybe this should be limited to JSON? Could the
stdlib json library be made to return objects that support this (maybe with
an option)?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQDXH3YTCV64KWHQZYE3XOEGY77POLTD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Kyle Stanley]

Raymond Hettinger wrote:
>  Yes, I get that.  Just want to point-out that working with heavily
nested dictionaries (typical for JSON) is no fun with square brackets and
quotation marks.

I can certainly agree with that sentiment, especially when working with
something like GraphQL that tends to return deeply nested JSON objects.
Repeatedly using [''] can get quite tiresome (and not look particularly
great) with something like this:

```
for pr_edge in pr_json['data']['user']['pullRequests']['edges']:
 for comment_edge in pr_edge['node']['comments']['edges']:
  commenter = comment_edge['node']['author']['login']
  ...
```

(Extracted from a personal side-project I worked on last year)

On Wed, Apr 15, 2020 at 12:10 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> [GvR]
> > We should not try to import JavaScript's object model into Python.
>
> Yes, I get that.  Just want to point-out that working with heavily nested
> dictionaries (typical for JSON) is no fun with square brackets and
> quotation marks.
>
>
> Raymond
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/G5SJKRQ7S5VY3JKLAVOTCCA7RSDUNWXS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DUL6QQ4VPGUSDKF4XBUTRTKJNRSOGY62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Glenn Linderman]

On 4/14/2020 9:25 PM, Guido van Rossum wrote:
On Tue, Apr 14, 2020 at 9:08 PM Raymond Hettinger 
mailto:raymond.hettin...@gmail.com>> wrote:


[GvR]
> We should not try to import JavaScript's object model into Python.

Yes, I get that.  Just want to point-out that working with heavily
nested dictionaries (typical for JSON) is no fun with square
brackets and quotation marks.


Yeah, I get that too. So maybe this should be limited to JSON? Could 
the stdlib json library be made to return objects that support this 
(maybe with an option)?


I replied in the issue, not noticing the discussion was happening here 
after getting interrupted with something else.


But I use my implementation (uploaded to the issue) for many things 
besides JSON. Such a feature is just too practical not to be Pythonic.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IGM424TB3L7FR536PFJZN2IYILGGA6YP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Cameron Simpson]

On 14Apr2020 21:25, Guido van Rossum  wrote:

On Tue, Apr 14, 2020 at 9:08 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

[GvR]
> We should not try to import JavaScript's object model into Python.

Yes, I get that.  Just want to point-out that working with heavily nested
dictionaries (typical for JSON) is no fun with square brackets and
quotation marks.


Yeah, I get that too. So maybe this should be limited to JSON? Could the
stdlib json library be made to return objects that support this (maybe with
an option)?


I find this feels slightly special purpose. Though admirably restrained.  
I think I dislike the need to detour thorough the json module to get 
such a feature.


Like many others, I recently implemented one of these 
__getattr__+__getitem__ SimpleNamespaces.  I'm hacking on some mappings 
which map dotted-names to values.  So the natural implementation is 
dicts or dict subclasses.  But I'm also feeding these to format strings, 
so I want to write:


 "Hi {dotted.name}"

so I've made a nesting of SimpleNamespace so that I can use nice dotted 
a.b.c.d type names in the format string. And because I'm using 
.format_map, the top level namespace also supports __getitem__.


Now, my dict subclass has a .ns() method returning one of the above 
SimpleNamespace subclasses, but I can readily imagine a utility function 
in collection that made such a thing.


Restricting that to only work via some contrived path through the JSON 
module seems... clunky.


Cheers,
Cameron Simpson 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LSS5XR2AU5NX62AWCWB5QDFOPATELP7L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Nathaniel Smith]

On Tue, Apr 14, 2020 at 9:26 PM David Mertz  wrote:
>
> I've written AttributeDict a fair number of times. Each time I write it from 
> scratch, which is only a few lines. And I only make a silly wore about 50% of 
> the time when I do so.

I've also written it a number of times, and never found a way to do it
that I was really happy with. (In particular, converting all sub-dicts
into AttributeDict is necessary to support a.b.c-style access, but if
c is itself a dict, then you end up leaking AttributeDict objects into
other parts of the code that might just be expecting a regular dict.)

These days I've given up on that approach and use Mahmoud's 'glom'
library instead: https://glom.readthedocs.io/

It has a ton of super-fancy features, but mostly I ignore those and
just write stuff like 'glom(json_obj, "a.b.c")' or maybe
'glom(json_obj, "a.b.c", default=None)'.

Like anything there are probably trade-offs and situations where
something like AttributeDict is better, but figured I'd throw that out
there as another option to consider.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q6U3AG3QRDBZU4RSV77CSYNJ62WJXYWY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvement to SimpleNamespace

[Glenn Linderman]

On 4/14/2020 10:09 PM, Cameron Simpson wrote:

On 14Apr2020 21:25, Guido van Rossum  wrote:

On Tue, Apr 14, 2020 at 9:08 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

[GvR]
> We should not try to import JavaScript's object model into Python.

Yes, I get that.  Just want to point-out that working with heavily 
nested

dictionaries (typical for JSON) is no fun with square brackets and
quotation marks.


Yeah, I get that too. So maybe this should be limited to JSON? Could the
stdlib json library be made to return objects that support this 
(maybe with

an option)?


I find this feels slightly special purpose. Though admirably 
restrained.  I think I dislike the need to detour thorough the json 
module to get such a feature.


Like many others, I recently implemented one of these 
__getattr__+__getitem__ SimpleNamespaces.  I'm hacking on some 
mappings which map dotted-names to values.  So the natural 
implementation is dicts or dict subclasses.  But I'm also feeding 
these to format strings, so I want to write:


 "Hi {dotted.name}"

so I've made a nesting of SimpleNamespace so that I can use nice 
dotted a.b.c.d type names in the format string. And because I'm using 
.format_map, the top level namespace also supports __getitem__.


Now, my dict subclass has a .ns() method returning one of the above 
SimpleNamespace subclasses, but I can readily imagine a utility 
function in collection that made such a thing.


Restricting that to only work via some contrived path through the JSON 
module seems... clunky.


Cheers,
Cameron Simpson 


Interesting comments. So .ns allows you to convert a nested dict to a 
dotted one "on demand" but the initial conversion only needs to do the 
top level object?


That sounds similar to the glom that Nathaniel brought to the attention 
of this thread, which I hadn't found before.  But glom requires bulkier 
syntax. When I read his example, I thought to myself "What if one could 
write his 'glom(json_obj, "a.b.c")' example as  json_obj['a.b.c']   or 
better as non_json_obj.a.b.c ?  I can imagine an implementation of my 
first "what-if" fully in the top-level object, but not of the second 
(but my internals imagination is probably limited).


Indeed, many of my use-cases are non-json, although a bunch are json also.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYPDZ4I65DWTS6F7FBNE2WGGG2DXZV6N/
Code of Conduct: http://python.org/psf/codeofconduct/