[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

2021-01-27 Thread Antoine Pitrou 
On Wed, 27 Jan 2021 11:05:28 +1100
Steven D'Aprano  wrote:
> On Tue, Jan 26, 2021 at 12:08:03PM -0800, Brett Cannon wrote:
> > On Tue, Jan 26, 2021 at 1:26 AM Antoine Pitrou  wrote:  
> 
> [...]
> > > Disagreed.  This is niche enough that it warrants a long but explicit
> > > name, rather than some ambiguous shortcut.
> > >  
> > 
> > I agree w/ Antoine that a more descriptive name for such a niche (but
> > useful!) attribute makes sense.  
> 
> This descriptive name is *literally incorrect*. By design, it doesn't 
> list modules. It only lists sub-packages and not sub-modules, to keep 
> the number of entries more managable.
> 
> (Personally, I don't think an extra hundred or two names makes that much 
> difference. Its going to be a big list one way or the other.)
> 
> So by the current rules, many stdlib modules are not included and the 
> name is inaccurate.

Ok, then "stdlib_package_names"? :-)

Regards

Antoine.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/NMDFLQDCXRQNUBMHTHTH37OMDPCCQYRZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

2021-01-27 Thread Steven D'Aprano 
On Wed, Jan 27, 2021 at 10:44:00AM +0100, Antoine Pitrou wrote:

> Ok, then "stdlib_package_names"? :-)

Heh :-)

I see your smiley, and I'm not going to argue about the name any 
further. I have my preference, but if the consensus is 
stdlib_module_names, so be it.

But I think the inconsistency between sub-modules and sub-packages is 
important. We should either list all sub-whatever or none of them, 
rather than only some of them.

-- 
Steve
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/4YYPKUGD4HCFXUNBYXFNNTDDOUF7ZRR2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

2021-01-27 Thread Victor Stinner 
Hi Steven,

That makes sense to me: I wrote
https://github.com/python/cpython/pull/24353 to exclude sub-package.

The change removes 12 sub-packages from sys.stdlib_module_names and
len(sys.stdlib_module_names) becomes 300 :-)

-"concurrent.futures",
-"ctypes.macholib",
-"distutils.command",
-"email.mime",
-"ensurepip._bundled",
-"lib2to3.fixes",
-"lib2to3.pgen2",
-"multiprocessing.dummy",
-"xml.dom",
-"xml.etree",
-"xml.parsers",
-"xml.sax",

With that name, names of sys.stdlib_module_names don't contain "." anymore.

So to check if "email.message" is a stdlib module name, exclude the
part after the first dot, and check if "email" is in
sys.stdlib_module_names. In practice, it is not possible to add a
sub-package or a sub-module to a stdlib module, so this limitation
(excluding sub-packages and sub-modules) sounds reasonable to me.

Victor


On Wed, Jan 27, 2021 at 1:09 AM Steven D'Aprano  wrote:
> This descriptive name is *literally incorrect*. By design, it doesn't
> list modules. It only lists sub-packages and not sub-modules, to keep
> the number of entries more managable.
>
> (Personally, I don't think an extra hundred or two names makes that much
> difference. Its going to be a big list one way or the other.)
>
> So by the current rules, many stdlib modules are not included and the
> name is inaccurate.
>
> If you're not going to list all the dotted modules of a package, why
> distinguish sub-modules from sub-packages? It is confusing and ackward
> to have only some dotted modules listed based on the **implementation**.
>
> (We need a good term for "things you can import that create a module
> *object* regardless of whether they are a *module file* or a *package*.
> I'm calling them a dotted module for lack of a better name.)
>
> By the current rules, many stdlib modules are not listed, and you can't
> see why unless you know their implementation:
>
>
> *  urllib - listed
> *  urllib.parse - not listed
>
> *  collections - listed
> *  collections.abc - not listed
>
> *  email - listed
> *  email.parser - not listed
> *  email.mime - listed  # Surprise!
>
>
> So we have this weird situation where an implementation detail of the
> dotted module (whether it is a file `package/module.py` or
> `package/module/__init__.py`) determines whether it shows up or not.
>
> And because the file system structure of a module is not part of its
> API, that implementation detail could change without warning.
>
> I think that either of:
>
> 1. list *all* package dotted modules regardless of whether they are
>implemented as a sub-module or sub-package;
>
> 2. list *no* package dotted modules, only the top-level package;
>
> would be better than this inconsistent hybrid of only listing some
> dotted modules.
>
> (Excluding the test modules is fine.)

-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/UW6F2MRC5RNOLEJJI64BALENK7R7UYA2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New sys.module_names attribute in Python 3.10: list of all stdlib modules

2021-01-27 Thread Victor Stinner 
On Wed, Jan 27, 2021 at 1:16 AM Steven D'Aprano  wrote:
> Right. This is (I think) Steve's point: the list is inaccurate, because
> the existence of 'winsound' in the stdlib_module_names doesn't mean that
> the module 'winsound' exists.

This point is addressed by the definition of the list:
sys.stdlib_module_names documentation.

"It is the same on all platforms. Modules which are not available on
some platforms and modules disabled at Python build are also listed.
All module kinds are listed: pure Python, built-in, frozen and
extension modules. Test modules are excluded."

https://docs.python.org/dev/library/sys.html#sys.stdlib_module_names

As I wrote previously, there are use cases which *require* the list
being the same on all platforms. Moreover, in practice, it's quite
hard to build a list of available stdlib module names. You need to
build extension modules, try to implement them, then rebuild the list
of module which requires to rebuild Python. It's not convenient. Also,
there are different definition of "available". For example, "import
multiprocessing" can fail on some platforms if there is no lock
implementation available. It's not because it's installed on the
system that the import will work for sure.

IMO the only reliable way to check if a module can be imported... is
to import it. And then you hit again the issue of import side effects.

There are different ways to filter sys.stdlib_module_names list to
only list "available" modules. Try import, pkgutil.iter_modules() or
pkgutil.walk_packages(). IMO it should remain out of the scope of
sys.stdlib_module_names.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/IKFK6CTTYTWD2VFH36AIN5IGS66KSMFA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

2021-01-27 Thread Nick Coghlan 
As far as I'm aware, the design is in a potentially acceptable state, I
just stalled out completely on the boring bits of finishing the
implementation of the write-through proxy:

* implement & test the rest of the mutable mapping methods
* refactor to properly share code with the odict implementation instead of
copying & pasting it

So a co-author would definitely be most welcome, given I've been
procrastinating on that part for literally years at this point.

Cheers,
Nick.


On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:

> Sounds good to me. Have you talked to Nick?
>
> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>
>> Hi,
>>
>> PEP 558 seems to be dormant, if not abandoned.
>>
>> There are at least two open issues for bugs resulting from the currently
>> weird and inefficient behavior of `f_locals` and `locals()`.
>> See https://bugs.python.org/issue30744 for an example of undesirable
>> behaviour.
>>
>> PEP 588, or something like it, would fix those.
>>
>> I'd be happy to take over the PEP, or write a new one.
>> I like PEP 588, although I would propose a simplification.
>>
>> The PEP mentions "tracing mode" and changes behavior according to
>> whether a program is in "tracing mode" or not. I'd like to remove this
>> distinction.
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/[email protected]/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> --
> --Guido (mobile)
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/J2DQDKEYVH3IQBEXEJLBUEQ4GYHU6X4F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

2021-01-27 Thread Nick Coghlan 
Note that PEP 558 already doesn't change behaviour in tracing mode any more
though - that idea didn't survive the first round of review.

Cheers,
Nick.

On Wed, 27 Jan 2021, 9:58 pm Nick Coghlan,  wrote:

> As far as I'm aware, the design is in a potentially acceptable state, I
> just stalled out completely on the boring bits of finishing the
> implementation of the write-through proxy:
>
> * implement & test the rest of the mutable mapping methods
> * refactor to properly share code with the odict implementation instead of
> copying & pasting it
>
> So a co-author would definitely be most welcome, given I've been
> procrastinating on that part for literally years at this point.
>
> Cheers,
> Nick.
>
>
> On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:
>
>> Sounds good to me. Have you talked to Nick?
>>
>> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>>
>>> Hi,
>>>
>>> PEP 558 seems to be dormant, if not abandoned.
>>>
>>> There are at least two open issues for bugs resulting from the currently
>>> weird and inefficient behavior of `f_locals` and `locals()`.
>>> See https://bugs.python.org/issue30744 for an example of undesirable
>>> behaviour.
>>>
>>> PEP 588, or something like it, would fix those.
>>>
>>> I'd be happy to take over the PEP, or write a new one.
>>> I like PEP 588, although I would propose a simplification.
>>>
>>> The PEP mentions "tracing mode" and changes behavior according to
>>> whether a program is in "tracing mode" or not. I'd like to remove this
>>> distinction.
>>>
>>> Cheers,
>>> Mark.
>>> ___
>>> Python-Dev mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/[email protected]/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> --
>> --Guido (mobile)
>>
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/GJSORMCNKKZUAXHJQOBBOQUKIHE2RX22/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

2021-01-27 Thread Nick Coghlan 
Hmm, I need to review the state of my PEP PRs, as what's in the main repo
definitely isn't up to date. The tracing mode distinction should be long
gone, but is still mentioned in the text on python.org.

Cheers,
Nick.

On Wed, 27 Jan 2021, 10:03 pm Nick Coghlan,  wrote:

> Note that PEP 558 already doesn't change behaviour in tracing mode any
> more though - that idea didn't survive the first round of review.
>
> Cheers,
> Nick.
>
> On Wed, 27 Jan 2021, 9:58 pm Nick Coghlan,  wrote:
>
>> As far as I'm aware, the design is in a potentially acceptable state, I
>> just stalled out completely on the boring bits of finishing the
>> implementation of the write-through proxy:
>>
>> * implement & test the rest of the mutable mapping methods
>> * refactor to properly share code with the odict implementation instead
>> of copying & pasting it
>>
>> So a co-author would definitely be most welcome, given I've been
>> procrastinating on that part for literally years at this point.
>>
>> Cheers,
>> Nick.
>>
>>
>> On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:
>>
>>> Sounds good to me. Have you talked to Nick?
>>>
>>> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>>>
 Hi,

 PEP 558 seems to be dormant, if not abandoned.

 There are at least two open issues for bugs resulting from the
 currently
 weird and inefficient behavior of `f_locals` and `locals()`.
 See https://bugs.python.org/issue30744 for an example of undesirable
 behaviour.

 PEP 588, or something like it, would fix those.

 I'd be happy to take over the PEP, or write a new one.
 I like PEP 588, although I would propose a simplification.

 The PEP mentions "tracing mode" and changes behavior according to
 whether a program is in "tracing mode" or not. I'd like to remove this
 distinction.

 Cheers,
 Mark.
 ___
 Python-Dev mailing list -- [email protected]
 To unsubscribe send an email to [email protected]
 https://mail.python.org/mailman3/lists/python-dev.python.org/
 Message archived at
 https://mail.python.org/archives/list/[email protected]/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
 Code of Conduct: http://python.org/psf/codeofconduct/

>>> --
>>> --Guido (mobile)
>>>
>>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/G6K3VT33NR2XWZRAPN3XHZJ5R24H7IUV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

2021-01-27 Thread Nick Coghlan 
On Tue, 26 Jan 2021 at 01:05, Mark Shannon  wrote:
> The PEP mentions "tracing mode" and changes behavior according to
> whether a program is in "tracing mode" or not. I'd like to remove this
> distinction.

OK, I've checked the status of things from an actual computer rather
than my phone, and it turns out this is just confusing wording on my
part: the PEP still makes it sound like the "tracing mode" distinction
will persist into the future, even though the actual goal now is to
eliminate that distinction.

I've made a new PR to the PEP to try to clear that up:
https://github.com/python/peps/pull/1783/files

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/4KATOFB2YMC7VUETL24FG2OMUKR6Q57V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Does the C-API include C structs not explicitly documented?

2021-01-27 Thread Mark Shannon 

Hi everyone,

Are C structs part of the C-API if the struct is only included in a 
header, but not documented?


I have always assumed not, but it isn't clear.

This question arose in https://github.com/python/cpython/pull/24298

Obviously, any structs listed in 
https://docs.python.org/3/c-api/structures.html are part of the API


Thoughts?

Cheers,
Mark.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/S45H4YOOHCFRGLDIU4XRTPRANP5D2KSD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Does the C-API include C structs not explicitly documented?

2021-01-27 Thread Guido van Rossum 
Unfortunately the public headers act as de-facto documentation.

On Wed, Jan 27, 2021 at 8:10 AM Mark Shannon  wrote:

> Hi everyone,
>
> Are C structs part of the C-API if the struct is only included in a
> header, but not documented?
>
> I have always assumed not, but it isn't clear.
>
> This question arose in https://github.com/python/cpython/pull/24298
>
> Obviously, any structs listed in
> https://docs.python.org/3/c-api/structures.html are part of the API
>
> Thoughts?
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/S45H4YOOHCFRGLDIU4XRTPRANP5D2KSD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/YM65CFSTTI2UF2YB4SSDE5JI4QAH6IPD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] What is the pyclbr public API?

2021-01-27 Thread Terry Reedy 
pyclbr is the stdlib module browser (once just class browser, hence the 
name).  The doc
https://docs.python.org/3/library/pyclbr.html#module-pyclbr, which I 
revised in 2017, documents readline and readline_ex as the public call 
interface.  The functions return a hierarchical structure that includes 
Function and Class instances.  (Both subclass pyclbr._Object.)  The doc 
I revised already omitted signatures for these classes and just listed 
the attributes of instances.  Just before that listing, I added "Users 
are not expected to create instances of these classes."


https://bugs.python.org/issue38307 and
https://github.com/python/cpython/pull/24348
propose to add an end_lineno attribute. Since pyclbr is now, since last 
November, based on the ast tree and since the relevant ast nodes have an 
end_line attribute, the proposal amounts to copying that attribute, 
along with others, instead of ignoring it.


The patch proposes to add 'end_lineno' after existing start 'lineno' in 
the _Object, Function, and Class signatures.  I prefer doing this, 
rather than adding the new parameter at the end of the list, because a) 
being the the logical place would make the code easier to read, and b) 
new names as the end of the signature, follow optional arguments, would 
have to be optional, whereas


My question is whether the omission of signatures and my added sentence 
sufficiently defines the signatures of these classes (in particular, the 
argument order) as private to approve the patch as is?



--
Terry Jan Reedy
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/SQWNNGY7WADHFGAIQZRIMPPLYJGIV4TZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: What is the pyclbr public API?

2021-01-27 Thread Guido van Rossum 
It's likely that the additions are going to break someone's code; the
question seems to be do we care, since the constructors are undocumented?

But shouldn't we just document the signatures, so that in the future we're
not going to make the same mistake?

I also wonder if some judicious use of keyword-only args might help users
who currently call the old constructors catch the breakage where it's
easily diagnosed (i.e. at the call site) instead of once the object is
instantiated (e.g. "why is end_lineno/parent the wrong type")?

Perhaps even add some type checks to catch obvious mistakes (could be as
simple as `assert isinstance(end_lineno, int)").

On Wed, Jan 27, 2021 at 2:36 PM Terry Reedy  wrote:

> pyclbr is the stdlib module browser (once just class browser, hence the
> name).  The doc
> https://docs.python.org/3/library/pyclbr.html#module-pyclbr, which I
> revised in 2017, documents readline and readline_ex as the public call
> interface.  The functions return a hierarchical structure that includes
> Function and Class instances.  (Both subclass pyclbr._Object.)  The doc
> I revised already omitted signatures for these classes and just listed
> the attributes of instances.  Just before that listing, I added "Users
> are not expected to create instances of these classes."
>
> https://bugs.python.org/issue38307 and
> https://github.com/python/cpython/pull/24348
> propose to add an end_lineno attribute. Since pyclbr is now, since last
> November, based on the ast tree and since the relevant ast nodes have an
> end_line attribute, the proposal amounts to copying that attribute,
> along with others, instead of ignoring it.
>
> The patch proposes to add 'end_lineno' after existing start 'lineno' in
> the _Object, Function, and Class signatures.  I prefer doing this,
> rather than adding the new parameter at the end of the list, because a)
> being the the logical place would make the code easier to read, and b)
> new names as the end of the signature, follow optional arguments, would
> have to be optional, whereas
>
> My question is whether the omission of signatures and my added sentence
> sufficiently defines the signatures of these classes (in particular, the
> argument order) as private to approve the patch as is?
>
>
> --
> Terry Jan Reedy
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/SQWNNGY7WADHFGAIQZRIMPPLYJGIV4TZ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/64UDFAO7472BVQRABDKVXIIDT42RZBKJ/
Code of Conduct: http://python.org/psf/codeofconduct/