Re: [Python-Dev] Policy on refactoring/clean up

[Kyle]

In itself, the code clean up you have done is a good thing in the sense
that you re-organized things and in my understanding, they look good now.
In some of the teams I've been granted to work, there was a rule stating
that whenever a dev would work on an enhancement/bugfix, he would create a
separate ticket that would track all code refactoring related to his work.
The strategy is quite different here and the suggestion was to have your
refactoring be part of an enhancement/bugfix. The most valuable argument in
favor of all the current reviewers is Guido van Rossum's comment on the
effect of having git blame made harder.
For some of the projects I'm currently working on, we have code cleanup
teams where members are affected expressly to refactoring code. We might
end up needing such things in the future but for now, it is wiser to leave
the code as is and focus on fixing actual problems.

-- 
*Q::Drone's Co-Founder and Co-CEO*
 *Hervé "Kyle" MUTOMBO*

*Envoyé depuis le web.*
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Brett Cannon wrote:
>  I agree with Raymond that if the calendar module was following the leading
> underscore practice (which we should probably encourage all new modules to 
> follow for
> consistency going forward) then I think the module should be updated to keep 
> the practice
> going.
> -Brett

Rather than it being on a case-by-case basis, would it be reasonable to 
establish a universal standard across stdlib for defining modules as public to 
apply to older modules as well? I think that it would prove to be quite 
beneficial to create an explicit definition of what is considered public. If we 
don't, there is likely to be further confusion on this topic, particularly from 
users. 

There would be some overhead cost associated with ensuring that every 
non-public function is is proceeded by an underscore, but adding public 
functions to __all__ could safely be automated with something like this 
(https://bugs.python.org/issue29446#msg287049): 

__all__ = [name for name, obj in globals().items() if not name.startswith('_') 
and not isinstance(obj, types.ModuleType)]

or a bit more lazily:

__all__ = [name for name in globals() if not name.startswith('_')]

Personally, I think the benefit of avoiding confusion on this issue and 
providing consistency to users would far outweigh the cost of implementing it.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QIBJF3EFHN2REFJEQPMPGL6JTAJT56GZ/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Serhiy Storchaka wrote:
> Either we establish the rule that all non-public names must be 
> underscored, and do mass renaming through the whole stdlib. Or allow to 
> use non-underscored names for internal things and leave the sources in 

Personally, I would be the most in favor of doing a mass renaming through 
stdlib, at least for any public facing modules (if they don't start with an 
underscore, as that already implies the entire module is internal). Otherwise, 
I have a feeling similar issues will be brought up repeatedly by confused 
end-users. 

This change would also follow the guideline of "Explicit is better than 
implicit" by explicitly defining any function in a public-facing module as 
private or public through the existence or lack of an underscore. There would 
be some cost associated with implementing this change, but it would definitely 
be worthwhile if it settled the public vs private misunderstandings.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CUKFA46FAYGJAAKXZJN6COGOSOPHAGX2/

[Python-Dev] Re: The order of operands in the comparison

[Kyle Stanley]

Serhiy Storchaka wrote:
> Thank you. The majority of the code uses needle on the right. There are 
> just 6 places where it is on the left, and the half of them look 
> copy-pasted, and in one place the C code differs from the corresponding 
> Python implementation. There is an order more places where needle is on 
> the right (and it is always on the right in the Python code). So I 
> prefer to fix these few places to get rid of the minor inconsistency.

If there are only a few places where the needle is on the left and there's not 
a specific functionality requirement/limitation in those locations which 
prevent it from being on the right, it's better to have consistency. Is there 
any particular benefit from having the needle on the left side, or was it just 
a personal preference from the developer who first implemented the code in 
those 6 locations?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYQ5YFUWLGULRUX5T36KD6EJEKKY3L22/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Brett Cannon wrote:
> Same would happen with a rename where people's code suddenly broke. We
> don't do renames on purpose without a proper deprecation cycle and doing that
> en-mass would be extremely disruptive.

Good point, this would probably have to be a gradual change if it did happen, 
rather than being at all once. If it were to happen with a proper deprecation 
cycle and clear communication, I think it would result in significantly less 
confusion overall and provide a higher degree of consistency across stdlib in 
the long term.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FYV3Q3YW2KWZGZ7VKMF2AXY3DUNDHWE6/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Upon further consideration and reading your response, I'm starting to think 
that the proposal to perform a mass renaming across stdlib might have been a 
bit too drastic, even if it was done over a longer period of time. Thanks for 
the detailed explanation of the costs, that significantly improved my 
understanding of the situation.

My primary motivation was to provide more explicit declaration of public vs 
private, not only for the purpose of shifting the responsibility to the 
authors, but also to shift the liability of using private members to the user. 

>From my perspective, if the communication is 100% clear that a particular 
>function is not public, the developers are able to make changes to it more 
>easily without being as concerned about the impact it will have on users. 
>Nothing prevents the users from using it anyways, but if a change that occurs 
>to a private function breaks their functionality, it's completely on them. 
>With the current system, users can potentially make the argument that they 
>weren't certain that it the function or module in question was private. Being 
>concerned about breaking the functionality for users on non-public functions 
>seems to entirely defeat the purpose of them. 

I also dislike the idea of adding the underscores or dealing with it on a 
case-by-case basis, due to the inconsistency it would provide across stdlib. In 
some cases the inconsistency might be necessary, but I'd rather avoid it if 
possible. 

Also, is the rule  "unless explicitly documented public, all 
imports are private even if not prefixed with an underscore" officially stated 
anywhere, or is it mostly implied? Personally, I think that it should be 
explicitly stated in a public manner if it's the methodology being followed.

A solid alternative proposal would also be Barry's public decorator proposal: 
https://public.readthedocs.io/en/latest/. I remember him saying that it was 
largely rejected by the community when it was proposed, but I'm not certain as 
to why. It would be far easier to implement something like this than it would 
be to rename all of the non-public functions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OA7KTZFYJKKNGLX6GRGNMUGBZ34MDMWC/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Barry Warsaw wrote:
> This leads to the second problem, which is that it’s too easy for the __all__ 
> to get
> out of sync with the module’s contents. Often a function or class is renamed, 
> removed, or
> added without the __all__ being updated. 

IMO, this seems to be the best part of the @public decorator, at least from a 
general user's perspective. Manually having to update __all__ anytime something 
is added, removed, or has its name modified adds an extra step to easily 
forget. Also, those coming from other languages would be far more likely to 
recognize the significance of @public, the primary purpose of the decorator is 
quite clear based on it's name alone.

Barry Warsaw wrote:
> My package has a C version.  If public() were a builtin (which I’ve 
> implemented)
> it wouldn’t have that much import time overhead.

Is there a way we could estimate the approximate difference in overhead this 
would add using benchmarks? If it's incredibly trivial, I'd say it's worthwhile 
in order to make the public API members more explicitly declared, and provide a 
significant QoL improvement for maintaining __all__. While we should strive to 
optimize the language as much as possible, as far as I'm aware, Python's main 
priority has been readability and convenience rather than pure performance.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J4JPURDHBKXH5B6Z52G3GYRHYPZ4HUCD/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Steve Dower wrote:
> So I apologise for mentioning that people care about import performance. 
> Let's ignore them/that issue for now and worry instead about making sure 
> people (including us!) know what the canonical reference for 
> public/internal is.

Good point, the discussion about __all__, adding the @public decorator, and 
worrying about performance impacts might be jumping too far ahead. 

For now, if most of the core devs are in agreement with the current unwritten 
rule of  "unless explicitly documented public, all imports are private even if 
not prefixed with an underscore", I think the first priority should be to 
document it officially somewhere. That way, other developers and any potential 
confused users can be referred to it.

It might not be the best long term solution, but it would require no code 
changes to be made and provide a written canonical reference for 
differentiating public vs private. I'm not certain as to where the most 
appropriate location for the rule would be, let me know if anyone has any 
suggestions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5XFBD2DECM234BYIEVCFQXCZP57S4FSY/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

I find myself in agreement with Inada (https://bugs.python.org/issue12445), in 
that comparing the values view between two dictionaries by itself would not be 
particularly useful for enough people to warrant implementing the comparison. 
In most situations when using the data structure, it is only useful to either 
compare the keys and values with ``d0.items() == d1.items()`` or just the keys 
with ``d0.keys() == d1.keys()``. 

The values are generally not particularly useful without the corresponding 
keys, so I'm actually somewhat curious as to the motivation of creating the 
function ``dict.values()``. But, if for any reason someone actually had to 
compare only the values (I can't imagine the reason), they could compare them 
by converting them to a list: ``list(d0.values()) == list(d1.values())``. It 
adds an extra step, but I don't think enough people would make use of something 
like this to justify adding the direct comparison with ``d0.values() == 
d1.values())``. 

However, I agree that the current behavior of just returning ``False`` is quite 
misleading, regardless of whether or not implementing an accurate comparison 
between the values views would be worthwhile. I'm not sure as to what the most 
appropriate behavior would be, but since it's using ``__eq__``, 
[NotImplemented](https://docs.python.org/3/library/constants.html#NotImplemented)
 seems appropriate. 

Another alternative would be to return ``None``. A note in the docs for 
[NotImplementedError](https://docs.python.org/3/library/exceptions.html#NotImplementedError)
 states "It [NotImplementedError] should not be used to indicate that an 
operator or method is not meant to be supported at all – in that case either 
leave the operator / method undefined or, if a subclass, set it to None".
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KX2TG7ZPVOWKUNDILV74YNFTBZTJHUP5/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

Serhiy Storchaka wrote:
> Is there any precedence of raising an exception in the equality comparison?
> Does 3 == "3" returning False make more sense to you?

Personally, I don't find ``3 == "3"`` to be an equivalent comparison to 
``d0.values() == d1.values()``. Generally, it makes sense when comparing two 
items of different types, they are not going to be equivalent (except in cases 
such as ``3 == 3.0``, but in that case they are both subtypes of numeric). 

I don't know that an exception would be the best behavior to suit this 
situation (or for anything using ``__eq__`` for that matter), but returning 
``False`` seems to be a bit misleading. Instead, I think that either returning 
the ``NotImplemented`` constant or ``None`` would provide far more useful 
information to the user, without the hindrance of causing an exception. I'm 
leaning more favorably towards ``NotImplemented`` because it explicitly tells 
the user "Hey, that equality comparison isn't implemented".
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SJKDNGBIEBCL67NWL7CPGTVNSY2QLXBD/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

Eric V. Smith wrote:
>That makes things worse. Now the comparison is always true in a boolean 
>context. And presumably you'd want __ne__ to also return >NotImplemented, 
>so then both __eq__ and __ne__ would be true, since >bool(NotImplemented) 
>is True.

Eric V Smith wrote:
7/25/2019 6:00 AM, Eric V. Smith wrote:
>I might have to take that back. I hadn't factored in what the == and != 
>machinery does, beyond calling __eq__ or __ne__.

Based on the behavior in this example class, it looks like this would still 
function appropriately, despite the value of bool(NotImplemented):

>>>class A:
   def __eq__(self, other):
   return NotImplemented
>>>a = A()
>>>b = A()
>>> a == b
False
>>> a != b
True

As you said, I believe it has to do with the underlying behavior of __eq__ and 
__ne__. However, this does somewhat lead to the same surface level issue of a 
False ultimately being returned. The NotImplemented makes the intention a bit 
more obvious if someone were to look at the __eq__ method for dict.values(), 
but otherwise it might still be the same issue. I'm somewhat curious as to why 
`a == b` doesn't directly return NotImplemented instead of False. Perhaps the 
underlying behavior makes it a pain to return anything other than True, False, 
or None (that's purely speculation on my part, I haven't looked further into 
it).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HTFXIN7L2UBEJ26AY2BTF5FEYA7MVUEX/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

Brett Cannon wrote:
> You're correct that I misspoke, but I personally still think a doc change
> is the best solution.

I would agree that a doc change should occur if it is decided that the current 
behavior is appropriate, but I would like to mention that in the current 
[documentation for 
`object.__eq__()`](https://docs.python.org/3/reference/datamodel.html#object.__eq__),
 it states: "A rich comparison method may return the singleton `NotImplemented` 
if it does not implement the operation for a given pair of arguments". 

Wouldn't returning `NotImplemented` be far more explicit to the user, in terms 
to directly telling them that the equality assessment between two dictionary 
views is not implemented? In general, I find this to be far more informative 
than simply returning False. At a surface level, users may assume that False 
would imply that there was an actual assessment of equality being performed. 

This may not be an established precedent for other similar equality 
assessments, but I don't think the `NotImplemented` constant is utilized as 
much as it could be. It seems to be particularly well suited for addressing 
this situation.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XHIXFB3BFOKQTWGUBUE5AJWHWAXXIGRD/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

Serhiy Storchaka wrote:
> Actually, the == operator cannot return NotImplemented.

Thanks for the clarification. What is the reason for this limitation and is it 
only possible for the `==` operator to return one of `None`, `False`, or 
`True`? It seems like it would be useful for it to be able to return 
`NotImplemented` in situations such as this.

Also, I think that I may have had some misconceptions with regards to the 
relationship between the `__eq__()` method and the `==` operator. I know they 
are not the same, but isn't the result of the `==` operator based on a 
transformation of the result from `__eq__()`?

As far as I can tell, the equality of two dictionary views are assessing used 
[`PyObject 
dictrich_compare`](https://github.com/python/cpython/blob/544fa15ea1b7b73068319bdb217b684e2fd7bacc/Objects/dictobject.c#L4040).
 Wouldn't it be possible to perform a conditional check if the view on the left 
side of the comparison is a values view and if so, use 
`Py_RETURN_NOTIMPLEMENTED`?

Apologies if I'm completely off base here, my experience and understanding of 
the underlying C-API is quite limited. I've been programming with Python for 
quite some time, but I only started learning the C-API once I became interested 
in contributing to CPython.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/72XO7ZWL6XZLQGDSHFSBENBHK5OG7HD4/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

Terry Reedy wrote:
> Given the absence of a consensus on when values() views should be 
> considered equal, I strongly agree.  I strongly oppose raising an exception.

I am with you regarding the strong opposition regarding the raising of an 
exception. I don't think that the `==` operator should raise an exception, 
doing so is excessively obstructive to the user.

I'm not certain that returning `False` is the best behavior, but based on what 
I've gathered from the discussion so far there has been nothing suggested that 
would be a viable alternative. I had initially proposed returning 
`NotImplemented`, but upon further assessment, that would still end up 
returning `False` when using the `==` operator. As a result, leaving it as is 
and addressing the behavior in the docs seems to be the most appropriate 
solution.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JAEML64F62M7ZB3U656CBE4YXQRU74QF/

[Python-Dev] Re: Comparing dict.values()

[Kyle Stanley]

> I want a real-world application which requires it.
> Without a strong use case, I think the discussion is just wasting time.

I would have to agree. Initially I was in support of changing the behavior, but 
upon reading the responses of several core developers and further 
consideration, the most appropriate course of action seems to be updating the 
docs. I have not seen any relevant applications where it would be useful to 
compare the values view between dictionaries, but I agree that the behavior of 
returning `False` might be confusing without any mention of it.

I [opened a PR](https://github.com/python/cpython/pull/14954) which mentions 
this behavior in the relevant documentation, but further explanation of why 
this occurs might be appropriate. I'm not certain as to whether or not further 
explanation is needed in this situation though.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D6XWQDTNQWXFD5ZKOSM3RTBZEFBXN6KM/

[Python-Dev] Re: 🍝 New keyword in bpo: `newcomer friendly`

[Kyle Stanley]

> Essentially those "easy" issues aren't so easy,
> and we're starting over.

Is "easy" still going to be used for intermediate level issues or
will it be no longer used? Apologies if this was already discussed
in the initial thread, I don't have access to core-mentorship.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBZUMAZPLGAJUWDOYN66RX6FW5PAZPCN/

[Python-Dev] Re: 🍝 New keyword in bpo: `newcomer friendly`

[Kyle Stanley]

Not sure if this would be the right place to mention it,
but the side menu url/button "Easy issues" in bpo
should probably be renamed to "Newcomer friendly"
and the "dispname" property in the url could also be
similarly updated. This could be done by changing the
url from: 

[current](https://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50&nosy=31554)
 
=>
[updated](https://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Newcomer%20friendly&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50&nosy=31554)

Apologies if this should be posted somewhere other
than the mailing list, but I figured there probably
wouldn't be an issue tracker for the issue tracker, 
that would be some serious BPO inception and
quite the paradox.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GDPOWKDKOBVRUAG5EHGVC5E6EKRCYEP5/

[Python-Dev] Re: 🍝 New keyword in bpo: `newcomer friendly`

[Kyle Stanley]

Clarification: The url change is only for updating the
"dispname" property, which is shown at the top of 
the page. It would change the header from:

"List of issues - Easy issues"
=>
"List of issues - Newcomer friendly"

Also, I realized that I accidentally included the "nosy"
property at the end, which sets it to only show
anything that I'm on the nosy list for. The "keywords"
property would also have to be updated, it looks like
6 currently represents Easy, not sure what the
value for Newcomer friendly would be.

If the Easy keyword is going to stick around, it might
be better to just have a separate Newcomer friendly
link on the side menu.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AEOGHA2EJ4QOFXAC4CO5OHK4AWO6XLRR/

[Python-Dev] Re: [core-workflow] Re: Re: %G🍝%@ New keyword in bpo: `newcomer friendly`

[Kyle Stanley]

Stephen J. Turnbull wrote:
> In general, it's not obvious to me how much extra burden we want to
> put on triagers.  This tag has basically the same problems that "easy"
> does, except that it deters experienced developers from snatching up
> easy issues.  I think we should see how this works out in practice
> before reducing the number of issues tagged and at the same time
> increasing burden.
> Steve

That's why I think we should continue to use the "easy" tag to
describe easier issues targeted towards those with some basic
familiarity, but with less experience and "newcomer friendly" to
be the easier version of "easy". Newcomer friendly should
optimally be reserved for incredibly quick/trivial fixes that serve
as an introduction to the process more than an actual issue
which needs to be addressed.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ILMPXTBFHIHCSC4ZDNDR4GNGKSSFVD24/

[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

[Kyle Stanley]

+1 to remove support for Solaris going forward. 4 years is plenty of time
to wait for someone to volunteer to maintain it, IMO. So my preference
would be for option 3 to remove it now, but I wouldn't be opposed to option
2 either w/ deprecating support and waiting a couple versions to remove it.
I'm only opposed to option 1, since it seems very likely that it will just
continue to stagnate at this point.

On Thu, Oct 29, 2020 at 5:49 PM Victor Stinner  wrote:

> Hi,
>
> I propose to drop the Solaris support in Python to reduce the Python
> maintenance burden:
>
>https://bugs.python.org/issue42173
>
> I wrote a draft PR to show how much code could be removed (around 700
> lines in 65 files):
>
>https://github.com/python/cpython/pull/23002/files
>
> In 2016, I asked if we still wanted to maintain the Solaris support in
> Python, because Solaris buildbots were failing for longer than 6
> months and nobody was able to fix them. It was requested to find a
> core developer volunteer to fix Solaris issues and to set up a Solaris
> buildbot.
>
>
> https://mail.python.org/archives/list/python-dev@python.org/thread/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/#NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS
>
> Four years later, nothing has happened. Moreover, in 2018, Oracle laid
> off the Solaris development engineering staff. There are around 25
> open Python bugs specific to Solaris.
>
> I see 3 options:
>
> * Current best effort support (no change): changes only happen if a
> core dev volunteers to review and merge a change written by a
> contributor.
>
> * Schedule the removal in 2 Python releases (Python 3.12) and start to
> announce that Solaris support is going to be removed
>
> * Remove the Solaris code right now (my proposition): Solaris code
> will have to be maintained outside the official Python code base, as
> "downstream patches"
>
>
> Solaris has a few specific features visible at the Python level:
> select.devpoll, os.stat().st_fstype and stat.S_ISDOOR().
>
> While it's unclear to me if Oracle still actively maintains Solaris
> (latest release in 2018, no major update since 2018), Illumos and
> OpenSolaris (variants or "forks") still seem to be active.
>
> In 2019, a Solaris blog post explains that Solaris 11.4 still uses
> Python 2.7 but plans to migrate to Python 3, and Python 3.4 is also
> available. These two Python versions are no longer supported.
>
> https://blogs.oracle.com/solaris/future-of-python-on-solaris
>
> The question is if the Python project has to maintain the Solaris
> specific code or if this code should now be maintained outside Python.
>
> What do you think? Should we wait 5 more years? Should we expect a
> company will offer to maintain the Solaris support? Is there a
> motivated core developer to fix Solaris issue? As I wrote, nothing has
> happened in the last 4 years...
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VDD7NMEDFXMOP4S74GEYJUHJRJPK2UR3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N3CQPEPNENRC6L7NIALN3JGA3O33UDD3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Who is target reader of tutorial?

[Kyle Stanley]

I can't speak for all of the members of the upcoming documentation WG, but
as someone that will be on it (based on our discussions at the recent core
dev sprint), my personal vote would be for keeping it as a comprehensive
guide for beginners of Python. Detailed enough that it covers the
fundamentals, but skips out on more advanced topics and info that is likely
going to be less applicable to most new users.

I would consider __cause__ and __context__ to fall under the category of
being "useful to know, but far from essential for beginners". You can have
a decent understanding of most python programs without ever knowing about
those two dunders.  So I would definitely be in favor of removing the
mention of __cause__ and not adding __context__, in my opinion it adds
extra unneeded complexity.

That being said, I'm not opposed to something like having something like a
footnote at the bottom of the section that links to a more advanced topic
(maybe an exception how-to?). It should be at the bottom of the section so
it doesn't add extra cognitive overhead for those trying to grasp the
basics, but allows for those who are interested to get a more in-depth
understanding if they want to. I would suggest redirecting the contributor
who proposed those changes to working on something like an exception
how-to; assuming that's something they're interested in.

On Thu, Nov 5, 2020 at 2:07 PM Brett Cannon  wrote:

> A documentation WG is going to be formed which will be in a better
> position to answer this, so until that WG is started I think we should keep
> the tutorial aimed towards beginners.
>
> On Thu, Nov 5, 2020 at 1:13 AM Inada Naoki  wrote:
>
>> Hi, all.
>>
>> Since "How To" guide is not organized well, it is very tempting to
>> write all details in tutorial.
>> I have seen may pull requests which "improve" tutorial by describe
>> more details and make
>> the tutorial more perfect.
>>
>> But adding more and more information into tutorial makes it slower to
>> learn Python by reading tutorial.
>> 10+ years ago, Python is far less popular in Japan and reading
>> tutorial is the best way to learn Python to me.
>>
>> But now Python is popular and there are many free/paid good books and
>> tutorials on the Web.
>> Some of them would be more new user friendly than official tutorial.
>> Then, should official Python tutorial become detailed guide to Python?
>> Or should we keep new user learning Python as targeted reader?
>>
>> There is ongoing issue for example: https://bugs.python.org/issue42179
>>
>> Chaining exception was added in tutorial.  Current tutorial mention to
>> `__cause__` attribute.
>> https://docs.python.org/3/tutorial/errors.html#exception-chaining
>>
>> bpo-42179 proposes to add mention to `__context__` to make the
>> tutorial more accurate about implicit chaining.
>> And https://github.com/python/cpython/pull/23160 is the pull request
>> to mention `__context__`.
>>
>> On the other hand, I want to remove confusion by removing mention to
>> `__cause__`.
>> Because I don't think `__context__` and `__cause__` is important for new
>> users.
>> See https://github.com/python/cpython/pull/23162 for my proposal.
>>
>> Regards,
>>
>> --
>> Inada Naoki  
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/MXMEFFYB6JXAKSS36SZ7DX4ASP6APWFP/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IWW2YBLJK4T3OWSKDUDVDVXPWDGIFWTC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FWQWF6BNVILXLYGZ7VVFBWP7OV23URSD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please do not remove random bits of information from the tutorial

[Kyle Stanley]

On Sun, Nov 8, 2020 at 8:17 PM Inada Naoki  wrote:

> OK. Since checking all mails in the long thread is tedious job, I will
> pick some up and leave a comment in the b.p.o.
>

Personally, I think that just linking to the python-dev thread in bpo
(and/or PR) is adequate for most cases, especially if there was a fairly
strong consensus pointing in one direction.

(More detail can be helpful of course, but if one can only spare enough
time to just place a link on the issue, it does quite a lot by itself IMO.)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AF2QAEPVWPSWS6ANXXUHUUMFKR5THPC3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please do not remove random bits of information from the tutorial

[Kyle Stanley]

 Rather than trying to specifically transform the existing tutorial into a
guide exclusively aimed at beginners, I think that we should use the
guideline of: "Is this useful information in 95% of real-world use cases or
does it have a strong niche purpose that will be useful at *some *point for
significant number of Python users?" If not, my opinion is that it should
be removed and replaced with a footnote link for more information (if the
existing documentation is too technical, we can have it linked to a
separate HOWTO which explains it in a way that's easier to parse).

On Sat, Nov 7, 2020 at 9:16 PM Inada Naoki  wrote:

> I think `from exc` syntax is not new Python users should know.
> Documenting implicit chaining is enough for 99% use cases, and `from
> None` covers 0.99% of the rest.
> So I considered removing explicit chaining (e.g. `from exc`) from the
> section too.
> But I kept it, because it's a "syntax showcase" even though it will
> give more noise to beginners.
>
> And deleting `__cause__` is not solery for "didactic" reason, nor
> "loosing something precious."
> As written in the b.p.o. issue, mention only `__cause__` "lose some
> precious".
> We need to describe `__context__` and `__suppress_context__` too.
> But all of them are documented in library/exceptions.html.
> Removing `__cause__` and adding a link to library/exceptions.html
> makes sense more than documenting all.
>

So, I think that in a case where as Inada mentioned above, it's not going
to be needed for 99% of use cases, my opinion is that the extra information
is just consuming valuable real estate in the tutorial that adds a higher
burden to readers trying to learn Python for little gain. I'm less
concerned about removing parts that are "too difficult" and more concerned
with removing parts that are never realistically going to be useful for the
vast majority of users (outside of something like implementing the
internals of some ultra-niche library).

__cause__ is a great example of this, but I would be hesitant to remove
explicit chaining; that would require much more consideration because it
has a significantly better chance of being relevant for users or at least
appearing in code at some point without specifically trying to find it. So
I agree with keeping the explicit exception chaining part.

There shouldn't be a concern that we'll be "removing random bits of
information from the tutorial", any change to the tutorial will be made
intentionally and with much consideration. Also, even as a core dev that
has spent a decent amount of time reviewing CPython documentation PRs, I
would personally wait until having 3+ opinions prior to merging any
removals of information or other substantial changes in the tutorial (with
at least one other core dev, preferably two), and encourage others to do
the same since the tutorial is a very fundamental part of the docs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2XKNMFRSK2YUDFDCQFX7XLIHUFQOLTAH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v2: Explicit constraint patterns *without* question marks in the syntax

[Kyle Stanley]

On Sat, Nov 14, 2020 at 7:54 AM Nick Coghlan  wrote:

> On Sat, 14 Nov 2020 at 09:51, Greg Ewing 
> wrote:
> >
> > On 14/11/20 7:45 am, Brandt Bucher wrote:
> > > with (using your own syntactic flavor):
> > > ```
> > > case >first, *>middle, >last:
> > >  rebuilt = first, *middle, last
> > > case {"key": >value, **>rest}:
> > >  rebuilt = {"key": value, **rest}
> > > case Point(x=>a, y=>b):
> > >  rebuilt = Point(x=a, y=b)
> >
> > I think this is a case where syntax matters. To my eyes this
> > looks far less confusing:
> >
> > case ?first, *?middle, ?last:
> >   rebuilt = first, *middle, last
> > case {"key": ?value, **?rest}:
> >   rebuilt = {"key": value, **rest}
> > case Point(x=?a, y=?b):
> >   rebuilt = Point(x=a, y=b)
>
> Based on the discussion in this thread, v3 of PEP 642 is going to
> propose spelling these as:
>
> case first, *middle, last:
>  rebuilt = first, *middle, last
> case {"key" as value, **rest}:
>  rebuilt = {"key": value, **rest}
> case Point(x as a, y as b):
>  rebuilt = Point(x=a, y=b)
>
> I'm in agreement with the PEP 634 authors that we want to keep
> sequence pattern matching consistent with iterable unpacking as far as
> name binding is concerned. Mapping patterns and class patterns are
> both completely new though, and I think the readability problem there
> is quite similar to the one that existed with walrus patterns in PEP
> 622, and thus amenable to a similar solution (i.e. spell the capture
> patterns for those cases with `as`, not `:` or `=`).
>

FWIW, I'd like to add my +1 to usage of "as" for spelling class capture
patterns. This is by far the clearest and easiest to read form I've seen
thus far, and I suspect that it would be the easiest to explain to users
already familiar with usage of "as" from other areas in Python. A new
feature being as distinguishable as possible and easy to explain to
existing users is very important in my book, and based on the responses, I
think that the current "=" form used in PEP 634 for matching class patterns
would be substantially more difficult for users to mentally parse and
understand compared to "as".

It's also worth considering new Python users that might have general OO
experience but not with robust pattern matching (e.g. Java-heavy
backgrounds). I could definitely see "case Point(x=a, y=b):" being confused
for instantiation, whereas usage of "as" makes it more clear that something
else is happening (hopefully leading them to search around for more info
about Python pattern matching).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VK6YAMK5NSOZRNRDHLHL7UH67LGVL3HU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v2: Explicit constraint patterns *without* question marks in the syntax

[Kyle Stanley]

On Sun, Nov 15, 2020 at 1:56 AM Chris Angelico  wrote:

> On Sun, Nov 15, 2020 at 4:28 PM Kyle Stanley  wrote:
> >
> > FWIW, I'd like to add my +1 to usage of "as" for spelling class capture
> patterns. This is by far the clearest and easiest to read form I've seen
> thus far, and I suspect that it would be the easiest to explain to users
> already familiar with usage of "as" from other areas in Python. A new
> feature being as distinguishable as possible and easy to explain to
> existing users is very important in my book, and based on the responses, I
> think that the current "=" form used in PEP 634 for matching class patterns
> would be substantially more difficult for users to mentally parse and
> understand compared to "as".
> >
> > It's also worth considering new Python users that might have general OO
> experience but not with robust pattern matching (e.g. Java-heavy
> backgrounds). I could definitely see "case Point(x=a, y=b):" being confused
> for instantiation, whereas usage of "as" makes it more clear that something
> else is happening (hopefully leading them to search around for more info
> about Python pattern matching).
> >
>
> case Point(x=as a, y=as b):
>
> That doesn't read well to me.
>
> Or is there some other spelling of 'as' that makes better sense to you?
>

The post from Nick that I was primarily replying to used the following
spelling: "case Point(x as a, y as b):".
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBSWERRHKBYAB7OW5HS3E4FGVTXDTJUP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please explain how to migrate when a function is removed, thanks ;-)

[Kyle Stanley]

Thanks for bringing attention to this, Victor, and to Ken Jin (GH:
Fidget-Spinner) for the PR. I've just completed reviewing and merging the
PR, so hopefully anyone affected will now have a more clear idea of how to
migrate their asyncio code to 3.10. Having the porting method explicitly
documented certainly helps to smooth the version transition process and
reduce headaches. :-)

Going forward, I'll try to make more of an active effort to ensure any
potentially incompatible changes I'm involved with include a clear method
of porting documented in their respective whatsnew. It can be easy to
forget at times that a seemingly minor fix which is intuitively clear to
the authors of a change may not be as clear to those not involved with it,
regardless of how difficult the fix actually is.

On Tue, Jan 19, 2021 at 12:03 PM Victor Stinner  wrote:

> A PR was proposed to document the removal of the loop parameter:
> https://github.com/python/cpython/pull/24256
>
> Victor
>
> On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner 
> wrote:
> >
> > Hi,
> >
> > We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
> > are facing many Python 3.10 incompatible changes. Most changes were
> > planned with a deprecation period, but, as usual, projects are not
> > tested with DeprecationWarning treated as errors, and so tons of
> > packages now fail to build.
> >
> > I'm not here to talk about DeprecationWarning, but how we communicate
> > on incompatible changes made on purpose. For example, What's New in
> > Python 3.8 announces: "In asyncio, the explicit passing of a loop
> > argument has been deprecated and will be removed in version 3.10". As
> > expected, the parameter was removed in Python 3.10 (bpo-42392), but I
> > cannot see anything in What's New in Python 3.10. The problem is that
> > I don't know how to fix projects broken by this change. I'm not
> > complaining about this specific change, but more generally.
> >
> > I strongly suggest to well document incompatible changes. The bare
> > minimum is to mention them in "Porting to Python 3.10" section:
> > https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
> >
> > A link to the bpo sometimes helps to understand how to port code. But
> > I would really appreciate if authors of incompatible changes would
> > explain how to add Python 3.10 support to existing projects, without
> > losing support for older Python versions. Not just "this function is
> > now removed, good luck!" :-)
> >
> > I didn't touch asyncio for at least 1 year, so I don't know what
> > happens if I remove a loop argument. Does an application remain
> > compatible with Python 3.6 without passing loop?
> >
> > I know that I made multiple incompatible changes myself and I'm sure
> > that the documentation should probably be enhanced, but I also expect
> > others to help on that! ;-)
> >
> > Please think about people who have to port 4000 Python projects to
> Python 3.10!
> >
> > ---
> >
> > The best would be to ship a tool which adds Python 3.10 support to
> > existing projects without losing support with Python 3.6. Maybe
> > something like https://github.com/asottile/pyupgrade could be used for
> > that? pyupgrade looks more specific to the Python syntax, than the
> > usage of the stdlib.
> >
> > I wrote such tool to add Python 3.10 support to C extensions without
> > losing support with Python 2.7. It relies on a header file
> > (pythoncapi_compat.h) which provides new C API functions on old Python
> > versions.
> > https://github.com/pythoncapi/pythoncapi_compat
> >
> > For example, it replaces "obj->ob_refcnt = refcnt;" with
> > "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
> > 3.8 and older (function added to Python 3.9)
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RGSEPSSPDIVKWYKACONKRUFBEXQ5YKGC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please explain how to migrate when a function is removed, thanks ;-)

[Kyle Stanley]

On Jan 20, 2021 at 9:51 PM Chros Jerdonek  wrote:

> Is there / would it make sense to have a section analogous to "Porting to
> Python X" that covers "Make All DeprecationWarnings Go Away in X"? If we
> had such a section, the "Porting to" section could be constructed by
> copying the relevant bits from that section in the previous release.


+1 from me to include this section in 3.10 and future whatsnew.

On Thu, Jan 21, 2021 at 2:17 PM Mariatta  wrote:

> I agree that when we land a feature that introduces incompatible change
> like this, as part of the PR it should also include updating the
> documentation on how to migrate.
> I would think that the feature should not be merged unless the doc has
> been updated too.
>
> Perhaps we should include this explicitly in devguide, as one of the
> things to consider when reviewing pull requests.
> Do we have such a guide/doc yet, on how to review CPython pull requests?
>

This seems like it would be a good location to include that information:
https://devguide.python.org/pullrequest/#how-to-review-a-pull-request

Maybe also a brief mention in the checklist for
https://devguide.python.org/committing/#is-the-pr-ready-to-be-accepted.
Step 9 currently has: "Was “What’s New” updated (as appropriate)?", which
could have a sub-point to mention "Incompatible changes should include
migration information in their respective "What's New".".
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E7NYJ5DUQM46GXKCQTUWRXBJUQ5DOAHR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] The importance of mental health

[Kyle Stanley]

Hey all,

In these last few months, I have been in the process of healing from some
pretty heavy past trauma. And now that I am on the road to recovery, I want
to share my journey with the Python community in hopes that it may reach
those that are struggling with their own mental health battles, as many of
us are during these dark and difficult times.

Trigger warning that it includes a decent amount of highly personal
content, so only check it out if you are okay with that:
https://discuss.python.org/t/break-from-open-source/6372/7?u=aeros.

To anyone that would limit my employment opportunities as a result of
having had these struggles, *that's perfectly okay*. I kept the post in the
private section because I was originally in fear of discriminate. However,
I have reached an important conclusion: *I would not want to work for your
company if you discriminate against people who have overcome past struggles*
.

-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3H6JIULWOTFYMJLWOBFO7PM4RAOEL74U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Farewell, for now :)

[Kyle Stanley]

Hi all,

Last week, I started a new thread on discuss.python.org
<https://discuss.python.org/t/farewell-for-now/8918?u=aeros> about my
intention to take a further extended break from open source to continue my
mental health healing process. Just wanted to announce it in the other
channels as well since I know that not everyone has the bandwidth to keep
up with more than just the MLs.

In the thread, I discussed my intention to pursue the path of becoming a
Buddhist monk for some time, and recently detailed my adventures at a local
Thai temple. Check it out if you are interested. :)

With Loving Regards,
-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5KN37I5PSMLSBEER3NWWVQCP7DBYPD3Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 661: Sentinel Values

[Kyle Stanley]

 As someone who's had to make use of the pattern `_sentinel = object()` a
few times within stdlib code, I'd like to give a strong +1 for the proposal
to add a new `sentinel()` function. This is much more intuitive, easier to
look up, etc. From my experience, it's a common enough pattern to be well
worth the addition.

Thanks for the proposal, Tal. :)

On Sun, Jun 6, 2021 at 4:14 AM Tal Einat  wrote:

> Hi,
>
> I have prepared a PEP proposing adding a stdlib function for defining
> sentinel values. You can find the PEP here:
>
> https://www.python.org/dev/peps/pep-0661/
>
> The discussion is happening in the discourse server:
>
> https://discuss.python.org/t/pep-661-sentinel-values/9126
>
> To avoid splitting the discussion, *please redirect your comments there*
> instead of replying to this thread.
>
> Have a nice week,
> - Tal Einat
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/3QNLVLSMVSQ5MGLXRIQ5QM4BA5OJCVEN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UM7EPUUSQPJAEU6QCA3FUOH3ULNUSXGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why list.sort() uses mergesort and not timsort?

[Kyle Stanley]

On Sun, Jun 6, 2021 at 7:09 PM Dan Stromberg  wrote:

> I've got a comparison of sort algorithms in both Cython and Pure Python
> (your choice) at:
> https://stromberg.dnsalias.org/~strombrg/sort-comparison/
> ...including a version of timsort that is in Cython or Pure Python.
>

Thanks for sharing the graphs. I found the performance of radix sort in
particular to be interesting to see mapped out visually, and have never
heard of "shellsort" prior to now. :)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BVHSMLAOJKVDXJO37LF5WY7OEM6D5YWX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is the Python review process flawed?

[Kyle Stanley]

On Thu, Jul 1, 2021 at 10:49 AM Victor Stinner  wrote:

> What happens usually is that some modules have no maintainer. Once you
> merge a single change in a module, boom! You are now the new
> maintainer for your entire life time! More and more people will ask
> you to look at their "very important" bug blocking their production
> and their very specific use case. You will get more and more reviews
> request: "since you merged a single change 12 months ago, I'm sure
> that you will love to review my precius bugfix! Come on, it's
> trivial!". Now you realize that you don't know the design of the
> module. You don't know its history. You know nothing, and the entire
> world now have very high expectation, since you merged an obvious and
> trivial change.
>
“The ancient Oracle said that I was the wisest of all the Greeks. It is
because I alone, of all the Greeks, know that I know nothing.” ~  Socrates

The same could largely be applied to maintaining open source (especially a
codebase as large as CPython). The truth is that at a fundamental level,
we're all (any software developer) just making educated guesses as to
whether or not patches are suitable for merging, or whether or not a bug
fix will actually work. Tests make us a little more certain, but it is
still a guess. So the problem is really just the unrealistic expectation
that there exists people who are able to make a decision with absolute 100%
certainty that no regressions or future unexpected issues will occur as a
result of the change.

Also, for fun, here's a pythonized version of the quote:
“[Guido] said that I was the wisest of all the [pythonistas]. It is because
I alone, of all the [pythonistas], know that I know nothing.”
:)

With loving-kindness,
-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SGDPAXGBI2627HFBK7NV5GQWDXDSRR4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bug report

[Kyle Stanley]

On Wed, Jul 21, 2021 at 12:19 PM Nguyen Do Minh Duc <
gd.minhduc1...@gmail.com> wrote:

> Hi,
>
> When I find what's new in 3.10 beta in
> https://docs.python.org/whatsnew/3.10.html
> It redirected me to https://docs.python.org/3/whatsnew/3.10.html which
> shows 404 error not found nginx. Can you fix this?
>

Thanks for the report! I don't believe the code and/or config for the web
server hosting docs.python.org is a public repo (may be mistaken), so I'll
CC the director of infrastructure, Ee W. Durbin. I suspect they will know
who to forward this to (or be able to fix it).

Best Regards,
-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAJM6PFMX2DXOGPEC6I2QATAUT2GNJXC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: python-dev thread w/ Marco Sulla

[Kyle Stanley]

I can agree with the general premise of what Antoine is saying, but to me
even as a non-participant, the following quote from the thread Brett linked
seems a clear CoC violation:

I repeat, even the worst AI will understand from the context what I meant.
> But let me do a very rude example:
>
> What if I said to Steven "I pretend immediate suck my $%#$"? Do you think
> you and the others will not understand the sense? :D
>

This at least indicates a lengthy timeout is needed, imo.

On Mon, Aug 16, 2021 at 1:01 PM Antoine Pitrou  wrote:

> On Mon, 16 Aug 2021 09:47:13 -0700
> Brett Cannon  wrote:
> >
> https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/
> >
> > I can't be objective with Marco as I believe we have recorded issues with
> > him previously (as with Steven if you take Marco's initial side with
> this).
> >
> > The thing that pushed me over the edge to report this was
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/O3JB3FE33KMT3OHZCVH3XO6VNJTGH5NL/
> > .
> >
>
> I think a private appeal to the two or three participants to remain
> cool, nice and polite would be welcome here.  Also an appeal to
> think twice before posting... (is what you're going to say really going
> to benefit the community?)
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OUKC6WY5VCAD2ZIKQZQUHOP5LUTSDMF7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L5SEEI5DLPU2UMGERSKTQCMVCG4TAAQI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bug fixed in the bugs.python.org OAuth-based authentication: user logged as the wrong account

[Kyle Stanley]

Thanks for the fix! This could have caused some serious issues, so glad we
were able to address it ahead of time.

On Mon, Sep 13, 2021 at 5:06 AM Victor Stinner  wrote:

> Hi,
>
> A bug has been identified and *fixed* in the OAuth-based
> authentication code used on the Python bug tracker bugs.python.org
> (BPO) to log in with GitHub, Launchpad or Google. Under some
> conditions, it was possible to be logged as another person account. We
> are only aware of a single user affected by the issue. We are not
> aware of any account takeover.
>
> All bugs at bugs.python.org are public: being logged as the wrong
> account cannot give access to private bugs. The main risk is if an
> attacker could be logged as an administrator (the "Coordinator" role)
> which allows to change the bug tracker configuration and to change
> accounts (add/remove roles, see/change the email address, etc.). We
> are not aware of any abuse.
>
> All OAuth accounts have been removed in the database to fully fix the
> issue. Users using OAuth-based authentication must associate again
> (once) their GitHub, Launchpad or Google account with their BPO
> account.
>
> A BPO account contains the following information: Name, Login Name,
> GitHub Name, Organisation, Timezone, Homepage, Contributor Form
> Received, Is Committer, E-mail address, Alternate E-mail addresses.
> All fields but Name and Timezone are hidden to other accounts, only
> coordinators can see all fields of other accounts. You can check in
> the "Your Details" page for the your account change log.
>
> Thanks Ammar Askar, Berker Peksağ and Ee Durbin who fixed the bug!
>
> Source code of bugs.python.org (Roundup fork):
> https://github.com/psf/bpo-tracker-cpython
>
> The OAuth-based authentication is an extension written for
> bugs.python.org. The bug report and its fix:
>
> * https://github.com/python/bugs.python.org/issues/64
> *
> https://github.com/psf/bpo-tracker-cpython/commit/0a32e072aafca20c0bf51cf16fb6a7328cdd720a
>
> Report issues with bugs.python.org:
> https://github.com/python/bugs.python.org/issues
>
> To report sensitive issues, write to: secur...@python.org
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CIXIB6EMN7HOPMXFJI7EBK7V7OPK4E7H/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OH7V7GJ6GTQJM3OBIXZ72IZXA4KSLVVH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Kyle Stanley]

I'd suggest both: briefer, easier to read write up for average user in
docs, more details/semantics in informational PEP. Thanks for working on
this, Petr!

On Tue, Nov 2, 2021 at 2:07 PM David Mertz, Ph.D. 
wrote:

> This is an amazing document, Petr. Really great work!
>
> I think I agree with Marc-André that putting it in the actual Python
> documentation would give it more visibility than in a PEP.
>
> On Tue, Nov 2, 2021, 1:06 PM Marc-Andre Lemburg  wrote:
>
>> On 01.11.2021 13:17, Petr Viktorin wrote:
>> >> PEP: 
>> >> Title: Unicode Security Considerations for Python
>> >> Author: Petr Viktorin 
>> >> Status: Active
>> >> Type: Informational
>> >> Content-Type: text/x-rst
>> >> Created: 01-Nov-2021
>> >> Post-History:
>>
>> Thanks for writing this up. I'm not sure whether a PEP is the right place
>> for such documentation, though. Wouldn't it be more visible in the
>> standard
>> Python documentation ?
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Experts (#1, Nov 02 2021)
>> >>> Python Projects, Coaching and Support ...https://www.egenix.com/
>> >>> Python Product Development ...https://consulting.egenix.com/
>> 
>>
>> ::: We implement business ideas - efficiently in both time and costs :::
>>
>>eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>Registered at Amtsgericht Duesseldorf: HRB 46611
>>https://www.egenix.com/company/contact/
>>  https://www.malemburg.com/
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/FSFG2B3LCWU5PQWX3WRIOJGNV2JFW4AU/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6PHPDZRCYNA44NHSHXPBL7QMWXMHXWGU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6OET4CKEZIA34PAXIJR7BUDKT2DPX2DG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

[Kyle Stanley]

On Sat, Nov 13, 2021 at 5:04 PM  wrote:

>
>
> def 𝚑𝓮𝖑𝒍𝑜():
>
> try:
>
> 𝔥e𝗅𝕝𝚘︴ = "Hello"
>
> 𝕨𝔬r𝓵ᵈ﹎ = "World"
>
> ᵖ𝖗𝐢𝘯𝓽(f"{𝗵ｅ𝓵𝔩º_}, {𝖜ₒ𝒓lⅆ︴}!")
>
> except 𝓣𝕪ᵖｅ𝖤𝗿ᵣ𝖔𝚛 as ⅇ𝗑c:
>
> 𝒑rℹₙₜ("failed: {}".𝕗𝗼ʳᵐªｔ(ᵉ𝐱𝓬))
>
>
>
> if _︴ⁿ𝓪𝑚𝕖__ == "__main__":
>
> 𝒉eℓˡ𝗈()
>
>
>
>
>
> # snippet from unittest/util.py
>
> _𝓟Ⅼ𝖠𝙲𝗘ℋ𝒪Lᴰ𝑬𝕽﹏𝕷𝔼𝗡 = 12
>
> def _𝔰ʰ𝓸ʳ𝕥𝙚𝑛(𝔰, p𝑟𝔢fi𝖝𝕝𝚎𝑛, ｓᵤ𝑓𝗳𝗂𝑥𝗹ₑ𝚗):
>
> ˢ𝗸ｉ𝗽 = 𝐥ｅ𝘯(𝖘) - ｐr𝚎𝖋𝐢x𝗅ᵉ𝓷 - 𝒔𝙪ffｉ𝘅𝗹𝙚ₙ
>
> if sｋi𝘱 > _𝐏𝗟𝖠𝘊𝙴H𝕺Ｌ𝕯𝙀𝘙﹏L𝔈𝒩:
>
> 𝘴 = '%s[%d chars]%s' % (𝙨[:𝘱𝐫𝕖𝑓𝕚ｘℓ𝒆𝕟], ₛ𝚔𝒊p, 𝓼[𝓁𝒆𝖓(
> 𝚜) - 𝙨𝚞𝒇fix𝙡ᵉ𝘯:])
>
> return ₛ
>

0_o color me impressed, I did not think that would be legal syntax. Would
be interesting to include in a textbook, if for nothing else other than to
academically demonstrate that it is possible, as I suspect many are not
aware.

-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ZA2KK46JHENKSPB52RMXBAQT7CP3Q6A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Steering Council elections.

[Kyle Stanley]

On Mon, Nov 15, 2021 at 10:49 AM Thomas Wouters  wrote:

>
> Just a reminder that the nomination period for the next SC ends *today*
> (AoE), so if you're intending to nominate (yourself or someone else),
> please get those posts in. (No need for a long post before the deadline, it
> can be expanded later.) We currently have the 4 incumbents, and nobody
> else, so please consider who you would like to see on the SC.
>

Thanks for the reminder, Thomas. Although I have nobody to nominate or
consider myself to have the capacity and experience to self-nominate for
SC, it is great to see that the incumbents desire to have a wide pool of
applicants for the longevity of Python; rather than being on the SC
indefinitely.

Best Regards,
-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PHBQHDHQJMJPT26EPJMLANIPWQ6W2GFM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Remove asyncore, asynchat and smtpd modules

[Kyle Stanley]

I think it would be fine to wait just one release, until 3.12. Makes no
substantial maintenance difference and maybe easier for users with more
advanced notice, especially for module removal.

I also wonder if maybe we should scale delay between dep -> removal based
on maintenance burden estimate, rather than 2 versions for all. Module
removal certainly takes more effort to adjust in code vs simple function
name change with 1:1 replacement.

-- 
--Kyle R. Stanley, Python Core Developer (what is a core dev?
<https://devguide.python.org/coredev/>)
*Pronouns: they/them **(why is my pronoun here?*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F52UKISVFFGBNCL2AZ4XVX2KL35O6ZNH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please be careful about changing PEPs post-submission to the SC

[Kyle Stanley]

FWIW that seems like a reasonable approach, at least to me.

On Thu, Nov 18, 2021, 5:29 PM Guido van Rossum  wrote:

> I know PEP 646 was one of these. In our defense, we *did* notify the SC
> that there was a pending issue (
> https://github.com/python/steering-council/issues/59#issuecomment-951728233),
> although at the time we didn't anticipate it to become such a contentious
> discussion between the PEP authors. (Though, while contentious, it's still
> a minor edge case in the PEP, and I don't think it would affect the SC's
> position which way we eventually end up going.)
>
> I'm guessing that the recommended approach in such a case is just to close
> the SC issue and reopen it once the PEP is updated?
>
> On Thu, Nov 18, 2021 at 12:04 PM Brett Cannon  wrote:
>
>> This is a personal plea (i.e. not coming from the SC at all), but in the
>> last month we have had PEPs changed twice post-submission to the SC. That's
>> a big time sink as we take multiple meetings to discuss a PEP and having
>> things change underneath us causes us to have to re-evaluate our
>> discussions (and I know I pretty much start thinking about PEPs once they
>> are submitted, whether we are actively discussing them or not and I'm
>> probably not the only SC member who does this).
>>
>> I know no one did this maliciously or anything, but since it's happened
>> twice now I just want to ask people be cognizant of this. Please reach out
>> to the SC if you want to make a change so we can discuss whether we think
>> it will help/hurt the PEP, etc. and we are also not taken off-guard by
>> things shifting (assume we don't monitor the commits and PRs to the peps
>> repo, so unless you explicitly say, "hold on", we won't realize discussions
>> are ongoing in a PR or anything). If that means withdrawing your PEP for
>> consideration for a while that's totally fine and it won't hurt your
>> chances of acceptance once you're at a stable state with your PEP.
>>
>> Once again, this is a personal ask and no one is mad at anyone. I'm just
>> asking people be very clear in communicating with us when they want to make
>> a change to a PEP or they have suddenly have an open issue they are still
>> discussing after they open an issue in the steering-council repo for us to
>> review a PEP and need us to stop considering their PEP for a while.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ZLC3H2XLEYJLFV3TRQ2EWRKRGZZ7DRMC/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WJOTJ6V7XKRMDESEF3BD4DHOP4C3TRNK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/44UCT4MC2NGSHSWBHOBSB363G3RPA5IW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: peps, devguide, voters repositories: master branch renamed to main

[Kyle Stanley]

Thanks for the update! Although possibly a minor inconvenience to some, I
am certainly of the belief that our choice in language has a subtle, yet
powerful effect on our perception of the world around us, so although some
may see it as needless, I think small changes such as "master" to "main"
are necessary; assuming we want to evolve beyond archaic standards and
become more inclusive over time.

On Sat, Dec 4, 2021 at 4:07 AM Victor Stinner  wrote:

> Hi,
>
> The "master" branch of the following Python GitHub repositories have
> been renamed to "main":
>
> * devguide
> * peps
> * voters
>
> For the rationale of the rename, see:
> https://sfconservancy.org/news/2020/jun/23/gitbranchname/
>
> If you already have a Git checkout of one of these repositories, you
> can rename your local master branch with these commands (where "orign"
> is the github.com/python/... remote):
> ---
> git branch -m master main
> git fetch origin
> git branch -u origin/main main
> git remote set-head origin -a
> ---
>
> Tell me if you have any issue with the branch rename.
>
> Thanks Brett, Ee, Mariatta and the Steering Council who helped to
> rename these branches ;-)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WUGZX7KW3B33LM7R7O2ON6NRQFDGCTSW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IUKOKO2YJNXPNVYLIJ5RFUPPOZBK6FU6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Sad news from Zurich

[Kyle Stanley]

May he rest in peace, I can't even fathom the impact PIL has had on Python
over the years.

On Fri, Dec 10, 2021 at 3:20 PM Guido van Rossum  wrote:

> A former core dev who works at Google just passed the news that Fredrik
> Lundh (also known as Effbot) has died.
>
> Fredrik was an early Python contributor (e.g. Elementtree and the 're'
> module) and his enthusiasm for the language and community were inspiring
> for all who encountered him or his work. He spent countless hours on
> comp.lang.python answering questions from newbies and advanced users alike.
>
> He also co-founded an early Python startup, Secret Labs AB, which among
> other software released an IDE named PythonWorks. Fredrik also created the
> Python Imaging Library (PIL) which is still THE way to interact with images
> in Python, now most often through its Pillow fork. His effbot.org site
> was a valuable resource for generations of Python users, especially its
> Tkinter documentation.
>
> Fredrik's private Facebook page contains the following message from
> November 25 by Ylva Larensson (translated from Swedish):
>
> """
>
> It is with such sorrow and pain that I write this. Fredrik has left us
> suddenly.
>
> """
>
> A core dev wrote: "I felt privileged to be able to study Fredrik's code
> and to read his writing. He was a huge asset to our community in the early
> days. I enjoyed his sense of humor as well. I'm sad that he passed away."
>
> We will miss him.
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> python-committers mailing list -- python-committ...@python.org
> To unsubscribe send an email to python-committers-le...@python.org
> https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-committ...@python.org/message/36Q5QBILL3QIFIA3KHNGFBNJQKXKN7SD/
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YAIZPWDPVWD74ZTA3Z7G25VFQ2URAU2Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 572 TargetScopeError

[Kyle Stanley]

Barry Warsaw wrote:
> The PEP doesn’t really go into the rationale for why a new exception is being 
> defined,
> and in the issue I’ve argued that we should just raise SyntaxError in those 
> cases.  To me,
> “TargetScopeError” is pretty obscure and doesn’t give users an obvious clue 
> as to what’s
> going wrong, without searching the interwebs.

Barry Warsaw wrote:
> Serhiy points out that we have IndentationError and TabError subclasses of 
> SyntaxError,
> but those feel different to me because the exception names themselves are 
> clear and
> descriptive, and lead to obvious actionable remedies.

I find myself in agreement that SyntaxError should be raised for the above 
reasons, as
long as the error message can adequately explain to the user that it's an issue 
with the
scope they're referring to:

``SyntaxError: Invalid scope defined by walrus operator: 'i := 10 for i in 
range(5)'``

To make it a bit more more succinct, "by walrus operator" could be potentially 
removed. 

I don't think there's anything inherently wrong with the users looking up the 
name of the
exception for more information, but it should not be required to gain a basic
understanding of what the error is referring to.

Also, I'm not entirely convinced that this would be a frequent enough 
occurrence to justify
creating a subclass of SyntaxError instead of simply using a custom message to
describe the issue.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZE3JNHAW46QCOAYQRNKYUABFK5YEJCZH/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Nick Coghlan wrote:
> It's not an unwritten rule, as it already has its own subsection in
> PEP 8: 
> https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces
> The main question in this thread is what to do about standard library
> modules that were written before those documented guidelines were put
> in place, and hence have multiple internal APIs that lack the leading
> underscore (and I don't think that's a question with a generic
> answer).
> Cheers,
> Nick.

Oh I see, thanks for the clarification. I've read over most of PEP 8 a few 
times at this point, but somehow I missed this part: "All undocumented
interfaces should be assumed to be internal". Apologies for that.

Personally, I think the stdlib modules which were written before the rule
should be gradually updated, with each one being it's own issue with
the respective experts for each of the modules carefully monitoring the changes.

It would also be appropriate to provide any user attempting to import
a module that is going to be prepended with an underscore with
warnings, and at least a couple of versions to update their code.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ICAXVGLOHF7H3GSL7OONI3Y6XDBBB54Z/

[Python-Dev] Re: What is a public API?

[Kyle Stanley]

Kyle Stanley wrote:
> It would also be appropriate to provide any user attempting to import
> a module that is going to be prepended with an underscore with
> warnings, and at least a couple of versions to update their code.

Clarification: When I mentioned prepending a module with an 
underscore, I meant for functions and classes within the module, 
not the module itself. 

It might be difficult to implement this in a way which does
not cause an excessive number of warnings, but I think it's
definitely worthwhile to aim towards having a fully consistent
standard for differentiating public and private interfaces
across all of stdlib.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2O7PZB5BGRUPPTPH4Q4MQIXUWT734BQY/

[Python-Dev] Inline links in Misc/NEWS entries

[Kyle Stanley]

Recently, on Discuss, I created a new topic: 
https://discuss.python.org/t/should-news-entries-contain-documentation-links/2127

However, many may not have the time to read the full post or don’t regularly 
check the core workflow category on Discuss, so I'll provide a shortened 
version here.

During my experience thus far reviewing PRs on GitHub, I've noticed a 
significant degree of inconsistency when it comes to the usage of inline links 
with reST and Sphinx in Misc/NEWS entries. Since many of my contributions have 
involved documentation changes, I've familiarized myself most of the syntax.

Many of the features in markup languages provide visual modifications rather 
than functional ones. However, with the inline markup supported by reST and 
processing from Sphinx, there's a functional improvement as well. For example, 
the usage of :func:\`\` (escaped for mailman) provides a link to 
the relevant docs for the function. 

In the context of news entries, this allows readers to click on a function, 
method, class, etc for more information. This can be useful if it's something 
they're not familiar with, or when the changes affected the docs. For those who 
are familiar with the structure of docs.python.org, the cross link to the docs 
may not seem at all necessary. 

However, for readers that are either newer or not familiar with the structure, 
they might be led astray into 2.7 docs or an entirely wrong page. This happens 
especially frequently when using external search engines.

I'm not at all suggesting that every PR author should be required to use it or 
know all of the reST constructs. However, I would like for everyone to be aware 
of the potential usefulness of including inline links in news entries, and 
mention it in the devguide.

Also, I would like to understand why some developers dislike including it, even 
when the reST syntax is provided. The majority of authors so far would add my 
suggestion to their PR, but there have been some that don't want anything 
besides plaintext in their news entry.

Personally, I think it provides further inclusiveness to readers of all levels 
of experience and QoL at a very minimal cost.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HTAFGTQIZJQUCU6QCVF3KFD3VFGFOBWV/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> * The BODY section should be a single paragraph of English text in ReST
format. It should not use the following ReST markup features:
> * section headers
> * comments
> * directives, citations, or footnotes
> * Any features that require significant line breaks, like lists,
definition lists, quoted paragraphs, line blocks, literal code blocks, and
tables.

As far as I can tell, the usage of inline markup for links with something
like :func:\`os.listdir\` would not be in violation of this.

Also, as a clarification, what I was describing was apparently just Sphinx,
not reST: https://devguide.python.org/documenting/#id4. For some reason, I
was under the impression that the syntax was reST and then Sphinx processed
the markup to find a matching link. I'm not certain as to whether or not
blurb supports Sphinx. If not, I wouldn't mind assisting with the process
of adding support for it, I think it's worthwhile to include inline links
when appropriate.

If not, the link could also be provided with the reST cross-link
markup: :ref:\`label\`
as long as the section has a corresponding label. This doesn't seem like it
uses any of the above restricted reST features. But if that's also an
issue, a more explicit reST hyper link could be provided with: \`Link text <
http://target>\`_. However, direct links are probably the most likely to
become deprecated or lead to dead ends, so the other options would be more
preferable for less maintenance.

Thanks,
Kyle Stanley

On Tue, Aug 13, 2019 at 2:41 AM Mariatta  wrote:

> > I would like to understand why some developers dislike including it,
> even when the reST syntax is provided.
>
> This has something to do with the use of blurb/blurb-it. Both tools
> specifically say "single paragraph with simple ReST markup".
>
> Further reading blurb's source code, it says the format of the news blurb
> should be as follows:
>
>   * The BODY section should be a single paragraph of English text
> in ReST format.  It should not use the following ReST markup
> features:
>   * section headers
>   * comments
>   * directives, citations, or footnotes
>   * Any features that require significant line breaks,
> like lists, definition lists, quoted paragraphs, line blocks,
> literal code blocks, and tables.
>
>
> Perhaps Larry has more context on why the news entry should be "simple"?
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YTST7DKDTD3K6PIUE345J25B2CMJORLY/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> Note that there is an open devguide issue that requests improvements in
the description of news entry processing

Good to know, I'll look further into this issue and see if I can help with
it.

> In general, using many Python-specific Sphinx markup entities is fine:
browsing though the consolidated blurb files for previous releases (and the
resultant changelog html) shows uses of entities like :func: and :class:.

Since it's okay to use Python-specific Sphinx, should we encourage the
usage of it when appropriate, such as when the links could provide helpful
information to users?

The primary purpose of me creating this topic was because there seems to be
some sentiment that it's perfectly fine to exclusively use plaintext in the
news entries. Especially in cases where authors have rejected suggestions
to adding the Sphinx markup in their PRs. There seems to be some sentiment
that it's perfectly fine to exclusively use plaintext in every news entry.
Personally, I think it's a bit more nuanced, and that the links can
sometimes be very helpful for readers.

> Readers interested in more detail can click on the link to the bpo issue
for the full discussion and links to PRs and merges.

Similarly to clicking on the bpo issue, users being able to also click on
the link for functions and classes for more details also allow the news
entry to be more succinct. Especially for changes involving complex
modules, such as asyncio. But even for more simple changes, it shows newer
readers where they can find more information on the more commonly used
functions.

Also, a related question: would it be helpful for contributors to look
through news entries for the latest stable and beta releases, and add
inline links where they could be useful for readers?

I would be more than happy to help with this. A large part of the reason
why I started contributing is because I've always been very pleased with
the quality of Python's documentation and helpfulness towards newer users.
It was my first programming language 5+ years ago. My primary goals are to
further improve the documentation and other resources for newer users of
the language.

Thanks,
Kyle Stanley


On Tue, Aug 13, 2019 at 3:58 AM Ned Deily  wrote:

> On Aug 13, 2019, at 00:20, Kyle Stanley  wrote:
> > On Tue, Aug 13, 2019 at 2:41 AM Mariatta 
> wrote:
> > > I would like to understand why some developers dislike including it,
> even when the reST syntax is provided.
> >
> > This has something to do with the use of blurb/blurb-it. Both tools
> specifically say "single paragraph with simple ReST markup".
> >
> > Further reading blurb's source code, it says the format of the news
> blurb should be as follows:
> >   * The BODY section should be a single paragraph of English text
> > in ReST format.  It should not use the following ReST markup
> > features:
> >   * section headers
> >   * comments
> >   * directives, citations, or footnotes
> >   * Any features that require significant line breaks,
> > like lists, definition lists, quoted paragraphs, line blocks,
> > literal code blocks, and tables.
>
> Note that there is an open devguide issue that requests improvements in
> the description of news entry processing:
> https://github.com/python/devguide/issues/358
>
> Release managers are responsible for reviewing and editing, as necessary,
> the changelog files that are produced in the docset for each release from
> the individual blurb news items.  In general, using many Python-specific
> Sphinx markup entities is fine: browsing though the consolidated blurb
> files for previous releases (and the resultant changelog html) shows uses
> of entities like :func: and :class:.  See, for example:
>
>
> https://raw.githubusercontent.com/python/cpython/3.7/Misc/NEWS.d/3.7.4rc1.rst
>
> https://docs.python.org/3/whatsnew/changelog.html#python-3-7-4-release-candidate-1
>
> Another thing that we look for is brevity.  In general, news entries
> should be short, ideally one to three sentences.  They should not go into
> great detail as the point of the changelog is to provide a high-level
> overview of the changes going into each release.  Readers interested in
> more detail can click on the link to the bpo issue for the full discussion
> and links to PRs and merges.
>
> --
>   Ned Deily
>   n...@python.org -- []
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7CK6RRERHSLEZDFBT2AFRJ3N7FNUSRIX/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> There seems to be some sentiment that it's perfectly fine to exclusively
use plaintext in every news entry.

Oops, that third sentence was a typo, I didn't intend to repeat myself
there.

On Tue, Aug 13, 2019 at 6:31 PM Kyle Stanley  wrote:

> > Note that there is an open devguide issue that requests improvements in
> the description of news entry processing
>
> Good to know, I'll look further into this issue and see if I can help with
> it.
>
> > In general, using many Python-specific Sphinx markup entities is fine:
> browsing though the consolidated blurb files for previous releases (and the
> resultant changelog html) shows uses of entities like :func: and :class:.
>
> Since it's okay to use Python-specific Sphinx, should we encourage the
> usage of it when appropriate, such as when the links could provide helpful
> information to users?
>
> The primary purpose of me creating this topic was because there seems to
> be some sentiment that it's perfectly fine to exclusively use plaintext in
> the news entries. Especially in cases where authors have rejected
> suggestions to adding the Sphinx markup in their PRs. There seems to be
> some sentiment that it's perfectly fine to exclusively use plaintext in
> every news entry. Personally, I think it's a bit more nuanced, and that the
> links can sometimes be very helpful for readers.
>
> > Readers interested in more detail can click on the link to the bpo issue
> for the full discussion and links to PRs and merges.
>
> Similarly to clicking on the bpo issue, users being able to also click on
> the link for functions and classes for more details also allow the news
> entry to be more succinct. Especially for changes involving complex
> modules, such as asyncio. But even for more simple changes, it shows newer
> readers where they can find more information on the more commonly used
> functions.
>
> Also, a related question: would it be helpful for contributors to look
> through news entries for the latest stable and beta releases, and add
> inline links where they could be useful for readers?
>
> I would be more than happy to help with this. A large part of the reason
> why I started contributing is because I've always been very pleased with
> the quality of Python's documentation and helpfulness towards newer users.
> It was my first programming language 5+ years ago. My primary goals are to
> further improve the documentation and other resources for newer users of
> the language.
>
> Thanks,
> Kyle Stanley
>
>
> On Tue, Aug 13, 2019 at 3:58 AM Ned Deily  wrote:
>
>> On Aug 13, 2019, at 00:20, Kyle Stanley  wrote:
>> > On Tue, Aug 13, 2019 at 2:41 AM Mariatta 
>> wrote:
>> > > I would like to understand why some developers dislike including it,
>> even when the reST syntax is provided.
>> >
>> > This has something to do with the use of blurb/blurb-it. Both tools
>> specifically say "single paragraph with simple ReST markup".
>> >
>> > Further reading blurb's source code, it says the format of the news
>> blurb should be as follows:
>> >   * The BODY section should be a single paragraph of English text
>> > in ReST format.  It should not use the following ReST markup
>> > features:
>> >   * section headers
>> >   * comments
>> >   * directives, citations, or footnotes
>> >   * Any features that require significant line breaks,
>> > like lists, definition lists, quoted paragraphs, line blocks,
>> > literal code blocks, and tables.
>>
>> Note that there is an open devguide issue that requests improvements in
>> the description of news entry processing:
>> https://github.com/python/devguide/issues/358
>>
>> Release managers are responsible for reviewing and editing, as necessary,
>> the changelog files that are produced in the docset for each release from
>> the individual blurb news items.  In general, using many Python-specific
>> Sphinx markup entities is fine: browsing though the consolidated blurb
>> files for previous releases (and the resultant changelog html) shows uses
>> of entities like :func: and :class:.  See, for example:
>>
>>
>> https://raw.githubusercontent.com/python/cpython/3.7/Misc/NEWS.d/3.7.4rc1.rst
>>
>> https://docs.python.org/3/whatsnew/changelog.html#python-3-7-4-release-candidate-1
>>
>> Another thing that we look for is brevity.  In general, news entries
>> should be short, ideally one to three sentences.  They should not go into
>> great detail as the point of the changelog is to p

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> The ability to effectively use Python-specific Sphinx features in NEWS
entries is a relatively new feature

Ah, I think that would likely explain some of the inconsistent responses
then. I think it would be worthwhile to add an example of using Sphinx and
a brief explanation of when it can be useful in news entries. This would
likely help improve awareness of the feature.

> But other people may have other opinions on the matter.  We could
undoubtedly spend a lot of time tidying up old NEWS entries
> and, at some point, that time might be better spent on other things, like
helping with the "What's New" documents for upcoming
> releases or just fixing bugs.  But I think there could be a lot of
benefit for a moderate bit of touching up.

I would definitely agree that at some point, it could end up being a bit
overkill (especially for further back versions). However, with regards to
prioritizing different projects, such as tidying news entries vs fixing
bugs, I think a large part of that comes down to each developer's personal
priorities. Also, from my experience, PRs which involve tidying up
documentation have a tendency to be a bit easier to review, so in general
they would end up being less time consuming anyways.

Also, thanks for the tips! I'll definitely look into helping with this.

Thanks,
Kyle Stanley

On Tue, Aug 13, 2019 at 7:30 PM Ned Deily  wrote:

> On Aug 13, 2019, at 15:31, Kyle Stanley  wrote:
> > > In general, using many Python-specific Sphinx markup entities is fine:
> browsing though the consolidated blurb files for previous releases (and the
> resultant changelog html) shows uses of entities like :func: and :class:.
> >
> > Since it's okay to use Python-specific Sphinx, should we encourage the
> usage of it when appropriate, such as when the links could provide helpful
> information to users?
> >
> > The primary purpose of me creating this topic was because there seems to
> be some sentiment that it's perfectly fine to exclusively use plaintext in
> the news entries. Especially in cases where authors have rejected
> suggestions to adding the Sphinx markup in their PRs. There seems to be
> some sentiment that it's perfectly fine to exclusively use plaintext in
> every news entry. Personally, I think it's a bit more nuanced, and that the
> links can sometimes be very helpful for readers.
>
> The ability to effectively use Python-specific Sphinx features in NEWS
> entries is a relatively new feature so I think that is the primary reason
> it is not encouraged more: many people don't realize you can now do this.
> Back in the day, NEWS files were just treated as plaintext, not reST.  This
> is still the case for Python 2.7 NEWS entries as the 2.7 docset was never
> modified to produce an HTML changelog as the Python 3.x docsets do.  (Of
> course, that will soon be a non-issue when 2.7 reaches end-of-life status
> in 2020.)
>
> > Also, a related question: would it be helpful for contributors to look
> through news entries for the latest stable and beta releases, and add
> inline links where they could be useful for readers?
>
> We haven't done a lot of that in the past but I don't see a reason not to,
> especially since it is easy on most systems to generate the changelog by
> building the html docs in the source tree:
>
> https://devguide.python.org/documenting/#using-make-make-bat
>
> and then opening the resultant docs in your browser as a file (the html
> docs are self-contained with regard to static files and do not need a
> webserver).
>
> But other people may have other opinions on the matter.  We could
> undoubtedly spend a lot of time tidying up old NEWS entries and, at some
> point, that time might be better spent on other things, like helping with
> the "What's New" documents for upcoming releases or just fixing bugs.  But
> I think there could be a lot of benefit for a moderate bit of touching up.
>
> One important note: to avoid confusion, only edit the blurb NEWS rst files
> in the branch for that release, i.e. edit Misc/NEWS.d/3.8.0b1.rst in the
> 3.8 branch, not in the master branch.
>
> --
>   Ned Deily
>   n...@python.org -- []
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SSNKSLSCN5GQ5Z27235VVXA5UBFTU3YW/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> Could a bot be written to suggest upgraded markup for Misc/NEWS entries
as a PR?

Potentially, but to some degree, the decision of whether or not to use the
Sphinx markup should be determined on a case by case basis, rather than
indiscriminately applying the markup to every possible candidate.
Generally, I've tried to only recommend to usage of the markup when the
inline link would provide a potential benefit to the readers. Adding the
markup to every possible candidate would result in an excessive number of
irrelevant links and potentially lead to more dead links to clean up in the
future as well.

Thanks,
Kyle Stanley

On Tue, Aug 13, 2019 at 11:30 PM Wes Turner  wrote:

> Could a bot be written to suggest upgraded markup for Misc/NEWS entries as
> a PR?
>
> Most e.g. class, method, and function references probably need ReST markup
> AND a more specific fully-qualified reference?
>
> Can class, method, and function lookups be done with the existing Sphinx
> API index?
>
> On Tuesday, August 13, 2019, Ned Deily  wrote:
>
>> On Aug 13, 2019, at 15:31, Kyle Stanley  wrote:
>> > > In general, using many Python-specific Sphinx markup entities is
>> fine: browsing though the consolidated blurb files for previous releases
>> (and the resultant changelog html) shows uses of entities like :func: and
>> :class:.
>> >
>> > Since it's okay to use Python-specific Sphinx, should we encourage the
>> usage of it when appropriate, such as when the links could provide helpful
>> information to users?
>> >
>> > The primary purpose of me creating this topic was because there seems
>> to be some sentiment that it's perfectly fine to exclusively use plaintext
>> in the news entries. Especially in cases where authors have rejected
>> suggestions to adding the Sphinx markup in their PRs. There seems to be
>> some sentiment that it's perfectly fine to exclusively use plaintext in
>> every news entry. Personally, I think it's a bit more nuanced, and that the
>> links can sometimes be very helpful for readers.
>>
>> The ability to effectively use Python-specific Sphinx features in NEWS
>> entries is a relatively new feature so I think that is the primary reason
>> it is not encouraged more: many people don't realize you can now do this.
>> Back in the day, NEWS files were just treated as plaintext, not reST.  This
>> is still the case for Python 2.7 NEWS entries as the 2.7 docset was never
>> modified to produce an HTML changelog as the Python 3.x docsets do.  (Of
>> course, that will soon be a non-issue when 2.7 reaches end-of-life status
>> in 2020.)
>>
>> > Also, a related question: would it be helpful for contributors to look
>> through news entries for the latest stable and beta releases, and add
>> inline links where they could be useful for readers?
>>
>> We haven't done a lot of that in the past but I don't see a reason not
>> to, especially since it is easy on most systems to generate the changelog
>> by building the html docs in the source tree:
>>
>> https://devguide.python.org/documenting/#using-make-make-bat
>>
>> and then opening the resultant docs in your browser as a file (the html
>> docs are self-contained with regard to static files and do not need a
>> webserver).
>>
>> But other people may have other opinions on the matter.  We could
>> undoubtedly spend a lot of time tidying up old NEWS entries and, at some
>> point, that time might be better spent on other things, like helping with
>> the "What's New" documents for upcoming releases or just fixing bugs.  But
>> I think there could be a lot of benefit for a moderate bit of touching up.
>>
>> One important note: to avoid confusion, only edit the blurb NEWS rst
>> files in the branch for that release, i.e. edit Misc/NEWS.d/3.8.0b1.rst in
>> the 3.8 branch, not in the master branch.
>>
>> --
>>   Ned Deily
>>   n...@python.org -- []
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/WQNYODDNTH6MFPBNJZZJR6AFFTD5CYBG/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4UIY6UDE6V2TEKN2CY6EXJG5MITXIQFJ/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> Also, for IDLE, news entries to idlelib/NEWS.txt
> where markup, as opposed to unicode, is noise.

Interesting, I actually wasn't aware of the distinction for idlelib/NEWS. I
can imagine that Sphinx constructs such as :func:, :meth:, and :class:
would not be nearly as useful there. However, I can imagine reST being
occasionally useful.

Based on a recent feature that was added to the IDLE, I could imagine
explicit inline reST links being somewhat useful for something like this:

Add support for displaying line numbers. See `IDLE Options menu <
https://docs.python.org/3/library/idle.html#options-menu-shell-and-editor
>`_.

The inline link would appear to readers as "IDLE Options menu" and allow
them to click on it to navigate to the corresponding documentation for more
information on the feature.

> Bottom line: I would rather a knowledgeable editor prettify the blurbs
> in a consistent manner after I am done with them.  To me, this is a
> place where specializations pays.

I completely agree. I was mainly addressing situations where PR authors
were rejecting or disregarding suggestions to add the markup in the news
entry from those who are knowledgeable of the Sphinx constructs/roles. It
wouldn't be reasonable to expect all PR authors to be able to properly
utilize every supported markup feature.

This is a rare occurrence, but I've had it happen a couple of times
recently. Based on the responses so far, it likely occurred due to some
developers not being aware that Misc/NEWS supported Sphinx roles and some
of the basic features of reST. That's completely understandable if it's a
newer feature.

Hopefully this discussion and any potential updates to the devguide will
improve awareness of the feature, and provide instructions on when it's
appropriate to utilize.

Thanks,
Kyle Stanley

On Wed, Aug 14, 2019 at 3:31 AM Terry Reedy  wrote:

> On 8/13/2019 6:31 PM, Kyle Stanley wrote:
>
> > The primary purpose of me creating this topic was because there seems to
> > be some sentiment that it's perfectly fine to exclusively use plaintext
> > in the news entries. Especially in cases where authors have rejected
> > suggestions to adding the Sphinx markup in their PRs. There seems to be
> > some sentiment that it's perfectly fine to exclusively use plaintext in
> > every news entry. Personally, I think it's a bit more nuanced, and that
> > the links can sometimes be very helpful for readers.
>
> Beyond what Ned said, (news markup is relatively new), people may be
> uncertain what is allowed and when appropriate.  Also, there is some
> situation for me where markup seems to be a nuisance and looks like it
> is introducing an error.  So I have changed unicode quotes and removed
> some rst markup.  Also, for IDLE, news entries to idlelib/NEWS.txt.
> where markup, as opposed to unicode, is noise.
>
> Bottom line: I would rather a knowledgeable editor prettify the blurbs
> in a consistent manner after I am done with them.  To me, this is a
> place where specializations pays.
>
> --
> Terry Jan Reedy
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/53QDMBQHW2S6F7JI4YSGAYYKJOVOIFQF/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5TXWQ7ARKDV73MNEIED35PZDAL6T2L2/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> I suggest slightly expanding the part about NEWS formatting in the dev
guide, and specifically have the example include appropriate uses of roles,
and a link to the list of available roles.

Yeah definitely, it was my intention to mention this in the devguide,
particularly with adding an example of the Sphinx roles being used and
explaining appropriate usage. I hadn't thought of linking to the list of
roles (https://devguide.python.org/documenting/#id4), but that's definitely
a good idea to include. I was just waiting for everyone to get a chance to
provide feedback on the topic before expanding the devguide.

After the devguide is updated, I was also planning on adding the markup to
3.8's Misc/NEWS entries (in the appropriate branch, as Ned recommended),
and then work on the 3.9. I'll probably split it into several smaller PRs
so it's easier to review.

On Thu, Aug 15, 2019 at 3:10 AM Tal Einat  wrote:

> I suggest slightly expanding the part about NEWS formatting in the dev
> guide, and specifically have the example include appropriate uses of roles,
> and a link to the list of available roles.
>
> https://devguide.python.org/committing/#what-s-new-and-news-entries
>
> On Thu, Aug 15, 2019 at 3:39 AM Kyle Stanley  wrote:
>
>> > Also, for IDLE, news entries to idlelib/NEWS.txt
>> > where markup, as opposed to unicode, is noise.
>>
>> Interesting, I actually wasn't aware of the distinction for idlelib/NEWS.
>> I can imagine that Sphinx constructs such as :func:, :meth:, and :class:
>> would not be nearly as useful there. However, I can imagine reST being
>> occasionally useful.
>>
>> Based on a recent feature that was added to the IDLE, I could imagine
>> explicit inline reST links being somewhat useful for something like this:
>>
>> Add support for displaying line numbers. See `IDLE Options menu <
>> https://docs.python.org/3/library/idle.html#options-menu-shell-and-editor
>> >`_.
>>
>> The inline link would appear to readers as "IDLE Options menu" and allow
>> them to click on it to navigate to the corresponding documentation for more
>> information on the feature.
>>
>> > Bottom line: I would rather a knowledgeable editor prettify the blurbs
>> > in a consistent manner after I am done with them.  To me, this is a
>> > place where specializations pays.
>>
>> I completely agree. I was mainly addressing situations where PR authors
>> were rejecting or disregarding suggestions to add the markup in the news
>> entry from those who are knowledgeable of the Sphinx constructs/roles. It
>> wouldn't be reasonable to expect all PR authors to be able to properly
>> utilize every supported markup feature.
>>
>> This is a rare occurrence, but I've had it happen a couple of times
>> recently. Based on the responses so far, it likely occurred due to some
>> developers not being aware that Misc/NEWS supported Sphinx roles and some
>> of the basic features of reST. That's completely understandable if it's a
>> newer feature.
>>
>> Hopefully this discussion and any potential updates to the devguide will
>> improve awareness of the feature, and provide instructions on when it's
>> appropriate to utilize.
>>
>> Thanks,
>> Kyle Stanley
>>
>> On Wed, Aug 14, 2019 at 3:31 AM Terry Reedy  wrote:
>>
>>> On 8/13/2019 6:31 PM, Kyle Stanley wrote:
>>>
>>> > The primary purpose of me creating this topic was because there seems
>>> to
>>> > be some sentiment that it's perfectly fine to exclusively use
>>> plaintext
>>> > in the news entries. Especially in cases where authors have rejected
>>> > suggestions to adding the Sphinx markup in their PRs. There seems to
>>> be
>>> > some sentiment that it's perfectly fine to exclusively use plaintext
>>> in
>>> > every news entry. Personally, I think it's a bit more nuanced, and
>>> that
>>> > the links can sometimes be very helpful for readers.
>>>
>>> Beyond what Ned said, (news markup is relatively new), people may be
>>> uncertain what is allowed and when appropriate.  Also, there is some
>>> situation for me where markup seems to be a nuisance and looks like it
>>> is introducing an error.  So I have changed unicode quotes and removed
>>> some rst markup.  Also, for IDLE, news entries to idlelib/NEWS.txt.
>>> where markup, as opposed to unicode, is noise.
>>>
>>> Bottom line: I would rather a knowledgeable editor prettify the blurbs
>>> in a consistent manner after

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> a) this seems to be "well-defined", and imho, suitable as "easy", etc..

Part of this could potentially be suitable as an "easy" issue, but I'm not
certain that it would make for a good first PR, since it is involving a
fairly important and widely used section of the devguide. Many newcomers
may not be familiar with what would be considered "appropriate usage" of
the Sphinx roles in news entries. When I think of issues that would be good
first PRs involving documentation, I generally think of grammar/spelling
improvements or perhaps more objective fixes. This section might be a
little bit too subjective though.

> b) isn't this something we want new people to be more aware of (as you
> said, you have been working with this for a year)

I think you might be confusing me with someone else, as I've only been
contributing to Python since June of this year. I've built up some amount
of experience from PRs I've submitted as well as PR reviews, but I would
still consider myself to be a more recent contributor. If I come across as
having more experience though, that's certainly a good thing. (:

> c) it is an area (Documentation) I have clearly 'missed' as I focused on
> 'other things', and, with myself and many projects I have worked on over
> the years - Documentation seems to come in last. Getting new (and
> newish, as myself) working here only makes us better suited for review
> in the future.

I definitely agree that we should try to come up with ways to improve the
interaction with the documentation (particularly with the devguide). A
great way to do that can be cleaning up existing sections, but for reasons
previously stated, I'm not certain that this issue is particularly well
suited for that. It's "well-defined" from feedback in the mailing list
discussion, but I think it would benefit from being written by someone with
some experience in news entries, reST and Sphinx markup, and documentation
in general.

However, a great way that anyone could help out would be through providing
feedback on the PR once it's open. Providing feedback on PRs is great way
to improve experience in an area, and also helps to deal with a major
bottleneck for Python development in general. We have far more people
submitting PRs than reviewers.

Once I open the PR for it in the devguide, I'll be sure to send a message
in this topic which includes a link to it. It would greatly benefit from
review from those who are less familiar with documentation. That will help
to ensure the section is easy to understand for the intended target
audience. Of course, it would also benefit from those are are experienced
as well, to ensure it is as accurate as possible.

Thanks,
Kyle Stanley

On Fri, Aug 16, 2019 at 4:48 AM Michael  wrote:

> On 16/08/2019 05:31, Kyle Stanley wrote:
> > Yeah definitely, it was my intention to mention this in the devguide,
> > particularly with adding an example of the Sphinx roles being used and
> > explaining appropriate usage. I hadn't thought of linking to the list of
> > roles (https://devguide.python.org/documenting/#id4), but that's
> definitely
> > a good idea to include. I was just waiting for everyone to get a chance
> to
> > provide feedback on the topic before expanding the devguide.
> >
> > After the devguide is updated, I was also planning on adding the markup
> to
> > 3.8's Misc/NEWS entries (in the appropriate branch, as Ned recommended),
> > and then work on the 3.9. I'll probably split it into several smaller PRs
> > so it's easier to review.
>
> There has been "a lot" of discussion re: things for new contributors to
> do and learn.
>
> a) this seems to be "well-defined", and imho, suitable as "easy", etc..
> b) isn't this something we want new people to be more aware of (as you
> said, you have been working with this for a year)
> c) it is an area (Documentation) I have clearly 'missed' as I focused on
> 'other things', and, with myself and many projects I have worked on over
> the years - Documentation seems to come in last. Getting new (and
> newish, as myself) working here only makes us better suited for review
> in the future.
>
> So, I guess this is an area where you could "mentor", perhaps create
> "issues" that specify the "paragraphs", or whatever you think are
> appropriate 'chunks' to make review sensible (if not also easier).
>
> Michael
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ROXFQX7ZDXFH5YRNGYZU3KCF7GR5QQLL/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

Based on the feedback from the ML thread and Discuss topic, I created a new
PR on python/devguide which expands the "What's New and News Entries"
section of "committing.rst" to include Sphinx roles:
https://github.com/python/devguide/pull/525/files

I would greatly appreciate feedback from contributors and developers less
experienced with documentation to ensure that the section is easy to
understand for the intended target audience. Also, review from those who
are experienced with documentation and news entries is needed to ensure it
is as accurate as possible.

Thanks,
Kyle Stanley

On Mon, Aug 12, 2019 at 10:24 PM Kyle Stanley  wrote:

> Recently, on Discuss, I created a new topic:
> https://discuss.python.org/t/should-news-entries-contain-documentation-links/2127
>
> However, many may not have the time to read the full post or don’t
> regularly check the core workflow category on Discuss, so I'll provide a
> shortened version here.
>
> During my experience thus far reviewing PRs on GitHub, I've noticed a
> significant degree of inconsistency when it comes to the usage of inline
> links with reST and Sphinx in Misc/NEWS entries. Since many of my
> contributions have involved documentation changes, I've familiarized myself
> most of the syntax.
>
> Many of the features in markup languages provide visual modifications
> rather than functional ones. However, with the inline markup supported by
> reST and processing from Sphinx, there's a functional improvement as well.
> For example, the usage of :func:\`\` (escaped for mailman)
> provides a link to the relevant docs for the function.
>
> In the context of news entries, this allows readers to click on a
> function, method, class, etc for more information. This can be useful if
> it's something they're not familiar with, or when the changes affected the
> docs. For those who are familiar with the structure of docs.python.org,
> the cross link to the docs may not seem at all necessary.
>
> However, for readers that are either newer or not familiar with the
> structure, they might be led astray into 2.7 docs or an entirely wrong
> page. This happens especially frequently when using external search engines.
>
> I'm not at all suggesting that every PR author should be required to use
> it or know all of the reST constructs. However, I would like for everyone
> to be aware of the potential usefulness of including inline links in news
> entries, and mention it in the devguide.
>
> Also, I would like to understand why some developers dislike including it,
> even when the reST syntax is provided. The majority of authors so far would
> add my suggestion to their PR, but there have been some that don't want
> anything besides plaintext in their news entry.
>
> Personally, I think it provides further inclusiveness to readers of all
> levels of experience and QoL at a very minimal cost.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HTAFGTQIZJQUCU6QCVF3KFD3VFGFOBWV/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2VI4OZI7YTFUXW4LUETY3QYZ3TU6BU7J/

[Python-Dev] Re: Inline links in Misc/NEWS entries

[Kyle Stanley]

> the talk I gave last month at PyOhio (
https://www.pyohio.org/2019/presentations/137)

> Right now I have the slides up as a view-only share in my Google Drive (
bit.ly/bskinn-pyohio2019-intersphinx). Perhaps it would be useful to link
to them? Or, to host a PDF someplace more permanent, and link to that?

Thanks for the links, I'll definitely take a look through them for myself
to see if I can gain any additional information. I've been using Sphinx and
reST a decent amount recently, but I'm definitely not the most experienced.

Also, there's a PR open for providing netlify previews for CPython PRs:
https://github.com/python/cpython/pull/15288. From what I understand, we'll
be recommending for PR authors to use that to preview documentation
changes, particularly when any rich formatting is involved.

> - [ ] And a *link*/"reference" from the "extensive cross-references"
paragraph of https://www.sphinx-doc.org/en/master/ to the cross-referencing
docs

We currently link to the Sphinx docs in the dedicated section, "Additional
Markup Constructs":
https://devguide.python.org/documenting/#additional-markup-constructs. The
section in the PR was only meant to provide a very brief summary for using
Sphinx in NEWS entries. I included a link to the "Additional Markup
Constructs" section for readers looking for additional information.

It might be worthwhile to create a new subsection for links to external
resources and guides. If accepted, the best location in the devguide would
probably be 7.4.12 (assuming that no other sections are added in the
meantime). However, that proposal would have to be in it's own topic,
rather than in this one.


On Mon, Aug 19, 2019 at 12:33 PM Wes Turner  wrote:

> - [ ] A page or section in the Sphinx docs would be helpful for many, I
> think
>
> https://github.com/sphinx-doc/sphinx/tree/master/doc
>
> ``__
>
>
> .. index:: Implicit reference
> .. _implicit-reference:
>
> Implicit ref
> ---
>
> `implicit ref`_
>
> `anchor text `_
>
> `implicit-reference`_
>
> `implicit reference`_
>
> :func:`modname.funcname`
>
> :meth:`modname.Classname.methname`
>
> :class:`modname.Classname`
>
> And, FWIW:
>
> :cite:`cpython37`
>
> .. bibliography:: references.bib
>
> References.bib
> --
> .. code:: latex
>
> @techreport{cpython37,
> title="Python 3.7 Documentation",
> author="Python Software Foundation",
> year=1999,
> howpublished = "\url{https://docs.python.org/3.7/}";,
> }
>
>
>
> - [ ] And/Or examples on the roles docs page
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#ref-role
>
> - [ ] And a *link*/"reference" from the "extensive cross-references"
> paragraph of https://www.sphinx-doc.org/en/master/ to the
> cross-referencing docs
>
> - [ ] And how to do parenthetical citations of / reference CPython with
> e.g. bibtex [with the `.. bibliography::` directive and :cite:`refkey` role
> from sphinxcontrib-bibtex]
>
> There are lots of great Sphinx primers; awesome-sphinxdoc could link to
> them to help everyone:
> https://github.com/yoloseem/awesome-sphinxdoc
>
> On Monday, August 19, 2019,  wrote:
>
>> Citing from the current (19 Aug 2019) version of this PR (
>> https://github.com/python/devguide/pull/525/files#diff-50cb76bbe8ae3fcd4170dc6e8d9d6b3fR225-R226
>> ):
>>
>> > Before using any Sphinx roles, ensure that a corresponding entry exists
>> within the documentation.
>>
>> At the risk of crass self-promotion, the talk I gave last month at PyOhio
>> (https://www.pyohio.org/2019/presentations/137) provides step-by-step
>> instructions for how to find the information needed to craft a working
>> Sphinx cross-reference, along with some of the tooling that's available.
>>
>> Right now I have the slides up as a view-only share in my Google Drive (
>> bit.ly/bskinn-pyohio2019-intersphinx). Perhaps it would be useful to
>> link to them? Or, to host a PDF someplace more permanent, and link to that?
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/IK5XP7CPZDF2FSFG55JRKXHEXNTOMUTB/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/SSMNWK7YF6WRT2NO4C2IQ6PGD2XVJOZT/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
h

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Kyle Stanley]

Abhilash Raj wrote:
> What I am coming from is that what one has permissions to do and what one
> should do has been different. We caution people with permissions to use
them
> judiciously.

>From my understanding, triagers should only close PRs or issues when they
are entirely invalid (for non-subjective reasons, such as something that
physically can't be merged) or when they are significantly outdated. Closing
PRs for anything remotely subjective, such as deciding if a documentation
change should be added or if a new feature is actually needed should be
only determined by the core developers.

Triagers can suggest the closing of those PRs, but a core dev should make
the final decision. If a mistake is made (or a triager misuses their
permissions), the PR could be reopened. Mistakes are bound to occur at some
point, but if a single triager is repeatedly doing so or is clearly
misusing their permissions, they should be removed.

Also, there's not an easy way for us to customize the exact permissions to
prevent triagers from being able to close issues, we are limited the
template permissions that GitHub laid out in
https://help.github.com/en/articles/repository-permission-levels-for-an-organization#repository-access-for-each-permission-level
under Triage (beta). So, it's kind of an all or nothing deal. There's also
some minor things that we can't do that would be helpful, such as the
ability to directly edit PR titles or adding reviewers. For more details,
there was a topic on Discuss addressing this:
https://discuss.python.org/t/permissions-for-the-python-traige-team/2191.

Brett Cannon wrote:
> closing stale PRs that have not received a timely response

Is there any existing criteria for roughly defining a "stale PR" that
should be closed, or what you would personally consider to be a "timely
response"? So far, I've avoided doing this as I didn't want to accidentally
close anything that could still be valid or useful.

My idea of a stale PR would be something that hasn't received a response or
comment for 3+ months, hasn't been updated in two or more releases prior to
the latest development one, or was recreated. An obvious candidate I can
think of would be this one: https://github.com/python/cpython/pull/117, as
it has been awaiting changes since Feb 20, 2018 and was recreated here:
https://github.com/python/cpython/pull/14976. Even if it hadn't been
recreated, I think it would still be a good example as of a stale PR, since
it had changes requested for over a year with no response from the author
since then.

Regards,
Kyle Stanley

On Thu, Aug 22, 2019 at 2:25 PM Brett Cannon  wrote:

> Abhilash Raj wrote:
> > On Thu, Aug 22, 2019, at 6:06 AM, Victor Stinner wrote:
> > > Hi,
> > > Oh, I just wrote a similar email to python-committers, I didn't notice
> > > that Mariatta wrote to python-dev and python-committers.
> > >
> https://mail.python.org/archives/list/python-committ...@python.org/message/5.
> ..
> > > My worry is more about closing pull requests.
> > > Le 21/08/2019 à 22:13, Raymond Hettinger a écrit :
> > > The capabilities of a triager mostly look good
> > > except for "closing PRs and issues".  This is a superpower that has
> traditionally been
> > > reserved for more senior developers because it grants the ability to
> shut-down the work of
> > > another aspiring contributor.  Marking someone else's suggestion as
> rejected is the most
> > > perilous and least fun aspect of core development.  Submitters tend to
> expect their idea
> > > won't be rejected without a good deal of thought and expert
> consideration.   Our bar for
> > > becoming a triager is somewhat low, so I don't think it makes sense to
> give the authority
> > > to reject a PR or close an issue.
> > > Closing an issue (on GitHub) is not new compared to the previous
> > > "Developer" role on bugs.python.org. When I gave the bug triage
> > > permission to a contributor, I always warned them that closing an
> issue
> > > is the most risky operation. I asked them to ask me before doing that.
> > > In practice, I don't recall a triagger who closed an issue, but
> someone
> > > else complained that the issue should be reopened. In a few specific
> > > cases, the original reporter was in disagreement with everybody else
> and
> > > didn't understand why their issue was not a bug and will not be fixed,
> > > but it wasn't an issue about triaggers ;-)
> > > The risk of closing an issue by mistake is quite low, since the bug
> > > remains in the database, it's trivial to reopen. Closed bugs can be
> > > found using Google 

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Kyle Stanley]

Kyle Stanley wrote:
> as it has been awaiting changes since Feb 20, 2018

Clarification: The author addressed the suggested changes on Feb 19, 2017
but had made no response after Brett added the ``awaiting changes`` label.

As a related question, would it be okay for triagers to manually add the
"awaiting changes" label on PRs that are suspected to be stale? I think
there are some obvious cases where it wouldn't be required (no response
from the author in over a year), but this might be an effective way of
addressing more recently inactive PRs (2-3+ months). If the author doesn't
respond within a month after the label is added, the PR could be closed.

On Thu, Aug 22, 2019 at 5:56 PM Kyle Stanley  wrote:

> Abhilash Raj wrote:
> > What I am coming from is that what one has permissions to do and what
> one
> > should do has been different. We caution people with permissions to use
> them
> > judiciously.
>
> From my understanding, triagers should only close PRs or issues when they
> are entirely invalid (for non-subjective reasons, such as something that
> physically can't be merged) or when they are significantly outdated. Closing
> PRs for anything remotely subjective, such as deciding if a documentation
> change should be added or if a new feature is actually needed should be
> only determined by the core developers.
>
> Triagers can suggest the closing of those PRs, but a core dev should make
> the final decision. If a mistake is made (or a triager misuses their
> permissions), the PR could be reopened. Mistakes are bound to occur at some
> point, but if a single triager is repeatedly doing so or is clearly
> misusing their permissions, they should be removed.
>
> Also, there's not an easy way for us to customize the exact permissions to
> prevent triagers from being able to close issues, we are limited the
> template permissions that GitHub laid out in
> https://help.github.com/en/articles/repository-permission-levels-for-an-organization#repository-access-for-each-permission-level
> under Triage (beta). So, it's kind of an all or nothing deal. There's also
> some minor things that we can't do that would be helpful, such as the
> ability to directly edit PR titles or adding reviewers. For more details,
> there was a topic on Discuss addressing this:
> https://discuss.python.org/t/permissions-for-the-python-traige-team/2191.
>
> Brett Cannon wrote:
> > closing stale PRs that have not received a timely response
>
> Is there any existing criteria for roughly defining a "stale PR" that
> should be closed, or what you would personally consider to be a "timely
> response"? So far, I've avoided doing this as I didn't want to accidentally
> close anything that could still be valid or useful.
>
> My idea of a stale PR would be something that hasn't received a response
> or comment for 3+ months, hasn't been updated in two or more releases prior
> to the latest development one, or was recreated. An obvious candidate I can
> think of would be this one: https://github.com/python/cpython/pull/117,
> as it has been awaiting changes since Feb 20, 2018 and was recreated here:
> https://github.com/python/cpython/pull/14976. Even if it hadn't been
> recreated, I think it would still be a good example as of a stale PR, since
> it had changes requested for over a year with no response from the author
> since then.
>
> Regards,
> Kyle Stanley
>
> On Thu, Aug 22, 2019 at 2:25 PM Brett Cannon  wrote:
>
>> Abhilash Raj wrote:
>> > On Thu, Aug 22, 2019, at 6:06 AM, Victor Stinner wrote:
>> > > Hi,
>> > > Oh, I just wrote a similar email to python-committers, I didn't
>> notice
>> > > that Mariatta wrote to python-dev and python-committers.
>> > >
>> https://mail.python.org/archives/list/python-committ...@python.org/message/5.
>> ..
>> > > My worry is more about closing pull requests.
>> > > Le 21/08/2019 à 22:13, Raymond Hettinger a écrit :
>> > > The capabilities of a triager mostly look good
>> > > except for "closing PRs and issues".  This is a superpower that has
>> traditionally been
>> > > reserved for more senior developers because it grants the ability to
>> shut-down the work of
>> > > another aspiring contributor.  Marking someone else's suggestion as
>> rejected is the most
>> > > perilous and least fun aspect of core development.  Submitters tend
>> to expect their idea
>> > > won't be rejected without a good deal of thought and expert
>> consideration.   Our bar for
>> > > becoming a triager is somewhat low, so I d

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Kyle Stanley]

Mariatta wrote:
> It still requires earning trust from at least one core developer who will
approve their request, which I don't actually believe is an easy > thing to
do.

Agreed, it may be a low bar in comparison to the 66% approval required for
becoming a core developer, but it's definitely not trivial to achieve
(especially for newer contributors). I think that approval from 2 core devs
would be reasonable as well though. So far, the 3 candidates who were
approved to join the triage team have had support from multiple core devs.
If the requirements are going to be modified, it would be easier to do so
while the team is still new. It would be less fair to future candidates if
those currently on the team wouldn't have met the new requirements.

Mariatta wrote:
> All "awaiting ..." labels are meant to be added automatically by
bedevere-bot. Even core devs should not try adding/removing them >
manually. The "awaiting change" is meant to be added only after core devs
reviewed the PR and requested change to it, so it doesn't > make sense for
a triager to add this label. It requires a core dev's decision.

That was my previous impression, but then I saw that Brett applied it
manually in this older PR to check if the author was still active:
https://github.com/python/cpython/pull/117#issuecomment-367187316. That's
why I figured it was worthwhile to ask about it.

The most common and usual case for the "awaiting changes" label is for it
to be automatically applied by bedevere-bot after a core dev requests
changes, but it seems like it would also be an effective means to measure
whether or not a PR's author is still active (if it's been several months
since the last update from them).

On Thu, Aug 22, 2019 at 8:24 PM Mariatta  wrote:

> The capabilities of a triager mostly look good except for "closing PRs and
>> issues".  This is a superpower that has traditionally been reserved for
>> more senior developers because it grants the ability to shut-down the work
>> of another aspiring contributor.  Marking someone else's suggestion as
>> rejected is the most perilous and least fun aspect of core development.
>> Submitters tend to expect their idea won't be rejected without a good deal
>> of thought and expert consideration.
>
>
> If most core devs prefer that triagers don't ever close PR/issue, then
> let's document that and let the triagers know of these boundaries. I'd like
> to be able to trust the triagers and let them help us, instead of
> restricting their movement.
>
> Personally I think it's ok for them to close PR/issue, especially invalid
> ones. That's why we need help.
> If a core dev disagree with the decision, it is easy enough to re-open the
> PR.
>
> Anyways, I have created a devguide issue for documenting the
> guidelines/ethiquette for closing PR/issue:
> https://github.com/python/devguide/issues/527
>
> Our bar for becoming a triager is somewhat low,
>
>
>  It still requires earning trust from at least one core developer who will
> approve their request, which I don't actually believe is an easy thing to
> do.
>
> is it possible to adjust permissions in
>> the Python project?
>
>
> Nothing we can do. You can try contacting GitHub about it.
>
> My worry is that it *might* require more trust in a
>> contributor before giving them the triager role.
>
>
> Yes, but I don't see this as a bad thing. If we want to make it more
> strict by saying at least 2 or more core devs approving, that's ok with me
> too.
>
> We could, if we really really really wanted to. Any of the bots can
>> be programmed to look for closed PRs from people in triage team (and
>> not the author of the PR) and make policy decision to re-open, ping
>> the relevant core dev, remind the person who did it not do it.
>>
>> I don't think we should though do it though.
>
>
> I also don't think we should do it.
>
> would it be okay for triagers to manually add the "awaiting changes" label
>> on PRs that are suspected to be stale?
>
>
> All "awaiting ..." labels are meant to be added automatically by
> bedevere-bot. Even core devs should not try adding/removing them manually.
> The "awaiting change" is meant to be added only after core devs reviewed
> the PR and requested change to it, so it doesn't make sense for a triager
> to add this label. It requires a core dev's decision.
>
>
> ᐧ
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HA5UFFO2TYKLPEKSQL5FIK3WC26NWN6Y/

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Kyle Stanley]

> That was Brett's bot for backfilling the "awaiting ..." label to PRs
created before we had any "awaiting .." label. A script was used to
> automatically determine the stage of the PR and apply appropriate label.

Ah, that makes sense. I wasn't aware that personal GitHub accounts were
previously used by bots before the current dedicated bots were implemented.
Thanks for the clarification, feel free to disregard my earlier suggestion
then.

On Thu, Aug 22, 2019 at 11:02 PM Mariatta  wrote:

> That was my previous impression, but then I saw that Brett applied it
>> manually in this older PR to check if the author was still active:
>> https://github.com/python/cpython/pull/117#issuecomment-367187316.
>
>
> That was Brett's bot for backfilling the "awaiting ..." label to PRs
> created before we had any "awaiting .." label. A script was used to
> automatically determine the stage of the PR and apply appropriate label.
>
>
> https://mail.python.org/archives/list/python-dev@python.org/thread/UYMPA4OFWCJPFPASDUZYGXTH6ZGM6RJZ/
>
>
> ᐧ
>>>
>> ᐧ
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M3RWXJSDJJBO6P4GVAKVSSC5NKYEBIPD/

[Python-Dev] Re: Announcing the new Python triage team on GitHub

[Kyle Stanley]

> For instance, I don't think a triager should be able to edit other users'
comments or lock conversations, but I'm afraid GitHub
> doesn't provide that level of granularity (correct?).

You are correct in that GitHub does not allow customization of the
permissions, but those with the  "Triage" permission are unable to edit
comments or lock conversations. In order to do either of those, the "Write"
permission is required. Compared to a normal contributor, triagers are only
able to add labels, close, reopen, and assign PRs/Issues.

On Fri, Aug 23, 2019 at 2:01 PM Giampaolo Rodola' 
wrote:

> If the concern is pruning old/stale/invalid issues/PRs, I think the
> proposal to add a specific "awaiting-close" (or similar) label should work.
> These issues can periodically be handled by a core-dev who can click on the
> label, then evaluate and bulk-close them in one shot if necessary. Also, it
> seems to me that including "devguide" and "core-workflow" repos is
> unnecessary since they are low-traffic administrative repos related to our
> internal workflow, they are attended by a small portion of the core-dev
> team only and labels don't play a central role as in the cpython repo.
> Please let's keep in mind that this role implies higher responsibilities
> than a usual contributor, and since the bar for being promoted is
> considerably lower than a core-dev's, triager permissions should reflect
> that and remain minimal. For instance, I don't think a triager should be
> able to edit other users' comments or lock conversations, but I'm afraid
> GitHub doesn't provide that level of granularity (correct?).
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2YYB6MAJ55U2U4KTKINFDSB4USJ3NMZY/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZR6STQ3FCSKPS5BRJRTVEZ637NDDYTX5/

[Python-Dev] Re: Python for Windows (python-3.7.4.exe) location confusing

[Kyle Stanley]

Terry Jan Reedy wrote:
> "This installer also installs the multiversion py.exe launcher if doing 
so would not be a downgrade.  If you have admin access, we recommend 
installing one py.exe for all users.  See xxx for how to use py.exe.
[ ] Install py.exe for all users.

+1, Particularly on this part. If the user installing Python has administrative 
access, I don't see much of a reason for them to have to install a separate 
launcher and installation. Recommending a global py.exe install for them seems 
to be far more straight forward. 

I'm not certain as to whether or not we are able to collect statistics on this, 
but I would not be at all surprised if the majority of users installing Python 
had administrative access on the device they're installing it on. While I think 
the ability to allow non-admin users to install packages in their own install 
is very useful, it should not come at the expense of making the process 
confusing for those with administrative privileges.

Steve Dower wrote:
> It also means 
that regular users can install packages without needing to be admin, and 
without corrupting other user's installs.

Does this have any advantage over using a virtual environment? I can imagine 
this might be more simple for new users (since they don't have to do anything 
additional), but setting up a virtual environment to use for a separate set of 
packages is very straightforward nowadays with venv.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WDQRQ5ZJDGABRQUKZ7C6WMIS5PEAAROA/

[Python-Dev] Re: The Python 2 death march

[Kyle Stanley]

Benjamin, what are you thoughts on usage of the "needs backport to 2.7"
label? For most of the PRs I've reviewed I tend to avoid adding it myself,
but I've seen it used periodically. It seems to be used rather infrequently
(
https://github.com/python/cpython/pulls?q=is%3Apr+label%3A%22needs+backport+to+2.7%22+is%3Amerged),
but I'm not entirely clear on when it should be used, particularly for
documentation-related PRs.

Also, is there an official date when the label will be removed? Removing
the backport label seems to generally be at the discretion of the release
manager for that version, but I was curious as to whether it would be
removed on the sunset date or prior to then. Without the label, backport
PRs can still be opened manually of course, but removing it would probably
help to discourage 2.7 backports.

On Mon, Sep 9, 2019 at 9:38 AM Benjamin Peterson 
wrote:

> Hi all,
> It's finally time to schedule the last releases in Python 2's life. There
> will be two more releases of Python 2.7: Python 2.7.17 and Python 2.7.18.
>
> Python 2.7.17 release candidate 1 will happen on October 5th followed by
> the final release on October 19th.
>
> I'm going to time Python 2.7.18 to coincide with PyCon 2020 in April, so
> attendees can enjoy some collective catharsis. We'll still say January 1st
> is the official EOL date.
>
> Thanks to Sumana Harihareswara, there's now a FAQ about the Python 2
> sunset on the website: https://www.python.org/doc/sunset-python-2/
>
> Regards,
> Benjamin
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/APWHFYQDKNVYQAK3HZMBGQIZHAVRHCV2/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NYIVL47NMGMSWTF5FQCZQF2ZWPA6I2SH/

[Python-Dev] Re: static variables in CPython - duplicated _Py_IDENTIFIERs?

[Kyle Stanley]

> How easy would it be to search the sources and the docs for the ones
that are currently public but not documented?

Comparing the docs and .c files for functions that are not documented
would be fairly trivial, but it's very difficult to define something as
being
public if it's not documented, since that's the main definition used. If
it's not
in the documentation, there's not a definitive way tell that it's public
(AFAIK).

The presence or lack of the name starting with an underscore can sometimes
be an indicator, but that's not consistently reliable.

On Mon, Sep 23, 2019 at 7:59 PM MRAB  wrote:

> On 2019-09-23 21:22, Vinay Sajip via Python-Dev wrote:
> > OK - but that's just one I picked at random. There are others like it -
> what would be the process for deciding which ones need to be made private
> and moved? Should an issue be raised to track this?
> >
> It's not really a question of which ones should be made private, but of
> which ones should be remain public. If it's mentioned in the docs, then
> it's public, otherwise it should be private.
> How easy would it be to search the sources and the docs for the ones
> that are currently public but not documented?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4KUPJBNXXL7VD6JPAC7YCCTD56A5OUAZ/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U3WTOSLDLIZBIB4IJPJ4LSMWQNLY7W2V/

[Python-Dev] Adding GitHub usernames to the developer log

[Kyle Stanley]

Recently, Brett updated the developer log in the devguide
(https://devguide.python.org/developers/) to fetch the names of each core
developer and the date they were given commit privileges from the private
python-committers repository.

I think it would also be quite useful to include GitHub usernames on that list.
Currently, the only list that contributors can find the GitHub usernames for
each core developer is through the committers list on bpo. Since we will be
moving away from bpo (PEP 581), we should have a comprehensive list that is
separate from that platform.

The motivation behind creating a a new topic for this issue was Brett's
response to my comment in the PR that updated the devguide
(https://github.com/python/devguide/pull/533#issuecomment-532405907).
Essentially, if no core developers have an issue with having their GitHub
username posted on the devguide, we can move forward with adding it.


Another related but more long term project is adding the GitHub usernames
to the experts index (https://devguide.python.org/experts/). This is more
involved because the bpo nosy list currently pulls from the experts index,
meaning the nosy list is dependent on the specific formatting used.

To address this, I opened a PR a couple of months ago which would add a .json
file containing the data from the experts index
(https://github.com/python/devguide/pull/517), based on the discussion in the
related issue (https://github.com/python/devguide/issues/507). If any available
core developers are experienced with structuring .json files, I would greatly
appreciate any feedback.

The next step would be converting the nosy list script to use the new .json
file instead of the experts index page, so that we could adjust the page
to also include GitHub usernames. Optimally, the contents in the experts
index would be pulled from the .json file automatically so any changes only
have to be made to a single location.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FPCSKVI6DXYV5FKT7BJZXVVIXYN2M52Y/

[Python-Dev] Re: Adding GitHub usernames to the developer log

[Kyle Stanley]

> But I will say that listing our GitHub usernames with our real names is
not required to tell who is a core developer. In GitHub's UI there are
multiple places it will tell you if a person is a member of the
repository/team

Within the GitHub UI if you're not a member of the organization, it tells
you who's a member of the org, but it does _not_ tell you what specific
team they're on. There are close to 100 people that are in the Python
organization on various teams but not a member of python-core. This
includes anyone on the more recently created python-triage team or any of
the PSF teams. I only became aware of this more recently, I'm not sure as
to whether or not it's a configurable setting that is modifiable by the
organization admins.

To verify this, simply log out of your GitHub account and go into a PR with
multiple organization members. Hovering over their name will show "Member
of Python" but not the actual name of the team(s) they're on. For a recent
example, logout of your GitHub account, go to PR-16375 (
https://github.com/python/cpython/pull/16375), then compare the hover cards
between myself and Yury. Many people have their role within the
organization mentioned in their GitHub bio (which appears on the hover
card), but this is not always the case.

It's also very useful when trying to find a specific core developer,
particularly when trying to locate the GitHub usernames for specific
experts. Locating the GitHub usernames of the experts is a bit of a
separate issue, but having the usernames listed on the developer log would
at least make it easier to do so without relying on bpo.

On Wed, Sep 25, 2019 at 12:48 PM Brett Cannon  wrote:

> I'm personally fine listing GItHub usernames in the devguide's developer
> list, but I'm also not trying to be anonymous on GitHub.
>
> But I will say that listing our GitHub usernames with our real names is
> not required to tell who is a core developer. In GitHub's UI there are
> multiple places it will tell you if a person is a member of the
> repository/team (e.g. hover cards, each comment a teammate makes).
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KOKMLHX4RSCWPCUR3YEQBVRB6R3VJOVS/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SGKRKMWXOAUW26332QAK2456QLDWL3AP/

[Python-Dev] Re: Adding GitHub usernames to the developer log

[Kyle Stanley]

> devguide now uses developers.rst which is generated from the following
private file:
https://github.com/python/voters/blob/master/python-core.toml

Ah good to know. I don't have access to the voters repo so I wasn't sure
about the exact format or location of the file.

> Guessing a GitHub identifier from a real name is not really easy.

Yeah that's my main concern, particularly for new contributors trying to
find or identify a specific core developer.

For some people it's quite easy when it matches up with their real name
(such as yourself and Brett for example). But for others it's definitely
not obvious, especially when their real name isn't included in any part of
their username. I admit that I'm guilty of this myself with using
"aeros167" instead of "kstanley" or something along those lines (I had my
GitHub account for a decent while before I started contributing to Python).
This would be a non-issue with a public listing of core developer real
names -> GitHub usernames that isn't dependent on bpo.

On Wed, Sep 25, 2019 at 7:12 AM Victor Stinner  wrote:

> Le mer. 25 sept. 2019 à 08:24, Kyle Stanley  a écrit :
> > Recently, Brett updated the developer log in the devguide
> > (https://devguide.python.org/developers/) to fetch the names of each
> core
> > developer and the date they were given commit privileges from the private
> > python-committers repository.
>
> devguide now uses developers.rst which is generated from the following
> private file:
> https://github.com/python/voters/blob/master/python-core.toml
>
> The TOML file contains developer identifiers for:
>
> * GitHub
> * bugs.python.org (Roundup)
> * discuss.python.org (Discourse)
>
> > The motivation behind creating a a new topic for this issue was Brett's
> > response to my comment in the PR that updated the devguide
> > (https://github.com/python/devguide/pull/533#issuecomment-532405907).
> > Essentially, if no core developers have an issue with having their GitHub
> > username posted on the devguide, we can move forward with adding it.
>
> I'm in favor of making the GitHub identifiers public since it's part
> of the trust relationship between core developers and contributors.
> Some operations in our workflow *requires* a core developer on GitHub
> pull requests. So it's good to be able to check who are core
> developers on GitHub.
>
> Guessing a GitHub identifier from a real name is not really easy.
>
> The GitHub identifer can be *guessed* using the public bugs.python.org
> data. For example, I'm user 2377 on bugs.python.org which shows that
> my GitHub identifier is vstinner:
> https://bugs.python.org/user2377
>
> "Is Committer [hidden]" doesn't help to check if it's real or a fake
> account :-/ Some core developers have multiple bugs.python.org
> accounts.
>
> Note: I changed my bugs.python.org and GitHub identifiers one year ago
> from "haypo" to "vstinner" :-)
>
> It seems like https://github.com/orgs/python/teams/python-core/members
> is private.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NKDS3ELJH3N3ARWJH2FAMBJ7KX4J3JKC/

[Python-Dev] Re: Currently working on the release of Python 3.8.0rc1

[Kyle Stanley]

> Can you help with the "What's New" document?

After seeing this, I went through the whatsnew for 3.8 and found a couple
of typos and broken links (from Sphinx roles that were slightly off). I
opened a PR to fix as many as I was able to:
https://github.com/python/cpython/pull/16535. Raymond will be the one to
make the final decision as the editor, but I would appreciate it if anyone
could help with verifying the changes and providing feedback. I'd like to
have the PR finished a good bit before October 14th to avoid last minute
additions. Having additional reviewers would likely help to accelerate the
process.

On Mon, Sep 30, 2019 at 3:53 AM Łukasz Langa  wrote:

> Team,
> amazing job on getting us back on track over the weekend.
>
> *Thank you*
> All release blockers and deferred release blockers solved. And there was
> relatively little additional activity on the branch -- as expected at this
> point! Thank you for this, it will help get the release candidate out on
> time.
>
> *I'm working on cutting RC1 today*
> Hopefully all sanity checks, as well as building the source tarball and
> the binary installers for macOS and Windows will all work out fine and
> we'll be seeing RC1 out tonight.
>
> *RC2 and the date of 3.8.0 gold*
> If we manage to avoid the need for RC2, we will be able to release 3.8.0
> on October 14th. If we need an RC2, that will slip by a week. I hope we
> won't. Ideally RC1 should be identical codewise to 3.8.0.
>
> To help our chances, please avoid any source-related activity on the 3.8
> branch from now until the release of 3.8.0. Yes, that also counts for bug
> fixes unless they are critical. (Yeah, it might be a bit annoying but
> nobody wants to be the person who introduced a last minute regression in a
> major release.)
>
> Note I didn't say I forbid any activity. I have no power over you,
> actually. More importantly though, I trust your judgement if you assess
> some bug is bad enough the fix absolutely has to get into 3.8.0. Moreover,
> I specifically said source-related activity because...
>
> 3.8.0 is important. To some users it will be the first and the last
> release in the 3.8 series they will see.
>
> *We need you to focus on the docs now*
> Are all your changes properly documented?
> Did you notice other changes you know of to have insufficient
> documentation?
> Can you help with the "What's New" document?
>
> anxiously configure-&&-makingly y'rs
> - Ł
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQ66MSEXD5PN5VVTIJA3E4U32MNIIMIL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Need help to fix test_asyncio issues

[Kyle Stanley]

> Recently, I started to experiment "./python -m test [options] -F
-j100" to attempt to reproduce some tricky race conditions: -j100
spawns 100 worker processes in parallel and -F stands for --forever
(run tests in loop and stop at the first failure).

Interesting, even as someone who has recently worked within
asyncio a decent amount recently, I was entirely unaware of
this method of producing race conditions in the tests. If continued
experimentation shows this to be a reliable means of reproducing
various race conditions, it might be worth adding this to
https://devguide.python.org/runtests/.

Thanks for sharing!


On Mon, Oct 21, 2019 at 7:08 AM Victor Stinner  wrote:

> Hi,
>
> Right now, there are 14 open issues with "test_asyncio" in the title.
> Many test_asyncio tests have race conditions. I'm trying to fix them
> one by one, but it takes time, and then new tests are added with new
> race condition :-( For example, the following new test is failing
> randomly on Windows:
>
> "Windows: test_asyncio: test_huge_content_recvinto() fails randomly
> with ProactorEventLoop" is failing randomly since 6 months:
> https://bugs.python.org/issue36732
>
> test_asyncio uses more and more functional tests which is a good
> thing. In the early days of asyncio, most tests mocked more than half
> of asyncio to really be "unit test". But at the end, the test tested
> more mocks than asyncio... The problem of functional tests is that
> it's hard to design them properly to avoid all race conditions,
> especially when you consider multiplatform (Windows, macOS, Linux,
> FreeBSD, etc.).
>
> It would help me if someone could try to investigate these issues,
> provide a reliable way to reproduce them, and propose a fix. (Simply
> saying that you can reproduce the test and that you would like to work
> on an issue doesn't really help, sorry.)
>
> Recently, I started to experiment "./python -m test [options] -F
> -j100" to attempt to reproduce some tricky race conditions: -j100
> spawns 100 worker processes in parallel and -F stands for --forever
> (run tests in loop and stop at the first failure). I was surprised
> that my Fedora 30 didn't burn in blame. In fact, the GNOME desktop
> remains responsible even with a system load higher than 100. The Linux
> kernel (5.2) is impressive! Under such high system load (my laptop has
> 8 logical CPUs), race conditions are way more likely.
>
> The problem of test_asyncio is that it's made of 2160 tests, see:
>
>./python -m test test_asyncio --list-cases
>
> You may want to only run a single test case (class) or even a single
> test method: see --match option which can be used multiple times to
> only run selected test classes or selected test methods. See also
> --matchfile which is similar but uses a file. Example:
>
> $ ./python -m test test_asyncio --list-cases > cases
> # edit cases
> $ ./python -m test test_asyncio --matchfile=cases
>
> test_asyncio is one of the most unstable test: I'm getting more and
> more buildbot-status emails about test_asyncio... likely because we
> fixed most of the other race conditions which is a good thing ;-)
>
> Some issues look to be specific to Windows, but it should be possible
> to reproduce most issues on Linux as Linux. Sometimes, it's just that
> some specific Windows buildbot workers are slower than other buildbot
> workers.
>
> Good luck ;-)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGEOKWD3PBWIL2LPZWZ6MMRANN5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7K4Z3QJTZCRALCIIFEJSQXVCMBYU37J2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Awareness creation of a new IDE

[Kyle Stanley]

> You don't need permission to support Python in a commercial IDE. It's
open source.

Adding to this, see
https://docs.python.org/3/license.html#terms-and-conditions-for-accessing-or-otherwise-using-python
for the complete licensing information on Python.

On a somewhat related note to the licensing page, I just noticed that the
link to Zope's website (http://www.zope.com) is no longer valid. They
currently use a .org TLD instead of .com (https://www.zope.org). Mind if I
open a PR to fix/update the link in Doc/licensing.rst?

On Mon, Oct 21, 2019 at 1:53 PM Guido van Rossum  wrote:

> Hi Lasan,
>
> You don't need permission to support Python in a commercial IDE. It's open
> source. If you use the Python logo, make sure it refers to Python and that
> you don't claim to be endorsed by the PSF, that's all we ask.
>
> Good luck!
>
> --Guido
>
> On Mon, Oct 21, 2019 at 9:41 AM Contact - Google 
> wrote:
>
>> Dear Sir / Madam ,
>>
>> I'm Lasan Nishshanka, CEO and the Founder of Clenontec. We are creating
>> an IDE. It's name is Bacend Studio. We're creating this for the Windows
>> Operating System. This IDE is designed to perform special functions such
>> as App Development, Software Development, Games Development, Machine
>> Learning, Artificial Intelligence etc.
>> We will send you this letter to get permission to add the PYTHON
>> Programming Language to our IDE. If you can give it permission, please
>> tell us. We await a speedy reply from you.
>>
>> Company Web Site : https://www.clenontec.com/
>>
>> E Mail : clenon...@gmail.com
>>
>> Thank You.
>>
>>
>> --
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LKL5DNJAGAQLNPVJF5DVGIBITZZ2DMBA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJ5DAQYXZXFUR6NF7BGRFNYQDY2CAW6R/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RTSYN4K3FD5WX4GANRMUB6QL6YZPAADF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Allow custom headers in `http.server`

[Kyle Stanley]

> Probably python-list/comp.lang.python mailing list/news group is the
best place

Since this involves a potential suggested change (adding a feature to
assign custom headers to http.server), python-ideas or the "Ideas" section
of https://discuss.python.org/ would also be appropriate.

On Sat, Oct 26, 2019 at 11:05 AM Oleg Broytman  wrote:

> Hello.
>
>This mailing list is to work on developing Python (adding new
> features to Python itself and fixing bugs); if you're having problems
> learning, understanding or using Python, please find another forum.
> Probably python-list/comp.lang.python mailing list/news group is the
> best place; there are Python developers who participate in it; you may
> get a faster, and probably more complete, answer there. See
> https://www.python.org/community/ for other lists/news groups/fora.
> Thank you for understanding.
>
> On Sat, Oct 26, 2019 at 03:29:03PM +0300, Alex Yursha <
> alexyur...@gmail.com> wrote:
> > Hi CPython maintainers,
> >
> > I need to test my CORS setup and looking for a possibility to set a
> > custom *Access-Control-Allow-Origin
> > *header in http.server. As of now, there is no such feature. Are you
> > interested in me writing a patch to contribute a feature of setting
> custom
> > headers directly to `http.server`?
>
>You can override method ``send_headers`` of the class
> ``HTTPRequestHandler`` and add your own headers. See an example at
>
> https://docs.aws.amazon.com/polly/latest/dg/example-Python-server-code.html
>
> > Best,
> > - Alex
>
> Oleg.
> --
> Oleg Broytmanhttps://phdru.name/p...@phdru.name
>Programmers don't die, they just GOSUB without RETURN.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/SSJL2BEQNII23ZGMLAXWYJYKTK35H6TO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EUCMXIWKK4ICGAIOZ457LD7QUEDU7TTB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: A much better tokenize.untokenize function

[Kyle Stanley]

> This is all irrelevant if you haven't signed the CLA*.

IMO, this shouldn't be an argument against the proposed changes, as the CLA
is fairly straightforward and only takes 24-48 hours to process. Unless the
OP specifically is unable to or doesn't want to sign the CLA, it won't be a
significant concern. Most new authors usually open their first PR without
having the CLA already signed, then have the CLA signed and processed while
waiting for review from a core developer.

On Mon, Nov 4, 2019 at 11:48 AM Ethan Furman  wrote:

> On 11/04/2019 03:42 AM, Edward K. Ream wrote:
> > On Sun, Nov 3, 2019 at 7:23 PM Edward K. Ream wrote:
>
> > Let's be clear, there is no reason for python libraries to include every
> little code gem. However, I have two motivations adding the gem to the
> tokenize module:
>
> This is all irrelevant if you haven't signed the CLA*.
>
> --
> ~Ethan~
>
>
> * https://www.python.org/psf/contrib/contrib-form/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/F6MZYCE3FRHAIBFCSMVQ2JTKQDRO5TAJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3ZC5VFJGGJBJE467HQ5E3KX346EC6OEN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Implementation of PEP-0604

[Kyle Stanley]

> I agree with Nick, this is indeed big, but not impossible. If you are not
sure yet whether you will work on implementation, you can focus on
polishing the PEP text, and then if it is accepted and you will decide to
give implementation to someone else, we will find a volunteer.

Unless I'm misunderstanding something, it seems that the OP is under the
impression that because they authored the PEP, they're obligated and
responsible for the entire implementation. Instead, we should be able to
relatively easily split up the implementation into multiple self-contained
PRs. The OP can still work on some of them if they want to and are able to,
but other contributors can work on other parts of it.

The easiest way of doing this would likely be separating each
self-contained PR into a bullet point in a bpo issue with a brief
description (similar to the layout posted above), and then the OP can work
through them as they have time to do so. Any other contributor can simply
pick up where the OP left off, or on other parts of it while the OP is
working on something else. This would of course be after/if the PEP is
accepted.

On Mon, Nov 4, 2019 at 7:33 AM Ivan Levkivskyi  wrote:

> On Thu, 31 Oct 2019 at 23:20, Nick Coghlan  wrote:
>
>> [...snip...]
>>
>>>
>>> It's not enough to move the typing classes, I must move
>>> functools.lru_cache() and dependencies, collections.abs.Mapping and
>>> dependencies, and track the frame level.
>>>
>>> *It's too big for me.*
>>>
>>>
>> It's certainly not an easy project to tackle.
>>
>> For some of the specific points you raise though, you wouldn't translate
>> the existing Python code directly to C.
>>
>
> I agree with Nick, this is indeed big, but not impossible. If you are not
> sure yet whether you will work on implementation, you can focus on
> polishing the PEP text, and then if it is accepted and you will decide to
> give implementation to someone else, we will find a volunteer.
>
> --
> Ivan
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AKXBKSBYESEJVKAAPCXHG4Z5EWOS54MM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YYAKCVANFGRJAVGML4WJ225M5ANYZFDD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP proposal to limit various aspects of a Python program to one million.

[Kyle Stanley]

  > The number of live coroutines in a running interpreter.

Could you further elaborate on what is meant by "live coroutines"? My
guesses (roughly from most likely to least likely) would be:

1) All known coroutine objects in a state of either CORO_RUNNING or
CORO_SUSPENDED, but *not* CORO_CREATED or CORO_CLOSED.
2) Coroutine objects that are currently running, in a state of CORO_RUNNING
3) Coroutines objects being awaited, in a state of CORO_SUSPENDED
4) All known coroutine objects in a state of either CORO_CREATED,
CORO_RUNNING, or CORO_SUSPENDED
5) All known coroutine objects in any state
6) Total declared coroutines

Just so we're all on the same page, I'm referring to a "coroutine object"
as the object returned from the call `coro()`; whereas a "coroutine" is the
coroutine function/method `async def coro` (or the deprecated
generator-based coroutines).

It probably wouldn't be as much of a concern to only allow 1M running
coroutines at one time, but it might be an issue to only allow for there to
be 1M known coroutine objects in any state within a given interpreter.
Particularly for servers that run nearly indefinitely and handle a
significant number of concurrent requests.

On Tue, Dec 3, 2019 at 11:24 AM Mark Shannon  wrote:

> Hi Everyone,
>
> I am proposing a new PEP, still in draft form, to impose a limit of one
> million on various aspects of Python programs, such as the lines of code
> per module.
>
> Any thoughts or feedback?
>
> The PEP:
> https://github.com/markshannon/peps/blob/one-million/pep-100.rst
>
> Cheers,
> Mark.
>
>
> Full text
> *
>
> PEP: 100
> Title: The one million limit
> Author: Mark Shannon 
> Status: Active
> Type: Enhancement
> Content-Type: text/x-rst
> Created: 03-Dec-2019
> Post-History:
>
>
>
> Abstract
> 
> This PR proposes a limit of one million (1 000 000) for various aspects
> of Python code and its implementation.
>
> The Python language does not specify limits for many of its features.
> Not having any limit to these values seems to enhance programmer freedom,
> at least superficially, but in practice the CPython VM and other Python
> virtual
> machines have implicit limits or are forced to assume that the limits are
> astronomical, which is expensive.
>
> This PR lists a number of features which are to have a limit of one
> million.
> If a language feature is not listed but appears unlimited and must be
> finite,
> for physical reasons if no other, then a limit of one million should be
> assumed.
>
>
> Motivation
> ==
>
> There are many values that need to be represented in a virtual machine.
> If no limit is specified for these values, then the representation must
> either be inefficient or vulnerable to overflow.
> The CPython virtual machine represents values like line numbers,
> stack offsets and instruction offsets by 32 bit values. This is
> inefficient, and potentially unsafe.
>
> It is inefficient as actual values rarely need more than a dozen or so
> bits to represent them.
>
> It is unsafe as malicious or poorly generated code could cause values to
> exceed 2\ :sup:`32`.
>
> For example, line numbers are represented by 32 bit values internally.
> This is inefficient, given that modules almost never exceed a few
> thousand lines.
> Despite being inefficent, is is still vulnerable to overflow as
> it is easy for an attacker to created a module with billions of newline
> characters.
>
> Memory access is usually a limiting factor in the performance of modern
> CPUs.
> Better packing of data structures enhances locality and reduces memory
> bandwith,
> at a modest increase in ALU usage (for shifting and masking).
> Being able to safely store important values in 20 bits would allow
> memory savings
> in several data structures including, but not limited to:
>
> * Frame objects
> * Object headers
> * Code objects
>
> There is also the potential for a more efficient instruction format,
> speeding up interpreter dispatch.
>
> Rationale
> =
>
> Imposing a limit on values such as lines of code in a module, and the
> number of local variables,
> has significant advantages for ease of implementation and efficiency of
> virtual machines.
> If the limit is sufficiently large, there is no adverse effect on users
> of the language.
>
> By selecting a fixed but large limit for these values,
> it is possible to have both safety and efficiency whilst causing no
> inconvience to human programmers
> and only very rare problems for code generators.
>
> One million
> ---
>
> The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
> (65535) for many program
> elements similar to those covered here.
> This limit enables limited values to fit in 16 bits, which is a very
> efficient machine representation.
> However, this limit is quite easily exceeded in practice by code
> generators and
> the author is aware of existing Python code that already exceeds 2\
> :sup:`16` lines of code.
>
> A limit of one million fits 

[Python-Dev] Re: Deprecating the "u" string literal prefix

[Kyle Stanley]

> BTW, I think 2to3 can help to move from 2&3 code to 3-only code.

> Instead, we can do:

> * Don't recommend u-prefix except in Python 2&3 code.
> * Provide a tool to remove the u-prefix.

+1, this seems like the smoothest way of handling it and has very minimal
impact on users. In 5+ years from now as u-strings become increasingly
rare, we can consider a long term deprecation process. In the meantime, any
tools we can provide to automate the process of converting them will make
the transition that much easier.

On Wed, Dec 4, 2019 at 12:41 AM Inada Naoki  wrote:

> On Wed, Dec 4, 2019 at 11:49 AM Ned Batchelder 
> wrote:
> >
> > On 12/3/19 8:13 PM, Inada Naoki wrote:
> > > I think it is too early to determine when to remove it.
> > > Even only talking about it causes blaming war.
> >
> > Has anyone yet given a reason to remove it?
>
> Note that "never" is included in ”when".
> I didn't promoting removing it.  I just said let's stop discussion about
> it.
>
>
> > It will change working code
> > into broken code.  Why do that?
>
> Of course, everyone in this thread understands it.
> No one proposes remove it now.
>
> On the other hand, we remove some parts from Python language
> and the stdlib regularly to keep Python clean.
> All removal will break some code.  That's why we have a deprecation period.
>
> Currently, u-prefix is very widely used.  It shouldn't be removed anytime
> soon.
> And I agree that we shouldn't raise DeprecationWarning for now.
>
> But how about 5, 10, and 20 years later?  No one knows it.
> Let's stop discussing it.  It can not be productive.
>
> Instead, we can do:
>
> * Don't recommend u-prefix except in Python 2&3 code.
> * Provide a tool to remove the u-prefix.
>
> Regards,
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EVKCCO5KMOGEEFMSSY2PZRVGT2LDOB5K/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AKUXJR6BO4WTO7SLN3ES6PCWNO5H62ZU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Travis CI for backports not working.

[Kyle Stanley]

Steve Dower wrote:
> As a related aside, I've been getting GitHub Actions support together
> (which I started at the sprints).

Would adding support for GitHub Actions make it easier/faster to
temporarily disable and re-enable specific CI services when they're having
external issues? IIUC, that seems to be the primary concern to address.

Note that I'm not particularly well acquainted with GitHub Actions, other
than briefly looking over https://github.com/features/actions.

On Fri, Dec 6, 2019 at 1:01 PM Steve Dower  wrote:

> On 06Dec2019 0620, Victor Stinner wrote:
> > What's the status? Right now, I see Travis CI jobs passing on 3.7,
> > 3.8 and master branches so I don't understand the problem. Maybe the
> > issue has been fixed and Travis CI can be made mandatory again?
>
> They've been passing fine for me too, I'm not quite sure what the issue
> was.
>
> As a related aside, I've been getting GitHub Actions support together
> (which I started at the sprints). My test PR is at
> https://github.com/zooba/cpython/pull/7 if anyone wants to check out
> what it could look like. Feel free to leave comments there.
>
> (One of these days I'll have to join core-workflow I guess...)
>
> Cheers,
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/MTD2EQC3BWEGEV3LNKPF3KEJBSQN24LS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PEHYMEYY2VGW5SKU6SR6RUTIQQ3C7JP4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we require all deprecations to have a removal version that we follow through on?

[Kyle Stanley]

Victor Stinner wrote:
> Isn't it already the current unwritten deprecation process?

Personally, I don't think we should rely on an unwritten process for
something as important and potentially breaking as a deprecation process.
Regardless of the outcome of this discussion, I think we should try to
fully write out the process in the devguide; while still providing some
amount of flexibility.

Victor Stinner wrote:
> I'm not sure about enforcing the removal. Some APIs were deprecated,
> but it was unsure if we really wanted to remove them. Or because we
> had no idea how long it would take for developers to upgrade to the
> new functions.
...
> I would suggest to schedule removal, but not require it too strongly ;-)

Would it be reasonable to require an minimum amount of versions to be
specified (such as n versions ahead), but provide flexibility in terms to
delaying the removal, as needed? IMO, it would be more convenient for users
to have a "minimum removal" version in mind, rather than an unscheduled
deprecation that could have a version+2 (deprecation and removal in 2
versions) assigned at any point in time.

This can be difficult sometimes when we're not sure what n versions from
now will actually be called (such as with 3.10 and 4.0), but I don't think
it would be an issue to state something like this:
"func() has been deprecated, and will be removed in no sooner than 4
versions after 3.9"

instead of:
"func() has been deprecated, and will be removed in 3.13"

for long term deprecations. By "long term", I mean more than two versions.

Victor Stinner wrote:
> I know that it's an unpopular opinion, but I would strongly prefer
> Python 4 to behave exactly as any another Python 3.x release in term
> of deprecation. If I exaggerate, I would prefer that Python 4.0 would
> have *less* incompatible changes than the last Python 3.x release.

My understanding was that we were specifically waiting on considering a
Python 4.0 until there were major enough improvements/changes (such as the
"GILectomy", for example) to justify some degree of incompatibility. If it
would behave exactly the same as a standard 3.x release in terms to
incompatible changes (or have less), what would be the purpose of making a
major version change in the first place? I don't see an issue with
indefinitely continuing the 3.x line in the meantime.

Victor Stinner wrote:
> I like to remove deprecated features at the beginning of a dev cycle
> to get users feedback earlier. If they are removed late in the dev
> cycle, there is a higher risk that a problematic incompatible change
> goes through a 3.x.0 final release and so cannot be reverted anymore.

I agree, the earlier a deprecation occurs in a given release cycle, the
easier it is for users to provide feedback and then potentially prepare for
the change. Perhaps we could establish some form of guideline, for example:
"When possible, the removal of previously deprecated features should occur
as early as possible in a version's dev cycle, preferably during the alpha
phases. This provides users more time to provide feedback and plan for the
potential removal".

On Fri, Dec 6, 2019 at 10:58 AM Victor Stinner  wrote:

> Hi,
>
> Le mer. 27 nov. 2019 à 19:40, Brett Cannon  a écrit :
> > What do people think of the idea of requiring all deprecations
> specifying a version that the feature will be removed in (which under our
> annual release cadence would be at least the third release from the start
> of the deprecation, hence the deprecation being public for 2 years)? And
> that we also follow through with that removal?
>
> Isn't it already the current unwritten deprecation process? The common
> case is to schedule the removal. Previously, it was common to wait 2
> releases before removing anything: pending deprecatation in 3.6,
> deprecation in 3.7, removal in 3.8. It means that developers are
> warned during 2 releases: 3 years (I'm optimistic and consider that
> developers pay attention to PendingDeprecationWarning). So deprecation
> during 3 releases with the new release cadence would fit the previous
> practice: 3 years ;-)
>
> Are you suggesting to start with a PendingDeprecationWarning? I recall
> a debate about the value of this warning which is quiet by default,
> whereas DeprecationWarning is now displayed in the __main__ module.
>
> I'm not sure about enforcing the removal. Some APIs were deprecated,
> but it was unsure if we really wanted to remove them. Or because we
> had no idea how long it would take for developers to upgrade to the
> new functions.
>
> For example, the C API for the old Unicode API using "Py_UNICODE*"
> type is deprecated since Python 3.3 but there is no concrete plan to
> remove it (just a vague "Python 4.0 removal" plan).
>
> I deprecated the support for bytes filenames on Windows in Python 3.3,
> but Steve Dower implemented his PEP 529 (use UTF-8 for bytes on
> Windows) in Python 3.6 and so the deprecation warnings have been
> simply removed!
>
> I would sugges

[Python-Dev] Re: Travis CI for backports not working.

[Kyle Stanley]

Steve Dower wrote:
> GitHub Actions *is* a CI service now, so my PR is actually using their
> machines for Windows/macOS/Ubuntu build and test.

Oh I see, that makes more sense. Thanks for the clarification.

Steve Dower wrote:
> It doesn't change the ease of enabling/disabling anything - that's still
> very easy if you're an administrator on our repo and impossible if
> you're not :)

Yeah I was aware that it required administrative permissions, I was just
under the impression that it wasn't a simple on/off toggle, and that there
wasn't an easy way to ensure failures were occurring due to external
factors. I suppose that's a separate issue though.

Steve Dower wrote:
> And you don't need a separate login to rerun checks - it's just a simple
button in the
> GitHub UI.

That would be especially great, and much better than closing+reopening the
PR. Restarting the checks is typically a last resort since intermittent
regression test failures are a concern, but it's highly useful when the
failure is occurring due to external factors (such as a recently merged PR
or issues with the CI service itself).

On Fri, Dec 6, 2019 at 1:33 PM Steve Dower  wrote:

> On 06Dec2019 1023, Kyle Stanley wrote:
> > Steve Dower wrote:
> >  > As a related aside, I've been getting GitHub Actions support together
> >  > (which I started at the sprints).
> >
> > Would adding support for GitHub Actions make it easier/faster to
> > temporarily disable and re-enable specific CI services when they're
> > having external issues? IIUC, that seems to be the primary concern to
> > address.
> >
> > Note that I'm not particularly well acquainted with GitHub Actions,
> > other than briefly looking over https://github.com/features/actions.
>
> GitHub Actions *is* a CI service now, so my PR is actually using their
> machines for Windows/macOS/Ubuntu build and test.
>
> It doesn't change the ease of enabling/disabling anything - that's still
> very easy if you're an administrator on our repo and impossible if
> you're not :)
>
> However, it does save jumping to an external website to view logs, and
> over time we can improve the integration so that error messages are
> captured properly (but you can easily view each step). And you don't
> need a separate login to rerun checks - it's just a simple button in the
> GitHub UI.
>
> Cheers,
> Steve
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KB7F5JNUILUH3H2DJ3T7NGZAKQXMK76D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

> (b) Why limit coroutines? It's just another Python object and has no
operating resources associated with it. Perhaps your definition of
coroutine is different, and you are thinking of OS threads?

This was my primary concern with the proposed PEP. At the moment, it's
rather trivial to create one million coroutines, and the total memory taken
up by each individual coroutine object is very minimal compared to each OS
thread.

There's also a practical use case for having a large number of coroutine
objects, such as for asynchronously:

1) Handling a large number of concurrent clients on a continuously running
web server that receives a significant amount of traffic.
2) Sending a large number of concurrent database transactions to run on a
cluster of database servers.

I don't know that anyone is currently using production code that results in
1 million coroutine objects within the same interpreter at once, but
something like this definitely scales over time. Arbitrarily placing a
limit on the total number of coroutine objects doesn't make sense to me for
that reason.

OS threads on the other hand take significantly more memory. From a recent
(but entirely unrelated) discussion where the memory usage of threads was
brought up, Victor Stinner wrote a program that demonstrated that each OS
thread takes up approximately ~13.2kB on Linux, which I verified on kernel
version 5.3.8. See https://bugs.python.org/msg356596.

For comparison, I just wrote a similar program to compare the memory usage
between 1M threads and 1M coroutines:

```
import asyncio
import threading
import sys
import os

def wait(event):
event.wait()

class Thread(threading.Thread):
def __init__(self):
super().__init__()
self.stop_event = threading.Event()
self.started_event = threading.Event()

def run(self):
self.started_event.set()
self.stop_event.wait()

def stop(self):
self.stop_event.set()
self.join()

def display_rss():
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")

async def test_mem_coros(count):
print("Coroutine memory usage before:")
display_rss()
coros = tuple(asyncio.sleep(0) for _ in range(count))
print("Coroutine memory usage after creation:")
display_rss()
await asyncio.gather(*coros)
print("Coroutine memory usage after awaiting:")
display_rss()

def test_mem_threads(count):
print("Thread memory usage before:")
display_rss()
threads = tuple(Thread() for _ in range(count))
print("Thread memory usage after creation:")
display_rss()
for thread in threads:
thread.start()
print("Thread memory usage after starting:")
for thread in threads:
thread.run()
print("Thread memory usage after running:")
display_rss()
for thread in threads:
thread.stop()
print("Thread memory usage after stopping:")
display_rss()

if __name__ == '__main__':
count = 1_000_000
arg = sys.argv[1]
if arg == 'threads':
test_mem_threads(count)
if arg == 'coros':
asyncio.run(test_mem_coros(count))

```
Here are the results:

1M coroutine objects:

Coroutine memory usage before:
VmRSS: 14800 kB
Coroutine memory usage after creation:
VmRSS:651916 kB
Coroutine memory usage after awaiting:
VmRSS:   1289528 kB

1M OS threads:

Thread memory usage before:
VmRSS: 14816 kB
Thread memory usage after creation:
VmRSS:   4604356 kB
Traceback (most recent call last):
  File "temp.py", line 60, in 
test_mem_threads(count)
  File "temp.py", line 44, in test_mem_threads
thread.start()
  File "/usr/lib/python3.8/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

(Python version: 3.8)
(Linux kernel version: 5.13)

As is present in the results above, 1M OS threads can't even be ran at
once, and the memory taken up just to create the 1M threads is ~3.6x more
than it costs to concurrently await the 1M coroutine objects. Based on
that, I think it would be reasonable to place a limit of 1M on the total
number of OS threads. It seems unlikely that a system would be able to
properly handle 1M threads at once anyways, whereas that seems entirely
feasible with 1M coroutine objects. Especially on a high traffic server.

On Mon, Dec 9, 2019 at 12:01 PM Guido van Rossum  wrote:

> I want to question two specific limits.
>
> (a) Limiting the number of classes, in order to potentially save space in
> object headers, sounds like a big C API change, and I think it's better to
> lift this idea out of PEP 611 and debate the and cons separately.
>
> (b) Why limit coroutines? It's just another Python object and has no
> operating resources associated with it. Perhaps your definition of
> coroutine is different, and you are thinking of OS threads?
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

> This logic doesn't seem much different than would be for coroutines...
Just need to wait for larger systems...

> With 100k threads started we're only using 8G memory, there are plenty of
systems today with more than 80G of RAM

Well, either way, I think it's still a solid argument against imposing the
1M limit on coroutines. Arguing in favor or against 1M OS threads wasn't
the primary message I was trying to convey, it was just to demonstrate that
1M coroutines could be created and awaited concurrently on most current
systems (w/ ~1.3GB+ available RAM and virtual memory).

But I think there's a reasonable question of practicality when it comes to
running 1M OS threads simultaneously. For especially high volumes of
concurrent tasks, OS threads are generally not the best solution (for
CPython, at least). They work for handling a decent number IO-bound tasks
such as sending out and processing network requests, but coroutine objects
are significantly more efficient when it comes to memory usage.

For the usage of child processes, we have watcher implementations that
don't use OS threads at all, such as the recently added PidfdChildWatcher (
https://docs.python.org/3.9/library/asyncio-policy.html#asyncio.PidfdChildWatcher).
There are also others that don't spawn a new thread per process.

That being said, you are correct in that at some point, the memory usage
for running 1M simultaneous OS threads will be perfectly reasonable. I'm
just not sure if there's a practical reason to do so, considering that more
efficient means of implementing parallelism are available when memory usage
becomes a significant concern.

Of course, the main question here is: "What benefit would imposing this
particular limit to either coroutines objects or OS threads provide?".

Personally, I'm not entirely convinced that placing a hard limit of 1M at
once for either would result in a significant benefit to performance,
efficiency, or security (mentioned in the PEP, as a reason for imposing the
limits). I could see it being more useful for other areas though, such as
lines of code or bytecode instructions per object.

I just think that placing a limit of 1M on current coroutine objects would
not be reasonable. But between the two, I think a limit of 1M on OS threads
is *more* reasonable in comparison.


On Mon, Dec 9, 2019 at 10:17 PM Khazhismel Kumykov  wrote:

>
>
> On Mon, Dec 9, 2019, 18:48 Kyle Stanley  wrote:
>
>> > (b) Why limit coroutines? It's just another Python object and has no
>> operating resources associated with it. Perhaps your definition of
>> coroutine is different, and you are thinking of OS threads?
>>
>> This was my primary concern with the proposed PEP. At the moment, it's
>> rather trivial to create one million coroutines, and the total memory taken
>> up by each individual coroutine object is very minimal compared to each OS
>> thread.
>>
>> There's also a practical use case for having a large number of coroutine
>> objects, such as for asynchronously:
>>
>> 1) Handling a large number of concurrent clients on a continuously
>> running web server that receives a significant amount of traffic.
>> 2) Sending a large number of concurrent database transactions to run on a
>> cluster of database servers.
>>
>> I don't know that anyone is currently using production code that results
>> in 1 million coroutine objects within the same interpreter at once, but
>> something like this definitely scales over time. Arbitrarily placing a
>> limit on the total number of coroutine objects doesn't make sense to me for
>> that reason.
>>
>> OS threads on the other hand take significantly more memory. From a
>> recent (but entirely unrelated) discussion where the memory usage of
>> threads was brought up, Victor Stinner wrote a program that demonstrated
>> that each OS thread takes up approximately ~13.2kB on Linux, which I
>> verified on kernel version 5.3.8. See https://bugs.python.org/msg356596.
>>
>> For comparison, I just wrote a similar program to compare the memory
>> usage between 1M threads and 1M coroutines:
>>
>> ```
>> import asyncio
>> import threading
>> import sys
>> import os
>>
>> def wait(event):
>> event.wait()
>>
>> class Thread(threading.Thread):
>> def __init__(self):
>> super().__init__()
>> self.stop_event = threading.Event()
>> self.started_event = threading.Event()
>>
>> def run(self):
>> self.started_event.set()
>> self.stop_event.wait()
>>
>> def stop(self):
>> self.stop_event.set()
>> self.join()
>>

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

> I also would suggest for PEP 611 that any limits are discoverable (maybe
in sys) so it can be used by other implementations like Jython.

I agree, I think that sys would likely be the most reasonable place to read
these limits from. Also, it seems like a good location for setting of the
limits, if that becomes an option. This would go along well with the
existing sys.getrecursionlimit() and sys.setrecursionlimit().

In general, this proposal would be much easier to consider if the limits
were customizable. I'm not sure if it would be reasonable for all of the
options, but it would at least allow those who have a legitimate use case
for going beyond the limits (either now or in the future) to still be able
to do so.

On Mon, Dec 9, 2019 at 8:51 PM Jim Baker  wrote:

> I was thinking the same thing. We should distinguish limits with respect
> to the codegen process, which seem reasonable, vs runtime. Classes and
> coroutines are objects, and like objects in general, the program should
> have the option of filling its heap with any arbitrary objects. (Whether
> wise or not, this design is not for us to arbitrarily limit. For example, I
> recall that Eve Online is/was running large numbers of stackless
> coroutines, possibly well in excess of 1M.)
>
> For some comparison:
> Note the JVM has it made easier to tune the use of the native heap for
> class objects since Java 8, in part to relax earlier constraints around
> "permgen" allocation - by default, class objects are automatically
> allocated from the heap without limit (this is managed by "metaspace"). I
> suppose if this was a tunable option, maybe it could be useful, but
> probably not - Java's ClassLoader design is prone to leaking classes, as we
> know from our work on Jython. There's nothing comparable to my knowledge
> for why this would be the case for CPython class objects more than other
> objects.
>
> I also would suggest for PEP 611 that any limits are discoverable (maybe
> in sys) so it can be used by other implementations like Jython. There's no
> direct correspondence between LOC and generated Python or Java bytecode,
> but it could possibly still be helpful for some codegen systems. Jython is
> limited to 2**15 bytes per method due to label offsets, although we do have
> workarounds for certain scenarios, and could always compile, then run
> Python bytecode for large methods. (Currently we use CPython to do that
> workaround compilation, thanks!)
>
> Lastly, PEP 611 currently erroneously conjectures that "For example,
> Jython might need to use a lower class limit of fifty or sixty thousand
> becuase of JVM limits."
>
> - Jim
>
>
> On Mon, Dec 9, 2019 at 9:55 AM Guido van Rossum  wrote:
>
>> I want to question two specific limits.
>>
>> (a) Limiting the number of classes, in order to potentially save space in
>> object headers, sounds like a big C API change, and I think it's better to
>> lift this idea out of PEP 611 and debate the and cons separately.
>>
>> (b) Why limit coroutines? It's just another Python object and has no
>> operating resources associated with it. Perhaps your definition of
>> coroutine is different, and you are thinking of OS threads?
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/CJO36YRFWCTEUUROJVXIQDMWGZBFAD5T/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RQIVRB4YHNQZ2M7GCXRRCOZHZSOSNSZL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S22M4XH6PY6VKH446FDWC4JT2PFEDHL2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please be more precise when commenting on PEP 611.

[Kyle Stanley]

Chris Angelico wrote:
> We have people who believe that a bit
> mask will slow things down, others who claim that improved cache
> locality will speed things up, and Mark asks us to please justify our
> objections with numbers. But surely it's up to Mark to show numbers
> first?

+1. While it would be helpful for criticisms of the PEP to be more specific
and provide more of a cost-benefit analysis, the burden of proof for
demonstrating the benefits provided are ultimately up to the author(s) of
the PEP. We require far more justification for making a change to impose
the limits in the first place than we do for maintaining the status quo.

Personally, I don't think it would be reasonable to impose any of these
limits without some form of concrete evidence that doing so will provide an
improvement; in any combination of efficiency, performance, and/or
security, as the PEP suggests would happen. But, I don't think exact
numbers are needed. (I.E. adding these limits will improve performance
across all Python programs by an average of 22.359%!).

On Mon, Dec 9, 2019 at 7:15 PM Chris Angelico  wrote:

> On Tue, Dec 10, 2019 at 10:51 AM Steven D'Aprano 
> wrote:
> >
> > On Mon, Dec 09, 2019 at 02:12:37PM -0800, Nathaniel Smith wrote:
> > > > > On 09/12/2019 2:15 pm, Chris Angelico wrote:
> > > > You: "We should limit things. Stuff will be faster."
> > > > Others: "Really? Because bit masking is work. It'll be slower."
> >
> > I'm not an expert, but the impression I've got from various discussions
> > on performance over the last few years is that the single biggest
> > bottleneck for CPU performance is memory locality. Cache misses are so
> > expensive, and CPU instructions so fast, that memory locality is king
> > and the cost of bit masking is insignificant. In other words, worrying
> > about the cost of bit masking in C code is so very 1990s.
> >
> > I could be wrong of course: I'm not an expert. And I don't think we
> > should take it for granted that this is the case, unless some experts on
> > modern CPUs speak up and say that Mark is so obviously correct that a
> > demonstration is unnecessary.
>
> And the speculation continues.
>
> > > > You: "Maybe we limit it somewhere else, whatever. It'll be faster."
> >
> > That's a totally unfair and inaccurate representation of Mark's
> > position. The PEP doesn't say "let's put in arbitrary limits in random
> > places for the lols", he proposed seven concrete limits and gave reasons
> > for why he expects that they will improve memory efficiency, safety,
> > performance or all three.
>
> *He expects that*. That's what I'm talking about. We have lots of
> speculation and no evidence either way.
>
> > Having said that though, I think you are right that the PEP could do
> > with a bit more detail on the current status quo and existing limits,
> > and how the proposed changes will improve safety and memory use.
>
> Exactly. Yes, I know that I massively oversimplified things in that
> post. But you nonetheless acknowledge here that we are *still* quite
> lacking in any actual evidence. We have people who believe that a bit
> mask will slow things down, others who claim that improved cache
> locality will speed things up, and Mark asks us to please justify our
> objections with numbers. But surely it's up to Mark to show numbers
> first?
>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YN2BDJFPGHTEZJRDN7LXNDAGXWXB6XRA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EDZOEKHLF52MG3JJ6HIVHLCGXQJRIM34/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

Steve D'Aprano wrote:
> As I understand the PEP, the proposal is to change a bunch of C-level
> structs which currently contain 32-bit fields into 20-bit fields. To
> change that, you would need to recompile with new struct definitions and
> build a new interpreter binary.
>
> If I'm right, making this configurable at *build-time* could be
> possible, but that's probably a bad idea. That's like the old "narrow
> versus wide Unicode" builds. It doubles (at least!) the amount of effort
> needed to maintain Python, for something which (if the PEP is correct)
> benefits nearly nobody but costs nearly everyone.

Ah, if that's case I'll withdraw the suggestion for allowing the limits to
be configurable through sys then. Thanks for the clarification.

I have some basic understanding of how C structs work, but I'll admit that
it's far from an area that I'm knowledgeable about. My feedback is mostly
based on experience as a user and involvement with stdlib development, not
as a C developer. I don't have a strong understanding of the implementation
details behind the limits.

Although if they're not (reasonably) configurable by most users, I would
consider that to further reinforce my previous statement that we should
have some form of concrete evidence; to prove that imposing the limits will
provide tangible benefits to the vast majority of Python users.

On Tue, Dec 10, 2019 at 4:45 AM Steven D'Aprano  wrote:

> On Mon, Dec 09, 2019 at 11:27:56PM -0500, Kyle Stanley wrote:
>
> > I agree, I think that sys would likely be the most reasonable place to
> read
> > these limits from. Also, it seems like a good location for setting of the
> > limits, if that becomes an option. This would go along well with the
> > existing sys.getrecursionlimit() and sys.setrecursionlimit().
> >
> > In general, this proposal would be much easier to consider if the limits
> > were customizable. I'm not sure if it would be reasonable for all of the
> > options, but it would at least allow those who have a legitimate use case
> > for going beyond the limits (either now or in the future) to still be
> able
> > to do so.
>
> Someone will correct me if I'm wrong, but my reading of the PEP tells me
> that these are structural limits in the interpreter's internal data
> structures, so they can't be configured at runtime.
>
> The impression I get is that you believe that the proposal is to add a
> bunch of runtime checks like this:
>
> # pseudo-code for the module-loading code
> lines = 0;
> while 1
> {
> read and process line of code
> lines += 1;
> if lines >= 100:
> GOTO error condition
> }
>
> In that case, it would be easy to change the 100 constant to a value
> that can be configured at runtime. You wouldn't even need to quit the
> running interpreter.
>
> As I understand the PEP, the proposal is to change a bunch of C-level
> structs which currently contain 32-bit fields into 20-bit fields. To
> change that, you would need to recompile with new struct definitions and
> build a new interpreter binary.
>
> If I'm right, making this configurable at *build-time* could be
> possible, but that's probably a bad idea. That's like the old "narrow
> versus wide Unicode" builds. It doubles (at least!) the amount of effort
> needed to maintain Python, for something which (if the PEP is correct)
> benefits nearly nobody but costs nearly everyone.
>
>
>
> --
> Steven
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Q2UOEO3VOLG762WJ5N7LMJGX6ULUE6JR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V5T3GYC5UB4HICCSNHXXUKIZ4KV4YWQC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Dropping coroutines from PEP 611.

[Kyle Stanley]

With the removal of the coroutine limit, I think that PEP 611 will be a lot
easier to consider. Personally, I've gone from -1 to +0. But, with the
class limit also being rather controversial, I think the PEP would benefit
significantly from backing that up with some form of analysis on memory
usage in existing code.

For example, in addition to the existing section "Total number of classes
in a running interpreter", where it mentions a potential reduction in
overhead on a Python object from 40 bytes to 16 bytes: I think this
argument would be much stronger with an example of production code. You
could perform a rough estimate on how much memory is currently being used
on objects in the code example, and mention how much could be potentially
saved by imposing the 1M class limit.

In general though, I think adding some practical examples to the PEP and
explaining how they could benefit from the limits would make it much more
convincing. At the present moment, it might be a bit too abstract.

On Wed, Dec 11, 2019 at 5:19 AM Mark Shannon  wrote:

> Hi all,
>
> Thanks again for all your feedback on PEP 611.
> So far the biggest pushback on limits has been for the number of
> classes, line count and coroutines.
>
> Of those, I have been convinced that a limit on the number of coroutines
> would the hardest to workaround. In addition the justification for
> limiting coroutines is probably the weakest.
>
> Consequently, I'm dropping the limit on the number of coroutines from
> the PEP.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6ESJKTW4CP42KWJVSEEYJABGU73T4UPA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AK7XRUXJMFKGR6MKYHQM4HQGI57BZFMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

Andrew Svetlov wrote:
> > Not sure how that works?  Each client has an accepted socket, which is
> > bound to a local port number, and there are 65536 TCP port numbers
> > available.  Unless you're using 15+ coroutines per client, you probably
> > won't reach 1M coroutines that way.
>
> I'm sorry, but the accepted socket has the same local port number as
> the listening one.
> Routing is performed by (local_ip, local_port, remote_ip, remote_port)
quad.

IIRC, that combination is typically referred to as the "port address" (IP
addr + port num source, IP addr + port num destination). All four are are
required in TCP, and in UDP, the IP and source port are optional. So in
UDP, this could potentially just be (remote_ip, remote_port).

Also, it's possible to bind multiple AF_INET (or AF_INET6) sockets to a
single port address by using the SO_REUSEPORT socket option, which we
discussed recently in bpo-37228 (https://bugs.python.org/issue37228). The
only requirement is that the same UID is used for each socket bound to the
same port address (from my understanding, SO_REUSEPORT is typically used
for binding a single process to multiple listening sockets).

TL;DR: It's definitely possible to have more than one client per TCP port.

I'll admit that I've never personally seen production code that uses
anywhere near 1M coroutine objects, but we shouldn't limit users from doing
that without a good reason. At the present moment, it's rather trivial to
create 1M coroutine objects on any system with ~1.3GB+ available main
memory (see my code example in
https://mail.python.org/archives/list/python-dev@python.org/message/WYZHKRGNOT252O3BUTFNDVCIYI6WSBXZ/
).

There's also the infamous "10M" problem, of accepting 10 million concurrent
clients without significant performance issues. This is mostly theoretical
at the moment, but there's an article that explains how the problem could
be addressed by using 10M goroutines: https://goroutines.com/10m. I see no
reason why this couldn't be potentially translated into Python's
coroutines, with the usage of an optimized event loop policy such as uvloop.

But, either way, Mark Shannon removed the 1M coroutine limit from PEP 611,
due to it having the least strong argument out of all of the proposed
limits and a significant amount of push-back from the dev community.

Andrew Svetlov wrote:
> The listening socket can accept hundreds of thousands of concurrent
> client connections.
> The only thing that should be tuned for this is increasing the limit
> of file descriptors.

The default soft limit of file descriptors per process on Linux is 1024
(which can be increased), but you could exceed a per-process limitation of
file descriptors by forking child processes. I'm have no idea what the
realistic maximum limit of global FDs would be for most modern servers
though, but here's the upper bound limit on Linux kernel 5.3.13:

[aeros:~]$ cat /proc/sys/fs/file-max
9223372036854775807

My system's current hard limit of file descriptors is much lower, but is
still fairly substantial:

[aeros:~]$ ulimit -nH
524288

I recall reading somewhere that per additional 100 file descriptors, it
requires approximately 1MB of main memory. Based on that estimate, 1M FDs
would require ~10GB+. This doesn't seem unreasonable to me, especially on a
modern server. But I'd imagine the actual memory usage depends upon how
much data is being buffered at once through the pipes associated with each
FD, but I believe this can be limited through the FD option F_SETPIPE_SZ (
https://linux.die.net/man/2/fcntl).

Note: I was unable to find a credible source on the minimum memory usage
per additional FD, so clarification on that would be appreciated.

On Wed, Dec 11, 2019 at 5:06 PM Andrew Svetlov 
wrote:

> On Wed, Dec 11, 2019 at 11:52 PM Antoine Pitrou 
> wrote:
> >
> > On Mon, 9 Dec 2019 21:42:36 -0500
> > Kyle Stanley  wrote:
> >
> > >
> > > There's also a practical use case for having a large number of
> coroutine
> > > objects, such as for asynchronously:
> > >
> > > 1) Handling a large number of concurrent clients on a continuously
> running
> > > web server that receives a significant amount of traffic.
> >
> > Not sure how that works?  Each client has an accepted socket, which is
> > bound to a local port number, and there are 65536 TCP port numbers
> > available.  Unless you're using 15+ coroutines per client, you probably
> > won't reach 1M coroutines that way.
> >
>
> I'm sorry, but the accepted socket has the same local port number as
> the listening one.
> Routing is performed by (local_ip, local_port, remote_ip, remote_port)
> quad.
>
> The listening socket can accept hundred

[Python-Dev] Re: PEP 611 -- why limit coroutines and classes?

[Kyle Stanley]

Antoine Pitrou wrote:
> 1M concurrent database transactions?  Does that sound reasonable at
> all?  Your database administrator probably won't like you.

I agree that 1M concurrent transactions would not be reasonable for the
vast majority of database configurations, I didn't mean to specifically
imply that 1M would be something reasonable today. That's why I said "a
large number of concurrent transactions" instead of specifically saying "1M
concurrent transactions". I honestly don't know if any databases are close
to capable of handling that many transactions at once, at least not at the
present time.

But, although we may think it's ridiculous today, who's to say that won't
be an occurrence in the future? Imagine a scenario where there was a
massive super-cluster of database servers, that performed real-time update
transactions every time a single item was checked in and out of some global
inventory system.

Currently, something like this would likely have to be implemented through
batching, where x number of updates have to be queued or x amount of time
has to pass before the next transaction is started. Alternatively, each
facility or region would have its own local database that synchronizes with
the global database every so often. But, there could be a significant
advantage in having a near-perfectly synchronized global inventory system,
which would only be possible if it was updated in real-time. IMO, the max
number of concurrent transactions that the a database system can handle at
once is a very clear application of Moore's Law.

My point being is that I don't want to arbitrarily restrict how many
coroutine objects can exist at once without having a strong reason for
doing so AND having a limit that's reasonable in the long term. 1M would
have the advantage of being easy to remember, but other than that I see no
reason why that should specifically be the limit for the max number of
coroutines. As Guido mentioned at the start of the thread, a coroutine
object is "just another Python object and has no operating resources
associated with it". Other than main memory usage, there's no other
external limit to the max number of coroutine objects that can exist at
once.

Note: Although coroutines were already dropped from PEP 611, I felt that
this response was still worthwhile to write. I suspect that the topic of
"coroutine object limits" is likely to come up again in the future.


On Wed, Dec 11, 2019 at 4:46 PM Antoine Pitrou  wrote:

> On Mon, 9 Dec 2019 21:42:36 -0500
> Kyle Stanley  wrote:
>
> > > (b) Why limit coroutines? It's just another Python object and has no
> > operating resources associated with it. Perhaps your definition of
> > coroutine is different, and you are thinking of OS threads?
> >
> > This was my primary concern with the proposed PEP. At the moment, it's
> > rather trivial to create one million coroutines, and the total memory
> taken
> > up by each individual coroutine object is very minimal compared to each
> OS
> > thread.
> >
> > There's also a practical use case for having a large number of coroutine
> > objects, such as for asynchronously:
> >
> > 1) Handling a large number of concurrent clients on a continuously
> running
> > web server that receives a significant amount of traffic.
>
> Not sure how that works?  Each client has an accepted socket, which is
> bound to a local port number, and there are 65536 TCP port numbers
> available.  Unless you're using 15+ coroutines per client, you probably
> won't reach 1M coroutines that way.
>
> > 2) Sending a large number of concurrent database transactions to run on a
> > cluster of database servers.
>
> 1M concurrent database transactions?  Does that sound reasonable at
> all?  Your database administrator probably won't like you.
>
> > something like this definitely scales over time. Arbitrarily placing a
> > limit on the total number of coroutine objects doesn't make sense to me
> for
> > that reason.
>
> There are a lot of arbitrary limits inside a computer system.  You just
> aren't aware of them because you don't hit them in practice.  Claiming
> that limits shouldn't exist is just pointless.
>
> Regards
>
> Antoine.
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/O3ZODXHEIJ2SM5SZBOVJ4PIAQMSYNXEJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PYUSGK4UDMGGDJPQ6E6YHKG2O5D5DTEL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Travis CI for backports not working.

[Kyle Stanley]

Victor Stinner wrote:
> What is the issue? Can someone please open a bug report at
https://bugs.python.org/ so I can try to investigate?

>From my understanding, it looks to be pyenv related and not something we
can fix on our end, at least based on the build logs:
https://travis-ci.org/python/cpython/jobs/624160244?utm_medium=notification&utm_source=github_status.
This was from a recent backport PR to 3.7 (backport also failing for 3.8
with similar issues).

On Thu, Dec 12, 2019 at 8:14 PM Victor Stinner  wrote:

> What is the issue? Can someone please open a bug report at
> https://bugs.python.org/ so I can try to investigate?
>
> Victor
>
> Le ven. 13 déc. 2019 à 02:05, Brett Cannon  a écrit :
> >
> > This is failing again, so I had to switch off Travis from being a
> requirement (again).
> >
> > I'm not not going to flip it back on until Travis has been stable for a
> month as I don't like being the blocker on stuff when I can help it. And if
> Travis isn't stable in a month then we might need to start talking about
> turning it off entirely as flaky CI is never useful.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YJRPHEDV6DRVMSXCORRDUDCEFVYP4QUI/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/REAVUXBHB7RSKVG4IZTPXYFDMJJF4TWB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/34HWMTFBGXFGAAN5LCHHGX3PMO4CGZBT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Travis CI for backports not working.

[Kyle Stanley]

> I.e. if someone breaks some test, is Travis-CI the only thing that keeps
the breakage from landing on master?

We still have Azure Pipelines as a mandatory check for PRs before they can
be merged to master, which includes a few additional platforms and hasn't
had any recent issues (AFAIK). I believe Steve Dower was also recently
experimenting with adding GitHub Actions as an additional CI service.

On Thu, Dec 12, 2019 at 10:16 PM Guido van Rossum  wrote:

> Does Travis-CI serve any purpose for us still? I.e. if someone breaks some
> test, is Travis-CI the only thing that keeps the breakage from landing on
> master?
>
> On Thu, Dec 12, 2019 at 5:46 PM Kyle Stanley  wrote:
>
>> Victor Stinner wrote:
>> > What is the issue? Can someone please open a bug report at
>> https://bugs.python.org/ so I can try to investigate?
>>
>> From my understanding, it looks to be pyenv related and not something we
>> can fix on our end, at least based on the build logs:
>> https://travis-ci.org/python/cpython/jobs/624160244?utm_medium=notification&utm_source=github_status.
>> This was from a recent backport PR to 3.7 (backport also failing for 3.8
>> with similar issues).
>>
>> On Thu, Dec 12, 2019 at 8:14 PM Victor Stinner 
>> wrote:
>>
>>> What is the issue? Can someone please open a bug report at
>>> https://bugs.python.org/ so I can try to investigate?
>>>
>>> Victor
>>>
>>> Le ven. 13 déc. 2019 à 02:05, Brett Cannon  a écrit :
>>> >
>>> > This is failing again, so I had to switch off Travis from being a
>>> requirement (again).
>>> >
>>> > I'm not not going to flip it back on until Travis has been stable for
>>> a month as I don't like being the blocker on stuff when I can help it. And
>>> if Travis isn't stable in a month then we might need to start talking about
>>> turning it off entirely as flaky CI is never useful.
>>> > ___
>>> > Python-Dev mailing list -- python-dev@python.org
>>> > To unsubscribe send an email to python-dev-le...@python.org
>>> > https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> > Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/YJRPHEDV6DRVMSXCORRDUDCEFVYP4QUI/
>>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>>
>>>
>>> --
>>> Night gathers, and now my watch begins. It shall not end until my death.
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/REAVUXBHB7RSKVG4IZTPXYFDMJJF4TWB/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/34HWMTFBGXFGAAN5LCHHGX3PMO4CGZBT/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TSM5LJZS2AJSFVIEW64PRJXEK7T5J3EM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Serhiy Storchaka wrote:
> Forbids calling str() without object if encoding or errors are
> specified. It is very unlikely that this can break a real code, so I
> propose to make it an error without a deprecation period.

+1, I suspect that nobody would intentionally pass an argument to the
encoding and/or errors parameter(s) without specifying an object. Returning
an empty string from this seems like it would cover up bugs rather than be
useful in any capacity.

Serhiy Storchaka wrote:
> 2. Make the first parameter of str(), bytes() and bytearray()
> positional-only.

+1, I don't think I've ever seen a single instance of code that passes the
first parameter, *object*, as a kwarg: str(object=obj). As long as the
other two parameters, *encoding* and *error*, remain keyword arguments, I
think this would make sense.

Serhiy Storchaka wrote:
> 3. Make encoding required if errors is specified in str(). This will
> reduce the number of possible combinations, makes str() more similar to
> bytes() and bytearray() and simplify the mental model: if encoding is
> specified, then we decode, and the first argument must be a bytes-like
> object, otherwise we convert an object to a string using __str__.

Hmm, I think this one might require some further consideration. But I will
say that the implicit behavior is not very obvious.

Isn't overly clear, implicit 'utf-8' conversion:
>>> str(b'\xc3\xa1', errors='strict')
'á'

Makes sense, and is highly explicit:
>>> str(b'\xc3\xa1', encoding='utf-8', errors='strict')
'á'

This is also fine ('strict' is a very reasonable default for *errors*)
>>> str(b'\xc3\xa1', encoding='utf-8')
'á'


On a related note though, I'm not a fan of this behavior:
>>> str(b'\xc3\xa1')
"b'\\xc3\\xa1'"

Passing a bytes object to str() without specifying an encoding seems like a
mistake, I honestly don't see how this ("b'\\xc3\\xa1'") would even be
useful in any capacity. I would expect this to instead raise a TypeError,
similar to passing a string to bytes() without specifying an encoding:
>>> bytes('á')
...
TypeError: string argument without an encoding

I'd much prefer to see something like this:
>>> str(b'\xc3\xa1')
...
TypeError: bytes argument without an encoding

Is there some use case for returning "b'\\xc3\\xa1'" from this operation
that I'm not seeing? To me, it seems equally, if not more confusing and
pointless than returning an empty string from str(errors='strict') or some
other combination of *errors* and *encoding* kwargs without passing an
object.

On Sun, Dec 15, 2019 at 9:10 AM Serhiy Storchaka 
wrote:

> Currently str() takes up to 3 arguments. All are optional and
> positional-or-keyword. All combinations are valid:
>
> str()
> str(object=object)
> str(object=buffer, encoding=encoding)
> str(object=buffer, errors=errors)
> str(object=buffer, encoding=encoding, errors=errors)
> str(encoding=encoding)
> str(errors=errors)
> str(encoding=encoding, errors=errors)
>
> The last three are especially surprising. If you do not specify an
> object, str() ignores values of encoding and errors and returns an empty
> string.
>
> bytes() and bytearray() are more limited. Valid combinations are:
>
> bytes()
> bytes(source=object)
> bytes(source=string, encoding=encoding)
> bytes(source=string, encoding=encoding, errors=errors)
>
> I propose several changes:
>
> 1. Forbids calling str() without object if encoding or errors are
> specified. It is very unlikely that this can break a real code, so I
> propose to make it an error without a deprecation period.
>
> 2. Make the first parameter of str(), bytes() and bytearray()
> positional-only. Originally this feature was an implementation artifact:
> before 3.6 parameters of a C implemented function should be either all
> positional-only (if used PyArg_ParseTuple), or all keyword (if used
> PyArg_ParseTupleAndKeywords). So str(), bytes() and bytearray() accepted
> the first parameter by keyword. We already made similar changes for
> int(), float(), etc: int(x=42) no longer works.
>
> Unlikely str(object=object) is used in a real code, so we can skip a
> deprecation period for this change too.
>
> 3. Make encoding required if errors is specified in str(). This will
> reduce the number of possible combinations, makes str() more similar to
> bytes() and bytearray() and simplify the mental model: if encoding is
> specified, then we decode, and the first argument must be a bytes-like
> object, otherwise we convert an object to a string using __str__.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YMIGWRUERUG66CKRJXDXNPCIDHRQJY6V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubs

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Chris Angelico wrote:
> ANY object can be passed to str() in order to get some sort of valid
> printable form. The awkwardness comes from the fact that str()
> performs double duty - it's both "give me a printable form of this
> object" and "decode these bytes into text".

While it does make sense for str() to be able to give some form of
printable form for any object, I suppose that I just don't consider something
like this:  "b'\\xc3\\xa1'" to be overly useful, at least for any practical
purposes. Can anyone think of a situation where you would want a string
representation of a bytes object instead of decoding it?

If not, I think it would be more useful for it to either:

1) Raise a TypeError, assume that the user wanted to decode the string but
forgot to specify an encoding
2) Implicitly decode the bytes object as UTF-8, assume the user meant
str(bytes_obj, encoding='utf-8')

Personally, I'm more in favor of (1) since it's much more explicit and
obvious, but I think (2) would at least be more useful than the current
behavior.

On Sun, Dec 15, 2019 at 8:13 PM Chris Angelico  wrote:

> On Mon, Dec 16, 2019 at 12:00 PM Kyle Stanley  wrote:
> > On a related note though, I'm not a fan of this behavior:
> > >>> str(b'\xc3\xa1')
> > "b'\\xc3\\xa1'"
> >
> > Passing a bytes object to str() without specifying an encoding seems
> like a mistake, I honestly don't see how this ("b'\\xc3\\xa1'") would even
> be useful in any capacity. I would expect this to instead raise a
> TypeError, similar to passing a string to bytes() without specifying an
> encoding:
> > >>> bytes('á')
> > ...
> > TypeError: string argument without an encoding
> >
> > I'd much prefer to see something like this:
> > >>> str(b'\xc3\xa1')
> > ...
> > TypeError: bytes argument without an encoding
> >
> > Is there some use case for returning "b'\\xc3\\xa1'" from this operation
> that I'm not seeing? To me, it seems equally, if not more confusing and
> pointless than returning an empty string from str(errors='strict') or some
> other combination of *errors* and *encoding* kwargs without passing an
> object.
> >
>
> ANY object can be passed to str() in order to get some sort of valid
> printable form. The awkwardness comes from the fact that str()
> performs double duty - it's both "give me a printable form of this
> object" and "decode these bytes into text". With an actual bytes
> object, I always prefer b.decode(...) to str(b, encoding=...). But the
> one-arg form of str() needs to be able to represent a bytes object in
> some way, just as it can represent an int, a Fraction, or a list.
>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZP7SXIDQOQVKUF66NVZPS3O4FN3A6DWA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBP2ZHCGV44IUP4LQEXO2UFEVJX6QNGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Eric V. Smith wrote:
> Debugging. I sometimes do things like: print('\n'.join(str(thing) for
thing in lst)), or various variations on this. This is especially useful >
when maybe something in the list is a bytes object where I was expecting a
string.
>
> I'm not saying it's the best practice, but calling str() on an object is
a currently a guaranteed way of making a string out of it, and I
> don't think we can change it.

I could see that being useful actually. Regardless of "best practices",
it's reasonably common to indiscriminately convert a large sequence of
objects into strings for basic inspection purposes. There may be better
means of debugging, but I wouldn't want to prevent that option entirely for
bytes objects.

But, I suspect that backwards compatibility might be too much of a concern
here for the change to be worthwhile either way. Adding the TypeError or
even gradual deprecation would more than likely lead to a decent amount of
code breakage and maintenance; and changing it to implicitly perform a
UTF-8 encoding would very likely cause some confusion and debugging
difficulties for those who frequently inspect via string conversion.

Thanks for the insight.

On Mon, Dec 16, 2019 at 3:43 AM Eric V. Smith  wrote:

> On 12/16/2019 3:05 AM, Kyle Stanley wrote:
>
> Chris Angelico wrote:
> > ANY object can be passed to str() in order to get some sort of valid
> > printable form. The awkwardness comes from the fact that str()
> > performs double duty - it's both "give me a printable form of this
> > object" and "decode these bytes into text".
>
> While it does make sense for str() to be able to give some form of
> printable form for any object, I suppose that I just don't consider something
> like this:  "b'\\xc3\\xa1'" to be overly useful, at least for any practical
> purposes. Can anyone think of a situation where you would want a string
> representation of a bytes object instead of decoding it?
>
> Debugging. I sometimes do things like: print('\n'.join(str(thing) for
> thing in lst)), or various variations on this. This is especially useful
> when maybe something in the list is a bytes object where I was expecting a
> string.
>
> I'm not saying it's the best practice, but calling str() on an object is a
> currently a guaranteed way of making a string out of it, and I don't think
> we can change it.
>
> Eric
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5B46FTPOWDWNWDD7UL2HCDGSVPCSUUR3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IRF6PY33LQAQXGBGBEHVWZFOAUQV7J6D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Serhiy Storchaka wrote:
> It is not more confusing that returning "". By
> default str() returns the same as repr(), unless we made the object
> having other string representation.

Yeah, I suppose not. But that does raise of question of why bytes objects
were made to have a specific form of string representation in the first
place, instead of the generic object address repr. I suspect that it might
be for historical or arbitrary reasons.

But, that's likely an entirely different topic. I'll leave it at that so I
don't derail the main topic.

Serhiy Storchaka wrote:
> You can get an error here if you run Python with -bb. This is a
> temporary option to catch common errors of porting from Python 2.

Huh, interesting.



On Mon, Dec 16, 2019 at 3:59 AM Serhiy Storchaka 
wrote:

> 16.12.19 02:55, Kyle Stanley пише:
> > I'd much prefer to see something like this:
> >  >>> str(b'\xc3\xa1')
> > ...
> > TypeError: bytes argument without an encoding
> >
> > Is there some use case for returning "b'\\xc3\\xa1'" from this operation
> > that I'm not seeing? To me, it seems equally, if not more confusing and
> > pointless than returning an empty string from str(errors='strict') or
> > some other combination of *errors* and *encoding* kwargs without passing
> > an object.
>
> It is not more confusing that returning "". By
> default str() returns the same as repr(), unless we made the object
> having other string representation.
>
> You can get an error here if you run Python with -bb. This is a
> temporary option to catch common errors of porting from Python 2.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RRG4Q7BQWLIYNYNKJGE4BASFWTQ3P7PK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y4EPTAME4QRLLBGQBSH6YYADBYGPMLMV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Inada Naoki wrote:
> If we find it broke some software, we can step back to regular
> deprecation workflow.
> Python 3.9 is still far from beta yet.  That's why I'm +1 on these
proposals.

IMO, since this would be changing a builtin function, we should at least
use a version+2 deprecation cycle (in this case, removal in 3.11)
regardless of reported breakages.

Especially if there's no _substantial_ security, efficiency, or performance
reason for immediate prevention of str() without passing an object (while
specifying *encoding* and/or *error)  or making *object* a positional only
argument.

On Mon, Dec 16, 2019 at 4:31 AM Inada Naoki  wrote:

> On Mon, Dec 16, 2019 at 6:25 PM Inada Naoki 
> wrote:
> >
> > +1 for 1 and 2.
> >
>
> If we find it broke some software, we can step back to regular
> deprecation workflow.
> Python 3.9 is still far from beta yet.  That's why I'm +1 on these
> proposals.
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HWNLBBHSVB5NRQC6ESQQNCQQ2EYUMW27/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YOOL6KM6JQWPSJ5O65IXWERIIDVPD3RU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Parameters of str(), bytes() and bytearray()

[Kyle Stanley]

Kyle Stanley wrote:
> or making *object* a positional only argument.

Typo: I meant "positional only parameter", not "argument".

On Mon, Dec 16, 2019 at 4:39 AM Kyle Stanley  wrote:

>
> Inada Naoki wrote:
> > If we find it broke some software, we can step back to regular
> > deprecation workflow.
> > Python 3.9 is still far from beta yet.  That's why I'm +1 on these
> proposals.
>
> IMO, since this would be changing a builtin function, we should at least
> use a version+2 deprecation cycle (in this case, removal in 3.11)
> regardless of reported breakages.
>
> Especially if there's no _substantial_ security, efficiency, or
> performance reason for immediate prevention of str() without passing an
> object (while specifying *encoding* and/or *error)  or making *object* a
> positional only argument.
>
> On Mon, Dec 16, 2019 at 4:31 AM Inada Naoki 
> wrote:
>
>> On Mon, Dec 16, 2019 at 6:25 PM Inada Naoki 
>> wrote:
>> >
>> > +1 for 1 and 2.
>> >
>>
>> If we find it broke some software, we can step back to regular
>> deprecation workflow.
>> Python 3.9 is still far from beta yet.  That's why I'm +1 on these
>> proposals.
>>
>> --
>> Inada Naoki  
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/HWNLBBHSVB5NRQC6ESQQNCQQ2EYUMW27/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KAQZGR37QU6BS6UVSW4H7F4MDOYFY5ZG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should set objects maintain insertion order too?

[Kyle Stanley]

Larry Hastings wrote:
> a hypothetical collections.OrderedSet would probably work just fine.

I'm also in agreement with adding a collections.OrderedSet. There certainly
seems to be a practical use case for a Set that preserves insertion order,
but with significant push back to modifying the existing Set implementation
(primarily due to performance concerns).

If later down the road an improvement to OrderedSet is made that gives it
equal or better average performance across the board compared to Set, we
can consider changing the default implementation *if* there's significant
demand for it.

Larry Hastings wrote:
> And he'd probably use it too, as that makes the code clearer / easier to
read.  If you read code using an OrderedSet, you know what the intent was
and what the semantics are.  If you see code using a dict but setting the
values to None, you think "that's crazy" and now you have a puzzle to
solve.  Let's hope those comments are accurate!

This is an excellent point. My first thought if someone were to be using a
Dict with all values set to None would be: why not use a Set here? As Larry
said, the need for preservation of insertion order would have to be
explicitly described in the code comments. Realistically speaking, code
comments are not something that can be consistently relied upon, especially
if it's part of some boilerplate format that gets replicated or if the
container is used in multiple locations. I'd much rather have a dedicated
data structure that clearly describes what it does based on the name alone,
IMO that's a million times better for readability purposes.

Also, this is mostly speculation since I haven't ran any benchmarks for an
OrderedSet implementation, but I strongly suspect that OrderedSet will end
up having better average performance for add, pop, and update than trying
to use a dictionary with None values (assuming it's implemented in C or at
least with a C accelerator).

Not to mention allowing the ability to update (for both addition and
removal) based on intersections and unions across ordered sets, which
currently isn't possible to do with dictionaries (at least not directly
with |=, &=, -=. and ^=). I'm not sure how much of a practical use case
this would have for ordered sets specifically, but it's a nice bonus.

On Sat, Dec 21, 2019 at 7:59 AM Larry Hastings  wrote:

>
> (mixing together messages in the thread, sorry threaded-view readers)
>
> On 12/19/19 3:15 PM, Tim Peters wrote:
>
> [Nick]
>
> I took Larry's request a slightly different way:
>
> [...]
>
> Only Larry can answer whether that would meet his perceived need.  My
> _guess_ is that he wouldn't know OrderedSet existed, and, even if he
> did, he'd use a dict with None values anyway (because it's less hassle
> and does everything he wanted).
>
>
> At last, something I can speak knowledgeably about: Larry's use case!
> Regarding Larry, I'd say
>
>1. his use case was small enough that almost anything maintaining
>order would have worked, even a list,
>2. he often *does have* a pretty good idea what goodies are salted
>away in the Python standard library, and
>3. a hypothetical collections.OrderedSet would probably work just
>fine.  And he'd probably use it too, as that makes the code clearer /
>easier to read.  If you read code using an OrderedSet, you know what the
>intent was and what the semantics are.  If you see code using a dict but
>setting the values to None, you think "that's crazy" and now you have a
>puzzle to solve.  Let's hope those comments are accurate!
>
>
> Also, speaking personally, at least once (maybe twice?) in this thread
> folks have asked "would YOU, Mr Larry, really want ordered sets if it meant
> sets were slightly slower?"
>
> The first thing I'd say is, I'm not sure why people should care about
> what's best for me.  That's sweet of you!  But you really shouldn't.
>
> The second thing is, my needs are modest, so the speed hit we're talking
> about would likely be statistically insignificant, for me.
>
> And the third thing is, I don't really put the set() API through much of a
> workout anyway.  I build sets, I add and remove items, I iterate over them,
> I do the occasional union, and only very rarely do I use anything else.
> Even then, when I write code where I reach for a fancy operation like
> intersection or symmetric_difference--what tends to happen is, I have a
> rethink and a refactor, and after I simplify my code I don't need the fancy
> operations anymore.  I can't remember the last time I used one of these
> fancy operations where the code really stuck around for a long time.
>
> So, personally?  Sure, I'd take that tradeoff.  As already established, I
> like that dict objects maintain their order, and I think it'd be swell if
> sets maintained their order too.  I suspect the performance hit wouldn't
> affect *me* in any meaningful way, and I could benefit from the
> order-maintaining semantics.  I bet it'd have other benefits

[Python-Dev] Re: Should set objects maintain insertion order too?

[Kyle Stanley]

> To begin with, please compare performance of dict-with-None-values to
that of a set, for operations that can be expressed by both (e.g. both have
update()).

Good idea. It would be useful to have a baseline comparison. I emboldened
the comparisons with a notable difference.

Tested on:
Version: Python 3.8.0
OS: Linux kernel 5.4.5
CPU: Intel i5-4460

Initialization (about the same):
>>> timeit.timeit("set(i for i in range(1000))", number=100_000)
4.878508095000143
>>> timeit.timeit("{i: None for i in range(1000)}", number=100_000)
4.9192055170024105

Membership (about the same):
>>> timeit.timeit("random.randint(0, 999) in s", setup="import random; s =
set(i for i in range(1000))", number=10_000_000)
7.992674231001729
>>> timeit.timeit("random.randint(0, 999) in d", setup="import random; d =
{i: None for i in range(1000)}", number=10_000_000)
8.032404395999038

*Add* (much faster for dicts):
>>> timeit.timeit("s = set(); s.add(0)", number=100_000_000)
13.330938750001224
>>> timeit.timeit("d = {}; d[0] = None", number=100_000_000)
5.788865385999088

*Update* (much faster for updating sets):
>>> timeit.timeit("s.update(other_s)", setup="s = set(); other_s = set(i
for i in range(1000))", number=1_000_000)
6.2428832090008655
>>> timeit.timeit("d.update(other_d)", setup="d = {}; other_d = {i: None
for i in range(1000)}", number=1_000_000)
13.031371458000649

*Create list from keys* (faster for sets):
>>> timeit.timeit("list(s)", setup="s = set(i for i in range(1000))",
number=10_00_000)
5.24364303600305
>>> timeit.timeit("list(d)", setup="d = {i: None for i in range(1000)}",
number=10_00_000)
6.303017043999716

Removal isn't easily comparable, since s.pop(), s.discard(), d.pop(), and
d.popitem() all behave quite differently. Also, I'm sure these benchmarks
could be improved significantly, particularly with the "Add" ones. This
should provide a decent general idea though.

On Mon, Dec 23, 2019 at 8:12 PM Guido van Rossum  wrote:

> To begin with, please compare performance of dict-with-None-values to that
> of a set, for operations that can be expressed by both (e.g. both have
> update()).
>
> On Mon, Dec 23, 2019 at 18:08 Kyle Stanley  wrote:
>
>> Larry Hastings wrote:
>> > a hypothetical collections.OrderedSet would probably work just fine.
>>
>> I'm also in agreement with adding a collections.OrderedSet. There
>> certainly seems to be a practical use case for a Set that preserves
>> insertion order, but with significant push back to modifying the existing
>> Set implementation (primarily due to performance concerns).
>>
>> If later down the road an improvement to OrderedSet is made that gives it
>> equal or better average performance across the board compared to Set, we
>> can consider changing the default implementation *if* there's
>> significant demand for it.
>>
>> Larry Hastings wrote:
>> > And he'd probably use it too, as that makes the code clearer / easier
>> to read.  If you read code using an OrderedSet, you know what the intent
>> was and what the semantics are.  If you see code using a dict but setting
>> the values to None, you think "that's crazy" and now you have a puzzle to
>> solve.  Let's hope those comments are accurate!
>>
>> This is an excellent point. My first thought if someone were to be using
>> a Dict with all values set to None would be: why not use a Set here? As
>> Larry said, the need for preservation of insertion order would have to be
>> explicitly described in the code comments. Realistically speaking, code
>> comments are not something that can be consistently relied upon, especially
>> if it's part of some boilerplate format that gets replicated or if the
>> container is used in multiple locations. I'd much rather have a dedicated
>> data structure that clearly describes what it does based on the name alone,
>> IMO that's a million times better for readability purposes.
>>
>> Also, this is mostly speculation since I haven't ran any benchmarks for
>> an OrderedSet implementation, but I strongly suspect that OrderedSet will
>> end up having better average performance for add, pop, and update than
>> trying to use a dictionary with None values (assuming it's implemented in C
>> or at least with a C accelerator).
>>
>> Not to mention allowing the ability to update (for both addition and
>> removal) based on intersections and unions across ordered sets, which
>> currently isn't possible to

[Python-Dev] Re: Should set objects maintain insertion order too?

[Kyle Stanley]

Chris Angelico wrote:
> I think this is an artifact of Python not having an empty set literal.
> [snip]
> When both use the constructor call or both use a literal, the
> difference is far smaller. I'd call this one a wash.

Ah, good catch. I hadn't considered that it would make a substantial
difference, but that makes sense. Here's an additional comparison between
"{}"  and "dict()" to confirm it:

>>> timeit.timeit("{}", number=100_000_000)
2.1038335599987477
>>> timeit.timeit("dict()", number=100_000_000)
10.22558353268

Some more in-depth comparisons might be required for addition of single
items to the containers. I might experiment further with this at some point
in the next week or so, likely with implementing proper tests that omit the
time to initialize the container.








On Mon, Dec 23, 2019 at 10:09 PM Chris Angelico  wrote:

> On Tue, Dec 24, 2019 at 1:57 PM Kyle Stanley  wrote:
> > Add (much faster for dicts):
> > >>> timeit.timeit("s = set(); s.add(0)", number=100_000_000)
> > 13.330938750001224
> > >>> timeit.timeit("d = {}; d[0] = None", number=100_000_000)
> > 5.788865385999088
>
> I think this is an artifact of Python not having an empty set literal.
>
> >>> timeit.timeit("s = set(); s.add(0)", number=100_000_000)
> 13.275540543720126
> >>> timeit.timeit("d = dict(); d[0] = None", number=100_000_000)
> 13.044076398015022
> >>> timeit.timeit("d = {}; d[0] = None", number=100_000_000)
> 6.088695731014013
> >>> timeit.timeit("s = {1}; s.add(0)", number=100_000_000)
> 9.260965215042233
> >>> timeit.timeit("d = {1:2}; d[0] = None", number=100_000_000)
> 8.75433829985559
>
> When both use the constructor call or both use a literal, the
> difference is far smaller. I'd call this one a wash.
>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ODZYHNI57MFZD3I7TGP3B3HJTRX36KGB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/COUO6PFHUADHYP5KSHMPHPIUCOAXS56L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should set objects maintain insertion order too?

[Kyle Stanley]

Tim Peters wrote:
> Sorry!  A previous attempt to reply got sent before I typed anything :-(

No worries, I only saw the link in the footer to the PSF CoC, and I was
mildly concerned for a moment. My first thought was "Oh no, what did I
do?". Thanks for clearing that up (:

> The collision resolution strategy for sets evolved to be fancier than
> for dicts, to reduce cache misses.  This is important for sets because
> the _only_ interesting thing about an element wrt a set is whether or
> not the set contains it.   Lookup speed is everything for sets.

Huh, that's quite interesting. For some reason, I had assumed in the back
of my head (without giving it much thought) that the average collision rate
would be the same for set items and dict keys. Thanks for the useful
information.

> If you use a contiguous range of "reasonable" integers for keys, the
> integer hash function is perfect:  there's never a collision.  So any
> such test misses all the work Raymond did to speed set lookups.
> String keys have sufficiently "random" hashes to reliably create
> collisions, though.  Cache misses can, of course, have massive effects
> on timing.

Ah, I forgot to consider how the hash function actually works for
continuous integers. A better comparison to demonstrate the collision
differences would likely use random strings.

Also, I believe that max "reasonable" integer range of no collision is
(-2305843009213693951, 2305843009213693951), as defined in Include/pyhash.h
(
https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Include/pyhash.h#L28
).

> In the former case you're primarily measuring the time to look up the
> "add" method of sets (that's more expensive than adding 0 to the set).
> A better comparison would, e.g., move `add = s.add` to a setup line,
> and use plain "add(0)" in the loop.

Good point, there's also what Chris Angelico pointed out as well: dicts
have a significant advantage in this case for having a literal for
initialization. From my understanding, this ends up having a minimal
performance impact in most realistic code (similar to method lookup time),
but it makes a very substantial difference in this case since
initialization of the containers takes up a significant portion of each
iteration.

I suspect my initialization comparison might also be flawed for similar
reasons, but I'm glad that at least 3/5 of my comparisons were mostly
reasonable. So far, the performance difference between set.update() and
dict.update() were the most notable.

I'll likely spend some more time experimenting in the next couple of weeks
or so, and report back if I find any interesting results. In the meantime,
anyone can certainly feel free to improve upon my existing comparisons.


On Mon, Dec 23, 2019 at 11:03 PM Tim Peters  wrote:

> Sorry!  A previous attempt to reply got sent before I typed anything :-(
>
> Very briefly:
>
> > >>> timeit.timeit("set(i for i in range(1000))", number=100_000)
>
> [and other examples using a range of integers]
>
> The collision resolution strategy for sets evolved to be fancier than
> for dicts, to reduce cache misses.  This is important for sets because
> the _only_ interesting thing about an element wrt a set is whether or
> not the set contains it.   Lookup speed is everything for sets.
>
> If you use a contiguous range of "reasonable" integers for keys, the
> integer hash function is perfect:  there's never a collision.  So any
> such test misses all the work Raymond did to speed set lookups.
> String keys have sufficiently "random" hashes to reliably create
> collisions, though.  Cache misses can, of course, have massive effects
> on timing.
>
> > Add (much faster for dicts):
> > >>> timeit.timeit("s = set(); s.add(0)", number=100_000_000)
> > 13.330938750001224
> > >>> timeit.timeit("d = {}; d[0] = None", number=100_000_000)
> > 5.788865385999088
>
> In the former case you're primarily measuring the time to look up the
> "add" method of sets (that's more expensive than adding 0 to the set).
> A better comparison would, e.g., move `add = s.add` to a setup line,
> and use plain "add(0)" in the loop.
>
> That's it!
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LDAC4526CZZWFMXORSOKPW2PM6HZE252/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should set objects maintain insertion order too?

[Kyle Stanley]

> The only `OrderedSet` use I have seen in the wild is
https://github.com/python-hyper/uritemplate/search?q=orderedset&unscoped_q=orderedset
.

A more generalized Python code search across GitHub of "orderedset" returns
~500k results: https://github.com/search?l=Python&q=orderedset&type=Code .

Of course, a general code search across GitHub is by no means a clear or
reliable measure of actual usage, considering the number of duplicate
results. But, it's still useful for rough estimations.

On Thu, Jan 2, 2020 at 7:08 PM Brett Cannon  wrote:

> The only `OrderedSet` use I have seen in the wild is
> https://github.com/python-hyper/uritemplate/search?q=orderedset&unscoped_q=orderedset
> .
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GYIOTCKMKAYK2T6AALEDLZQVWCWLMXQD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W2SY6WM5QT6TZUBRUY7Y7TFIOMFUKY7S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python Language Summit at PyCon 2020

[Kyle Stanley]

On the attendance application, there is currently an incorrect link to the
informational page for the Language Summit: "
https://us.pycon.org/2020/events/language-summit/";.

Either the link could be changed to "
https://us.pycon.org/2020/events/languagesummit/"; or "
https://us.pycon.org/2020/events/language-summit/"; could be configured to
redirect towards the correct page.

On Tue, Jan 28, 2020 at 4:17 PM Mariatta  wrote:

> (cross posting to python-committers and python-dev)
>
> I'm happy to announce that the signups for Python Language Summit at PyCon
> 2020 is now open.
>
> Full details at: https://us.pycon.org/2020/events/languagesummit/
>
> *TL;DR*
>
> When: Wednesday, April 15, 2020, 9am–4pm (Note, we’re starting 1 hour
> earlier than usual!)
> Where: David L. Lawrence Convention Center, Pittsburgh, PA, room TBD
>
> Sign up to attend: https://forms.gle/Fg7ayhYTaY75J1r7A (closes Feb 29th,
> 2020 AoE)
> Sign up to discuss a topic: https://forms.gle/g4BXezH1Vcn7tLds5 (closes
> Feb 29th, 2020 AoE)
>
> *Who can attend*
>
> We welcome Python core developers, active core contributors to Python and
> alternative Python implementations, and anyone else who has a topic to
> discuss with core developers.
>
> *Who can propose a discussion topic*
>
> If you have discussion items; seeking consensus; awaiting decision on a
> PEP; needing help with your core dev work; or have specific questions that
> need answers from core developers, please submit a proposal. According to
> last year’s feedback, our audience prefer more discussions and shorter
> talks.
>
> To get an idea of past language summits, you can read past years' coverage:
> 2019:
> http://pyfound.blogspot.com/2019/05/the-2019-python-language-summit.html
> 2018: https://lwn.net/Articles/754152/
> 2017: https://lwn.net/Articles/723251/
>
> This year's event will be covered by A. Jesse Jiryu Davis again, and will
> be posted on PSF's blog.
> 
>
> Some changes to note this year:
> 1) We plan to start 1 hour earlier (9AM)
> 2) The room will have U-shaped table layout
>
> Thanks!
>
>
> Mariatta & Łukasz
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PUFQESFTN5ZUX64MKOO76LWS7W2N33M7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WCZIBRCUEKPFGPRLDTS4V23BCRB4Z6A2/
Code of Conduct: http://python.org/psf/codeofconduct/