[Python-ideas] Re: Arrow functions polyfill

[Barry]

> On 23 Feb 2021, at 18:08, Stéfane Fermigier  wrote:
> 
> 
> Both '%' and .format() support both positional and named arguments.
> 
> There are probably a few use cases for .format() (vs. f-strings) but overall 
> I don't believe there is much reasons left to prefer %.

I18n you can translate a f string.

Barry

> 
> Note that the existence, and popularity, of tools like flynt and pyupgrade 
> (that convert % and .format() directives to f-strings automatically) supports 
> this affirmation.
> 
> I found the 'un-fstring' project on pypi that does the reverse, but it's use 
> case, as advertised in the README, is clear: "Sometimes, unfortunately, you 
> need to write code that is compatible with Python 3.5"...
> 
>   S.
> 
> 
>> On Tue, Feb 23, 2021 at 6:50 PM Chris Angelico  wrote:
>> On Wed, Feb 24, 2021 at 4:13 AM Richard Damon  
>> wrote:
>> > As far as between % or .format(), I think the documentation is fairly
>> > clear that the % method is 'old' and if not 'formally' deprecated, is no
>> > longer considered the 'obvious' way to do it (even if some people will
>> > still do it that way for the simplest cases).
>> 
>> Not really - both forms have their places. You use .format() when you
>> need to be able to reorder arguments, you use percent formatting when
>> you want a compact and simple notation. It's like how we have both
>> string methods and regular expressions - neither one deprecates the
>> other.
>> 
>> ChrisA
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/UEZIQ56UYPDWQE3UU5BP72ADRKM2ZSNB/
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> -- 
> Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - 
> http://linkedin.com/in/sfermigier
> Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/
> Chairman, National Council for Free & Open Source Software (CNLL) - 
> http://cnll.fr/
> Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & 
> http://pydata.fr/
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/TTOMZH2L27DU3LCC7PSKNFQL2NA7KEKH/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/G32LOYMNBMFQX5RNL4HYFT5IEGZGWQ3B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Arrow functions polyfill

[Barry]

> On 23 Feb 2021, at 21:42, Barry  wrote:
> 
> 
> 
> 
>>> On 23 Feb 2021, at 18:08, Stéfane Fermigier  wrote:
>>> 
>> 
>> Both '%' and .format() support both positional and named arguments.
>> 
>> There are probably a few use cases for .format() (vs. f-strings) but overall 
>> I don't believe there is much reasons left to prefer %.
> 
> I18n you can translate a f string.

Sigh... you *cannot* translate an f string.

Barry

> 
> Barry
> 
>> 
>> Note that the existence, and popularity, of tools like flynt and pyupgrade 
>> (that convert % and .format() directives to f-strings automatically) 
>> supports this affirmation.
>> 
>> I found the 'un-fstring' project on pypi that does the reverse, but it's use 
>> case, as advertised in the README, is clear: "Sometimes, unfortunately, you 
>> need to write code that is compatible with Python 3.5"...
>> 
>>   S.
>> 
>> 
>>> On Tue, Feb 23, 2021 at 6:50 PM Chris Angelico  wrote:
>>> On Wed, Feb 24, 2021 at 4:13 AM Richard Damon  
>>> wrote:
>>> > As far as between % or .format(), I think the documentation is fairly
>>> > clear that the % method is 'old' and if not 'formally' deprecated, is no
>>> > longer considered the 'obvious' way to do it (even if some people will
>>> > still do it that way for the simplest cases).
>>> 
>>> Not really - both forms have their places. You use .format() when you
>>> need to be able to reorder arguments, you use percent formatting when
>>> you want a compact and simple notation. It's like how we have both
>>> string methods and regular expressions - neither one deprecates the
>>> other.
>>> 
>>> ChrisA
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at 
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/UEZIQ56UYPDWQE3UU5BP72ADRKM2ZSNB/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> 
>> -- 
>> Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - 
>> http://linkedin.com/in/sfermigier
>> Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/
>> Chairman, National Council for Free & Open Source Software (CNLL) - 
>> http://cnll.fr/
>> Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & 
>> http://pydata.fr/
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/TTOMZH2L27DU3LCC7PSKNFQL2NA7KEKH/
>> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZIFEA4I6AMKPOQBICFSGYB3JS2JD3ISN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: multiprocessing: hybrid CPUs

[Barry]

> On 18 Aug 2021, at 16:03, Chris Angelico  wrote:
> 
> On Thu, Aug 19, 2021 at 12:52 AM Marc-Andre Lemburg  wrote:
>> 
>>> On 18.08.2021 15:58, Chris Angelico wrote:
>>> On Wed, Aug 18, 2021 at 10:37 PM Joao S. O. Bueno  
>>> wrote:
>>>> 
>>>> So,
>>>> It is out of scope of Pythonmultiprocessing, and, as I perceive it, from
>>>> the stdlib as a whole to be able to allocate specific cores for each 
>>>> subprocess -
>>>> that is automatically done by the O.S. (and of course, the O.S. having an 
>>>> interface
>>>> for it, one can write a specific Python library which would allow this 
>>>> granularity,
>>>> and it could even check core capabilities).
>>> 
>>> Python does have a way to set processor affinity, so it's entirely
>>> possible that this would be possible. Might need external tools
>>> though.
>> 
>> There's os.sched_setaffinity(pid, mask) you could use from within
>> a Python task scheduler, if this is managing child processes (you need
>> the right permissions to set the affinity).
> 
> Right; I meant that it might require external tools to find out which
> processors you want to align with.
> 
>> Or you could use the taskset command available on Linux to fire
>> up a process on a specific CPU core. lscpu gives you more insight
>> into the installed set of available cores.
> 
> Yes, those sorts of external tools.
> 
> It MAY be possible to learn about processors by reading /proc/cpuinfo,
> but that'd still be OS-specific (no idea which Unix-like operating
> systems have that, and certainly Windows doesn't).

And next you find out that you have to understand the NUMA details
of your system because the memory attached to the CPUs is not the same speed.

> 
> All in all, far easier to just divide the job into far more pieces
> than you have processors, and then run a pool.

As other already stated using a worker pool solves this problem for you.
All you have to do it break your big job into suitable small pieces.

Barry
> 
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/UQNSUSHUONT4AO6NJEPEUENQG2AINAMO/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/62AXMS62J2H7TBHANIXZTTS2RJPUZZ5Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Should bare logical comparisons raise a warning?

[Barry]

> On 25 Nov 2021, at 13:06, ehmatt...@gmail.com wrote:
> 
> Hi everyone,
> 
> I've been really impressed by the recent improvements to error messages in 
> Python 3.10. The specific suggestions in some error messages are going to 
> save countless hours of people's time. The benefits for beginners are fairly 
> obvious. But even for experienced programmers, the difference between seeing 
> a missing colon in an error message and just adding it where it should have 
> been vs spending 30 seconds or a minute figuring out what's missing at the 
> start of a loop adds up.
> 
> One common error that I haven't seen discussed is bare logical comparisons. 
> They're syntactically legal so they don't raise errors, but I have never seen 
> a real-world use case for one. Here's the simplest example I can come up with:
> 
> name = 'eric'
> print(name)
> 
> name == 'ever'

The __eq__ method is run which might have side effects.
Which may mean raising error may be a backwards compat issue,
Maybe it is a problem for linters to report as a problem.
I have not checked if they already do.

> print(name)
> 
> We want to assign a new value to an existing variable, but we accidentally 
> type a second equals sign. The output here makes the mistake fairly obvious, 
> but in longer programs it can take a while to spot this issue.
> 
> I know there are some editors and linters that flag this. Is there any reason 
> not to raise a warning or even an error at the language level? Is there any 
> reason to use a bare logical comparison in real-world code?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/4QVUWND6FFDWCD3DL3BR3763EQRAAI5D/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VTEHJONXDAV3SX2IOCIOM3Q43J3SRCQ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: easier writing to multiple streams

[Barry]

> On 25 Nov 2021, at 13:07, Eyal Gruss  wrote:
> 
> 
> hi
> 
> first post here :)
> i have a recurring use pattern where i want to save to file everything i 
> print to the screen. i have used the standard logging module and found it too 
> cumbersome for my simple use case, and i find myself changing all my 
> print()'s to custom myprint()'s. i am interested in suggesting several 
> additions to the standard libraries, as i feel this use case is common 
> enough, important enough, and currently painful enough to justify such 
> changes, and that all four of the following suggestions could be useful in a 
> complementary fashion.
> 
> 1. allow the print() "file" argument to take a list of streams (currently it 
> can take only a single stream). e.g:
> 
> with open('output.txt', 'w') as f:
>print('hello', file=[sys.stdout, f])

You can replace sys.stdout with your own object that prints to a list of 
streams.
I have used this in the past.

The implementation can be as simple as supplying an object with a write method 
that writes to each of the steams I turn.

Barry
> 
> this seems like the quickest and least resistant path for the user. need to 
> decide how multiple streams will work with the flush argument. 
> 
> 2. modify contextlib.redirect_stdout() to take an additional boolean argument 
> which we can call tee/replicate/duplicate/clone etc. when True it will 
> duplicate instead of just redirecting:
> 
> with open('output.txt', 'w') as f:
> with redirect_stdout(f, tee=True):  # will duplicate instead of 
> redirecting
> print('hello')
> 
> or we could add a new context manager contextlib.copy_stdout() 
> 
> 3. allow the contextlib.redirect_stdout() "new_target" argument to take a 
> list of streams, or allow specifying multiple arguments for multiple streams. 
> e.g:
> 
> with open('output.txt', 'w') as f:
> with redirect_stdout(sys.stdout, f):  # or redirect_stdout([sys.stdout, 
> f]) 
> print('hello')
> 
> this has the benefit over the tee argument of allowing more than one 
> additional stream, but you need to explicitly specify stdout. so would be 
> nice to have both modifications.
> 
> 4. add to the standard io library a new class which gives you the write 
> interface of a single stream, but is a wrapper that will write to multiple 
> streams:
> 
> with open('output.txt', 'w') as f:
>multi = io.OutputStreamCollection(sys.stdout, f)
>multi.write('hello\n') 
># or:  print('hello', file=multi)
> 
> this is similar to: 
> https://stackoverflow.com/questions/9130755/wrapper-to-write-to-multiple-streams,
>  and we need to decide about the rest of the stream methods.
> 
> eyal.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/7I55RDLFAV6A3K762TE3BW42WTTUE4ET/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/526YR6GSE3CNII27GQDVXVUGETNMF5BF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Official means to release the GIL in Python

[Barry]

> On 3 Dec 2021, at 15:23, TobiasHT  wrote:
> 
> The GIL has been a widely discussed topic in the Python community. It's has 
> it's advantages and disadvantages.
> I was suggesting that an official way be placed in the Python threading 
> module to release the GIL if one needs to perform some tasks that don't need 
> the GIL.
> 
> It could be something as simple as
> 
> thd = threading.Thread(target = encrypt, nogil = True)
> 
> I saw a nogil option in Cython, I just thought it would be great to have it 
> fully support in Python as we await the results from Sam Gross's paper.

You must hold the GIL to execute python code.
Only extensions written in C etc can release the GIL while they do
work that does not access any python API.

Cython generates C code I believe and can release the GIL because it
I’d not python code.

In other words this cannot be made to work.

Barry

> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/5PWD7MXW6EO3ZECUPNXUQHE6HBYZ4JPU/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZTHSAFW3E72TXVX6XOYGV5YMZS4IA7OD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Applications user/system directories functions

[Barry]

> On 15 Dec 2021, at 13:45, JGoutin via Python-ideas  
> wrote:
> 
> Hello,
> 
> The idea is to add 3 functions to get "config", "data" and "cache" 
> directories that are commonly used to store application files in user home / 
> system.
> 
> This look very straightforward to get theses directories path, but in 
> practices it depends on many factors like OS, environnement variables, user 
> or system dir.
> 
> For instance, with the "config" directory:
> * Linux, user: os.path.join(os.getenv("XDG_CONFIG_HOME", 
> os.path.expanduser("~/.config")), app_name)
> * Linux, system: os.path.join("/etc", app_name)
> * Windows, user: os.path.join(os.path.expandvars("%APPDATA%"), app_name)
> * Windows, system: os.path.join(os.path.expandvars("%CSIDL_COMMON_APPDATA%"), 
> app_name)
> 
> For linux, the full spec is here: 
> https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
> 
> I see many applications that just use "~/.app_name" to not have to handle 
> theses cases.
> 
> 
> The functions prototypes may look like and may be added to "shutil" (or 
> "os.path" ?):
> 
> def getcachedir(app_name: str=None, system: bool=False):
> 
> With
> * app_name: The application name
> * system: If the required directory is the systemd directory or user 
> direcotry.
> 
> 
> This may also be implemented as an external library, but I am not sure I 
> would like add add a dependency to my projects "just for this".
> 
> 
> I can implement this if people are interested with this feature.

I wrote this https://pypi.org/project/config-path/ to solve the problem for 
macOs, windows and unix.

There are subtle points to consider on each platform.

Barry



> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/MHEWO4U6SBDU7OU3JH4A62EWCANDM7I2/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LZP4ISNMJETVHCVMOTQR4YKBRY3RODEU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding sortedconatiners to Python or merge the ideas?

[Barry]

> On 3 Feb 2022, at 18:53, Abdur-Rahmaan Janhangeer  
> wrote:
> 
> Fine discussion but i wonder how it ended up over there.
> Hoping over. Thanks for pointers!

Please don’t top post.

The thinking on python dev seems to be that if a package is not needed by core 
python
then PyPI is the best place for it. I know you read python dev isn’t that what 
you
have sensed as well?

Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AKJQLJZ6KX5SBXSB52V6LMV4AWNOO45Q/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] File format for automatic and manual tests

[Barry]

> On 8 Aug 2018, at 12:08, Victor Porton  wrote:
> 
>> On 08/08/18 12:25, Barry Scott wrote:
>>> On Tuesday, 7 August 2018 22:57:51 BST Victor Porton wrote:
>>> This is an idea of a new PEP.
>>> 
>>> I propose to create a portable file format which will list command line
>>> options to run Python scripts with dual purpose:
>> At the moment I solve this problem with various solutions, depending on
>> requirements.
>> 
>> * use python unittest
>> * add a test target to a makefile.
>> * write a bash script to run the tests and diff output if required
>> * on windows do the same with CMD scripts
>> * use python to run python in a subprocess that run the tests.
>> 
>> Given all these ways to solve the problem what extra are you looking for?
> 
> As I notes, I want to keep it in sync with PyCharm debug targets.

Then work with pycharm to get what you need.

> 
> We need a standard to make PyCharm and others to conform to it.

No you need to convince pycharm it is worth implementing.

Barry


> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Off topic: 'strike a balance' - second language English

[Barry]

> On 20 Aug 2018, at 07:35, Jacco van Dorp  wrote:
> 
> I would consider conciseness and accuracy most important. Using jargon but 
> linking to accurate explanations would, in my not exactly humble opinion, be 
> the best way to go about it.

+1 my thoughts exactly.


> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Possible PEP regarding the use of the continue keyword in try/except blocks

[Barry]

> On 7 Jan 2019, at 03:06, Richard Damon  wrote:
> 
> For something like reading options from a config file, I would use a
> call that specifies the key and a value to use if the key isn't present,
> and inside that function I might use a try to handle any exception
> caused when processing the key, and it could return the default.

Most config file APIs I have used have has_section and has_key type functions 
that remove the need to catch exceptions.

What config file API are you using htat is missing this?

Barry


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] tkinter: time for round buttons?

[Barry]

> On 16 Jan 2019, at 20:32, Christopher Barker  wrote:
> 
>> On Wed, Jan 16, 2019 at 2:07 PM Barry Scott  wrote:
> 
>> aesthetic level go up a bit more
>>> 
> tkinter is a wrapper around TK -- so it's TK that would need to "modernize" 
>>> poor me, if only py had some really nice native gu
>>  pip install PyQy5
> 
> or pip install wxpython

I used to use wxPython, but it bugs and platform specific quirky drove me to 
port to PyQt and I have not regreted that decision.

> 
> native support to Windows, OS-X, *nix/GTK
> 
> The install story is SO much better now that the "built in" advantage of 
> tkinter is no longer such a big deal.
> 
> -Chris
> 
> -- 
> Christopher Barker, PhD
> 
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Implement POSIX ln via shutil.link and shutil.symlink

[Barry]

> On 29 May 2019, at 16:48, Steven D'Aprano  wrote:
> 
>> On Wed, May 29, 2019 at 10:07:38PM +0700, Tom Hale wrote:
>> They say silence is golden... But I'm still looking for some feedback on 
>> the below.
> 
> I think the first thing you need to do is respond to Serhiy's objection:
> 
>Sorry, but I do not understand what problem do you try to solve. If
>somebody can create a file named link_name between unlink and 
>symlink, he can also remove and create a file named link_name after 
>symlink.
> 
> If Serhiy is correct, that completely undermines the primary motive for 
> adding this functionality.

Serhiy, I think, is conflating two things.
1. How to write software robust aginst attack.
2. How to replace a symlink atomically.

The only reason 1 is a problem is that the application is not in control of its 
file space which I would suggest means you already lost.

I think the OP wants a soution to 2, which as I suggest can leverage the design 
work in the implementation of the GNU ln -sf command in python.

Barry



> 
> 
> -- 
> Steven
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Universal parsing library in the stdlib to alleviate security issues

[Barry]

> On 16 Jul 2019, at 04:47, Andrew Barnert via Python-ideas 
>  wrote:
> 
> How often do you need to parse a million URLs in your inner loop?

As it happens i work on code that would be impacted by such a slow down. It 
runs in production 24x7 and parses URLs on its critical path.

Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FFPNYUUZ4RKT47B24URSLTMFSMUGARIX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Universal parsing library in the stdlib to alleviate security issues

[Barry]

> On 21 Jul 2019, at 19:03, Steven D'Aprano  wrote:
> 
>> On Sun, Jul 21, 2019 at 08:48:49AM +0100, Barry Scott wrote:
>> 
>> I took at very quick look at bpo30500 and was struck by the comment 
>> that the code was working on a URL that had not been validated.
>> 
>> Validation of the URL would reject the URL before the parsing happens 
>> in this case. Was that the case?
> 
> Sorry, can you elaborate on that? How do you validate a URL without 
> attempting to parse it? You're surely not talking about looking it up in 
> a whitelist are you?

I was thinking about ensuring the the characters in the url are from the subset 
that is allowed. \n is not allowed for example. Yes agree you have a try to 
parse it.

Barry

> 
> 
> Thanks,
> 
> 
> 
> 
> -- 
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/BTACAKW4W57NZNO72IRLCQHFYDX4YSX3/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/P2PZ6L56YW6AFXMSAFG4OHRLKZBO6DQB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Using 'with' with extra brackets for nicer indentation

[Barry]

> On 18 Nov 2019, at 19:27, Brendan Barnwell  wrote:
> 
> On 2019-11-18 10:49, Random832 wrote:
>> I would say open() is arguably a wart. Had context managers existed
>> in the language all along, it should not be possible to do anything
>> that creates an open-ended future requirement to do something (i.e.
>> "require that the returned object gets closed at some stage") without
>> either using a with statement or writing code to manually call
>> __enter__ and __exit__. The likely most common case being a class
>> that holds an open file, which should really have its own context
>> manager that forwards to the file's.
> 
>I agree.  I think this discussion wouldn't be necessary if there were a 
> context manager that was like open() but opened the file in __enter__ and 
> closed it in __exit__.  I would go so far as to say I think it would be a 
> good idea to add such a context manager to Python and deprecate open().
> 
>When I teach Python, I don't really even explain open() in isolation to 
> students,  I just tell them to always use it with a with statement.  If I see 
> code that uses open() outside a with statement it's a red flag to me.  These 
> hypothetical cases where you might want to use open() outside a with 
> statement seem, to me, to be fairly esoteric.  The best policy is for every 
> file-opening operation to occur as part of a context manager,

Esoteric? I work with files that are open beyond the scope of a single block
of code all the time. Clearly I need to arrange to close the file eventually.
Why would I want the uglyness of __exit__ rather then close()?
I use with all the time if the open() is with linear code.

Barry




> 
> -- 
> Brendan Barnwell
> "Do not follow where the path may lead.  Go, instead, where there is no
> path, and leave a trail."
>   --author unknown
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/STCWSHQJWBDNZILIJPPW7HQJTGVMH6GJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IAVAIUBF73T5ON7N5JMU7DODGEJK4LP4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: frozen dataclasses attribute initialization

[Barry]

> On 13 Dec 2019, at 14:57, Arthur Pastel  wrote:
> 
> 
>> I think the simpler route to go there (allow the use of `self.attr=` until 
>> after ___post_init__ is run is simply
>> to add another flag attribute, that tells wether "initialization is over", 
>> and respect that flag
> 
> 
> We discussed this just before and having an extra instance attribute was 
> quite problematic.
> That's why I suggested to have an attribute set in the beginning of the 
> __init__ and deleted after the initialization is complete.
> 
> Anyways in both cases, there is still a problem when  the class uses 
> __slots__as @Eric V. Smith mentioned previously.

Hack warning:

Put the flag outside of the class? Say by putting the instance id() into a 
set().
Now it works for slots, just need to think about data races...

Barry


> 
> 
>> On Fri, Dec 13, 2019 at 2:50 PM Joao S. O. Bueno  
>> wrote:
>> I think the simpler route to go there (allow the use of `self.attr=` until 
>> after ___post_init__ is run is simply
>> to add another flag attribute, that tells wether "initialization is over", 
>> and respect that flag
>> in the added `__setattr__`. The hook calling `__post_init__` would then set 
>> that flag after it is done.
>> 
>> A 5 line change is enough to make this work for __post_init__:
>> 
>> ```
>> diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py 
>> index 91c1f6f80f..65d51ca0a9 100644 
>> --- a/Lib/dataclasses.py 
>> +++ b/Lib/dataclasses.py 
>> @@ -194,6 +194,11 @@ _PARAMS = '__dataclass_params__' 
>> # __init__. 
>> _POST_INIT_NAME = '__post_init__' 
>>  
>> + 
>> +# Flag used to allow field initialization in `__post_init__` 
>> +# and `__init__` for frozen classes 
>> +_FROZEN_INITIALIZATION_FLAG = '__dataclass_initialized__' 
>> + 
>> # String regex that string annotations for ClassVar or InitVar must match. 
>> # Allows "identifier.identifier[" or "identifier[". 
>> # https://bugs.python.org/issue33453 for details. 
>> @@ -517,6 +522,9 @@ def _init_fn(fields, frozen, has_post_init, self_name): 
>>   if f._field_type is _FIELD_INITVAR) 
>> body_lines.append(f'{self_name}.{_POST_INIT_NAME}({params_str})') 
>>  
>> +if frozen: 
>> +body_lines.append(f'{self_name}.{_FROZEN_INITIALIZATION_FLAG} = 
>> True') 
>> + 
>> # If no body lines, use 'pass'. 
>> if not body_lines: 
>> body_lines = ['pass'] 
>> @@ -552,13 +560,13 @@ def _frozen_get_del_attr(cls, fields): 
>> fields_str = '()' 
>> return (_create_fn('__setattr__', 
>>   ('self', 'name', 'value'), 
>> -  (f'if type(self) is cls or name in {fields_str}:', 
>> +  (f'if (type(self) is cls or name in {fields_str}) and 
>> getattr(self, "{_FROZEN_INITIALIZATION_F
>> LAG}", False):', 
>> ' raise FrozenInstanceError(f"cannot assign to field 
>> {name!r}")', 
>>f'super(cls, self).__setattr__(name, value)'), 
>>globals=globals), 
>> _create_fn('__delattr__', 
>>   ('self', 'name'), 
>> -  (f'if type(self) is cls or name in {fields_str}:', 
>> +  (f'if (type(self) is cls or name in {fields_str}) and 
>> getattr(self, "{_FROZEN_INITIALIZATION_F
>> LAG}", False):', 
>> ' raise FrozenInstanceError(f"cannot delete field 
>> {name!r}")', 
>>f'super(cls, self).__delattr__(name)'), 
>>globals=globals), 
>> (END)
>> ```
>> 
>> To get it working for `__init__` as well, however is a bit more complicated 
>> - as we
>> don't have control of the dataclass metaclass, the only way to put a hook to 
>> set the
>> flag after __init__ is run is to apply a decorator in the existing __init__ 
>> that would
>> do it. 
>> 
>> However, this might be a way to have this feature _IF_ people think it would
>> be worth it - (I personally would prefer yet another parameter to allow 
>> changing 
>> fields during frozen initialization, as that is, for me, an exceptional way 
>> of doing it
>> less simple than havi

[Python-ideas] Re: Enhancing Zipapp

[Barry]

> On 7 Jan 2020, at 02:40, Christopher Barker  wrote:
> 
> 
> I’m a bit unclear on how far this goes: is it just a bit more specific with 
> more meta-data standards?
> 
> Or are you aiming for something that will run without a Python install? 
> 
> Other issues:
> 
> Are you aiming for a bundle that can run on multiple platforms? If so, then 
> it’ll have to have a way to bundle multiple compiled extensions and select 
> the right ones at runtime.
> 
> If this Is essentially just zipapp with the ability to bundle dependencies, 
> then you could probably just do some sys.path hackery.
> 
> In any case, thus seems like something you could implement, and then see if 
> people find it useful.
> 
> BTW- I’m pretty sure we could simply specify that filenames are utf-8 and 
> we’d be good to go.

Have a look at this write up about the horror that is zip file name handling.

https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/

This has been a pain point at work.

Barry

> 
> -CHB
> 
> 
> 
> 
> 
>> On Mon, Jan 6, 2020 at 5:50 PM Abdur-Rahmaan Janhangeer 
>>  wrote:
>> 
>> 
>>> On Tue, 7 Jan 2020, 01:57 Barry Scott,  wrote:
>>> 
>>> 
>>> Please cover the pro's and con's of the alernatives that have been raised 
>>> as comments
>>> on this idea, as is usually done for a PEP style document.
>> 
>> 
>> Thanks, i don't have much experience writing peps and
>> if i don't bug you may i ask what "alternatives" refer to?
>> 
>>> Also beware that zip file format does not include the encoding of the files 
>>> that are in the
>>> zip file.
>> 
>> 
>> For the encoding of the contents, well since we are
>> packaging python code files, it's handling will be the same
>> as handling outside the zip file. It's handling is the
>> same as how zipapp handles things.
>> 
>>> This means that for practical purposes only ASCII filenames are portable 
>>> across
>>> systems. Is this limitation a problem for this proposal?
>> 
>> 
>> If we are talking about filenames, then i guess
>> ascii filenames are the way to go as you'd 
>> unnecessarily break things otherwise.
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/RVGFMDP3PG6TXFQGH7ISRLYM4FS5CO64/
>> Code of Conduct: http://python.org/psf/codeofconduct/
> -- 
> Christopher Barker, PhD
> 
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4ZU5L7HSYJNAXWQ437A5VJFV7NACQXIR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Enhancing Zipapp

[Barry]

> On 7 Jan 2020, at 01:48, Abdur-Rahmaan Janhangeer  
> wrote:
> 
> 
> 
> 
>> On Tue, 7 Jan 2020, 01:57 Barry Scott,  wrote:
>> 
>> 
>> Please cover the pro's and con's of the alernatives that have been raised as 
>> comments
>> on this idea, as is usually done for a PEP style document.
> 
> 
> Thanks, i don't have much experience writing peps and
> if i don't bug you may i ask what "alternatives" refer to?

You are offing up a competitor against python wheels, Py2app, py2exe etc 
packagers.
Explain the benefits and weaknesses compared to the existing alternatives.

You might want to look at pex that is mentioned in the pep you refer to.
The other mentioned app has seen no update sine 2013.

> 
>> Also beware that zip file format does not include the encoding of the files 
>> that are in the
>> zip file.
> 
> 
> For the encoding of the contents, well since we are
> packaging python code files, it's handling will be the same
> as handling outside the zip file. It's handling is the
> same as how zipapp handles things.

I replies seperaly about this problem.
> 
>> This means that for practical purposes only ASCII filenames are portable 
>> across
>> systems. Is this limitation a problem for this proposal?
> 
> 
> If we are talking about filenames, then i guess
> ascii filenames are the way to go as you'd 
> unnecessarily break things otherwise.

Barry___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/L6H6LGU2J32QA5I2Z6NCTGQGEZ2YQFCW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding a built-in data structure with binary search tree semantics

[Barry]

> On 16 Mar 2020, at 14:43, Ben Rudiak-Gould  wrote:
> 
> On Mon, Mar 16, 2020 at 12:57 AM Andrew Barnert  wrote:
>> Even if the extra indirection overhead turns out not to be an issue, just 
>> from the added complexity (to every possible implementation) it seems like 
>> it would be a bad idea to make that a requirement.
> 
> The only change needed to support O(1) copy is making modified copies
> of nodes instead of mutating them in place. The algorithms are
> otherwise the same.

So when I insert into the copy how do you avoid all the nodes being modified?

Barry

> 
> That said, it's possible that blist does something that's faster but
> more complicated. I haven't looked.
> 
>>> pyrsistent has O(1) copying and is maintained but it's terribly slow and 
>>> the interface is nonstandard (Haskellish instead of Pythonic).
>> 
>> It’s also neither sorted nor mutable, which means it’s pretty far from what 
>> you’re looking for here.
> 
> The distinction between mutability and immutability is an illusion.
> You can convert a blist-style data structure into a pyrsistent-style
> data structure with a wrapper that takes an O(1) copy of its internal
> blist object, mutates it, wraps it, and returns it. You can convert a
> pyrsistent-style data structure into a blist-style data structure with
> a wrapper that maintains an internal pyrsistent object and overwrites
> it with the modified copy after each operation. The complexity of the
> operations is unaffected and the underlying implementations can be
> identical.
> 
> The key to this equivalence is the O(1) copying of the mutable
> version. Pyrsistent could switch to a blist backend but not to a
> sortedcontainers backend unless sortedcontainers added that operation.
> 
> You're right that blist and pyrsistent don't have the same
> functionality as they stand. Pyrsistent's PMap and PSet are sorted but
> don't allow duplicates. The equivalent (modulo interface translation)
> of blist.sortedlist would be pyrsistent.PMultiSet.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/FYK4IJKS5ZGVMKFDFNYVQLDM4N6RLP4D/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QG3XO3DR5HLFUIG34T3LXOJCJMIYEE54/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: TLS session resumption

[Barry]

> On 15 Apr 2020, at 19:56, Ander Juaristi  wrote:
> 
> Hi list,
> 
> I haven't seen this being discussed, but TL;DR I just want to propose to 
> extend the ssl module so that TLS session state can be serialized to disk.

You might want to read this analysis of TLS session tickets in TLS 1.2.

https://blog.filippo.io/we-need-to-talk-about-session-tickets/

Tl;dr its not considered secure.

Barry

> 
> I see a similar issue was being discussed here [0], but cannot tell why it 
> (apparently) didn't get into. The link to the patch is broken.
> 
> TLS session resumption is currently supported, but only within the same 
> process. To the best of my knowledge, there is no way to save the TLS session 
> to a file to resume the TLS session later on. Please tell me how this is done 
> if I'm wrong.
> 
>#
># open a TLS connection
>#
>with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
>with context.wrap_socket(sock, server_hostname=hostname) \
>as ssock:
>ssock.connect((hostname, 443))
>ssl_session = ssock.session  # get TLS session
> 
>#
># later on, open a new connection resuming session
>#
>with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
>with context.wrap_socket(sock,
> server_hostname=hostname,
> session=ssl_session) as ssock:
>ssock.connect((hostname, 443))
>print("-- TLS session %s resumed --"
>  % ("WAS" if ssock.session_reused else "WAS NOT")
> 
> 
> However TLS sessions live for
> hours, if not days, and hence it may make sense to persist them
> to disk for later retrieval. I think this is a real world use case though I 
> have not any numbers to back this up.
> 
> The values currently exposed by SSLSession (id, time, timeout) are not enough 
> to resume a TLS session. There are a number of parameters that need to be 
> stored to resume a TLS session at a later time, some of which are not 
> currently retrievable from python's ssl module. For instance, the master 
> secret.
> 
> I'm not a fan of exposing the master secret directly with a getter as that 
> could be and will definitely be misused by someone in spite of the bright red 
> warnings, leading to disaster.
> 
> What OpenSSL currently does, as do other implementations, is to allow the 
> SSL_SESSION object to be serialized, with i2d_SSL_SESSION(). This function 
> will return a DER-encoded ASN.1 object containing all the needed state to 
> resume the TLS session. I do believe exposing this object from the ssl module 
> (as an opaque bytes object) would be a useful compromise with very little 
> chance of misuse.
> 
> One possible incarnation of this idea would be add two new methods to 
> SSLSession, to_der(), and from_der():
> 
>class SSLSession:
>def to_der():
>""" Returns OpenSSL's DER-encoded SSL_SESSION object """
> 
>@staticmethod
>def from_der(der_bytes):
>"""
>Creates a new SSLSession object from the given
>DER-encoded SSL_SESSION object
>"""
> 
> If this has already been discussed (likely), what are the reasons it was 
> rejected?
> 
> Thanks!
> 
> [0] https://bugs.python.org/issue19500
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/55RZJODLVNVP72FNE42GUZEO57RDE5W4/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XWDGAS52R3N2VL3QMBS6RFW4XEF7FZTD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding a "once" function to functools

[Barry]

> On 27 Apr 2020, at 16:01, Ethan Furman  wrote:
> 
> On 04/27/2020 05:09 AM, Steven D'Aprano wrote:
>> On Sun, Apr 26, 2020 at 07:48:10PM -0700, Ethan Furman wrote:
> 
>>> How many beginners know they want to call a function only once?
>> More than the number who know about LRU caches.
>> Ethan, are you objecting to a self-descriptive name because it is too
>> confusing for beginners and lru_cache isn't? Because that's the argument
>> you seem to be defending.
> 
> I'm objecting to using "beginners" as the basis for name choices for advanced 
> topics.
> 
> Aside from that, `once` is a fine name.  I'm sure it means that a function 
> can only be defined once, and subsequent definitions will either be ignored 
> or raise, right?
> 
>>>> @once
>... def my_unique_function(...):
>... do_something_cool()
>...
>>>> def something_else():
>... pass
>...
>>>> def my_unique_function(...):
>... # uh oh, duplicate!
>...
>RuntimeError - duplicate function name detected
> 

That needs a linter to spot ai think.

Or are you saying that once could be interested to mean I cannot do that 2nd 
def?

Barry
> --
> ~Ethan~
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/VD7ZSMAGG2BHDAFJ2M7GME2TIPFA25C7/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/46Y3L5EW5NDBTNFMPLCTTAXULMQUBMHY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: isolating user allocations

[Barry]

> On 20 Jul 2020, at 18:51, Wenjun Huang  wrote:
> 
> 
> Hi Guido,
> 
> It's great to hear from you directly :)
> 
> Sorry for not mentioning this earlier. The use case here is profiling. 
> Specifically, Scalene: https://github.com/emeryberger/scalene.
> 
> At the moment, Scalene does stack inspection to decide if an allocation is 
> from the user. If there are hooks helping with the differentiation, it's 
> arguably more efficient. Additionally, users might want to know about memory 
> allocated explicitly by their objects instead of by the interpreter internals.

In my day job and open source projects I care about memory leaks and fixing 
them.

If my code creates a list you call that user allocated?
If memory increases as part of python book keeping that is counted as something 
else?

Why is that interesting?

When looking for memory leaks I have not cared about that distinction.
Should I be caring? Am I missing something?

Barry

> 
> If the interpreter doesn't have this functionality already, it would probably 
> be difficult to add, as the existing allocation API is called throughout the 
> interpreter code base.
> 
> Thanks
> 
>> On Sun, Jul 19, 2020 at 10:39 PM Guido van Rossum  wrote:
>> What purpose do you have in mind for making this distinction? Even if it 
>> could be done easily (which I doubt), why would this be useful?
>> 
>>> On Sun, Jul 19, 2020 at 19:01  wrote:
>>> Hello,
>>> 
>>> The API provided by PEP 445 makes it possible to intercept allocation 
>>> requests through hooks, but it seems that both user allocations and 
>>> interpreter allocations are sent to the hooks.
>>> 
>>> Here, user allocations refer to those that are triggered explicitly by the 
>>> code (e.g. memory allocations to hold the integer created by x = 1), and 
>>> interpreter allocations refer to everything else (e.g. memory allocations 
>>> for internal states).
>>> 
>>> I've poked around a bit in the interpreter source code, and I think such 
>>> differentiations aren't being done at all, so all allocations are directed 
>>> to the same set of API. If that's indeed the case, why is the interpreter 
>>> implemented this way? Would it make sense to implement the differentiation?
>>> 
>>> Thanks
>>> ___
>>> Python-ideas mailing list -- python-ideas@python.org
>>> To unsubscribe send an email to python-ideas-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>>> Message archived at 
>>> https://mail.python.org/archives/list/python-ideas@python.org/message/HJQW2MWFP5AUWTRD5N2TVQWAIKIDFNJW/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> -- 
>> --Guido (mobile)
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/H33JNM67O3OYAGOLCDHURPFT6IDAU4ZH/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SF272PQLJTFAN66XECUXU7Z7NHQER7TG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: New clause in FOR and WHILE instead of ELSE

[Barry]

> On 21 Jul 2020, at 18:47, Rob Cliffe via Python-ideas 
>  wrote:
> 
> 
> On Mon, Jul 20, 2020 at 03:22 Jonathan Fine  wrote:
>> This is a continuation of my previous post to this thread.
>> 
>> Python's FOR ... ELSE ... , Raymond Hettinger has told us, has origins in 
>> some ideas of Don Knuth. 
> 
> That’s news to me (both that it’s due to Knuth and that Raymond said so). I 
> invented it without awareness of prior art, by reasoning about the similarity 
> between IF and WHILE (FOR followed from WHILE).
> 
> —Guido
> 
> See Raymond's video "Transforming Code into Beautiful, Idiomatic Python" at
> https://www.youtube.com/watch?v=OSGv2VnC0go
> from 15 min 50 sec to 18 min 57 sec.
> 
> On 20/07/2020 15:42, Guido van Rossum wrote:
>> Also, let me be clear that this feature will never be added to the language.
>> 
> With respect, that seems pretty dogmatic, given that for...else is one of the 
> most confusing features of Python.
> What would be so terrible about allowing, at minimum, `if not break:' as a 
> synonym for 'else:'?

1. Because that not what else mean today. Its elif never looped.
2. Because if after for is confusing. I can get behind elif as after for it  
pull work.

Barry

> 
> Best wishes
> Rob Cliffe
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/4H5SMUSIHBOWJUIAS7WO2BHEHJ5WYQKA/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HAM3ESIGTLEN4P4A7RCYBQAICV73P7H3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Tracebacks for C code in Python

[Barry]

> On 15 Aug 2020, at 20:15, Ram Rachum  wrote:
> 
> 
> Here's something that's been bugging me for years. I'll suggest something, 
> but since I'm a total newbie about this area, it's possible that everything 
> I'm saying is impossible or doesn't make sense.
> 
> I'm working with some Pandas code now, and there's an exception because I'm 
> doing something wrong. I get a traceback, but some of the frames are in pyd 
> files (C code I guess?) so I don't see the code for them.
> 
> This is frustrating, because the exception message isn't that clear, so I 
> would at least like to know what the code was trying to do when it got the 
> exception. Maybe this will give me more hints about what's going wrong.
> 
> Would it be possible to have Python tracebacks include code for C code that's 
> called in Python?

On Linux gdb can show the python and c code stack at the same time.

Barry

> 
> I know very little about how the C-to-Python interaction works, and I assume 
> we'd need something complicated like packaging the source code with the 
> binaries in some way that lets Python get the right line of C code to put in 
> the traceback. This can get complicated.
> 
> Do you think it's possible?
> 
> 
> Thanks,
> Ram.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/2EAJ6IZ5QHW6GNPIQQBOOZ2MT4WLBCQT/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HUEXNTQJE5OXJ5KYZB25H6LVSS5AFLM4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: argmax and argmin to python list

[Barry]

> On 30 Aug 2020, at 12:25, Filipp Bakanov  wrote:
> 
> 
> It's expected that python single iterating would be slower than two times C 
> iterating. Nevertheless one time C iterating will be probably faster than a 
> separate min + index.
> 
> >> I think one would want argmin() and argmax() to work with general iterables
> How is it supposed to work with set or dict or other iterables without clear 
> order?
> 
> My last case was to find the min index of the array part. index_min could 
> accept the starting element index, as index does. 
> 
> a = [1, 4, 1, 2, 3]
> print(a.index_min())  # 0
> print(a.index_min(1))# 2

Of course this could be implemented in C in python, but you are going to have to
convince the python developers that it’s important enough to want to maintain 
the
code.

Is your use of such a function so performance sensitive that you need this in C?
Have you considered writing this as a C extension for your own use?

Barry

> 
> 
> 
> On Sun, 30 Aug 2020 at 11:52, Barry Scott  wrote:
>> 
>> 
>>>> On 30 Aug 2020, at 09:03, Jeff Allen  wrote:
>>>> 
>>>> On 29/08/2020 14:17, Barry Scott wrote:
>>>>>> On 29 Aug 2020, at 13:42, Filipp Bakanov  wrote:
>>>>>> 
>>>>>> I'd like to propose adding argmax and argmin functions to the python 
>>>>>> list. These functions return the index of a maximum / minimum element of 
>>>>>> the list. Eg:
>>>>>> 
>>>>>> a = [1, 4, 2, 3]
>>>>>> print(a.argmax())  # 1
>>>>>> print(a.argmin())  # 0
>>>>>> 
>>>>>> It's a very popular request (based on stackoverflow 
>>>>>> https://stackoverflow.com/questions/16945518/finding-the-index-of-the-value-which-is-the-min-or-max-in-python
>>>>>>  ), and currently there is no elegant way to find it.
>>>>>> 
>>>>>> What do you think?
>>>>> 
>>>>> Just do this:
>>>>> 
>>>>> >>> a=[1,4,2,3]
>>>>> >>> min(a)
>>>>> 1
>>>>> >>> a.index(min(a))
>>>>> 0
>>>>> >>> a.index(max(a))
>>>>> 1
>>>>> 
>>>>> Barry
>>>>> 
>>>> This has the drawback of passing twice over the list. The following 
>>>> doesn't, but the complexity somewhat makes Filipp's point:
>>>> 
>>>> >>> min((e, i) for i, e in enumerate(a))[1]
>>>> 0
>>>> 
>>> That is 4x slower then my code for 1,000,000 items.
>>> 
>>> --- a.py --
>>> import sys
>>> import time
>>> import random
>>> 
>>> alg = sys.argv[1]
>>> size = int(sys.argv[2])
>>> 
>>> x = [random.randint(0, 1_000_000) for _ in range(size)]
>>> 
>>> start = time.time()
>>> 
>>> if alg == 'barry':
>>> 
>>> m = x.index(min(x))
>>> 
>>> elif alg == 'jeff':
>>> 
>>> m = min((e, i) for i, e in enumerate(x))[1]
>>> 
>>> end = time.time()
>>> 
>>> print( alg, end-start, (end-start)/size )
>>> -
>>> 
>>> Here is the output I got on my laptop.
>>> 
>>> barry 0.022754907608032227 2.2754907608032226e-08
>>> barry 0.03325295448303223 3.325295448303223e-08
>>> barry 0.034243106842041016 3.4243106842041016e-08
>>> barry 0.02784109115600586 2.784109115600586e-08
>>> jeff 0.13722586631774902 1.3722586631774904e-07
>>> jeff 0.1359708309173584 1.359708309173584e-07
>>> jeff 0.13658690452575684 1.3658690452575684e-07
>>> 
>>> I think one would want argmin() and argmax() to work with general 
>>> iterables, so I wonder if the stdlib would not be a better home than list 
>>> itself. I half expected it to be an itertools recipe. The advantage of a 
>>> recipe is that variations such as needing the last occurrence of the 
>>> minimum are easily accommodated.
>>> 
>> 
>> Surely its index_min() and index_max() not argmin() and argmax().
>> 
>> Barry
>> 
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/VMSSYHAOST7K3WNHO3JOQENI6G5CXZZ5/
>> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5VSXEETS2VFEAQSJTWUOJQ3JO5F7YMIG/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Efficient debug logging

[Barry]

> On 14 Feb 2017, at 17:39, Kyle Lahnakoski  wrote:
> 
> 
> Can you wrap the expensive functions in lambdas? And have your logger
> evaluate it, only if required?
> 
>> debugLog( ‘info is %r’ % (lambda: expensiveFunction(),) )

Interesting idea. I will bench mark and see what the cost of the lamba version 
is.
It would still be nice to have the code look cleaner without the lambda.

Barry

> 
> 
>> On 2017-02-14 10:51, Barry Scott wrote:
>> A common pattern I use is to have logging calls for debug and information 
>> with my applications.
>> The logging calls can be separately enabled and disabled.
>> 
>> For example:
>> 
>> debug_log_enabled = False
>> def debugLog( msg ):
>>  If debug_log_enabled:
>>print( ‘Debug: %s’ % (msg,) )
>> 
>> Then the caller can simple write:
>> 
>> def main():
>>  debugLog( ‘Start of main’ )
>> 
>> This is fine until the evaluation of the msg becomes expensive.
>> 
>>debugLog( ‘info is %r’ % (expensiveFunction(),) )
>> 
>> What would be nice is to be able to avoid evaluation the tuple of arguments 
>> if debug is
>> disabled as this can be expensive. I can write this:
>> 
>>if debug_log_enabled:  debugLog( ‘info is %r’ % (expensiveFunction(),) )
>> 
>> But that is a more code then I would like to write. And if the debug code is 
>> a performance problem cannot
>> be left in the production code.
>> 
>> I could combine the boolean and the log function by using a class to tidy up 
>> the implementation.
>> 
>> class DebugLog:
>>def __init__( self, enabled = False ):
>>self.enabled = enabled
>> 
>>def __bool__( self ):
>>return self.enabled
>> 
>>def __call__( self, msg ):
>>if self.enabled: print( ‘Debug: %s’ % (msg,) )
>> 
>> And call like this:
>> 
>>dbg_log = DebugLog()
>> 
>>   If dbg_log: dbg_log( ‘a debug message’ )
>> 
>> But I’d like to only write:
>> 
>>dbg_log( ‘a debug message’ )
>> 
>> And have the evaluation of the argument skipped unless its dbg_log is 
>> enabled.
>> 
>> I cannot see how to do this with python as it stands.
>> 
>> Something would have to be added to allow python to short circuit the 
>> argument tuple evaluation.
>> 
>> Maybe python can check for a special dunder on the class that know how to do 
>> this idiom, __if_true_call__?
>> 
>> Thoughts?
>> 
>> Barry
>> 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Efficient debug logging

[Barry]

> On 14 Feb 2017, at 17:03, Cory Benfield  wrote:
> 
> 
>> On 14 Feb 2017, at 15:51, Barry Scott  wrote:
>> And have the evaluation of the argument skipped unless its dbg_log is 
>> enabled.
>> 
>> I cannot see how to do this with python as it stands.
>> 
>> Something would have to be added to allow python to short circuit the 
>> argument tuple evaluation.
>> 
>> Maybe python can check for a special dunder on the class that know how to do 
>> this idiom, __if_true_call__?
>> 
> 
> Is there any reason logger.isEnabledFor(level), as shown in the docs 
> https://docs.python.org/2/library/logging.html#logging.Logger.isEnabledFor, 
> is not sufficient for this?

The point is that the cost of creating the msg argument can be very high.

At the point that logging decides to skip output it is to late to save the cost 
of creating the arg tuple.

Not that it is relivent for this idea bit logging's levels are too course for 
logging in complex applications. The app I am working on at the moment has 20 
seperate debug categories
that are independently enabled.

Barry

> Cory
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Efficient debug logging

[Barry]

> On 16 Feb 2017, at 21:03, Abe Dillon  wrote:
> 
> I personally don't see why you can't use floats for log levels, but I'm 
> interested to know how people are using logs such that they need dozens of 
> levels. That, however; is tangential to the discussion about conditional 
> execution of an expensive function.

Its not lots of levels its lots of categories.
I did add one level, INFOHEADER, but not for debug.

It you are interested in what I did you could look at this project:

https://github.com/barry-scott/scm-workbench

Have a look at:

https://github.com/barry-scott/scm-workbench/blob/master/Source/Scm/wb_scm_debug.py
https://github.com/barry-scott/scm-workbench/blob/master/Source/Common/wb_debug.py

Then look at almost any module and see the use of the _debug calls.

Barry




___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] get() method for list and tuples

[Barry]

> On 1 Mar 2017, at 01:26, Michel Desmoulin  wrote:
> 
> - you can iterate on both
Maybe, bit do you want the keys, values or (key, value) items? Keys being the 
deafult.
> - you can index both
Maybe as you cannot in the general case know the index. Need keys().
> - you can size both
Yes

I think this duck cannot swim or quack.

Barry


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] get() method for list and tuples

[Barry]

> On 1 Mar 2017, at 19:13, Abe Dillon  wrote:
> 
> Barry, you're taking the metaphor too far. Duct typing is about presenting a 
> certain interface. If your function takes an object that has a get(key, 
> default) method, the rest doesn't matter. That's the only way in which the 
> object needs to resemble a duck in your function.

In support of get() I found that list of ducks a poor agument for the reasons I 
stated.

That's not to say that get() on lists may well have value for other reasons, 
but I find the duck typing a very weak argument.

> 
> I'd like to +1 this proposal. It should be trivial to implement. It won't 
> break backward compatibility. It's intuitive. I can think of several places I 
> would use it. I can't think of a good reason not to include it.

I do not think I have encounted any use cases where I would have used this my 
self.
Maybe in command line processing, I have to dig in my repos and check. Might be 
able to short cut a length check with a get with default None or "".

Barry

> 
>> On Wed, Mar 1, 2017 at 12:06 PM, Barry  wrote:
>> 
>> > On 1 Mar 2017, at 01:26, Michel Desmoulin  
>> > wrote:
>> >
>> > - you can iterate on both
>> Maybe, bit do you want the keys, values or (key, value) items? Keys being 
>> the deafult.
>> > - you can index both
>> Maybe as you cannot in the general case know the index. Need keys().
>> > - you can size both
>> Yes
>> 
>> I think this duck cannot swim or quack.
>> 
>> Barry
>> 
>> 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Exploiting type-homogeneity in list.sort() (again!)

[Barry]

Can you assume that list of of type(list[0]) and use that type's optimised sort?
But in the optimised sort code check that the types are as required.
If you hit an element that is not of the required type then fall back to the 
unoptimised sort.

So part of list is sorted using optimised code and the sort is completed with 
the unoptimised code. Is that sematically clean?

Barry


> On 8 Mar 2017, at 17:08, Erik  wrote:
> 
>> On 08/03/17 11:07, Steven D'Aprano wrote:
>> I mentioned earlier that I have code which has to track the type of list
>> items, and swaps to a different algorithm when the types are not all the
>> same.
> 
> Hmmm. Yes, I guess if the expensive version requires a lot of isinstance() 
> messing or similar for each element then it could be better to have optimized 
> versions for homogeneous lists of ints or strings etc.
> 
>>> A list.is_heterogeneous() method
>>> could be implemented if it was necessary,
>> 
>> I would prefer to get the list item's type:
>> 
>> if mylist.__type_hint__ is float:
> 
> If you know the list is homogeneous then the item's type is "type(mylist[0])".
> 
> Also, having it be a function call gives an obvious place to put the 
> transition from "unknown" to known state if the tri-state hint approach was 
> taken. Otherwise, that would have to be hooked into the attribute access 
> somehow.
> 
> That's for someone who wants to try implementing it to decide and propose 
> though :)
> 
> E.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Adding an 'errors' argument to print

[Barry]

I took to using

 chcp 65001

This puts cmd.exe into unicode mode.

Of course the python 3.6 make this uneccesary i understand.

Barry


> On 24 Mar 2017, at 15:41, Ryan Gonzalez  wrote:
> 
> Recently, I was working on a Windows GUI application that ends up running 
> ffmpeg, and I wanted to see the command that was being run. However, the file 
> name had a Unicode character in it (it's a Sawano song), and when I tried to 
> print it to the console, it crashed during the encode/decode. (The encoding 
> used in cmd doesn't support Unicode characters.)
> 
> The workaround was to do:
> 
> 
> print(mystring.encode(sys.stdout.encoding, 
> errors='replace).decode(sys.stdout.encoding))
> 
> 
> Not fun, especially since this was *just* a debug print.
> 
> The proposal: why not add an 'errors' argument to print? That way, I could've 
> just done:
> 
> 
> print(mystring, errors='replace')
> 
> 
> without having to worry about it crashing.
> 
> --
> Ryan (ライアン)
> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
> http://refi64.com
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] ImportError raised for a circular import

[Barry]

> On 13 Jun 2017, at 20:13, Antoine Rozo  wrote:
> 
> But circular imports are sometimes needed in modules.
> For example when you have two classes in two different modules that reference 
> each other in their methods (and because you can't pre-declare classes like 
> in C++).

Really? It has always been a strong sign of a design bug in all the cases I 
have ever seen.
The example you suggest always fails when I accidentally write it.

Pylint will certainly shout loud that this case is an error.

Barry

> 
> 2017-06-13 20:30 GMT+02:00 Barry Scott :
>> Recently I fell into the trap of creating a circular import and yet again it 
>> took time to figure out what was wrong.
>> 
>> I'm wondering why the python import code does not detect this error and 
>> raise an exception.
>> 
>> I took a look at the code and got as far as figuring out that I would need 
>> to add the detection to the
>> python 3 import code. Unless I missed something I cannot get the detection 
>> without
>> modifying the core code as I could see no way to hook the process cleanly.
>> 
>> Is it reasonable idea to add this detection to python?
>> 
>> I am willing to work on a patch.
>> 
>> Barry
>> 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> Antoine Rozo
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] HTTP compression support for http.server

[Barry]

Does you code allow suporting more then gzip? For example Brotli compression is 
becoming inmportant for some web apps.

Barry

> On 24 Jul 2017, at 17:30, Chris Angelico  wrote:
> 
>> On Tue, Jul 25, 2017 at 2:20 AM, Chris Barker  wrote:
>> On Thu, Jul 20, 2017 at 12:15 AM, Pierre Quentel 
>> wrote:
>>> - if so, should it be supported by default ? It is the case in the PR,
>>> where a number of content types, eg text/html, are compressed if the user
>>> agent accepts the gzip "encoding"
>> 
>> 
>> I'm pretty wary of compression happening by default -- i.e. someone runs
>> exactly the same code with a newer version of Python, and suddenly some
>> content is getting compressed.
> 
> FWIW I'm quite okay with that. HTTP already has a mechanism for
> negotiating compression (Accept-Encoding), designed to be compatible
> with servers that don't support it. Any time a server gains support
> for something that clients already support, it's going to start
> happening as soon as you upgrade.
> 
> Obviously this kind of change won't be happening in a bugfix release
> of Python, so it would be part of the regular checks when you upgrade
> from 3.6 to 3.7 - it'll be in the NEWS file and so on, so you read up
> on it before you upgrade.
> 
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 550 v2

[Barry]

I'm not clear why there is a new_context_key which seems not to be a key.
It seems that the object is a container for a single value.

Key.set( value ) does not feel right.

Container.set( value ) is fine.

Barry


> On 16 Aug 2017, at 00:55, Yury Selivanov  wrote:
> 
> Hi,
> 
> Here's the PEP 550 version 2.  Thanks to a very active and insightful
> discussion here on Python-ideas, we've discovered a number of
> problems with the first version of the PEP.  This version is a complete
> rewrite (only Abstract, Rationale, and Goals sections were not updated).
> 
> The updated PEP is live on python.org:
> https://www.python.org/dev/peps/pep-0550/
> 
> There is no reference implementation at this point, but I'm confident
> that this version of the spec will have the same extremely low
> runtime overhead as the first version.  Thanks to the new ContextItem
> design, accessing values in the context is even faster now.
> 
> Thank you!
> 
> 
> PEP: 550
> Title: Execution Context
> Version: $Revision$
> Last-Modified: $Date$
> Author: Yury Selivanov 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 11-Aug-2017
> Python-Version: 3.7
> Post-History: 11-Aug-2017, 15-Aug-2017
> 
> 
> Abstract
> 
> 
> This PEP proposes a new mechanism to manage execution state--the
> logical environment in which a function, a thread, a generator,
> or a coroutine executes in.
> 
> A few examples of where having a reliable state storage is required:
> 
> * Context managers like decimal contexts, ``numpy.errstate``,
>  and ``warnings.catch_warnings``;
> 
> * Storing request-related data such as security tokens and request
>  data in web applications, implementing i18n;
> 
> * Profiling, tracing, and logging in complex and large code bases.
> 
> The usual solution for storing state is to use a Thread-local Storage
> (TLS), implemented in the standard library as ``threading.local()``.
> Unfortunately, TLS does not work for the purpose of state isolation
> for generators or asynchronous code, because such code executes
> concurrently in a single thread.
> 
> 
> Rationale
> =
> 
> Traditionally, a Thread-local Storage (TLS) is used for storing the
> state.  However, the major flaw of using the TLS is that it works only
> for multi-threaded code.  It is not possible to reliably contain the
> state within a generator or a coroutine.  For example, consider
> the following generator::
> 
>def calculate(precision, ...):
>with decimal.localcontext() as ctx:
># Set the precision for decimal calculations
># inside this block
>ctx.prec = precision
> 
>yield calculate_something()
>yield calculate_something_else()
> 
> Decimal context is using a TLS to store the state, and because TLS is
> not aware of generators, the state can leak.  If a user iterates over
> the ``calculate()`` generator with different precisions one by one
> using a ``zip()`` built-in, the above code will not work correctly.
> For example::
> 
>g1 = calculate(precision=100)
>g2 = calculate(precision=50)
> 
>items = list(zip(g1, g2))
> 
># items[0] will be a tuple of:
>#   first value from g1 calculated with 100 precision,
>#   first value from g2 calculated with 50 precision.
>#
># items[1] will be a tuple of:
>#   second value from g1 calculated with 50 precision (!!!),
>#   second value from g2 calculated with 50 precision.
> 
> An even scarier example would be using decimals to represent money
> in an async/await application: decimal calculations can suddenly
> lose precision in the middle of processing a request.  Currently,
> bugs like this are extremely hard to find and fix.
> 
> Another common need for web applications is to have access to the
> current request object, or security context, or, simply, the request
> URL for logging or submitting performance tracing data::
> 
>async def handle_http_request(request):
>context.current_http_request = request
> 
>await ...
># Invoke your framework code, render templates,
># make DB queries, etc, and use the global
># 'current_http_request' in that code.
> 
># This isn't currently possible to do reliably
># in asyncio out of the box.
> 
> These examples are just a few out of many, where a reliable way to
> store context data is absolutely needed.
> 
> The inability to use TLS for asynchronous code has lead to
> proliferation of ad-hoc solutions, which are limited in scope and
> do not support all required use cases.
> 
> Curre

Re: [Python-ideas] a sorting protocol dunder method?

[Barry]

> On 4 Dec 2017, at 20:12, Antoine Pitrou  wrote:
> 
> On Mon, 4 Dec 2017 19:37:02 +0000
> Barry Scott  wrote:
>> I wondered what the performance would be and tested the following code:
>> 
> [...]
>> 
>> it outputs this for my with python 3.6.0
>> 
>> 1
>> key 0.010628s  1 calls
>> lt 0.053690s 119886 calls
>> 
>> It seems that providing a key is ~5 times faster the depending on __lt__.
>> (I even used a short circuit to se if __lt__ could beat key).
> 
> Thanks for taking the time to write a benchmark.  I'm not surprised
> by the results (and your __lt__ method isn't even complicated: the gap
> could be very much wider).  There is more to Python performance than
> aggregate big-O algorithmic complexity.

I was surprised by the huge difference. I was expecting a closer race.

For the record I think that a __sort_key__ is not a good idea as it is so easy 
to
do as I did and define key methods on the class, without the limit of one such
method.

Barry


> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Repr of lambda

[Barry]

> On 21 Dec 2017, at 06:57, Chris Barker  wrote:
> 
> in theory, the "goal" is for eval(repr(obj)) to return an equivelent object

Is that really was the goal of repr? If true then we would not need pickle.

I have always assumed that repr of simple things aims to represent them
in just the way you would write them in python code. Repr of complex things
represents the obj as a useful summary.

Lamba seems to be in the complex end of things.

In debug logs I am often very interested in object identity and use the
0x123 as one way to know. Removing the unique id would be a regression
in my eyes.

Maybe what you would like to have is an explain function that given any
object tells you alll about it. help function does some of this I guess.

Barry

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add an UML class diagram to the collections.abc module documentation

[Barry]

After reading this thread I went off and read up about plantuml.
According to its docs it produces dot files that it reandes with graphviz.
So if graphviz will produce svg that part is solved.

Can you creat the plantuml file automatically from the python code?

Barry

On 2 Jan 2018, at 15:26, Yahya Abou 'Imran via Python-ideas 
 wrote:

>> Can I suggest that rather than manually producing or tweaking, and later
>> updating, the diagrams it might be better to spend a little time
>> annotating the source code [...]
>> While the diagrams produced might lack the elegance of manually produced
>> ones they would be much more useful as they would always be up to date
>> due to being produced, and updated, automatically.
> 
> I think it would be a lot of changes in the source code...
> 
> I would like to precise, so everybody can know it, the syntax of a .puml is 
> realy simple. For example:
> 
> test.puml:
> 
> @startuml
> 
> hide members
> show methods
> 
> abstract class Iterable {
>  {abstract} __iter__()
> }
> 
> abstract class Iterator {
>   {abstract} __next__()
>   ..
>   __iter__()
> }
> 
> Iterable <|-- Iterator
> 
> @enduml
> 
> Then:
> 
> $ plantuml test.puml
> 
> And you have test.png I joined.
> 
> 
> 
>> [...]
>> I am attaching the diagram produced for the full inheritance of
>> collections.abc as produced by doxygen/graphviz but I am sure that there
>> are some options that could make this more readable/useful.
>> 
>> Steve
> 
> It's a blank file that I have...
> 
> I've been struggling with those kind of tools these days, and realised that 
> it's a lot more work (and pain) than a plain text file.
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add an UML class diagram to the collections.abc module documentation

[Barry]

> On 2 Jan 2018, at 19:38, Brett Cannon  wrote:
> 
> 
> 
>> On Tue, 2 Jan 2018 at 05:25 Yahya Abou 'Imran 
>>  wrote:
>> At the end of the day, I found that plantuml is the most suitable tool for 
>> this.
> 
> Right, but when I look at http://plantuml.com/ I don't see any open source 
> code to guarantee
> it will be available in e.g. 5 years. (I really just see a lot of ads around 
> a free Java app).

I see fedora packages plantuml and says is lgpl3 licensed.

Barry

>  
>> Graphviz dot is interesting, but it doesn't feel natural to make class 
>> diagram with it, or at least it's less handy... I could bring several 
>> arguments to support this, but it's not the topic.
> 
> It's somewhat the topic to me, though, since people seem to have found the 
> diagrams helpful, which means defining how to sustainably maintain them so we 
> are willing to accept them into the documentation is important as that will 
> be the next step in accepting them into the documentation.
>  
>> 
>> Everybody wanting to try itself is welcome, but, *I* can take the commitment 
>> to maintain it over the years.
> 
> I personally appreciate that offer, but I also don't know you well enough to 
> be able to take that as a guarantee, hence why I'm trying to make sure the 
> tooling that is used will last for a very long time (next month will be the 
> 27th anniversary of Python's first public release, so anything we do may need 
> to last a while :) .
> 
> I know this is an annoying thing to be thinking about when you already have 
> the diagram done in plantuml, but sustaining this work for a long time is 
> part of maintaining open source.
> 
> -Brett
>  
>> 
>> Here are the 3 svg files witch are my last proposals for the moment:
>> 
>> https://gitlab.com/yahya-abou-imran/collections-abc-uml/blob/master/plantuml/base.svg
>> https://gitlab.com/yahya-abou-imran/collections-abc-uml/blob/master/plantuml/other_collections.svg
>> https://gitlab.com/yahya-abou-imran/collections-abc-uml/blob/master/plantuml/full.svg
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why CPython is still behind in performance for some widely used patterns ?

[Barry]

> On 20 Feb 2018, at 13:12, Nick Coghlan  wrote:
> 
>> On 20 February 2018 at 16:17, Antoine Pitrou  wrote:
>> On Mon, 19 Feb 2018 20:15:27 +0100
>> Stefan Behnel  wrote:
>>> Nick Coghlan schrieb am 02.02.2018 um 06:47:
>>>> to make the various extension module authoring tools
>>>> easier to discover, rather than having folks assuming that handcrafted
>>>> calls directly into the CPython C API is their only option.
>>> 
>>> Or even a competitive option. Tools like Cython or pybind11 go to great
>>> length to shave off every bit of overhead from C-API calls, commonly
>>> replacing high-level C-API functionality with macros and direct access to
>>> data structures. The C/C++ code that they generate is so complex and tuned
>>> that it would be infeasible to write and maintain something like that by
>>> hand, but it can perfectly be generated, and it usually performs visibly
>>> better than most hand-written modules, definitely much better than anything
>>> a non-expert could write.
>>> 
>>> Basically, by not learning the C-API you can benefit from all that highly
>>> tuned and specialised code written by C-API experts that the documentation
>>> doesn't even tell you about.
>> 
>> Doesn't the documentation ever mention Cython? It probably should (no
>> idea about pybind11, which I've never played with). Perhaps you can
>> open an issue about that?
> 
> We mention them in the Extending & Embedding guide, and link out to
> the page on packaging.python.org that describes them in more detail:
> https://docs.python.org/3/extending/index.html#recommended-third-party-tools

Can you add PyCXX to the list please?

Barry

> 
> Cheers,
> Nick.
> 
> P.S. There are also a number of open issues at
> https://github.com/pypa/python-packaging-user-guide/issues regarding
> additional projects that should be mentioned in
> https://packaging.python.org/guides/packaging-binary-extensions/
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations

[Barry]

> On 12 Mar 2018, at 21:56, George Fischhof  wrote:
> 
> 
> 
> 2018-03-12 22:16 GMT+01:00 Paul Moore :
>> On 12 March 2018 at 20:57, George Fischhof  wrote:
>> > Good day all,
>> >
>> > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to
>> > contain file operations.
>> >
>> > Here is the draft. What do you think about this?
>> 
>> I don't know for certain what I think I feel about the idea - in
>> general, it seems plausible. But I think you'll need to get down to
>> specifics in the PEP, exactly what functions do you suggest get added
>> to pathlib?
>> 
>> Paul
> 
> 
> Basically  file and directory operations: copy, rename, move, delete. With 
> all variants.
> I will collect them, and put intot PEP
> 

I thought that because of __fspath__ conversions pathlib can be use with any 
function that needs a path. So long as shutil is Pathlib friendly isn’t this 
asolv d problem?

No need to extend pathlib for every file op, just pass the existed file ops a 
Path.

Wasn’t this discussed at the time fspath was added?

Barry

> George
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2

[Barry]

> On 17 Mar 2018, at 10:42, George Fischhof  wrote:
> 
> Hi folks,
> 
> I added the list of functions to the proposal, here is the new version.
> 
> George
> 
> 
> 
> 
> PEP: 
> Title: Pathlib Module Should Contain All File Operations
> Author: George Fischhof 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 15-Mar-2018
> Python-Version: 3.8
> Post-History: 12-Mar-2018, 17-Mar-2018
> 
> 
> Abstract
> 
> 
> This PEP proposes pathlib module to be a centralized place for all
> file-system related operations.
> 
> 
> Rationale
> =
> 
> Right now we have several modules that contain functions related
> to file-system operations mainly the os, pathlib and shutil.
> For beginners it is quite hard to remember where can he / she find
> a function (copy resides in shutil, but the remove function can be
> found in the os module.  (And sometimes developers with moderate
> experience have to check the documentation as well.)
> 
> After the release of version 3.6 several methods became aware of
> path-like object.  There are only a few ones which does not support
> the path-like object.  After making these methods path-like object
> aware, these functions could be added to pathlib.
> 
> With functions in pathlib the developers should not have to think
> on which method (function) can be found in which module.
> 
> Makes the life easier.
> 
> 
> Implementation
> ==
> 
> For compatibility reasons the pathlib should contain wrappers to
> the original functions.  The original functions should remain
> at their original place.  (Or if pathlib contains the function, the
> original modules should have a wrapper to it.)
> 
> 
> List of mentioned functions
> ===
> 
> * os.link => path.hardlink_to
>   (should be named similarly to path.softlink_to)
> 
> * os.mkfifo
> 
> * os.readlink
> 
> * os.remove
> 
> * os.removedirs
>   (shutil.rmtree has the same functionalaty)

I think you will find these to are a lot different.

> 
> * os.truncate
> 
> * shutil.copyfileobj

This function does not take paths as arguments. I guess it does not belong here.
> 
> * shutil.copyfile
> 
> * shutil.copymode
> 
> * shutil.copystat
> 
> * shutil.copy
> 
> * shutil.copy2
> 
> * shutil.copytree with shutil.ignore_patterns
> 
> * shutil.move
> 
> * shutil.disk_usage
> 
> * shutil.chown

Is yout plan to also expose all the constants need to use chmod etc from 
pathlib?

> 
> All functions from os module accept path-like objects,
> and none of the shutil functions.

Should shutils be improved to accept path-like first?

Barry

> 
> 
> Copyright
> =
> 
> This document has been placed in the public domain.
> 
> 
> 
> ..
>Local Variables:
>mode: indented-text
>indent-tabs-mode: nil
>sentence-end-double-space: t
>fill-column: 70
>coding: utf-8
>End:
> 
> 
> 
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2

[Barry]

> On 17 Mar 2018, at 22:49, George Fischhof  wrote:
> 
> 
> 
> 2018. márc. 17. 21:34 ezt írta ("Barry" ):
> 
> 
>> On 17 Mar 2018, at 10:42, George Fischhof  wrote:
>> 
>> Hi folks,
>> 
>> I added the list of functions to the proposal, here is the new version.
>> 
>> George
>> 
>> 
>> 
>> 
>> PEP: 
>> Title: Pathlib Module Should Contain All File Operations
>> Author: George Fischhof 
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 15-Mar-2018
>> Python-Version: 3.8
>> Post-History: 12-Mar-2018, 17-Mar-2018
>> 
>> 
>> Abstract
>> 
>> 
>> This PEP proposes pathlib module to be a centralized place for all
>> file-system related operations.
>> 
>> 
>> Rationale
>> =
>> 
>> Right now we have several modules that contain functions related
>> to file-system operations mainly the os, pathlib and shutil.
>> For beginners it is quite hard to remember where can he / she find
>> a function (copy resides in shutil, but the remove function can be
>> found in the os module.  (And sometimes developers with moderate
>> experience have to check the documentation as well.)
>> 
>> After the release of version 3.6 several methods became aware of
>> path-like object.  There are only a few ones which does not support
>> the path-like object.  After making these methods path-like object
>> aware, these functions could be added to pathlib.
>> 
>> With functions in pathlib the developers should not have to think
>> on which method (function) can be found in which module.
>> 
>> Makes the life easier.
>> 
>> 
>> Implementation
>> ==
>> 
>> For compatibility reasons the pathlib should contain wrappers to
>> the original functions.  The original functions should remain
>> at their original place.  (Or if pathlib contains the function, the
>> original modules should have a wrapper to it.)
>> 
>> 
>> List of mentioned functions
>> ===
>> 
>> * os.link => path.hardlink_to
>>   (should be named similarly to path.softlink_to)
>> 
>> * os.mkfifo
>> 
>> * os.readlink
>> 
>> * os.remove
>> 
>> * os.removedirs
>>   (shutil.rmtree has the same functionalaty)
> 
> I think you will find these to are a lot different.
> 
> 
> From high level, it is only question of implementation: the main goal is to 
> remove a chain of directories

This is a PEP you and proposing details matter and this is one of the details 
tha you need be clear on.

> 
>> 
>> * os.truncate
>> 
>> * shutil.copyfileobj
> 
> This function does not take paths as arguments. I guess it does not belong 
> here.
> 
> 
> No path, it is true (right now), but it has sense to have this function among 
> the others.

What do you mena right now?

> 
> 
>> 
>> * shutil.copyfile
>> 
>> * shutil.copymode
>> 
>> * shutil.copystat
>> 
>> * shutil.copy
>> 
>> * shutil.copy2
>> 
>> * shutil.copytree with shutil.ignore_patterns
>> 
>> * shutil.move
>> 
>> * shutil.disk_usage
>> 
>> * shutil.chown
> 
> Is yout plan to also expose all the constants need to use chmod etc from 
> pathlib?
> 
> Of course, because this PEP is about conveniece and easyness.

Can you say so then and not leave it in doubt.

> 
>> 
>> All functions from os module accept path-like objects,
>> and none of the shutil functions.
> 
> Should shutils be improved to accept path-like first?
> 
> Maybe. The PEP should show the goal. This is just a task among others to 
> achieve the goal. But it can be done simultanuosly.
> 
> George
> 
> Barry
> 
>> 
>> 
>> Copyright
>> =
>> 
>> This document has been placed in the public domain.
>> 
>> 
>> 
>> ..
>>Local Variables:
>>mode: indented-text
>>indent-tabs-mode: nil
>>sentence-end-double-space: t
>>fill-column: 70
>>coding: utf-8
>>End:
>> 
>> 
>> 
>> 
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add shutil.chown(..., recursive=False)

[Barry]

> On 28 May 2018, at 21:23, Giampaolo Rodola'  wrote:
> 
> ...as in (not tested):
> 
> def _rchown(dir, user, group):
> for root, dirs, files in os.walk(dir, topdown=False):
> for name in files:
> chown(os.path.join(root, name), user, group)
> 
> def chown(path, user=None, group=None, recursive=False):
> if recursive and os.path.isdir(path):
> _rchown(dir, user, group)
> ...
> 
> It appears like a common enough use case to me ("chown -R path"). 
> Thoughts?

I wonder if it is very common.
Don’t you have to be root or use sudo chown?
In which case it is only python code running as root that could use this.

Barry
> 
> -- 
> Giampaolo - http://grodola.blogspot.com
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Calling python from C completely statically

[Barry]

> On 9 Jul 2018, at 21:11, Alberto Garcia  wrote:
> 
> Hi,
> 
> thank you for your response. I've downloaded the sources but I couldn't find 
> any documentation. In addition I see that there is not a single C/C++ file. 
> What I want to do is calling python from C.
> Am I missing something?

You need to grab the sources with an svn checkout. I guess you grabbed the 
binary kit.

https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/

Docs are in here.

https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/Installer/doc/

Barry

> Cheers
> 
> 
>> On Mon, Jul 9, 2018 at 12:20 PM Barry Scott  wrote:
>> I think you might find Gordon McMillian's installer interesting to look at. 
>> It has a lot
>> f the tech that I think you are looking for.
>> 
>> Works up to python 2.7. I ended up taking over when Gordon stopped 
>> maintaining
>> it and kept it going up to python 2.7. In principle the same ideas could be 
>> made to
>> work in python 3 I believe.
>> 
>>  https://sourceforge.net/projects/meinc-installer/
>> 
>> The zip file is appended to the end of the .EXE or unix ELF fie.
>> The boot strap knows how to import form the ZIP at the end of
>> the binary.
>> 
>> It also has a way to split out the .SO/.DLL files from the ZIP and
>> allow them to be loaded. Single EXE mode.
>> 
>> There are docs that explain how it works in the sources.
>> 
>> Barry
>> 
>> 
>> 
>>> On 9 Jul 2018, at 19:40, Alberto Garcia  wrote:
>>> 
>>> Does the zip need to reside in disk to be loaded. Or can it be loaded from 
>>> memory? I don't want it to be loaded from disk but from Memory
>>> 
>>> 
>>> 
>>>> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia  
>>>> wrote:
>>>> O  I guess you mean this:
>>>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>>>> 
>>>> Right?
>>>> 
>>>>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia  
>>>>> wrote:
>>>>> Thank you for your response,
>>>>> 
>>>>> I was thinking on creating that zip file with the content of the Lib 
>>>>> folder and having my c code to download it over the network and have it 
>>>>> in memory. 
>>>>> I guess that the zip file should have no compression at all right?
>>>>> 
>>>>> When you say that I need to use the cx_freeze approach what do you mean? 
>>>>> Can you point me to where they do that? 
>>>>> 
>>>>> And why changing sys.path again to the executable again? Which part of 
>>>>> the executable? 
>>>>> 
>>>>> I'll put my efforts in this.
>>>>> 
>>>>> Thank you 
>>>>> 
>>>>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan  wrote:
>>>>>> On 9 July 2018 at 03:10, Alberto Garcia  wrote:
>>>>>> > Hey there,
>>>>>> >
>>>>>> > Yes, the part of having the pyd modules built in in library is already 
>>>>>> > done.
>>>>>> > I followed the instructions in the README. What I would like to know 
>>>>>> > now is
>>>>>> > how to embed the non frozen python (py) modules. Can you guys please 
>>>>>> > point
>>>>>> > me in the right direction.
>>>>>> 
>>>>>> The gist is to:
>>>>>> 
>>>>>> 1. take the entire Lib directory and put it in a zip archive
>>>>>> 2. use the approach demonstrated in cx_freeze to point sys.path in
>>>>>> your static executable at that zip archive
>>>>>> 3. adjust your C code to point sys.path back at the executable itself,
>>>>>> and then combine your executable and the zip archive into a single
>>>>>> contiguous file (similar to what zipapp does with it's helper script
>>>>>> and app archive)
>>>>>> 
>>>>>> There are likely to still be rough edges when doing that, since this
>>>>>> isn't a well tested configuration. When all else fails, find the part
>>>>>> of the source code responsible for any error messages you're seeing,
>>>>>> and try to work out if there's a setting you can tweak to avoid
>>>>>> hitting that code path.
>>>>>> 
>>>>>> Cheers,
>>>>>> Nick.
>>>>>> 
>>>>>> -- 
>>>>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>>> -- 
>>>>> Alberto García Illera
>>>>> 
>>>>> GPG Public Key: https://goo.gl/twKUUv
>>>> -- 
>>>> Alberto García Illera
>>>> 
>>>> GPG Public Key: https://goo.gl/twKUUv
>>> 
>>> 
>>> -- 
>>> Alberto García Illera
>>> 
>>> GPG Public Key: https://goo.gl/twKUUv
>>> ___
>>> Python-ideas mailing list
>>> Python-ideas@python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
> 
> 
> -- 
> Alberto García Illera
> 
> GPG Public Key: https://goo.gl/twKUUv
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Calling python from C completely statically

[Barry]

> On 10 Jul 2018, at 19:12, Alberto Garcia  wrote:
> 
> Thank you Barry,
> 
> but I see that the the way yo do it is dropping files to disk. I want to 
> prevent any file to touch the disk except for the actual binary that has 
> everything embedded in it.

What file are you refering to? The zip is built onto the elf/exe.

BTW this is all off topic for python ideas. You might want to take this offline.


Barry

> 
> Cheers
> 
>> On Tue, Jul 10, 2018 at 10:25 AM Barry  wrote:
>> 
>> 
>>> On 9 Jul 2018, at 21:11, Alberto Garcia  wrote:
>>> 
>>> Hi,
>>> 
>>> thank you for your response. I've downloaded the sources but I couldn't 
>>> find any documentation. In addition I see that there is not a single C/C++ 
>>> file. What I want to do is calling python from C.
>>> Am I missing something?
>> 
>> You need to grab the sources with an svn checkout. I guess you grabbed the 
>> binary kit.
>> 
>> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/
>> 
>> Docs are in here.
>> 
>> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Installer/Installer/doc/
>> 
>> Barry
>> 
>>> Cheers
>>> 
>>> 
>>>> On Mon, Jul 9, 2018 at 12:20 PM Barry Scott  wrote:
>>>> I think you might find Gordon McMillian's installer interesting to look 
>>>> at. It has a lot
>>>> f the tech that I think you are looking for.
>>>> 
>>>> Works up to python 2.7. I ended up taking over when Gordon stopped 
>>>> maintaining
>>>> it and kept it going up to python 2.7. In principle the same ideas could 
>>>> be made to
>>>> work in python 3 I believe.
>>>> 
>>>>https://sourceforge.net/projects/meinc-installer/
>>>> 
>>>> The zip file is appended to the end of the .EXE or unix ELF fie.
>>>> The boot strap knows how to import form the ZIP at the end of
>>>> the binary.
>>>> 
>>>> It also has a way to split out the .SO/.DLL files from the ZIP and
>>>> allow them to be loaded. Single EXE mode.
>>>> 
>>>> There are docs that explain how it works in the sources.
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>> 
>>>>> On 9 Jul 2018, at 19:40, Alberto Garcia  wrote:
>>>>> 
>>>>> Does the zip need to reside in disk to be loaded. Or can it be loaded 
>>>>> from memory? I don't want it to be loaded from disk but from Memory
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia  
>>>>>> wrote:
>>>>>> O  I guess you mean this:
>>>>>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Common.c
>>>>>> 
>>>>>> Right?
>>>>>> 
>>>>>>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia  
>>>>>>> wrote:
>>>>>>> Thank you for your response,
>>>>>>> 
>>>>>>> I was thinking on creating that zip file with the content of the Lib 
>>>>>>> folder and having my c code to download it over the network and have it 
>>>>>>> in memory. 
>>>>>>> I guess that the zip file should have no compression at all right?
>>>>>>> 
>>>>>>> When you say that I need to use the cx_freeze approach what do you 
>>>>>>> mean? Can you point me to where they do that? 
>>>>>>> 
>>>>>>> And why changing sys.path again to the executable again? Which part of 
>>>>>>> the executable? 
>>>>>>> 
>>>>>>> I'll put my efforts in this.
>>>>>>> 
>>>>>>> Thank you 
>>>>>>> 
>>>>>>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan  wrote:
>>>>>>>> On 9 July 2018 at 03:10, Alberto Garcia  
>>>>>>>> wrote:
>>>>>>>> > Hey there,
>>>>>>>> >
>>>>>>>> > Yes, the part of having the pyd modules built in in library is 
>>>>>>>> > already done.
>>>>>>>> > I followed the instructions in the README. What I would like to know 
>>>>>>>> > now is

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry]

> On 17 Jul 2018, at 20:35, Eric Snow  wrote:
> 
>> On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou  wrote:
>> On Mon, 16 Jul 2018 18:00:37 +0100
>> MRAB  wrote:
>>> Could you explicitly share an object in a similar way to how you
>>> explicitly open a file?
>>> 
>>> The shared object's refcount would be incremented and the sharing
>>> function would return a proxy to the shared object.
>>> 
>>> Refcounting in the thread/process would be done on the proxy.
>>> 
>>> When the proxy is closed or garbage-collected, the shared object's
>>> refcount would be decremented.
>>> 
>>> The shared object could be garbage-collected when its refcount drops to
>>> zero.
>> 
>> Yes, I'm assuming that would be how shareable buffers could be
>> implemented: a per-interpreter proxy (with a regular Python refcount)
>> mediating access to a shared object (which could have an atomic /
>> thread-safe refcount).
> 
> Nice! That's exactly how I'm doing it. :)  The buffer protocol makes
> it easier, but the idea could apply to arbitrary objects generally.
> That's something I'll look into in a later phase of the project.
> 
> In both cases the tricky part is ensuring that the proxy does not
> directly mutate the object (especially the refcount).  In fact, the
> decref part above is the trickiest.  The trickiness is a consequence
> of our goals.  In my multi-core project we're aiming for not sharing
> the GIL between interpreters.  That means reaching and keeping proper
> separation between interpreters.  Notably, without a GIL shared by
> interpreters, refcount operations are not thread-safe.  Also, in the
> decref case GC would happen under the wrong interpreter (which is
> problematic for several reasons).
> 
> With this in mind, here's how I'm approaching the problem:
> 
> 1. interp A "shares" an object with interp B (e.g. through a channel)
>* the object is incref'ed under A before it is sent to B
> 2. the object is wrapped in a proxy owned by B
>* the proxy may not make C-API calls that would mutate the object
> or even cause an incref/decref
> 3. when the proxy is GC'd, the original object is decref'ed
>* the decref must happen in a thread in which A is running
> 
> In order to make all this work the missing piece is a mechanism by
> which the decref (#3) happens under the original interpreter.  At the
> moment Emily Morehouse and I are pursuing an approach that extends the
> existing ceval "pending call" machinery currently used for handling
> signals (see Py_AddPendingCall).  The new [*private*] API would work
> the same way but on a per-interpreter basis rather than just the main
> interpreter.  This would allow one interpreter to queue up a decref to
> happen later under another interpreter.

The decrement itself is not the problem, that can be made thread safe.

Do you mean that once the ref reaches 0 you have to make the delete happen on 
the original interpreter?

Barry

> 
> FWIW, this ability to decref an object under a different interpreter
> is a blocker right now for a number of things, including supporting
> buffers in PEP 554 channels.
> 
> -eric
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Barry]

>> On 17 Jul 2018, at 21:00, Eric Snow  wrote:
>> 
>> On Tue, Jul 17, 2018 at 1:44 PM Barry  wrote:
>> The decrement itself is not the problem, that can be made thread safe.
> 
> Yeah, by using the GIL.   Otherwise, please elaborate.  My
> understanding is that if the decrement itself were not the problem
> then we'd have gotten rid of the GIL already.

All processors have thread safe ways to inc and dec and test, integers without 
holding a lock.

That is the mechanism that locks themselves are built out of. You can use that 
to avoid holding the GIL until the ref count reaches 0.

In c++ they built it into the language with std::atomic_int, you would have to 
find the way to do this C, i don’t have an answer at my finger tips for C.

Barry

> 
>> Do you mean that once the ref reaches 0 you have to make the delete happen 
>> on the original interpreter?
> 
> Yep.  For one thing, GC can trigger __del__, which can do anything,
> including modifying other objects from the original interpreter (incl.
> decref'ing them).  __del__ should be run under the original
> interpreter.  For another thing, during GC containers often decref
> their items.  Also, separating the GIL between interpreters may mean
> we'll need an allocator per interpreter.  In that case the
> deallocation must happen relative to the interpreter where the object
> was allocated.
> 
> -eric
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Regex timeouts

[Barry]

> On 17 Feb 2022, at 01:04, Tim Peters  wrote:
> 
> [J.B. Langston  ]
>> Thanks for the conclusive answer.
> 
> Not conclusive - just my opinion. Which is informed, but not infallible ;-)
> 
>> I will checkout the regex library soon.
> 
> You may not realize how easy this is? Just in case: go to a shell and type
> 
> pip install regex
> 
> (or, on Windows, "python -m pip install regex" in a DOS box).
> 
> That's it. You're done. Now you can use regex. In some cases, you can
> put "import regex as re" at the top of a module  At worst, replace
> instances of "re" with "regex". Stay away from the new features, and
> it's highly compatible with Python;s re.

I suspect that like me what was meant is that checkout means read the docs
to understand regex features. The install is trivial.

Barry

> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/C35D2Z5GZSWN3T46OR5LECU7VG5YD3LQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KLSG7EECZ3UWH4KJ7PAAOEI37XWTKUNP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: shutil.copyfileobj to return number of bytes copied

[Barry]

> On 2 Mar 2022, at 13:40, Davis, Matthew via Python-ideas 
>  wrote:
> 
> 
> Hi,
> 
> Currently shutil.copyfileobj returns nothing.
> I would like to be able to find out how many bytes were copied.
> 
> Whilst most file-like objects have a .tell() which you could use, some don’t, 
> and .tell() is not guaranteed to measure the number of bytes, it could 
> measure with other units.

Can you use os.fstat() on one of the file objects?
Can you write a helper that gets the size from the object that do not have 
tell()?

Barry

> 
> I don’t think changing the return value from None to not-None would be 
> backwards incompatible.
>  
> We could also do the same change for all other shutil.copy* methods (except 
> copytree).
>  
> Looking at the code, this seems straightforward to implement.
>  
> Existing code:
> 
> ```
> def copyfileobj(fsrc, fdst, length=0):
> """copy data from file-like object fsrc to file-like object fdst"""
> # Localize variable access to minimize overhead.
> if not length:
> length = COPY_BUFSIZE
> fsrc_read = fsrc.read
> fdst_write = fdst.write
> while True:
> buf = fsrc_read(length)
> if not buf:
> break
> fdst_write(buf)
> ```
>  
> New code:
> 
> 
> ```
> def copyfileobj(fsrc, fdst, length=0):
> """copy data from file-like object fsrc to file-like object fdst"""
> # Localize variable access to minimize overhead.
> if not length:
> length = COPY_BUFSIZE
> fsrc_read = fsrc.read
> fdst_write = fdst.write
> bytes_copied = 0
> while True:
> buf = fsrc_read(length)
> if not buf:
> break
> fdst_write(buf)
> bytes_copied += len(buf)
> return bytes_copied
> ```
>  
> Regards,
> Matt
>  
>  
> Matthew Davis
> Data Analytics Senior Lead
> Telstra Energy – Decision Science & AI
> E: matthew.davi...@team.telstra.com | M: 0415762868
>  
>  
>  
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/ZKVI4DE7RLYQNLV3XYUUW24FOO57WB5P/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6C2D45ZBI3ZKBU322J7EC2O6RLJMM6KE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: suprocess.check_call and friends include STDERR in error message

[Barry]

> On 9 Mar 2022, at 14:42, mangelo...@gmail.com wrote:
> 
> I often wish to perform the action of `subprocess.check_output`, i.e. print 
> STDOUT live, and check the return code is 0 on exit. However  if there is an 
> error, I would like the STDERR to be printed. When looking at log files and 
> just seeing the return code is often not very beneficial.
> 
> E.g.:
> 
> ```py
> subprocess.check_call(['ping', '-c', '1', 'github.com2'])
> ```
> 
> Will generate this error:
> ```
> subprocess.CalledProcessError: Command '['ping', '-c', '1', 'github.com2']' 
> returned non-zero exit status 2.
> ```
> 
> Here we are left wondering why it failed, on real world examples it becomes 
> critical to be able to see what the actual error is. So I find I always have 
> to wrap such calls in something like this to be able to see STDERR:
> 
> ```
> import subprocess
> cmds = ['ping', '-c', '1', 'github.com2']
> result = subprocess.run(cmds, stderr=subprocess.PIPE)
> if result.returncode != 0:
>msg = result.stderr.decode().strip()
>raise subprocess.CalledProcessError(f"CALLED SUBPROCESS ERROR: Command: {' 
> '.join(cmds)}\nReturn code {result.returncode}\nSTDERR: {msg}\n")
> ```
> 
> With the above one gets to see that STDERR is:
> ```
> ping: github.com2: Temporary failure in name resolution
> ```

I think you are saying you can do what you want with subprocess.run().
What did you need a change to support?

I often wrap code around subprocess.run() to get the exact behaviour that
I need. Seems to work well.

Barry


> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/7PVHSJGKFUIV4ZJFWG7EU3DR6XAGAKQW/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7F6HKX3FRG6QWEJM57BTCKHKUE2T7BXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: mro and super don't feel so pythonic

[Barry]

> On 26 Mar 2022, at 20:19, Christopher Barker  wrote:
> 
> Also, super() actually calls the method on all the  superclasses (but not the 
> same one twice) -- so that right to left thing doesn't matter.

Super() calls the method on only one class. Not all.
It finds that one class by using the mro to find the one class.

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/N7PUDFLIFYS5VH4XPWLKFDRAFR6JND2R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: default as a keyword argument for dict.get and dict.pop

[Barry]

> On 8 Jun 2022, at 16:09, Rob Cliffe via Python-ideas 
>  wrote:
> 
> 
> 
>> On 08/06/2022 15:40, Eric V. Smith via Python-ideas wrote:
>> 
>>> On 6/7/2022 4:59 PM, Chris Angelico wrote:
>>> On Wed, 8 Jun 2022 at 00:36,  wrote:
>>>> Hello!
>>>> 
>>>> Do you know if there has been discussions around why is the default 
>>>> argument is positional only in the dict methods get and pop?
>>>> 
>>>> I think
>>>> 
>>>> ```
>>>> d.get(key, default=3)
>>>> ```
>>>> 
>>>> way more readable than
>>>> 
>>>> ```
>>>> d.get(key, 3)
>>>> ```
>>>> 
>>>> specially since max and min builtin functions use default as a keyword 
>>>> argument.
>>> With min and max, it MUST be a keyword argument, because positional
>>> arguments are the values to be compared. So I think the main reason is
>>> "because nobody ever bothered to do it". If there's enough value in
>>> it, that could probably be changed, although mere consistency alone
>>> isn't a very strong argument.
>> 
>> I suspect it's been this way because the API is so old. Now that we have 
>> Argument Clinic it would be easier to implement as a keyword argument. But 
>> there may also be a performance issue with keyword arguments vs. positional. 
>> People are touchy when it comes to dicts!
>> 
>> Eric
> True (and rightly so).  But all that is required is that there is no 
> significant slow-down when the positional argument is used.
> Does that sound feasible?
> People can always choose not to use the keyword form.


Won’t the code have to check for a keyword even if a positional is used and 
therefore it must be slower. Need a benchmark to know by how much.

I’m one of those people that notice uS slow does in access to some APIs.

Barry

> Best wishes
> Rob Cliffe
>> 
>> ___
>> Python-ideas mailing list -- python-ideas@python.org
>> To unsubscribe send an email to python-ideas-le...@python.org
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-ideas@python.org/message/73OOS3YPUDTZVWFJ5V44X66XZCNXRQV6/
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/LMVZXUZQAJ4PRKWGTQDP45FS5YDRKRZG/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SF2TJG6XL37UMZOQCPGZDV5XUDKCKETW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Generalized deferred computation in Python

[Barry]

> On 23 Jun 2022, at 08:27, Stephen J. Turnbull  
> wrote:
> 
> Barry Scott writes:
> 
>> I can think of ways to implement evaluation-on-reference, but they
>> all have the effect of making python slower.
> 
> Probably.
> 
>> The simple
>> 
>>a = b
>> 
>> will need to slow down so that the object in b can checked to see
>> if it need evaluating.
> 
> No, it doesn't.  Binding a name is special in many ways, why not this
> one too?  Or "a = a" could be the idiom for "resolve a deferred now",
> which would require the check for __evaluate_me_now__ as you say.  But
> such simple "a = b" assignments are not so common that they would be a
> major slowdown.  I would think the real problem would be the "oops" of
> doing "a = b" and evaluating a deferred you don't want to evaluate.
> But this isn't a completely new problem, it's similar to a = b = []
> and expecting a is not b.

Interest idea that ref does not auto evaluate in all cases.
I was wondering about what the compile/runtime can do it avoid the costs
of checking for an evaluation.
> 
> Now consider a = b + 0.  b.__add__ will be invoked in the usual way.
> Only if b is a deferred will evaluation take place.

But the act of checking if b is deferred is a cost I am concerned about.

> 
> So I don't really see the rest of Python slowing down much.

Once we have the PEP address it’s semantics in detail we can estimate the costs.

I would think that it’s not that hard to add the expected check into the python 
ceval.c
And benchmark the impact of the checks. This would not need a full 
implementation
of the deferred mechanism.

Barry
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DEBOPY6OIRRLLCO2SQDYXXM7UMXZYRMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add copy to pathlib

[Barry]

> On 18 Oct 2022, at 21:59, Chris Angelico  wrote:
> 
> On Wed, 19 Oct 2022 at 06:50, Todd  wrote:
>> 
>> Currently, pathlib supports pretty much all common filesystem operations. 
>> You can create, move, and delete both files and directories. One big 
>> omission is copying. You need shutil for that.
>> 
>> Whatever the original intent might have been behind pathlib, it is now 
>> effectively the standard tool for filesystem operations. As such, excluding 
>> copying alone from basic operations creates an unnecessary hurdle for users. 
>> It also increases the difficulty of doing such operations since there is no 
>> hint within the pathlib documentation on how to copy, new users basically 
>> need to google it to find out.  That is fine for less common operations, but 
>> far from ideal from a basic one like copying.
>> 
> 
> Ah. I would look at solving this the other way: since this really
> isn't a path operation (in the same sense that moving/renaming and
> deleting are), keep it in shutil, but improve discoverability with a
> docs reference.

I tend to think that if move is in pathlib then copy makes sense.

As with all things pathlib it is a reasonable question to ask where should 
additions stop.

Barry

> 
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/DD36UOFGIBHY63YDAVYKTR5CFMMD3GUC/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Z5YOAHHLNZ5BPC3VUBVQNZIBEWWUUVKI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add mechanism to check if a path is a junction (for Windows)

[Barry]

On 8 Nov 2022, at 17:56, Charles Machalow  wrote:I would argue that just because it was easy for one to implement doesn't mean it's easy for others. That is true of so many things in life!I would have had no idea how to implement this without extra Googling and confusion.Having the abstraction makes it easier for others.But anyone that is suitably motivated can implement this.Barry- Charlie Scott Machalow
On Tue, Nov 8, 2022 at 1:12 AM Eryk Sun  wrote:On 11/8/22, Charles Machalow  wrote:
>
> Funny enough in PowerShell, for prints an "l" for both symlinks and
> junctions.. so it kind of thinks of it as a link of some sort too I guess.

As does Python already in many cases. For example, os.lstat() doesn't
traverse a mount point (junction). On Windows, symlinks and mount
points are in a general category of name-surrogate reparse points.
os.lstat() doesn't traverse them.

If Python supported copying a mount point via
os.symlink(os.readlink(src), dst), I'd be reluctantly in favor of just
letting ntpath.islink() return true for a mount point, as a practical
measure for seamless cross-platform implementations of functions like
rmtree() and copytree(). In terms of POSIX that's nonsense, but not
really on Windows.

> Is it that much of a waste to just return False on posix? I mean it's a
> couple lines and just maintains api.. and in theory can be more clear to
> some.

I'm just thinking this through in terms of conceptual cost and
usefulness in the standard library relative to how easy it is to
implement one's own isjunction() or is_name_surrogate() test. Of
course, a lot of the os.path tests have simple implementations, such
as exists(), isdir() and isfile(). They're in the standard library
because they're commonly needed. The question is whether isjunction()
is needed enough generally to justify adding it.

___Python-ideas mailing list -- python-ideas@python.orgTo unsubscribe send an email to python-ideas-le...@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MOY5NNFQK5PMD5GQBADZE3COVT7ONA62/Code of Conduct: http://python.org/psf/codeofconduct/___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WKA3ZPPLHTY3K3CCADUYC3KZI76NYTTA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Better (?) PRNG

[Barry]

> On 14 Nov 2022, at 14:31, James Johnson  wrote:
> 
> 
> I wrote the attached python (3) code to improve on existing prng functions. I 
> used the time module for one method, which resulted in disproportionate odd 
> values, but agreeable means.
> 
> I used the hashlib module for the second. It is evident that the code is 
> amateur, but the program might result in better PRN generation. 
> 
> The "app" lends itself to checking, using statistical tools (see comments.)

Have you used any cryptographic tools to prove the quality of your PRNG?
What results did you get?
How does your PRNG compare to what python already has?

Without that this analysis this will be unlikely to be considered as a 
candidate for python stdlib.

Barry

> 
> I remain a fan,
> 
> James Johnson
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/VENAT2YTVYVHTBSEVHHMIURCU6MG2CEO/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HWQV4AKQAUM7CY4NWS2IRIVLRAYMKR3V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Enhancing variable scope control

[Barry]

> On 29 Nov 2022, at 14:34, Anony Mous  wrote:
> 
> 
> As it stands now, to create a local scope, you must define a function.
> 
> However, there are many cases where various things can reasonably be done 
> inline. Preparatory code is often done this way.
> 
> Good coding practice is generally accepted to be that variables are local if 
> at all possible. However, in direct, inline Python code, we're inherently 
> creating variables with a global scope.
> 
> We don't actually need a function for this kind of code; but a local scope 
> would often be valuable (as we see with lambda.) Moving things off into a 
> function can create a circumstance where we have to go looking for the 
> function. When something is a "one-shot" as in global preparatory code, 
> that's doing more work, and creating more indirection, than needs actually be 
> done. But global operations have their own pitfalls, one of which is variable 
> collisions. So Python sort of drives us to make "one-use" functions where 
> lambdas are insufficient to the case in order to control locality.
> 
> You can end up writing things like...
> 
> def PrepOne():
> for MyCount in range(0,10):
> pass
> 
> PrepOne()
> 
> ...which achieves the locality desired for variable MyCount, but is clunky. 
> It also clutters the global namespace with prepOne, which has no good reason 
> to exist.
> 
> So I'm thinking of something along these lines:
> 
> local:# <== or whatever syntax makes sense
> for MyCount in range(0,10):
> pass
> 
> So in that example, the code is inline - runs as part of the global, 
> sequential code execution - but the variable MyCount is local and goes "poof" 
> once the local: scope is exited.
> 
> This is the scoping equivalent of a pair of curly braces in c and c++. I 
> would like to see it be nestable, as scopes are in c/c++:
> 
> local:# <== or whatever syntax makes sense
> for MyCount in range(0,10):
> local:
> for YourCount in range(0,MyCount+1)
> pass
> 
> There, MyCount is available in both scopes (the first local: is an enclosing 
> scope) and YourCount is available only in the second local: scope.
> 
> Or, local: could be very strict, and require further syntax to incorporate a 
> previous scope, something like this:
> 
> local:# <== or whatever syntax makes sense
> for MyCount in range(0,10):
> local:
> incorporate MyCount # <== or whatever syntax makes sense
> for YourCount in range(0,MyCount+1)
> pass
> 
> Lastly, I suggest that the global keyword be used to expose global scope 
> variables, just as it is within def functions:
> 
> TheirCount = 10
> local:# <== or whatever syntax makes sense
> global TheirCount
> for MyCount in range(0,TheirCount):
> local:
> for YourCount in range(0,TheirCount)
> pass
> 
> This would also be useful within normal functions, and again allows the 
> program flow to be linear rather than calling a function-within-a-function.

This has been discussed here before and rejected.
The ways scoping works in python is very hard to change as you are asking for i 
think was a major reason to reject.

Barry


> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/3ZEM5FPYWI24G3EOM3HA2I4RKWCMR3J5/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3AUAAUN5E6CZDWPSUKKVUGFSGMF7WASB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Nonuniform PRNG?

[Barry]

On 7 Dec 2022, at 19:45, James Johnson  wrote:How do I join the “regular” Python list? I’m not academically qualified for python ideasTry out https://discuss.python.org that has lots going on.BarryOn Wed, Dec 7, 2022 at 10:05 AM David Lowry-Duda  wrote:I'm very sorry, I didn't meant to post this to python-ideas. I didn't 
pay enough attention. I'm sending the same thing to the regular python 
list.

On Wed, Dec 07, 2022 at 11:03:24AM -0500, David Lowry-Duda wrote:
>Inspired by the recent thread on PRNG, I began to wonder: suppose that 
>I had a pseudorandom number generator that attempted to generate a 
>nonuniform distribution. Suppose for instance that it was to generate 
>a 0 bit 2/3 of the time, and a 1 bit 1/3 of the time.
>
>How would one go about testing this PRNG against an idealized 
>(similarly biased) PRNG?
>
>- DLD
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5C6XQAKD2KLJNYQT6QUIDCJLDSY3ECPL/
Code of Conduct: http://python.org/psf/codeofconduct/
-- Truth causes consequences; consequences bring pain; pain exorcises guilt!
___Python-ideas mailing list -- python-ideas@python.orgTo unsubscribe send an email to python-ideas-le...@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HLSE3UTNQTG7ML3ZVBJAX7QCGPOFGZV7/Code of Conduct: http://python.org/psf/codeofconduct/___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HYAF6PMFRNNVYS3SQHFM4DM3EEEYWD6N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Proposal: -X importcache to supplement -X importtime for loaded modules

[Barry]

Try raising on https://discuss.python.org/ where you are more likely to have a 
python core dev comment on this idea.

Barry

> On 14 Feb 2023, at 15:12, anthony.flury via Python-ideas 
>  wrote:
> 
> 
> Personally,
> I am -1 on the name - I think '-X importtrace' or similar would be better I 
> think.
> 
> I am +100 on the functionality - Unless you put in log messages, tracking 
> imports is complex - so something like this I can see being very useful.
> 
> I know I don't make the decisions, but it seems if this really is only a 
> couple of lines, and all the test cases pass this (or something very similar) 
> should be a considered for 3.12.
> 
> Thank you Noah,
> 
> 
> 
> -- Original Message --
> From: "Noah Kim" 
> To: python-ideas@python.org
> Sent: Saturday, 11 Feb, 23 At 00:07
> Subject: [Python-ideas] Proposal: -X importcache to supplement -X importtime 
> for loaded modules
> 
> All,
> 
> I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X 
> importcache` (open to alternative naming). 
> 
> While `-X importtime` is incredibly useful for analyzing module import times, 
> by design, it doesn't log anything if an imported module has already been 
> loaded. `-X importcache` would provide additional output for every module 
> that's already been loaded: 
> 
> ```
> >>> import uuid
> import time: cached| cached |   _io
> import time: cached| cached |   _io
> import time: cached| cached |   os
> import time: cached| cached |   sys
> import time: cached| cached |   enum
> import time: cached| cached | _io
> import time: cached| cached | _io
> import time: cached| cached | collections
> import time: cached| cached | os
> import time: cached| cached | re
> import time: cached| cached | sys
> import time: cached| cached | functools
> import time: cached| cached | itertools
> import time:   151 |151 | _wmi
> import time: 18290 |  18440 |   platform
> import time:   372 |372 |   _uuid
> import time: 10955 |  29766 | uuid
> ```
> 
> In codebases with convoluted/poorly managed import graphs (and consequently, 
> workloads that suffer from long import times), the ability to record all 
> paths to an expensive dependency--not just the first-imported--can help 
> expedite refactoring (and help scale identification of this type of issue). 
> More generally, this flag would provide a more efficient path to tracking 
> runtime dependencies.
> 
> As a proof of concept, I was able to hack this functionality into `-X 
> importtime` by adding a couple lines to `import_ensure_initialized` in 
> `Python/import.c` (hence the output above). A separate flag is probably 
> desirable to preserve backwards compatibility. 
> 
> Looking forward to your feedback,
> Noah
> 
>  
> ___
> 
> Python-ideas mailing list -- python-ideas@python.org
> 
> To unsubscribe send an email to python-ideas-le...@python.org
> 
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> 
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/
> 
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/ENFJY2RP73IYPUSRVGHBPNELS5JVQLAA/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/W3KUOJXPIZKEAKO5NRDGFY25WNOVAMWZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Deprecate misleading escapes in strings

[Barry]

> On 16 Feb 2023, at 14:57, Arusekk  wrote:
> 
> Hi all!
> 
> I was writing a tutorial on the distinction between bytes and strings
> and why it is important, when I saw the root cause.  People coming from
> C, Perl, Python 2 and similar languages tend to misinterpret "\x90" for
> b"\x90" often.  My idea is that Python could deprecate string literals
> containing any non-ASCII codepoints specified in any way different from
> unicode or unicode escapes (\u, \U, \N).
> 
> (Actually I found that I started having the idea already back in 2021 on
> StackOverflow[1].  The question is an excellent example of what I mean.)
> 
> I would not go so far to follow JSON (disallowing \x11 and \222 escapes
> completely), but while writing "\x00" or "\0" is useful and widely used,
> "\x99" (and especially "\777"!) is probably marginal and definitely less
> explicit than "\u0099" (in the Zen of explicit better than implicit).
> Byte strings do not treat b"\u00ff" as b"\xff".
> 
> In the first part of implementing it, Python could raise a SyntaxWarning
> (or should it be DeprecationWarning? BytesWarning?), suggesting "\x99"
> to either become b"\u0099" or b"\x99", eventually promoting it to some
> equally helpful SyntaxError.  All of it could be hidden behind a feature
> like from __future__ import backslashes (one nice name I can think of).
> 
> The new regular expression for octals would be \\[01]?[0-7]{1,2} and
> \\x[0-7][0-9A-Fa-f] for hexadecimals, hopefully not confusing anyone,
> and not much more complex than the old ones.
> 
> In the meantime, probably between introducing a warning and changing it
> to become an error (the most reasonable timeline I can think of now),
> the default ascii() representation should eventually use the \u0099 form
> for all such codepoints, to keep the invariant of eval(ascii(x)) == x
> without syntax warnings.  repr() is also affected, but it is fortunately
> limited to the [\x80-\xa0\xad] range.  I mean [\u0080-\u00a0\u00ad] :-)
> 
> Another timeline would be to change the repr first, initially hidden
> under an interpreter flag or environment variable, then officially
> deprecate it in the documentation, then introduce the error guarded by
> from __future__ import backslashes or another flag, then make the repr
> use \u by default, then add the warning and finally make it always raise
> an error.
> As a precedent, breaking repr() was not a dealbreaker when introducing
> randomized seeds (even repr({"a", "b"}) is now unpredictable).
> 
> This would be of course a breaking change for a lot of unit tests, and
> stuff like pickle should probably support old syntax, delaying any such
> change until a new protocol comes (if it applies to the newest one ---
> quite sure it does not).  Such a breaking change must be used wisely.
> Other changes to octal escapes could be sneaked in, based on conclusions
> from the 2018 'Python octal escape character encoding "wats"' thread[2]
> (I like writing "\0" and "\4" though, just to make my opinion clear).
> If going the whole hog, the 2015 'Make non-meaningful backslashes
> illegal in string literals' thread[3] could be revived as well, maybe
> even with "\f\v" deprecated, "\e" = "\33" introduced and such.
> 
> Please let me know what you think, what else could break, and is it
> useful anywhere else apart from my use case, and what similar problems
> you have.

-1 i think you will break too much valid code.

This is valid and does not match your rules. ‘\x9b’ that is the ANSI CSI in 
8-bit.
In 7-bit it is ‘\x1b[‘.

Barry

> 
> Cheers,
> Arusekk
> 
> [1]: https://stackoverflow.com/q/64832281/3869724
> [2]: 
> https://mail.python.org/archives/list/python-ideas@python.org/thread/ARBCIPEQB32XBS7T3JMKUDIZ7BZGFTL6/
> [3]: 
> https://mail.python.org/archives/list/python-ideas@python.org/message/PJXKDJQT4XW6ZSMIIK7KAZ4OCDAO6DUT/
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/ITBFU4GPJJVXTHT57WNLASXKL4R4MPF5/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6Z62MV6LN3HIJ2UEPHMGCM25COWOB35U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Deprecate misleading escapes in strings

[Barry]

On 16 Feb 2023, at 19:45, Ben Rudiak-Gould  wrote:On Thu, Feb 16, 2023 at 8:09 AM Barry <ba...@barrys-emacs.org> wrote:This is valid and does not match your rules. ‘\x9b’ that is the ANSI CSI in 8-bit.
In 7-bit it is ‘\x1b[‘.Shouldn't that be b‘\x9b’?
No. I can work in unicode internally and only encode to bytes for output.Barry___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DL73X7VOXRSREL4MAKA3TMYDW647IJEG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: join() could add separators at start or end

[Barry]

> On 9 Mar 2023, at 00:37, Rob Cliffe via Python-ideas 
>  wrote:
> 
> Having had my last proposal shot down in flames, up I bob with another. 😁

See this discussion that has a nice solution proposed with the concat function.

Barry

> It seems to me that it would be useful to be able to make the str.join() 
> function put separators, not only between the items of its operand, but also 
> optionally at the beginning or end.
> E.g. '|'.join(('Spam', 'Ham', 'Eggs')) returns
> 'Spam|Ham|Eggs'
> but it might be useful to make it return one of
> '|Spam|Ham|Eggs'
> 'Spam|Ham|Eggs|'
> '|Spam|Ham|Eggs|'
> Again, I suggest that this apply to byte strings as well as strings.
> Going through the 3.8.3 stdlib I have found
> 24 examples where the separator needs to be added at the beginning
> 52 where the separator needs to be added at the end
>  4 where the separator needs to be added at the both ends
> I list these examples below.  Apologies if there are any mistakes.
> 
> While guessing is no substitute for measurement, it seems plausible that 
> using this feature
> where appropriate would increase runtime performance by avoiding 1 (or 2) 
> calls of str.__add__.
> This is perhaps more relevant when the separator is not a short constant 
> string,
> as in this example:
> Lib\email\_header_value_parser.py:2854:return 
> policy.linesep.join(lines) + policy.linesep
> Note also this example:
> Lib\site-packages\setuptools\command\build_ext.py:221: pkg = 
> '.'.join(ext._full_name.split('.')[:-1] + [''])
> where the author has used the unintuitive device of appending an empty string
> to a list to force join() to add an extra final dot, thereby avoiding 1 call 
> of str.__add__
> at the cost of 1 call of list.append.
> 
> What I cannot decide is what the best API would be.
> str.join() currently takes only 1 parameter, so it would be possible to add 
> an extra parameter or two.
> One scheme would be to have an atEnds parameter which could take values such 
> as
> 0=default behaviour  1=add sep at start  2=add sep at end  3=add sep at 
> both ends
> or
> 's'=add sep at start  'e'=add sep at end  'b'=add sep at both ends  
> (some) other=default behaviour
> Another would be to have 2 parameters, atStart and atEnd, which would both 
> default to False or 0.  E.g.
> '|'.join(('Spam', 'Ham', 'Eggs'), 1)== '|Spam|Ham|Eggs'
> '|'.join(('Spam', 'Ham', 'Eggs'), 0, 1) == 'Spam|Ham|Eggs|'
> Neither scheme results in particularly transparent usage, though no worse than
> s.splitlines(True) # What on earth is this parameter???
> 
> Corner case:
> What if join() is passed an empty sequence?  This is debatable,
> but I think it should return the separator if requested to add it
> at the beginning or end, and double it up if both are requested.
> This would preserve identities such as
> sep.join(seq, ) == sep + sep.join(seq)
> 
> Best wishes
> Rob Cliffe
> 
> EXAMPLES WHERE SEPARATOR ADDED AT START:
> 
> Lib\http\server.py:933:splitpath = ('/' + '/'.join(head_parts), tail_part)
> Lib\site-packages\numpy\ctypeslib.py:333:name += "_"+"_".join(flags)
> Lib\site-packages\numpy\testing\_private\utils.py:842: err_msg += '\n' + 
> '\n'.join(remarks)
> Lib\site-packages\pip\_vendor\pyparsing\core.py:2092-2095:
> out = [
> "\n" + "\n".join(comments) if comments else "",
> pyparsing_test.with_line_numbers(t) if with_line_numbers else 
> t,
> ]
> Lib\site-packages\pip\_vendor\requests\status_codes.py:121-125:
> __doc__ = (
> __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes))
> if __doc__ is not None
> else None
> )
> Lib\site-packages\reportlab\lib\utils.py:1093: self._writeln(' '+' 
> '.join(A.__self__))
> Lib\site-packages\reportlab\platypus\flowables.py:708:L = 
> "\n"+"\n".join(L)
> Lib\site-packages\twisted\mail\smtp.py:1647: r.append(c + b' ' +  b' 
> '.join(v))
> Lib\site-packages\twisted\protocols\ftp.py:1203:return (PWD_REPLY, 
> '/' + '/'.join(self.workingDirectory))
> Lib\site-packages\twisted\runner\procmon.py:424-426:
> return ('<' + self.__class__.__name__ + ' '
> + ' '.join(l)
>

[Python-ideas] Re: Make decorator-based context managers reentrant

[Barry]

> On 30 Mar 2023, at 08:10, Tobias Bengfort  wrote:
> 
> Hi,
> 
> I was wondering why decorator-based context managers are not reentrant or 
> even reusable.

Do you mean this use of reentrant? 
https://en.wikipedia.org/wiki/Reentrancy_(computing)


> 
> They could be made reusable by initializing the generator in __enter__() 
> instead of __init__(). They could be made reentrant by storing generatos in a 
> stack. A similar thing has been implemented for redirect_stdout in 
> 8e113b418df7d0c8480e1e2de29a385e1f31b15b.
> 
> I understand that not all generator-based context managers can be reentrant 
> by their nature. But currently it is impossible to make them reentrant, which 
> IMHO is an unnecessary restriction.
> 
> I could find very little discussion on this. Are there any strong reasons why 
> it was implemented this way or should I create a pull request to change it?
> 
> (My specific use case is a TUI application where I want to make sure that the 
> terminal is restored to a usable state on exit, but also need to step in and 
> out of the TUI on SIGTSTP/SIGCONT.)
> 
> thanks,
> tobias
> 
> https://docs.python.org/3/library/contextlib.html#single-use-reusable-and-reentrant-context-managers
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/K7M62JLBCR5USTJV5IRILJNQEYSJQ4QY/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/C7DQF5ZXLFVLVOJUCEPLHJ4MH4J5I2MV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Auto dedent -c arguments.

[Barry]

On 10 Apr 2023, at 02:18, Jonathan Crall  wrote:There's no question that there are lots of ways you can work around the issue (I think the `if 1:` method is the easiest and most portable, but it still is boilerplate which can be confusing if you don't know why it's there). The question is: would enough people benefit from this to add the feature to CPython?A PR that has an implementation might push this forward.I wonder if in the C code of -c treating a string that starts with \n could trigger the dedent?BarryOn Sun, Apr 9, 2023 at 4:09 PM Barry Scott <ba...@barrys-emacs.org> wrote:
  

  
  


On 04/04/2023 15:18, Jonathan Crall
  wrote:


  
  I have what I think is a fairly low impact quality
of life improvement to suggest for the python CLI.


When I'm not working in Python I tend to be working in
  bash. But often I want to break out and do something quick in
  Python. I find the `python -c ` CLI very useful for this. For
  one liners it's perfect. E.g.



NEW_VAR=$(python -c "import pathlib;
print(pathlib.Path('$MYVAR').parent.parent)")


And even if I want to do something multi-line it's pretty
  easy


NEW_VAR=$(python -c "
import pathlib
for _ in range(10):
    print('this is a demo, bear with
me')
")


But the problem is when I'm writing bash inside a function
  or some other nested code, I would like to have nice
  indentation in my bash file, but if I write something like
  this:

  
mybashfunc(){
    python -c "
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    "
}



I get `IndentationError: unexpected indent`. 


This means I have to write the function ugly like this:
  
  mybashfunc(){
    python -c "
import pathlib
for _ in range(10):
    print('this is a demo, bear with me')
"
}



Or use a helper function like this:
  
  codeblock()
{
    __doc__='
    copy-pastable implementation
    Prevents indentation errors in bash
    '
    echo "$1" | python -c "import sys; from textwrap import
dedent; print(dedent(sys.stdin.read()).strip('\n'))"
}
  

  
mybashfunc(){
    python -c $(codeblock "
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    ")
}



Or more recently I found that this is a low-impact
  workaround: 
  
mybashfunc(){
    python -c "if 1:
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    "
}



But as a certain Python dev may say: "There must be a
  better way."


Would there be any downside to the Python CLI automatically
  dedenting the input string given to -c? I can't think of any
  case off the top of my head where it would make a previously
  valid program invalid. Unless I'm missing something this would
  strictly make previously invalid strings valid.


Thoughts?
  

You can solve this with a small module. I named mine run_dedent
mkdir -p run_dedent
  echo pass >run_dedent/__init__.py
  cat <run_dedent/__main__.py
  import sys
  import textwrap
  cmd = textwrap.dedent(sys.argv[1])
  exec(cmd)
  EOF
    python3 -m run_dedent "
      import sys
          print(sys.version_info)
          "
Barry



  


-- 
  

  

  -Dr. Jon Crall (him)

  

  

  
  
  
  ___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/

[Python-ideas] Re: XDG Directory Support for PDB

[Barry]

> On 18 Apr 2023, at 14:20, Gustaf Waldemarson  
> wrote:
> 
> Hello,
> 
> I don't really use the Python debugger all that much, but recently I used it
> quite a bit for a small side-project. That said, it kept annoying me that 
> there
> were no history associated with the prompt, so I added a few lines to the
> `pdbrc` config file to store that for me. However, it seems like this file 
> must
> either exist in the user home directory, or the current working directory,
> neither of which were very convenient for me.
> 
> Additionally, I strongly dislike polluting my home directory with dotfiles, 
> thus
> I did a bit of researching to see if this could be fixed. Thus, below is a 
> small
> patch to PDB that adds support for **also** reading from configuration files 
> in
> some XDG directories (currently ${XDG_CONFIG_HOME}/python/pdbrc).
> 
> So I guess the question is, would anyone else find this useful enough to the
> point were I should write a PEP for it? As far as I can tell, Python haven't
> been using the XDG directories, so I suppose the naming of the subdirectory
> is really subject to a larger discussion.
> 
>From d4ea8fbba3ba6c7538ccecad2a32e6b3b2059e5d Mon Sep 17 00:00:00 2001
>From: Gustaf Waldemarson 
>Date: Thu, 13 Apr 2023 11:39:00 +0200
>Subject: [PATCH] Allow users to store PDB configuration in XDG directories.
> 
>---
> Lib/pdb.py | 22 +++---
> 1 file changed, 11 insertions(+), 11 deletions(-)
> 
>diff --git a/Lib/pdb.py b/Lib/pdb.py
>index a3553b345a..3f4ffd7719 100755
>--- a/Lib/pdb.py
>+++ b/Lib/pdb.py
>@@ -231,19 +231,19 @@ def __init__(self, completekey='tab', stdin=None, 
> stdout=None, skip=None,
> self.allow_kbdint = False
> self.nosigint = nosigint
> 
>-# Read ~/.pdbrc and ./.pdbrc
>+# Read user configuration from a set of paths.
> self.rcLines = []
> if readrc:
>-try:
>-with open(os.path.expanduser('~/.pdbrc'), 
> encoding='utf-8') as rcFile:
>-self.rcLines.extend(rcFile)
>-except OSError:
>-pass
>-try:
>-with open(".pdbrc", encoding='utf-8') as rcFile:
>-self.rcLines.extend(rcFile)
>-except OSError:
>-pass
>+for rcFileName in ('${XDG_CONFIG_HOME}/python/pdbrc',
>+   '~/.config/python/pdbrc',
>+   '~/.pdbrc',
>+   '.pdbrc'):
I would expect .pdbrc to be first.

>+rcPath = 
> os.path.expanduser(os.path.expandvars(rcFileName))
>+try:
>+with open(rcPath, encoding='utf-8') as rcFile:
>+self.rcLines.extend(rcFile)
>+    except OSError:
>+pass
> 
> self.commands = {} # associates a command list to breakpoint 
> numbers
> self.commands_doprompt = {} # for each bp num, tells if the prompt

Try raising a python bug and a PR with this code in it.
Seems reasonable change to support.

Barry

>-- 
>2.25.1
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/HWWWESO3KUCU54OU5FISD3RKLEZMKBP7/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WCO4GTZUYTHQ4CVS72OWEPEP22RO7PB7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: XDG Directory Support for PDB

[Barry]

On 20 Apr 2023, at 14:24, Gustaf Waldemarson  wrote:Interesting, are you both sure ./.pdbrc should be the first file? I was expectingthe order to always go from "most general" to "most specific", i.e.,"system" -> "home" -> "current". This way, it would be straightforwardto have general configuration in a "system"/"home" configuration which canbe overwritten in a more specialized one ("current").Oh i see you read all the files.Usually you need to stop looking once you find an rc file.BarryCoincidentally, this is also the order GDB is using:   https://sourceware.org/gdb/current/onlinedocs/gdb#StartupAny particular reason you guys prefer a different order?
> Not your job, but I bet you get "are there other stdlib applications 
> that would benefit from this change?" as feedback.  Eg, IDLE.Probably, but lets start the discussion somewhere!Best regards,GustafDen ons 19 apr. 2023 kl 17:11 skrev turnbull <turnb...@sk.tsukuba.ac.jp>:I apologize for the top-post and awkward trimming, my employer just 
switched to o365 and won't accomodate my existing mail client so I'm 
stuck with a crappy webmail client on this address. :-p

I agree with Barry on both points: seems like a good idea, but the 
search priority should be
./.pdbrc first.

Not your job, but I bet you get "are there other stdlib applications 
that would benefit from this change?" as feedback.  Eg, IDLE.

2023-04-19 02:50 に Barry さんは書きました:

>>    +            for rcFileName in ('${XDG_CONFIG_HOME}/python/pdbrc',
>>    +                               '~/.config/python/pdbrc',
>>    +                               '~/.pdbrc',
>>    +                               '.pdbrc'):
> I would expect .pdbrc to be first.
> 
>>    +                rcPath = 
>> os.path.expanduser(os.path.expandvars(rcFileName))
>>    +                try:
>>    +                    with open(rcPath, encoding='utf-8') as rcFile:
>>    +                        self.rcLines.extend(rcFile)
>>    +                except OSError:
>>    +                    pass
>> 
>>             self.commands = {} # associates a command list to 
>> breakpoint numbers
>>             self.commands_doprompt = {} # for each bp num, tells if 
>> the prompt
> 
> Try raising a python bug and a PR with this code in it.
> Seems reasonable change to support.
> 
> Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/45VRIN5V7ZJLG7HZC5P57VHQVMQ3JWCM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: XDG Directory Support for PDB

[Barry]

On 20 Apr 2023, at 16:52, Gustaf Waldemarson  wrote:> Oh i see you read all the files.> Usually you need to stop looking once you find an rc file.I guess that really depends on the program, but since pdb is modelled ongdb, I figured it made sense to do something similar :)Turns out the gdbinit behaviour is complex.From https://sourceware.org/gdb/onlinedocs/gdb/Initialization-Files.html#Initialization-Files2.1.4.3 Home directory initialization fileAfter loading the system wide initialization files GDB will look for an initialization file in the users home directory2. There are a number of locations that GDB will search in the home directory, these locations are searched in order and GDB will load the first file that it finds, and subsequent locations will not be checked.On non-Apple hosts the locations searched are:$XDG_CONFIG_HOME/gdb/gdbinit$HOME/.config/gdb/gdbinit$HOME/.gdbinitThen ./.gdbinit is loaded if . Is not ~/And you need different logic for macOS.Anyways, I'll prep a pull-request for the cpython repo, should be interestingto see what comes up.Best regards,GustafDen tors 20 apr. 2023 kl 17:05 skrev Barry <ba...@barrys-emacs.org>:On 20 Apr 2023, at 14:24, Gustaf Waldemarson <gustaf.waldemar...@gmail.com> wrote:Interesting, are you both sure ./.pdbrc should be the first file? I was expectingthe order to always go from "most general" to "most specific", i.e.,"system" -> "home" -> "current". This way, it would be straightforwardto have general configuration in a "system"/"home" configuration which canbe overwritten in a more specialized one ("current").Oh i see you read all the files.Usually you need to stop looking once you find an rc file.BarryCoincidentally, this is also the order GDB is using:   https://sourceware.org/gdb/current/onlinedocs/gdb#StartupAny particular reason you guys prefer a different order?
> Not your job, but I bet you get "are there other stdlib applications 
> that would benefit from this change?" as feedback.  Eg, IDLE.Probably, but lets start the discussion somewhere!Best regards,GustafDen ons 19 apr. 2023 kl 17:11 skrev turnbull <turnb...@sk.tsukuba.ac.jp>:I apologize for the top-post and awkward trimming, my employer just 
switched to o365 and won't accomodate my existing mail client so I'm 
stuck with a crappy webmail client on this address. :-p

I agree with Barry on both points: seems like a good idea, but the 
search priority should be
./.pdbrc first.

Not your job, but I bet you get "are there other stdlib applications 
that would benefit from this change?" as feedback.  Eg, IDLE.

2023-04-19 02:50 に Barry さんは書きました:

>>    +            for rcFileName in ('${XDG_CONFIG_HOME}/python/pdbrc',
>>    +                               '~/.config/python/pdbrc',
>>    +                               '~/.pdbrc',
>>    +                               '.pdbrc'):
> I would expect .pdbrc to be first.
> 
>>    +                rcPath = 
>> os.path.expanduser(os.path.expandvars(rcFileName))
>>    +                try:
>>    +                    with open(rcPath, encoding='utf-8') as rcFile:
>>    +                        self.rcLines.extend(rcFile)
>>    +                except OSError:
>>    +                    pass
>> 
>>             self.commands = {} # associates a command list to 
>> breakpoint numbers
>>             self.commands_doprompt = {} # for each bp num, tells if 
>> the prompt
> 
> Try raising a python bug and a PR with this code in it.
> Seems reasonable change to support.
> 
> Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GSAXO2JZK4EAWOLTM5SLDYJKQSBB4KJW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: XDG Directory Support for PDB

[Barry]

On 20 Apr 2023, at 16:52, Gustaf Waldemarson  wrote:> Oh i see you read all the files.> Usually you need to stop looking once you find an rc file.I guess that really depends on the program, but since pdb is modelled ongdb, I figured it made sense to do something similar :)Turns out the gdbinit behaviour is complex.2.1.4.3 Home directory initialization fileAfter loading the system wide initialization files GDB will look for an initialization file in the users home directory2. There are a number of locations that GDB will search in the home directory, these locations are searched in order and GDB will load the first file that it finds, and subsequent locations will not be checked.On non-Apple hosts the locations searched are:$XDG_CONFIG_HOME/gdb/gdbinit$HOME/.config/gdb/gdbinit$HOME/.gdbinitThen ./.gdbinit is loaded if . Is not ~/And you need different logic for macOS.Anyways, I'll prep a pull-request for the cpython repo, should be interestingto see what comes up.Best regards,GustafDen tors 20 apr. 2023 kl 17:05 skrev Barry <ba...@barrys-emacs.org>:On 20 Apr 2023, at 14:24, Gustaf Waldemarson <gustaf.waldemar...@gmail.com> wrote:Interesting, are you both sure ./.pdbrc should be the first file? I was expectingthe order to always go from "most general" to "most specific", i.e.,"system" -> "home" -> "current". This way, it would be straightforwardto have general configuration in a "system"/"home" configuration which canbe overwritten in a more specialized one ("current").Oh i see you read all the files.Usually you need to stop looking once you find an rc file.BarryCoincidentally, this is also the order GDB is using:   https://sourceware.org/gdb/current/onlinedocs/gdb#StartupAny particular reason you guys prefer a different order?
> Not your job, but I bet you get "are there other stdlib applications 
> that would benefit from this change?" as feedback.  Eg, IDLE.Probably, but lets start the discussion somewhere!Best regards,GustafDen ons 19 apr. 2023 kl 17:11 skrev turnbull <turnb...@sk.tsukuba.ac.jp>:I apologize for the top-post and awkward trimming, my employer just 
switched to o365 and won't accomodate my existing mail client so I'm 
stuck with a crappy webmail client on this address. :-p

I agree with Barry on both points: seems like a good idea, but the 
search priority should be
./.pdbrc first.

Not your job, but I bet you get "are there other stdlib applications 
that would benefit from this change?" as feedback.  Eg, IDLE.

2023-04-19 02:50 に Barry さんは書きました:

>>    +            for rcFileName in ('${XDG_CONFIG_HOME}/python/pdbrc',
>>    +                               '~/.config/python/pdbrc',
>>    +                               '~/.pdbrc',
>>    +                               '.pdbrc'):
> I would expect .pdbrc to be first.
> 
>>    +                rcPath = 
>> os.path.expanduser(os.path.expandvars(rcFileName))
>>    +                try:
>>    +                    with open(rcPath, encoding='utf-8') as rcFile:
>>    +                        self.rcLines.extend(rcFile)
>>    +                except OSError:
>>    +                    pass
>> 
>>             self.commands = {} # associates a command list to 
>> breakpoint numbers
>>             self.commands_doprompt = {} # for each bp num, tells if 
>> the prompt
> 
> Try raising a python bug and a PR with this code in it.
> Seems reasonable change to support.
> 
> Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZWIKD7QFIWTWTFPB6RAFJTZEYAHQZCU4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Thoughts on allowing Path objects in shlex.join

[Barry]

> On 12 May 2023, at 18:54, mitchell.negus...@gmail.com wrote:
> 
> At the moment, `shlex.join` raises 
> 
> ```
> TypeError: expected string or bytes-like object
> ```
> 
> when given `pathlib.Path` objects. Since `subprocess.run` and related 
> subprocess commands allow `Path` since Python 3.6 and 3.8 for Windows, it 
> seems intuitive to me that `shlex.join` be able to handle paths as well. This 
> has been briefly discussed before on the bug tracker (issue 89293, link 
> below), but with the caveat that it was the wrong venue, and should be 
> brought up on this list instead. That said, I can't find any record that it 
> ever made it over here, but I am curious to get this community's perspective. 
> 
> My understanding of arguments against in that discussion were that 
> `shlex.join` should not implicitly convert its arguments to strings, and that 
> `Path` objects were not special enough for an exception. However (and as 
> noted over there), `subprocess` takes the exact opposite approach, and has an 
> explicit `isinstance` check to allow and convert `os.PathLike` objects. I'm 
> assuming that some of the reasoning behind allow path-to-string conversions 
> there has to do with the prevalence of using paths as command line arguments, 
> and that seems to me like it could be rationale enough for why `Path` objects 
> warrant an exception to the "don't-convert-inputs-to-strings" unwritten rule 
> for `shlex.join`. 
> 
> Without getting into the specific details of implementation, I'd imagine that 
> changing this to just convert `os.PathLike` to strings in `shlex.join` would 
> not break much existing code. I find it rather unlikely—though I'll admit I 
> have no supporting evidence—that anyone is relying on `shlex.join` to catch 
> path objects that should be strings; I'd bet instead anyone using paths 
> already is just converting them to strings "manually" before passing them to 
> `shlex.join`.
> 
> I'm curious to hear more opinions on this though, since I'm guessing there 
> may be very valid reasons that others have for not wanting to move in this 
> direction that I'm totally oblivious to.
> 
> Thanks all!
> 
> 
> Issue 89283: https://github.com/python/cpython/issues/89293

You may find that you need to raise on as an idea on 
https://discuss.python.org/ as that is where the python code dev are more 
likely to read this idea.

Barry
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/UJYYYI3VQKXN46X5QQTAAS5RORIXJSMF/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4Q4DSM2IXXAH5ITF3J5ZFP34RMPCODZI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Mark a set of keys of a TypedDict as incompatible

[Barry]

On 17 May 2023, at 14:16, Sync In  wrote:I have a couple of {{{TypedDict}}}s for use in type hinting that are supposed to have some sets of incompatible keys.For example, if I need to specify that a particular {{{dict}}} can have {{{bar}}} or {{{baz}}} but not both of them, I can write two {{{TypedDict}}}s:{{{#!highlight pythonfrom typing import TypedDictclass Foo1(TypedDict):  bar: strclass Foo2(TypedDict):  baz: intFoo = Foo1 | Foo2foo_instance_1: Foo = {  # Works fine  'bar': 'foobar'}foo_instance_2: Foo = {  # Also fine  'baz': 42}foo_instance_3: Foo = {  # Warning: Expected type 'Foo1 | Foo2', got 'dict[str, str | int]' instead  'bar': 'foobar',  'baz': 42However, this doesn't scale very well if I were to have multiple sets; for example, a simple multiplication shows that three sets of 2, 3 and 4 keys each will result in 24 {{{TypedDict}}}s. A solution would be:{{{#!highlight pythonclass Foo(TypedDict):  bar: IncompatibleWith('baz', 'qux')[str]  baz: IncompatibleWith('bar', 'qux')[int]  qux: IncompatibleWith('bar', 'baz')[bool]}}}...or, with a decorator somewhat similar to {{{@overload}}}:{{{#!highlight python@incompatible('bar', 'baz', 'qux')# ...class Foo(TypedDict):  bar: str  baz: int  qux: bool}}}I asked this question on StackOverflow and was informed that such a feature doesn't exist. How about adding it to Python then?You may get a reply here but all the python type discussions take place on https://discuss.python.org these days. Suggest you ask there.Barry___Python-ideas mailing list -- python-ideas@python.orgTo unsubscribe send an email to python-ideas-le...@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TPILQMOHCWAD7GKDCREONOHZTFY5Z7YW/Code of Conduct: http://python.org/psf/codeofconduct/___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KXX2FASV4CXN2PSMC2RPX2NL5YH25WZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a .whitespace property to module unicodedata

[Barry]

> On 1 Jun 2023, at 19:10, David Mertz, Ph.D.  wrote:
> 
> %time unicode_whitespace = [chr(c) for c in range(0x) if 
> unicodedata.category(chr(c)) == "Zs"]

Try 0x10 to get all of unicode. 

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HMB57XHRGWRUJE4ZMULNBOGKD3ILH2DY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Warn when iterating over an already exhausted generator

[Barry]

> On 12 Jun 2023, at 16:55, BoppreH via Python-ideas  
> wrote:
> 
> Then the empty list creates hard-to-track bugs. I'm familiar with the 
> iterator protocol and why the behavior above happens, but couldn't it be 
> prevented?

I don’t think so. It is not always a bug that an iterator is empty.
For example this pattern:

args = iter(sys.argv)
progname = next(args)
mandatory_arg = next(args)
for arg in args:
 print(‘optional arg’, next(arg))

Your proposal will traceback for the for loop if there are no optional args.

Barry



___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/K4K32L226IAC22NOJLLWIHKC3WWF7SPG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Proposal for get_or function in Python dictionaries

[Barry]

On 15 Jul 2023, at 23:14, Dom Grigonis  wrote:I am well aware of that.I just argue that it is not very pleasant to read.Personally, I am not a big user of it.C’s although maybe doesn’t have words to read, but is much more pleasant in both conciseness and logical sequence.Maybe it is not a big issue for others, but if C’s style conditional _expression_ existed, it would significantly change the way I write things.I agree with you and avoid the python version, and refactor it out of code.As you say ii also find it hard to read, and familiarity is not helping.The C and rust versions I do use.BarrySo just checking. :)On 16 Jul 2023, at 00:06, Paul Moore  wrote:On Sat, 15 Jul 2023 at 21:09, Dom Grigonis  wrote:So I would vote for something similar to:result = bar is None ? default : barresult = default if bar is None else barPython has a conditional _expression_ already.Paul 
___Python-ideas mailing list -- python-ideas@python.orgTo unsubscribe send an email to python-ideas-le...@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QARSVXWITOLFAXZZJLIEBSPFO35PRDDS/Code of Conduct: http://python.org/psf/codeofconduct/___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QUUEWVIEBABWDJCCPCSQYNIGDS2MYGC4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: built in to clear terminal

[Barry Scott]

The simplest answer is

print('`\x1b[2J\x1b[H')

Are there any terminals that this does not work on that are in active use?

Is using curses that uses termcap needed these days?

Of course Windows is the outlier, but the new Windows Terminal
supports ANSI escapes sequences and utf-8.

I tested the above with Windows Terminal 1.4 on Windows 10
and it just works.

Otherwise os.system('cls') works for windows terminal and the old
windows console stuff.

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NPB5HWQ3YHPXUVBEGBDMBMLJRJ3FWOEC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: built in to clear terminal

[Barry Scott]

> On 22 Dec 2020, at 09:49, Barry Scott  wrote:
> 
> The simplest answer is
> 
>   print('`\x1b[2J\x1b[H')
> 
> Are there any terminals that this does not work on that are in active use?
> 
> Is using curses that uses termcap needed these days?
> 
> Of course Windows is the outlier, but the new Windows Terminal
> supports ANSI escapes sequences and utf-8.
> 
> I tested the above with Windows Terminal 1.4 on Windows 10
> and it just works.
> 
> Otherwise os.system('cls') works for windows terminal and the old
> windows console stuff.

It turns out that on Windows 10 it works for old console API if you do this:

-
import sys

def clear_terminal():
if sys.platform == 'win32':
import ctypes
kernel32 = ctypes.windll.kernel32
# turn on the console ANSI colour handling
kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)

sys.stdout.write('\x1b[2J' '\x1b[H')
-

The above should work in all but the old none ANSI terminals.

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7KA7OGZKLFG6DQDV7VO7DGKOYRG72Y5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: built in to clear terminal

[Barry Scott]

> On 22 Dec 2020, at 12:39, Eryk Sun  wrote:
> 
> On 12/22/20, Barry Scott  wrote:
>> 
>> import sys
>> 
>> def clear_terminal():
>>if sys.platform == 'win32':
>>import ctypes
>>kernel32 = ctypes.windll.kernel32
>># turn on the console ANSI colour handling
>>kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)
>> 
>>sys.stdout.write('\x1b[2J' '\x1b[H')
> 
> Here are some concerns I have:
> 
> * Does not support Windows 8
> * Does not support legacy console in Windows 10 (on the "options" tab)
> * Does not check for SetConsoleMode failure
> * Does not support a different active screen buffer
> * Assumes StandardOutput is a screen buffer for the current console
> * Assumes the current mode of the screen buffer is 3 or 7. New modes
> have been added, and even more may be added
> * Sets a global console setting that persists after Python exits
> 
snip...

Eryk,

Does this work for you:

import sys, os

def clear():
if sys.playform == 'win32':
os.system('cls')

else:
sys.stdout.write('\x1b[2J' '\x1b[H')


No need to address the list above because CLS does it.
No concern about CLS being a trojan becuse as a builtin to CMD and PowerShell
and it takes priority over cls.bat etc.

For all other platforms I'm assuming the VT100 level of ANSI escape sequences
is supported.

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5SCM3VKDQORXYAEWBURP7IULSUYB6RUR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: fsync-on-close io object

[Barry Scott]

> On 24 Dec 2020, at 17:15, Michael A. Smith  wrote:
> 
> With all the buffering that modern disks and filesystems do, a
> specific question has come up a few times with respect to whether or
> not data was actually written after flush. I think it would be pretty
> useful for the standard library to have a variant in the io module
> that would explicitly fsync on close.
> 
> You might be tempted to argue that this can be done very easily in
> Python already, so why include it in the standard io module?
> 
> 1. It seems to me that it would be better to do this in C, so for the
> folks who need to make a consistency > performance kind of choice,
> they don't have to sacrifice any additional performance.
> 2. Having it in the io library will call attention to this issue,
> which I think is something a lot of folks don't consider. Assuming
> that `close` or `flush` are sufficient for consistency has always been
> wrong (at its limits), but it was less likely to be a stumbling block
> in the past, when buffering was less aggressive and less layered and
> the peak size and continuous-ness of data streams was a more niche
> concern.
> 3. There are many ways to do this, and I think several of them could
> be subtly incorrect. In other words, maybe it can't be done very
> easily and correctly after all. Providing "obviously right" ways to do
> things is the baileywick of the standard library, isn't it?

I have used rename to make a new file appear atomically after it is closed
and I have used fsync to ensure records are on disk before a file is closed.

I've not needed fsync on close yet.

What is the use case that needs this?

Without a compelling use case I'm -1 on this.

Barry



> 
> Thanks for your consideration and feedback,
> Michael Smith
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/UOUS4BNSI6WFBPWMXHJF5IEAV2PQ47VN/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CNIIRBIEXL46FEUTVHSDU63ASBMHE72X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Off-topic: What costs NaN pounds for a null amount?

[Barry Scott]

> On 14 Jan 2021, at 17:14, Jonathan Fine  wrote:
> 
> Hi
> 
> There's interest here in arithmetic operations on NaN . I've just seen a 
> product listed as costing NaN pounds to buy a null amount. That was written 
> as £NaN/null.
> 
> The bargain item is Glade Shake & Vacuum Citrus, and you can see it at
> https://www.tesco.com/groceries/en-GB/products/253732570 
> <https://www.tesco.com/groceries/en-GB/products/253732570>
> 
> Searching on this site for 'NaN' brings up many entries for 
> https://en.wikipedia.org/wiki/Naan <https://en.wikipedia.org/wiki/Naan> bread.

Nice to know that there are well tested web sites out there.

Do you recall seeing sites in 2000 that showed the year date as:

06-01- 19100

Barry


> 
> with best regards
> 
> Jonathan
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/FIP7EUZ3E3KNCCK7XVWAS3GJY6KMX6C2/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LQ3LB7653DK7VLQYIUWBP6V3R5YZES4H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: PEP Draft: Build Dependency Specification for Manylinux Wheels

[Barry Scott]

> On 4 Jan 2021, at 14:27, Chris Antonellis  
> wrote:
> 
> `system_dependencies`
> -
> System dependencies to install with `yum` prior to building. 
> Entries are expected to be in `yum` `name-version` format.

If dnf is avaiable I think it is better to use dnf over yum.

So your intention is to install deps so that a package can to source compiler 
and linked?

That will only work for root.

You might want to output the yum/dnf command and ask the user to run that 
command
then retry the install.

Also not that you will need to know for each distro where to find the deps.
I do not think you can rely of package names across distros.

> 
> `environment_variables`
> ---
> Environment variables to set prior to building.
> 
> `python_dependencies`
> -
> Python libraries to install with `pip` prior to building.
> Will be installed for each version of python available in 
> `/opt/python/`.

At least with Centos 8 and fedora its /usr that python and its libs are 
installed into not /opt.
Are you thinking of SCL here?

Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/56QLHPL6DR2AO6UB24PS77D62LE75KRR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Adding `open_text()` builtin function. (relating to PEP 597)

[Barry Scott]

> On 23 Jan 2021, at 11:00, Steven D'Aprano  wrote:
> 
> On Sat, Jan 23, 2021 at 12:40:55AM -0500, Random832 wrote:
>> On Fri, Jan 22, 2021, at 20:34, Inada Naoki wrote:
>>> * Default encoding is "utf-8".
>> 
>> it might be worthwhile to be a little more sophisticated than this.
>> 
>> Notepad itself uses character set detection [it might not be 
>> reasonable to do this on the whole file as notepad does, but maybe the 
>> first 512 bytes, or the result of read1(512)?] when opening a file of 
>> unknown encoding, and msvcrt's "ccs=UTF-8" option to fopen will at 
>> least detect at the presence of UTF-8 and UTF-16 BOMs [and treat the 
>> file as UTF-16 in the latter case].
> 
> 
> I like Random's idea. If we add a new "open text file" builtin function, 
> we should seriously consider having it attempt to auto-detect the 
> encoding. It need not be as sophisticated as `chardet`.

I think that you are going to create a bug magnet if you attempt to auto
detect the encoding.

First problem I see is that the file may be a pipe and then you will block
until you have enough data to do the auto detect.

Second problem is that the first N bytes are all in ASCII and only later
do you see Windows code page signature (odd lack of utf-8 signature).

> That auto-detection behaviour could be enough to differentiate it from 
> the regular open(), thus solving the "but in ten years time it will be 
> redundant and will need to be deprecated" objection.
> 
> Having said that, I can't say I'm very keen on the name "open_text", but 
> I can't think of any other bikeshed colour I prefer.

Given the the functions purpose is to open unicode text use a name that
reflects that it is the encoding that is set not the mode (binary vs. text).

open_unicode maybe?

If you are teaching open_text then do you also need to have open_binary?

Barry

> 
> 
> -- 
> Steve
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/VAWFPIAA4WIVLIF4LFJ4OATJK6JDJS2N/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4LHLZ5QIBOCLIZUVYQ2UXAU6MEX6VMJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 30 Jan 2021, at 12:05, Inada Naoki  wrote:
> 
>> Where would Python look for a "configuration file like `pyvenv.cfg`" ?
> 
> I am not a Windows expert so I am not sure. But I think it should be
> the same directory where `python.exe` is in.


You can put the system default there but each user needs to have a file that 
they can control
to set the per user config.

py.exe uses %LOCALAPPDATA%\py.ini

I'd suggest that you could have a %LOCALAPPDATA%\python.ini.

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4UUIHD7Z7REJ63Y2NXMFKHKGP3GRL6KN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 2 Feb 2021, at 00:22, Inada Naoki  wrote:
> 
> On Tue, Feb 2, 2021 at 6:31 AM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
>> 
>>> On 30 Jan 2021, at 12:05, Inada Naoki  wrote:
>>> 
>>> Where would Python look for a "configuration file like `pyvenv.cfg`" ?
>>> 
>>> I am not a Windows expert so I am not sure. But I think it should be
>>> the same directory where `python.exe` is in.
>> 
>> You can put the system default there but each user needs to have a file that 
>> they can control
>> to set the per user config.
>> 
>> py.exe uses %LOCALAPPDATA%\py.ini
>> 
>> I'd suggest that you could have a %LOCALAPPDATA%\python.ini.
>> 
> 
> But what happen if user installed Python from python.org <http://python.org/> 
> and Python from conda?
> User may have two or more Pythons having the same version.

Apple showed the way by using reversed FQD's.

Python.org <http://python.org/> would use org.python.python.ini
Conda.io <http://conda.io/> would use io.conda.python.ini
barry-emacs.org <http://barry-emacs.org/>'s python would use 
org.barrys-emacs.python.python.ini

Further we would need to support multiple versions of python from the same org 
installed side-by-side.

Structure the .ini so that it has default settings and version specific 
settings.

---
[default]
utf8_mode = true

[3.8-64]
utf8-mode = false
---


> 
> In my idea, if user can not change the config of system installed
> Python, user still can create venv and change the setting for the
> venv.

That fine for the venv users but does not help the people that do not need/want 
to use venv.

Barry


> 
> -- 
> Inada Naoki  mailto:songofaca...@gmail.com>>

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UWFIHR5AOIO56RTJMLBBKUXW2FNJ67KR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 2 Feb 2021, at 19:45, Christopher Barker  wrote:
> 
> On Tue, Feb 2, 2021 at 11:12 AM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
>>>> Where would Python look for a "configuration file like `pyvenv.cfg`" ?
>>>> 
>>>> I am not a Windows expert so I am not sure. But I think it should be
>>>> the same directory where `python.exe` is in.
> 
> A small note here -- ideally there would be nothing Windows specific here. 
> Yes, UTF-8 mode is Windows only, but:
> 
> 1) should it be? I'm still unsure on this, but while the vast majority of 
> other platforms use UTF-8 -- maybe it would be more robust to have UTF-8 mode 
> available everywhere -- essentially, ignore the "system encoding", no matter 
> the system.
> 
> 2) perhaps UTF-8 mode isn't the only use-case for this -- it would be good to 
> have a way to have Python startup parameters that are installation / 
> environment specific -- that could help with other issues with "global" 
> configuration: PYTHONPATH, PYTHONHOME, PYTHONSTARTUP, other PYTHON* 
> environment variables.

Forgive me if I'm miss understanding your position.

Windows is successful because of its backward compatibility.
That includes big problems with text encoding in the modern world.

Aside: HTML 5 even has a encoding rule that acknowledges that web pages marked
utf-8 are really windows USA code page and show how to fall back!

Are you calling env vars global config?

But it is a Windows problem only.

I write code accepting that unix (*BSD, Linux), macOS and Windows have unique 
quirks.
To believe that if you ignore that reality is not a path that leads to 
happiness.

>>> 
>>> I'd suggest that you could have a %LOCALAPPDATA%\python.ini
>> But what happen if user installed Python from python.org 
>> <http://python.org/> and Python from conda?
>> User may have two or more Pythons having the same version.
> 
> and different environments, be they conda environments, pipenv, what have you.
>  
> Apple showed the way by using reversed FQD's.
> 
> Python.org <http://python.org/> would use org.python.python.ini
> Conda.io <http://conda.io/> would use io.conda.python.ini
> barry-emacs.org <http://barry-emacs.org/>'s python would use 
> org.barrys-emacs.python.python.ini
> 
> Further we would need to support multiple versions of python from the same 
> org installed side-by-side.
> Structure the .ini so that it has default settings and version specific 
> settings.
> 
> This would be pretty painful to manage -- it's a "bad idea" to have a single 
> configuration file that is being managed and updated by any number of 
> different tools. And those tools are managed by different groups of people.

Why would *tools* be messing with my personal config file?
No *tool* messes with py.exe config that I know of.

> 
> And this would be VERY hard for end users to manage -- as people installed 
> and uninstalled python versions and environments, they would get a very, very 
> messy global ini file.

End users are best served with good, practical defaults. Choosing those 
defaults is very hard
which is the point of this thread.

I do not advocate a global ini file. I do not think I asked for that.

I'm suggesting a mechanism that is identical to the way the py.exe is 
configured.

> 
> Also: "conda python" is not necessarily any different than python.org 
> <http://python.org/> python -- it's generally built exactly the same way -- 
> the only difference is how it's installed.

A distro will often patch in distro specific changes.
If conda needs to be configure independently from python.org 
<http://python.org/> kits this is an obvious requirement.

> 
> So it would be much better for the config file to be located inside the 
> Python installation: essentially a 1:1 relationship between the python 
> executable and the config file. So you know for sure that if you fire up the 
> python you are intending to, you will get the configuration that comes with 
> it.

You mean where it may well not be editable without admin privs?
That is a bad thing surely?

>  
> [3.8-64]
> utf8-mode = false
> 
> The key issue here is that the configuration is not "version number specific" 
> -- it's (or should be) application specific. And Python has had that issue 
> for ages: as a run-time system (for lack of a better word), each application 
> needs a different set of packages in various versions. And that's been 
> addressed with with "environments" of various sorts. So it would be good to 
> leverage that, and have a config file that goes right along with the 
> environ

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 3 Feb 2021, at 02:49, Christopher Barker  wrote:
> 
> Aside: HTML 5 even has a encoding rule that acknowledges that web pages marked
> utf-8 are really windows USA code page and show how to fall back!
> 
> But that doesn't depend ina. system setting does it? So I don't get your 
> point:

The bug in most (?) .net web apps apparently is that the .net libraries convert 
to the default
system locale and do not assume utf-8. The programmer has to explicity use 
utf-8.

So yes it does depend on a system setting.

I came across this in the HTML 5 specs because of working on web page content 
that did not decode
not because I'm a .NET developer.

I raise this as this seems to be the same problem that python faces with system 
locale conflicting
with the wider world using utf-8.

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QZYVQQX2PKO2OY3L2QRACJ5DKQY3KM53/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 3 Feb 2021, at 02:49, Christopher Barker  wrote:
> 


Rather than reply point by point I will summarise my input.

I think that utf-8 mode is a great idea.

I think that an .INI file in the style that py.exe uses is better then env var.

Env var on WIndows could be used but there can be surprises with the way
windows merges user and system env vars. Maybe that only with PATH that
is very odd.

I'm hoping that the solution implemented allows new users to get a great
experience and also that advanced users can get control of the mode.

Personally I'd prefer to have files that I edit to configure python then 
registry
keys. I can put files into git, not exmple. I have to work hard to manage 
registry
keys via git.

Barry
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/N4HV3CM3QAUNZA4XRP2PQTNNR2BV7H2V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 5 Feb 2021, at 03:11, Inada Naoki  wrote:
> 
> On Fri, Feb 5, 2021 at 6:17 AM Barry Scott  wrote:
>> 
>> Rather than reply point by point I will summarise my input.
>> 
>> I think that utf-8 mode is a great idea.
>> 
>> I think that an .INI file in the style that py.exe uses is better then env 
>> var.
>> 
>> Env var on WIndows could be used but there can be surprises with the way
>> windows merges user and system env vars. Maybe that only with PATH that
>> is very odd.
>> 
>> I'm hoping that the solution implemented allows new users to get a great
>> experience and also that advanced users can get control of the mode.
>> 
>> Personally I'd prefer to have files that I edit to configure python then 
>> registry
>> keys. I can put files into git, not exmple. I have to work hard to manage 
>> registry
>> keys via git.
>> 
> 
> I 100% agree with you. And pyvenv.cfg satisfies all your needs.
> 
> When compared pyvenv.cfg with py.ini-like new config file:
> 
> Cons:
> 
> * Need system privilege to change the setting of system installed Python.
>  * But user can install another Python, or create venv anyway.
> 
> Pros:
> 
> * The file is already supported.
>  * No need to lookup another file at startup.
> * No need to edit any file outside the install location.
>  * Easy to clean uninstall
>  * Portable app friendly
> * One file per environment
>  * Breaking the config file affects only one environment.
> 
> So I still prefer pyvenv.cfg.

I'm under the impression that new users will not create a venv.
Indeed I run a lot of python scripts outside of venv world.
I only use venv as part of my development pipe lines.

I not sure that a venv cfg file would not help.
But a python.ini could.

Barry


> 
> -- 
> Inada Naoki  
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7QALMNRYQDKZPGH2N5URXNCRO4F7CBEW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 5 Feb 2021, at 11:06, Inada Naoki  wrote:
> 
> On Fri, Feb 5, 2021 at 7:59 PM Barry Scott  wrote:
>> 
>> I'm under the impression that new users will not create a venv.
>> Indeed I run a lot of python scripts outside of venv world.
>> I only use venv as part of my development pipe lines.
>> 
>> I not sure that a venv cfg file would not help.
>> But a python.ini could.
>> 
> 
> python.exe lookup pyvenv.cfg even outside of venv.
> So we can write utf8mode=1 in pyvenv.cfg even outside of venv.

Oh I did not know that. I'm happy that a new file is not need for the
system wide setting.

> 
> The main limitation is that users can not write config file in install
> location when Python is installed for system, not for user.

This is the problem that I was thinking about when I proposed using
a py.ini like solution where the file is looked for in the users config folder.
I think that is the %LOCALAPPDATA% folder for py.exe.

As Chris points out in his summary of the issue.

How would this work for different version of python being installed and needing 
different config?
How would this work for python installed from different vendors?

Maybe the answer is that there is only one user defined override possible and 
all versions use it.

Also am I right to assume that the impact of these changes would only impact on 
Windows?

Barry

> 
> Regards,
> -- 
> Inada Naoki  
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WXYZFAGGBZGDPBKXBRIY6O4VUFA36APL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Make UTF-8 mode more accessible for Windows users.

[Barry Scott]

> On 5 Feb 2021, at 11:49, Inada Naoki  wrote:
> 
> On Fri, Feb 5, 2021 at 8:15 PM Barry Scott  wrote:
>> 
>>> 
>>> The main limitation is that users can not write config file in install
>>> location when Python is installed for system, not for user.
>> 
>> This is the problem that I was thinking about when I proposed using
>> a py.ini like solution where the file is looked for in the users config 
>> folder.
>> I think that is the %LOCALAPPDATA% folder for py.exe.
>> 
>> As Chris points out in his summary of the issue.
>> 
>> How would this work for different version of python being installed and 
>> needing different config?
> 
> Each installation have each config file.

I'm talking about the user's override of the system default.
The system default is the easy part.

> 
>> How would this work for python installed from different vendors?
>> 
> 
> Vendor installer should provide an option for it.
> 
>> Maybe the answer is that there is only one user defined override possible 
>> and all versions use it.
>> 
>> Also am I right to assume that the impact of these changes would only impact 
>> on Windows?
>> 
> 
> I think we don't have any reason to restrict this for Windows.
> But since this idea is proposed only for Windows users, only Windows
> installer will have "Enable UTF-8 mode" option.

I'm not sure that Linux and macOS suffer from this problem. Am I wrong to think 
that?

Barry


> 
> -- 
> Inada Naoki  
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CBQ6CYQLQM5KJ6CIBG7VQOIIN4FMN5V3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inadequate error reporting during function call setup stage

[Barry Scott]

> On 22 Feb 2021, at 10:15, Paul Sokolovsky  wrote:
> 
> It looks like:
> 
> Traceback (most recent call last):
>  File "pseudoc_tool.py", line 91, in 
>  File ".../xforms.py", line 25, in print
> TypeError: unexpected keyword argument 'noann'
> 
> - that makes clear that it's "print" function of "xforms.py" module,
> line 25, which got an unexpected keyword argument.

You are proposing to fake a stack frame that I have to know is not a stack 
frame but is in fact the location of the function in the exception?
I'm -1 on that as its confusing.

Having checked that its python code and not a C extension function you could 
use the info
in fn.__code__ to get the filename and line of where the function is defined 
and put that info into the exception.

Example of the info:
| >>> os.path.join.__code__


I use repr(fn.__code__) a lot when debugging complex code.

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YJ3RNTHGZWJ2DG2T5HFGXKTKFYN3CGAN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Deprecate/change the behaviour of ~bool

[Barry Scott]

> On 23 Feb 2021, at 22:10, Steven D'Aprano  wrote:
> 
> There are exactly 2**4 = 16 boolean operators of two variables. Python 
> only supports two: `and` and `or`. Plus a single unary operator `not` 
> (out of four possible unary operators). What makes xnor so special that 
> you want it to be an operator?

Python implements more then 2 of them:

True
False
not
and
or

https://en.wikipedia.org/wiki/Boolean_algebras_canonically_defined#Truth_tables

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XXN3NDW2KZDPQWRAF4L3MCAC7FP6DXYU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Barrier Object in asyncio lib

[Barry Scott]

> On 25 Feb 2021, at 13:14, Yves Duprat  wrote:
> 
> Hi,the list,
> 
> I'm wondering why Barrier object does not exist in the synchronization 
> primitives of the asyncio lib while it is present in threading and 
> multiprocessing libs ?
> This may not be the right place to ask this question, but I never found an 
> answer on the web.
> Thanks for your help.


I'm assuming that the barrier you are speaking of is the mechanism that is used 
to
synchronise threads/processes running in parallel to prevent data races.

With async code that is never an issue. Each function runs to completion 
uninterrupted.
There are no data races. Each time a async function runs it can know that the 
state of
the objects it uses will not be changed while it is running.

Barry



> 
> Yves
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7PWMUDUTLXYLNSXES7VMDQ26A3W/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPXNZH5KYK2BLHJXUFZF2DLFBLCBR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Barrier Object in asyncio lib

[Barry Scott]

> On 25 Feb 2021, at 17:15, Jonathan Slenders  wrote:
> 
> It does make sense to have a barrier synchronization primitive for asyncio.
> The idea is to make a coroutine block until at least X coroutines are waiting 
> to enter the barrier.
> This is very useful, if certain actions need to be synchronized.

I do most of my async coding with twisted where what you calling a barrier is a 
DeferredList.

The way its used is that you add in all the deferreds that you want to complete 
before you continue
into the list. Once all the deferered have competed the DefferedList completes 
and its callback is run.

Barry


> 
> Recently, I had to implement a barier myself for our use case. See code below:
> 
> It is simple to implement, but I too would like to have one for asyncio, in 
> order to be consistent with the concurrency primitives we have for threading.
> 
> Jonathan
> 
> 
> class Barier:
> """
> Make a coroutine block until there are at least X waiters.
> 
> Similar to the threading Barier objects but for asyncio:
> https://docs.python.org/3/library/threading.html#barrier-objects 
> <https://docs.python.org/3/library/threading.html#barrier-objects>
> """
> 
> def __init__(self, parties: int) -> None:
> self.parties = parties
> self._waiting: int
> self._event = asyncio.Event()
> 
> def add_one(self) -> None:
> self._waiting += 1
> if self._waiting == self.parties:
> self._event.set()
> 
> async def wait(self, worker: "Worker") -> None:
> """
>     Wait until all we have at least `parties` waiters.
> """
> self.add_one()
> await self._event.wait()
> 
> 
> 
> 
> Le jeu. 25 févr. 2021 à 16:42, Barry Scott  <mailto:ba...@barrys-emacs.org>> a écrit :
> 
> 
> > On 25 Feb 2021, at 13:14, Yves Duprat  > <mailto:ydup...@gmail.com>> wrote:
> > 
> > Hi,the list,
> > 
> > I'm wondering why Barrier object does not exist in the synchronization 
> > primitives of the asyncio lib while it is present in threading and 
> > multiprocessing libs ?
> > This may not be the right place to ask this question, but I never found an 
> > answer on the web.
> > Thanks for your help.
> 
> 
> I'm assuming that the barrier you are speaking of is the mechanism that is 
> used to
> synchronise threads/processes running in parallel to prevent data races.
> 
> With async code that is never an issue. Each function runs to completion 
> uninterrupted.
> There are no data races. Each time a async function runs it can know that the 
> state of
> the objects it uses will not be changed while it is running.
> 
> Barry
> 
> 
> 
> > 
> > Yves
> > ___
> > Python-ideas mailing list -- python-ideas@python.org 
> > <mailto:python-ideas@python.org>
> > To unsubscribe send an email to python-ideas-le...@python.org 
> > <mailto:python-ideas-le...@python.org>
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/ 
> > <https://mail.python.org/mailman3/lists/python-ideas.python.org/>
> > Message archived at 
> > https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7PWMUDUTLXYLNSXES7VMDQ26A3W/
> >  
> > <https://mail.python.org/archives/list/python-ideas@python.org/message/IAFAH7PWMUDUTLXYLNSXES7VMDQ26A3W/>
> > Code of Conduct: http://python.org/psf/codeofconduct/ 
> > <http://python.org/psf/codeofconduct/>
> > 
> ___
> Python-ideas mailing list -- python-ideas@python.org 
> <mailto:python-ideas@python.org>
> To unsubscribe send an email to python-ideas-le...@python.org 
> <mailto:python-ideas-le...@python.org>
> https://mail.python.org/mailman3/lists/python-ideas.python.org/ 
> <https://mail.python.org/mailman3/lists/python-ideas.python.org/>
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPXNZH5KYK2BLHJXUFZF2DLFBLCBR/
>  
> <https://mail.python.org/archives/list/python-ideas@python.org/message/B6WDPXNZH5KYK2BLHJXUFZF2DLFBLCBR/>
> Code of Conduct: http://python.org/psf/codeofconduct/ 
> <http://python.org/psf/codeofconduct/>

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IHSEWHYYJI3BQB4GTT7UCFAS5DOCRNVM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Barrier Object in asyncio lib

[Barry Scott]

> On 26 Feb 2021, at 08:31, Jonathan Slenders  wrote:
> 
> Barry,
> 
> What you describe sounds like `asyncio.gather(...)` if I understand correctly.
> 
> The thing with a Barier is that it's usable in situations where we don't know 
> the other tasks. Maybe there is no reference to them from the current scope. 
> Maybe they are even not yet created.
> It certainly can be done with a list of `asyncio.Future` and 
> `asyncio.gather(...)`, but that's a lot of boilerplate.
> 
> IMHO, Yves is right. For both asyncio and threading, we have Lock, Event, 
> Condition, Semaphore and BoundedSemaphore. Only Barier is missing among the 
> asyncio primitives. (RLock doesn't make sense.)

Why would you need locks for async? Is it to sync with things outside of the 
async process?

With the large and complex async app I work on there are no locks at all.

> (I guess we can probably go to bugs.python.org <http://bugs.python.org/> with 
> this proposal.)

Having shown that a Barrier for async is a missing piece it would be good to 
get a thumbs up here.

Barry

> 
> Jonathan
> 
> 
> 
> 
> 
> 
> Le jeu. 25 févr. 2021 à 23:38, Barry Scott  <mailto:ba...@barrys-emacs.org>> a écrit :
> 
> 
>> On 25 Feb 2021, at 17:15, Jonathan Slenders > <mailto:jonat...@slenders.be>> wrote:
>> 
>> It does make sense to have a barrier synchronization primitive for asyncio.
>> The idea is to make a coroutine block until at least X coroutines are 
>> waiting to enter the barrier.
>> This is very useful, if certain actions need to be synchronized.
> 
> I do most of my async coding with twisted where what you calling a barrier is 
> a DeferredList.
> 
> The way its used is that you add in all the deferreds that you want to 
> complete before you continue
> into the list. Once all the deferered have competed the DefferedList 
> completes and its callback is run.
> 
> Barry
> 
> 
>> 
>> Recently, I had to implement a barier myself for our use case. See code 
>> below:
>> 
>> It is simple to implement, but I too would like to have one for asyncio, in 
>> order to be consistent with the concurrency primitives we have for threading.
>> 
>> Jonathan
>> 
>> 
>> class Barier:
>> """
>> Make a coroutine block until there are at least X waiters.
>> 
>> Similar to the threading Barier objects but for asyncio:
>> https://docs.python.org/3/library/threading.html#barrier-objects 
>> <https://docs.python.org/3/library/threading.html#barrier-objects>
>> """
>> 
>> def __init__(self, parties: int) -> None:
>> self.parties = parties
>> self._waiting: int
>> self._event = asyncio.Event()
>> 
>> def add_one(self) -> None:
>> self._waiting += 1
>> if self._waiting == self.parties:
>> self._event.set()
>> 
>> async def wait(self, worker: "Worker") -> None:
>> """
>> Wait until all we have at least `parties` waiters.
>> """
>> self.add_one()
>> await self._event.wait()
>> 
>> 
>> 
>> 
>> Le jeu. 25 févr. 2021 à 16:42, Barry Scott > <mailto:ba...@barrys-emacs.org>> a écrit :
>> 
>> 
>> > On 25 Feb 2021, at 13:14, Yves Duprat > > <mailto:ydup...@gmail.com>> wrote:
>> > 
>> > Hi,the list,
>> > 
>> > I'm wondering why Barrier object does not exist in the synchronization 
>> > primitives of the asyncio lib while it is present in threading and 
>> > multiprocessing libs ?
>> > This may not be the right place to ask this question, but I never found an 
>> > answer on the web.
>> > Thanks for your help.
>> 
>> 
>> I'm assuming that the barrier you are speaking of is the mechanism that is 
>> used to
>> synchronise threads/processes running in parallel to prevent data races.
>> 
>> With async code that is never an issue. Each function runs to completion 
>> uninterrupted.
>> There are no data races. Each time a async function runs it can know that 
>> the state of
>> the objects it uses will not be changed while it is running.
>> 
>> Barry
>> 
>> 
>> 
>> > 
>> > Yves
>> > ___
>> > Python-ideas mailing list -- python-ideas@python.org 
>> > <mailto:python-ideas@python.org>
>> > To unsubscribe send an e

[Python-ideas] Re: Integer concatenation to byte string

[Barry Scott]

> On 1 Mar 2021, at 18:01, mmax42...@gmail.com wrote:
> 
> Currently, the only way to concatenate an integer to a bytes object is by 
> converting the integer to bytes with a function call before concatenating. 
> And there is no way to make a mutable bytes object without a function call. 
> 
> I propose an array-type string like the, or for the bytearray. It would work 
> as a mutable b-string, as 
> 
> foo = a"\x00\x01\x02abcÿ"   # a-string, a mutable bytes object.
> foo[0] = 123  # Item assignment
> foo+= 255 # Works the same as 
> bytesvariable+=b"123"
> foo+= a"\x255\x00"  # Concatenation with itself
> foo+= b"\x255\x00"  # Cross compatibility with bytes objects.
> 
> This would be processed the same as, or would be the bytearray,
>>>> type(a"\x00\x01\x02abcÿ")
> 

When I needed to do the creation of bytes objects from a mix of types the
struct.pack() method has been the obvious way to go.

What is the use case that leads to needing the above?

Barry


> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/H5W3TTDC633NCPFUYIMQ26ZWX7XM4W6S/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/77KYAIS7BAJ2N6XZWHOABUVBR66L7DTS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Integer concatenation to byte string

[Barry Scott]

We where no longer on the ideas list...

> On 2 Mar 2021, at 13:04, Memz  wrote:
> 
> There is no specific scenario it solves. The lack of efficiency of the timed 
> code should speak for itself. Non-mutable bytes is a limit of python, since 
> it's reliant on using function calls.
> 
> b"\x00\x00\x00\x00\x00"
> bytearray( b"\x00\x00\x00\x00\x00" )
> struct.pack("i",0,0,0,0,0)
> b"\x00\x00\x00\x00\x00"  + bytes([1,0,0,0,0])

You mean the above is what you timed?
It is not a realistic problem you are measuring.

You also did not share how you measured the the code.

If you are not experienced in benchmarking there are variables
that you must control for to have meaningful results.

> 
> All function calls take time and resources, it would be impossible to 
> streamline a function call to make it faster than building it in. This goes 
> for most languages, including C.

All python byte code is interpreted by calling functions. They take time and 
resources.

Barry

> 
> 
> On Tue, Mar 2, 2021 at 3:29 AM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
> 
> 
> > On 2 Mar 2021, at 02:03, Memz  > <mailto:mmax42...@gmail.com>> wrote:
> > 
> > >When I needed to do the creation of bytes objects from a mix of types the
> > >struct.pack() method has been the obvious way to go.
> > 
> > >What is the use case that leads to needing the above?
> > 
> > >Barry
> > 
> > The use of my suggestion is to reduce the reliance of function calls for 
> > bytes and bytearrays, which currently can only be done through function 
> > calls, and making it more efficient all-together. Here is a timed version 
> > of this: 
> > b-strings: 5-5.4 s
> > bytearray() function call:  67.9 s
> > struct.pack() 80.9 s 
> > b-string + bytes([1,...])  54.3 s
> 
> What code are you benchmarking? What problem does that code solve?
> 
> My experience with creating byte objects is that struct is the fastest way to 
> get the job done.
> But that could be becuase of the problems that I need bytes for.
> For example calling ioctl().
> 
> > 
> > Moving bytearray to a non-function call would overhaul and optimize code 
> > that works with bytes, increase flexibility, and reduce reliance on 
> > imports, including struct.pack(). There is a lot of code that could, and 
> > should be using bytearray but can't, because other, more slow and painful 
> > than should be methods are more efficient because of the function call 
> > nature of bytearray.
> 
> You are assuming that the problem is the function calls.
> Surely its the algorithms that lead to issues in most code that is slow?
> 
> Barry


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UBJUDDMGPGMQPIVF5JV4CVAJYSSY2F35/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Integer concatenation to byte string

[Barry Scott]

> On 2 Mar 2021, at 23:49, Steven D'Aprano  wrote:
> 
> 
> [Barry]
>> All python byte code is interpreted by calling functions. They take 
>> time and resources.
> 
> That's not entirely correct. Literals such as text strings, ints and 
> floats get compiled directly into the byte-code. Now of course there is 
> some overhead while executing the byte-code, but that doesn't include 
> the heavy cost of a Python function call.

I was thinking of the C functions that are executed in ceval.c to run the 
interpreter
for any byte code.

Barry


> 
> 
> -- 
> Steve
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/7GTPEGXDAQRKWITBAGYWCU3MNY6JJE6U/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/T25IDROJ7NIUUCQTSVTACMWATH7MBAM3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improved multi-tasking performance through deterministic GIL hand-off

[Barry Scott]

> On 10 May 2021, at 15:30, Sophist  wrote:
> 
> I don't know how many people will remember some work that David Beazley did 
> about a decade ago on how the GIL impacts multithreading performance - 
> essentially he instrumented the Python interpreter to log how multiple 
> threads competed for  the GIL and gave several presentations over the space 
> of 2-3 years. A couple of years ago I reached out to him with an idea on how 
> to significantly improve the way that Python handles multi-threading hand-off 
> of the GIL, but unfortunately he was not interested in pursuing this further. 
> I am raising it here in the hope that someone else would be interested in 
> implementing this.
> 
> In essence my idea is to stop Python handing off the GIL through a 
> competition between threads that are ready to run, and instead for Python to 
> implement a scheduler for the GIL which decides which thread should get the 
> GIL next and directly hands it over. 
> 
> Background
> 
> Here are the links to David Beazley's presentations:
> 
> 2009: Inside the Python GIL  - 
> https://www.youtube.com/watch?v=ph374fJqFPE
> 2010: Understanding the Python GIL- 
> https://speakerdeck.com/dabeaz/understanding-the-python-gil 
> https://www.youtube.com/watch?v=Obt-vMVdM8s
> 2011: Embracing the Global Interpreter Lock   - 
> https://speakerdeck.com/dabeaz/embracing-the-global-interpreter-lock 
> https://www.youtube.com/watch?v=fwzPF2JLoeU
> 2011: In Search of the Perfect Global Interpreter Lock - 
> https://speakerdeck.com/dabeaz/in-search-of-the-perfect-global-interpreter-lock
>  https://www.youtube.com/watch?v=5jbG7UKT1l4

Given this is very old information I think the first thing needed is to 
reproduce David's experiments and see if the 3.10 implementation has the same 
issues.

Have you done this already?

If you turn these slides into benchmark code that would make it easier to 
experiment with.

Benchmarks will need running on macOS, Windows, Linux at least.

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AUASYYYOMQGGWLHOPMP4J5RGRQTDYCAK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Improved multi-tasking performance through deterministic GIL hand-off

[Barry Scott]

> On 10 May 2021, at 20:23, Barry Scott  wrote:
> 
> 
> 
>> On 10 May 2021, at 15:30, Sophist  wrote:
>> 
>> I don't know how many people will remember some work that David Beazley did 
>> about a decade ago on how the GIL impacts multithreading performance - 
>> essentially he instrumented the Python interpreter to log how multiple 
>> threads competed for  the GIL and gave several presentations over the space 
>> of 2-3 years. A couple of years ago I reached out to him with an idea on how 
>> to significantly improve the way that Python handles multi-threading 
>> hand-off of the GIL, but unfortunately he was not interested in pursuing 
>> this further. I am raising it here in the hope that someone else would be 
>> interested in implementing this.
>> 
>> In essence my idea is to stop Python handing off the GIL through a 
>> competition between threads that are ready to run, and instead for Python to 
>> implement a scheduler for the GIL which decides which thread should get the 
>> GIL next and directly hands it over. 
>> 
>> Background
>> 
>> Here are the links to David Beazley's presentations:
>> 
>> 2009: Inside the Python GIL  - 
>> https://www.youtube.com/watch?v=ph374fJqFPE
>> 2010: Understanding the Python GIL- 
>> https://speakerdeck.com/dabeaz/understanding-the-python-gil 
>> https://www.youtube.com/watch?v=Obt-vMVdM8s
>> 2011: Embracing the Global Interpreter Lock   - 
>> https://speakerdeck.com/dabeaz/embracing-the-global-interpreter-lock 
>> https://www.youtube.com/watch?v=fwzPF2JLoeU
>> 2011: In Search of the Perfect Global Interpreter Lock - 
>> https://speakerdeck.com/dabeaz/in-search-of-the-perfect-global-interpreter-lock
>>  https://www.youtube.com/watch?v=5jbG7UKT1l4
> 
> Given this is very old information I think the first thing needed is to 
> reproduce David's experiments and see if the 3.10 implementation has the same 
> issues.
> 
> Have you done this already?
> 
> If you turn these slides into benchmark code that would make it easier to 
> experiment with.
> 
> Benchmarks will need running on macOS, Windows, Linux at least.

It looks like the GIL code has not changed in a long time.

But for 3.7 FORCE_SWITCHING is always defined that changes the GIL behaviour.

This comment in Python/ceval_gil.h explains what that does:

  - When a thread releases the GIL and gil_drop_request is set, that thread
 ensures that another GIL-awaiting thread gets scheduled.
 It does so by waiting on a condition variable (switch_cond) until
 the value of last_holder is changed to something else than its
 own thread state pointer, indicating that another thread was able to
 take the GIL.

 This is meant to prohibit the latency-adverse behaviour on multi-core
     machines where one thread would speculatively release the GIL, but still
 run and end up being the first to re-acquire it, making the "timeslices"
 much longer than expected.
 (Note: this mechanism is enabled with FORCE_SWITCHING above)

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RPRFINFGHOMYOVQSEA64PXOVGQGVURLQ/
Code of Conduct: http://python.org/psf/codeofconduct/