Re: [Python-Dev] summary: a Path protocol
On Fri, Apr 8, 2016 at 4:35 PM, Victor Stinner wrote: > Sorry, I don't have time to read the whole discussion. What is the problem > with adding a __str__ to pathlib? > > Victor Everything else has __str__ too, so you run the risk of open(["Hello", "World"], "w") working and doing something weird. Or of passing an open file object to something that was expecting a file name, and having *that* work too. Calling str(p) on something that ought to be either a Path or a string should raise an exception if given something else. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On 7 April 2016 at 03:26, Brett Cannon wrote: > WIth Ethan volunteering to do the work to help make a path protocol a thing > -- and I'm willing to help along with propagating this through the stdlib > where I think Serhiy might be interested in helping as well -- and a seeming > consensus this is a good idea, it seems like this proposal has a chance of > actually coming to fruition. > > Now we need clear details. :) Some open questions are: > > Name: __path__, __fspath__, or something else? __fspath__ > Method or attribute? (changes what kind of one-liner you might use in > libraries, but I think historically all protocols have been methods and the > serialized string representation might be costly to build) Method, as long as there's a helper function somewhere > Built-in? (name is dependent on #1 if we add one) os.fspath (alongside os.fsencode and os.fsdecode) (Putting this in a module low in the dependency stack makes it easy for other modules to access without pulling in all of pathlib's dependencies) > Add the method/attribute to str? (I assume so, much like __index__() is on > int, but I have not seen it explicitly stated so I would rather clarify it) Makes sense > Expand the C API to have something like PyObject_Path()? PyUnicode_FromFSPath, perhaps? The return type is well-defined here, so it can be done as an alternate constructor, and the C API counterparts of os.fsdecode and os.fsencode are PyUnicode functions (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) > Some people have asked for the pathlib PEP to have a more flushed out > reasoning as to why pathlib doesn't inherit from str. If Antoine doesn't > want to do it I can try to instil my blog post into a more succinct > paragraph or two and update the PEP myself. > > Is this going to require a PEP or if we can agree on the points here are we > just going to do it? If we think it requires a PEP I'm willing to write it, > but I obviously have no issue if we skip that step either. :) It's worth summarising in a PEP at least for communications purposes - very easy for folks that don't follow python-dev to miss otherwise. Plus my specific API suggestions are pretty different from Ethan's :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
Please write a new PEP. The topic looks to be discussed since many months by many different people on different mailing list. A PEP is a good standard to take a decision and it became clear that a decision must be taken for pathlib. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Other pathlib improvements? was: When should pathlib stop being provisional?
FYI the doc of the builtin functions is the #1 in stats of docs python.org. I also read this doc every week, even if I consider that I know well Python. IMHO it's not an issue to regulary read the doc. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
I like __fspath__ because it looks like os.fsencode() and os.fsdecode(). Please no builtin function, we have enough of them, but make sure that the __fspath__ is accepted in all functions expecting a filename. If you consider that a function would make your change simpler, I suggest to add os.fspath(): if isinstance(obj, str): return obj try: return obj.__fspath__ except AttributeError: raise TypeError(...) Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
I've made another attempt at Python sandboxing, which does something which I've not seen tried before - using the 'ast' module to do static analysis of the untrusted code before it's executed, to prevent most of the sneaky tricks that have been used to break out of past attempts at sandboxes. In short, I'm turning Python's usual "gentleman's agreement" that you should not access names and attributes that are indicated as private by starting with an underscore into a rigidly enforced rule: try and access anything starting with an underscore and your code will not be run. Anyway the code is at https://github.com/jribbens/unsafe It requires Python 3.4 or later (it could probably be made to work on Python 2.7 as well, but it would need some changes). I would be very interested to see if anyone can manage to break it. Bugs which are trivially fixable are of course welcomed, but the real question is: is this approach basically sound, or is it fundamentally unworkable? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On 8 April 2016 at 15:18, Jon Ribbens wrote: > I would be very interested to see if anyone can manage to break it. > Bugs which are trivially fixable are of course welcomed, but the real > question is: is this approach basically sound, or is it fundamentally > unworkable? What are the limitations? It seems to even block "import" which seems over-zealous (no import math?) Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On Fri, Apr 08, 2016 at 03:37:45PM +0100, Paul Moore wrote: > On 8 April 2016 at 15:18, Jon Ribbens > wrote: > > I would be very interested to see if anyone can manage to break it. > > Bugs which are trivially fixable are of course welcomed, but the real > > question is: is this approach basically sound, or is it fundamentally > > unworkable? > > What are the limitations? It seems to even block "import" which seems > over-zealous (no import math?) The restrictions are: Of the builtins, __import__, compile, globals, input, locals, memoryview, open, print, type and vars are unavailable (and some of the exceptions, but mostly because they're irrelevant). You cannot access any name or attribute which starts with "_", or is called "gi_frame" or "gi_code". You cannot use the "with" statement (although it's possible it might be safe for me to add that back in if I also disallow access to attributes called "tb_frame"). Importing modules is fundamentally unsafe because the untrusted code might alter the module, and the altered version would then be used by the containing application. My code has a "_copy_module" function which copies (some of) the contents of modules, so some sort of import functionality of a white-list of modules could be added using this, but there's no point in me going through working out which modules are safe to white-list until I'm vaguely confident that my approach isn't fundamentally broken in the first place. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On 8 April 2016 at 16:18, Jon Ribbens
wrote:
> I've made another attempt at Python sandboxing, which does something
> which I've not seen tried before - using the 'ast' module to do static
> analysis of the untrusted code before it's executed, to prevent most
> of the sneaky tricks that have been used to break out of past attempts
> at sandboxes.
>
> In short, I'm turning Python's usual "gentleman's agreement" that you
> should not access names and attributes that are indicated as private
> by starting with an underscore into a rigidly enforced rule: try and
> access anything starting with an underscore and your code will not be
> run.
>
> Anyway the code is at https://github.com/jribbens/unsafe
> It requires Python 3.4 or later (it could probably be made to work on
> Python 2.7 as well, but it would need some changes).
>
> I would be very interested to see if anyone can manage to break it.
> Bugs which are trivially fixable are of course welcomed, but the real
> question is: is this approach basically sound, or is it fundamentally
> unworkable?
>
If i'm not mistaken, this breaks out:
> exec('open("out", "w").write("a")', {})
because if the second argument of exec does not contain a __builtins__ key,
then a copy of the original builtins module is inserted:
https://docs.python.org/3/library/functions.html#exec
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On 04/08/2016 02:50 AM, Nick Coghlan wrote: Built-in? (name is dependent on #1 if we add one) os.fspath (alongside os.fsencode and os.fsdecode) I like this better. Add the method/attribute to str? (I assume so, much like __index__() is on int, but I have not seen it explicitly stated so I would rather clarify it) Makes sense What will this do? Return a Path or a str? I don't think we need either. Expand the C API to have something like PyObject_Path()? PyUnicode_FromFSPath, perhaps? The return type is well-defined here, so it can be done as an alternate constructor, and the C API counterparts of os.fsdecode and os.fsencode are PyUnicode functions (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) So this will do the same thing as os.fspath() at the C level, yes? It's worth summarising in a PEP at least for communications purposes - very easy for folks that don't follow python-dev to miss otherwise. Plus my specific API suggestions are pretty different from Ethan's :) *sigh* Okay. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Fri, 8 Apr 2016 at 08:33 Ethan Furman wrote: > On 04/08/2016 02:50 AM, Nick Coghlan wrote: > > >> Built-in? (name is dependent on #1 if we add one) > > > > os.fspath (alongside os.fsencode and os.fsdecode) > > I like this better. > > > >> Add the method/attribute to str? (I assume so, much like __index__() is > on > >> int, but I have not seen it explicitly stated so I would rather clarify > it) > > > > Makes sense > > What will this do? Return a Path or a str? I don't think we need either. > When I brought this up it was to return self. > > > >> Expand the C API to have something like PyObject_Path()? > > > > PyUnicode_FromFSPath, perhaps? The return type is well-defined here, > > so it can be done as an alternate constructor, and the C API > > counterparts of os.fsdecode and os.fsencode are PyUnicode functions > > (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) > > So this will do the same thing as os.fspath() at the C level, yes? > Yes. > > > > It's worth summarising in a PEP at least for communications purposes - > > very easy for folks that don't follow python-dev to miss otherwise. > > Plus my specific API suggestions are pretty different from Ethan's :) > > *sigh* Okay > Chris Angelico and I have been asked by Guido to work together to come up with a proposal after all the discussions are finished and it will most likely be a patch to the pathlib PEP. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On Fri, Apr 08, 2016 at 05:21:38PM +0200, Arthur Darcet wrote:
>If i'm not mistaken, this breaks out:
>> exec('open("out", "w").write("a")', {})
>because if the second argument of exec does not contain a __builtins__
>key, then a copy of the original builtins module is inserted:
>https://docs.python.org/3/library/functions.html#exec
Ah, that's a good point. I did think allowing eval/exec was a bit
ambitious. I've updated it to disallow passing namespace arguments to
them.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On 08/04/16 16:18, Jon Ribbens wrote:
I've made another attempt at Python sandboxing, which does something
which I've not seen tried before - using the 'ast' module to do static
analysis of the untrusted code before it's executed, to prevent most
of the sneaky tricks that have been used to break out of past attempts
at sandboxes.
In short, I'm turning Python's usual "gentleman's agreement" that you
should not access names and attributes that are indicated as private
by starting with an underscore into a rigidly enforced rule: try and
access anything starting with an underscore and your code will not be
run.
Anyway the code is at https://github.com/jribbens/unsafe
It requires Python 3.4 or later (it could probably be made to work on
Python 2.7 as well, but it would need some changes).
I would be very interested to see if anyone can manage to break it.
Bugs which are trivially fixable are of course welcomed, but the real
question is: is this approach basically sound, or is it fundamentally
unworkable?
That one is trivially fixable, but here goes:
async def a():
global c
c = b.cr_frame.f_back.f_back.f_back
b = a()
b.send(None)
c.f_builtins['print']('broken')
Also, if the point of giving me a subclass of datetime is to prevent
access to the actual class, that can be circumvented:
>>> real_datetime = datetime.datetime.mro()[1]
>>> real_datetime
But I'm not sure what good that is.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, Apr 8, 2016 at 2:50 AM, Nick Coghlan wrote: > On 7 April 2016 at 03:26, Brett Cannon wrote: > > > Method or attribute? (changes what kind of one-liner you might use in > > libraries, but I think historically all protocols have been methods and > the > > serialized string representation might be costly to build) > couldn't it be a property? > Method, as long as there's a helper function somewhere what has the helper function got to do with whether it's a method or attribute (would we call a property an attribute here?) > Built-in? (name is dependent on #1 if we add one) > > os.fspath (alongside os.fsencode and os.fsdecode) > > (Putting this in a module low in the dependency stack makes it easy > for other modules to access without pulling in all of pathlib's > dependencies) Iike that -- though still =0.5 on having one at all -- this is only going to be used by the stdlib and other path-using libraries, not user code -- is that that hard to call obj.__fspath__() ? > Add the method/attribute to str? (I assume so, much like __index__() is on > > int, but I have not seen it explicitly stated so I would rather clarify > it) > I thought the whole point off all this is that not any old string can be a path! (whereas any int can be an index). Unless we go with Chris A's suggestion that this be a more generic lossless string protocol, rather than just for paths. > It's worth summarising in a PEP at least for communications purposes - > very easy for folks that don't follow python-dev to miss otherwise. > I'd say add it to the existing pathlib PEP -- along with the extra discussion of why Path does not inherit from str. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2016-04-01 - 2016-04-08)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open5477 ( +6)
closed 32993 (+22)
total 38470 (+28)
Open issues with patches: 2381
Issues opened (23)
==
#26686: email.parser stops parsing headers too soon.
http://bugs.python.org/issue26686 opened by msapiro
#26687: Use Py_RETURN_NONE in sqlite3 module
http://bugs.python.org/issue26687 opened by berker.peksag
#26689: Add `has_flag` method to `distutils.CCompiler`
http://bugs.python.org/issue26689 opened by sylvain.corlay
#26692: cgroups support in multiprocessing
http://bugs.python.org/issue26692 opened by Satrajit Ghosh
#26693: Exception ignored in: http://bugs.python.org/issue26693 opened by skydoom
#26694: Disasembler fall with Key Error while disassemble obfuscated c
http://bugs.python.org/issue26694 opened by pulina
#26695: pickle and _pickle accelerator have different behavior when un
http://bugs.python.org/issue26695 opened by josh.r
#26696: Document collections.abc.ByteString
http://bugs.python.org/issue26696 opened by brett.cannon
#26697: tkFileDialog crash on askopenfilename Python 2.7 64-bit Win7
http://bugs.python.org/issue26697 opened by Eric Johnson
#26698: Tk DPI awareness
http://bugs.python.org/issue26698 opened by westley.martinez
#26699: locale.str docstring is incorrect: "Convert float to integer"
http://bugs.python.org/issue26699 opened by mark.dickinson
#26700: Make digest_size a class variable
http://bugs.python.org/issue26700 opened by rhettinger
#26701: Documentation for int constructor mentions __int__ but not __t
http://bugs.python.org/issue26701 opened by Robert Smallshire2
#26702: A better assert statement
http://bugs.python.org/issue26702 opened by barry
#26703: Socket state corrupts when original socket object goes out of
http://bugs.python.org/issue26703 opened by JoshN
#26704: unittest.mock.patch: Double patching instance method: Attribut
http://bugs.python.org/issue26704 opened by asottile
#26705: logging.Handler.handleError should be called from logging.Hand
http://bugs.python.org/issue26705 opened by palaviv
#26706: Update OpenSSL version in readme
http://bugs.python.org/issue26706 opened by scw
#26707: plistlib fails to parse bplist with 0x80 UID values
http://bugs.python.org/issue26707 opened by slo.sleuth
#26708: Constify C string pointers in the posix module
http://bugs.python.org/issue26708 opened by serhiy.storchaka
#26710: ConfigParser: Values in DEFAULT section override defaults pass
http://bugs.python.org/issue26710 opened by Marc.Abramowitz
#26711: Fix comparison of plistlib.Data
http://bugs.python.org/issue26711 opened by serhiy.storchaka
#26712: Unify (r)split(), (l/r)strip() method tests
http://bugs.python.org/issue26712 opened by martin.panter
Most recent 15 issues with no replies (15)
==
#26706: Update OpenSSL version in readme
http://bugs.python.org/issue26706
#26700: Make digest_size a class variable
http://bugs.python.org/issue26700
#26699: locale.str docstring is incorrect: "Convert float to integer"
http://bugs.python.org/issue26699
#26697: tkFileDialog crash on askopenfilename Python 2.7 64-bit Win7
http://bugs.python.org/issue26697
#26696: Document collections.abc.ByteString
http://bugs.python.org/issue26696
#26695: pickle and _pickle accelerator have different behavior when un
http://bugs.python.org/issue26695
#26694: Disasembler fall with Key Error while disassemble obfuscated c
http://bugs.python.org/issue26694
#26692: cgroups support in multiprocessing
http://bugs.python.org/issue26692
#26672: regrtest missing in the module name
http://bugs.python.org/issue26672
#26669: time.localtime(float("NaN")) does not raise a ValueError on al
http://bugs.python.org/issue26669
#26667: Update importlib to accept pathlib.Path objects
http://bugs.python.org/issue26667
#26665: pip is not bootstrapped by default on 2.7
http://bugs.python.org/issue26665
#26663: asyncio _UnixWritePipeTransport._close abandons unflushed writ
http://bugs.python.org/issue26663
#26661: python fails to locate system libffi
http://bugs.python.org/issue26661
#26660: tempfile.TemporaryDirectory() cleanup exception on Windows if
http://bugs.python.org/issue26660
Most recent 15 issues waiting for review (15)
=
#26712: Unify (r)split(), (l/r)strip() method tests
http://bugs.python.org/issue26712
#26711: Fix comparison of plistlib.Data
http://bugs.python.org/issue26711
#26708: Constify C string pointers in the posix module
http://bugs.python.org/issue26708
#26707: plistlib fails to parse bplist with 0x80 UID values
http://bugs.python.org/issue26707
#26706: Update OpenSSL version in readme
http://bugs.python.org/issue26706
#26705: logging.Handler.handleError should be called from logging.Ha
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
Nick Coghlan wrote: > On 7 April 2016 at 03:26, Brett Cannon wrote: >> >> Name: __path__, __fspath__, or something else? > > __fspath__ > I think I might like this dunder name because it does not clutter the list of regular methods and attributes, and is perhaps more pythonic. >> Method or attribute? (changes what kind of one-liner you might use in >> libraries, but I think historically all protocols have been methods and the >> serialized string representation might be costly to build) > > Method, as long as there's a helper function somewhere As a further minor benefit of it being a method, it may be easier to distinguish it from from `__path__`, which is an iterable attribute. >> Built-in? (name is dependent on #1 if we add one) > > os.fspath (alongside os.fsencode and os.fsdecode) > > (Putting this in a module low in the dependency stack makes it easy > for other modules to access without pulling in all of pathlib's > dependencies) Strong +1 on putting it in os. This should also be implemented in DirEntry, instances of which are "yielded" by os.scandir. Also, you have a strong case regarding naming with the 'fs' prefix. It is also easier to read fspath as f-s-path than it is to read ospath as o-s-path, because ospath could also be pronounced as a single (meaningless?) word. I'm still thinking a little bit about 'pathname', which to me sounds more like a string than fspath does [1]. It would be nice to have the string/path distinction especially when pathlib adoption grows larger. But who knows, maybe somewhere in the far future, no-one will care much about fspath, fsencode, fsdecode or os.path. >> Add the method/attribute to str? (I assume so, much like __index__() is on >> int, but I have not seen it explicitly stated so I would rather clarify it) > > Makes sense If added to str, it should also be added to bytes. But will that then return str or bytes? See also the next point. > Expand the C API to have something like PyObject_Path()? > > PyUnicode_FromFSPath, perhaps? The return type is well-defined here, > so it can be done as an alternate constructor, and the C API > counterparts of os.fsdecode and os.fsencode are PyUnicode functions > (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) What about DirEntry, which may have a bytes representation? I would expect the function return type of os.fspath to be Union[str, bytes], unless bytes pathnames are decoded with surrogate escapes. [1] https://mail.python.org/pipermail/python-ideas/2016-April/039595.html PS. I have been reading this list occasionally on the google groups mirror, and I now subscribed to it just to send this. (BTW, I probably broke the thread, as I did not have Nick's email in my inbox to reply to. Sorry about that.) I'll have to mention that I was surprised, to say the least, to find that the pathlib discussion had moved here from python-ideas, where I had mentioned I was working on a proposal. Then, I also found that the solution discussed here was seemingly an improved version of what I had proposed on python-ideas somewhat earlier [1], but did not get any reactions to. While I can only make guesses about what happened, these kinds of things easily make you go from "Hey, maybe I'll be able to do something to improve Python!" to "These people don't seem to want me here or appreciate my efforts.". Not to accuse anyone in particular; just to let people know. Anyway, I somehow got sucked into thinking deeply about pathlib etc. (which I do use). Not that I really have much at stake here, except spending ridiculous amounts of time thinking about paths, mainly during my Easter holidays and after that. I really had a hard time explaining to friends and family what the heck I was doing ;). ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens wrote: > Anyway the code is at https://github.com/jribbens/unsafe > It requires Python 3.4 or later (it could probably be made to work on > Python 2.7 as well, but it would need some changes). Not being a security expert, I'm not the best one to try to break it maliciously; but I can break things accidentally. Pull request sent through. :) ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On 04/08/2016 08:41 AM, Brett Cannon wrote: On Fri, 8 Apr 2016 at 08:33 Ethan Furman wrote: >> Brett previously queried: Add the method/attribute to str? (I assume so, much like __index__() is on int, but I have not seen it explicitly >>> stated so I would rather clarify it) What will this do? Return a Path or a str? I don't think >> we need either. When I brought this up it was to return self. Okay, thanks. Chris Angelico and I have been asked by Guido to work together to come up with a proposal after all the discussions are finished and it will most likely be a patch to the pathlib PEP. Cool. I wasn't looking forward to that part. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On 04/08/2016 09:04 AM, Chris Barker wrote: On Fri, Apr 8, 2016 at 2:50 AM, Nick Coghlan wrote: Method, as long as there's a helper function somewhere what has the helper function got to do with whether it's a method or attribute (would we call a property an attribute here?) Built-in? (name is dependent on #1 if we add one) os.fspath (alongside os.fsencode and os.fsdecode) [...] this is only going to be used by the stdlib and other > path-using libraries, not user code -- is that that hard to > call obj.__fspath__() ? 1) user code may call it 2) folks who write libraries are still users ;) 3) using __dunder__s directly is usually poor form. I thought the whole point off all this is that not any old string can be a path! (whereas any int can be an index). Unless we go with Chris A's suggestion that this be a more generic lossless string protocol, rather than just for paths. That does seem to be a valid point against str.__fspath__. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: > I'm still thinking a little bit about 'pathname', which to me sounds > more like a string than fspath does [1]. I like that a lot - or even "__pathstr__" or "__pathstring__" after all, we're making a big deal out of the fact that a path is *not a string*, but rather a string is a *representation* (or serialization) of a path. > If added to str, it should also be added to bytes. ouch! not sure I want to go there, though... > I'll have to mention that I was surprised, to > say the least, to find that the pathlib discussion had moved here from > python-ideas, where I had mentioned I was working on a proposal. ... > While I can only make > guesses about what happened, these kinds of things easily make you go > from "Hey, maybe I'll be able to do something to improve Python!" to > "These people don't seem to want me here or appreciate my efforts.". > For the record, this is pretty rare -- and it was announced on -ideas that the discussion had started up here -- maybe you missed that post? I think in this case, there were ideas over on -ideas, but then it was decided (by whom, who knows?) that the goal of supporting PAth in the stdlib was decided upon, so it was time to talk implementation, rather than ideas -- thus python-dev. In fact, the implementation turned out to be less straightforward than originally thought, so maybe it should have stayed on -ideas, but there you go. > Not to accuse anyone in particular; just to let people know. Anyway, I > somehow got sucked into thinking deeply about pathlib etc. (which I do > use). Not that I really have much at stake here, except spending > ridiculous amounts of time thinking about paths, mainly during my > Easter holidays and after that. I really had a hard time explaining to > friends and family what the heck I was doing ;). speaking only for me - thanks for your contribution -- I'm glad you found the discussion here. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On Fri, Apr 08, 2016 at 05:49:12PM +0200, Marcin Kościelnicki wrote:
> On 08/04/16 16:18, Jon Ribbens wrote:
> That one is trivially fixable, but here goes:
>
> async def a():
> global c
> c = b.cr_frame.f_back.f_back.f_back
>
> b = a()
> b.send(None)
> c.f_builtins['print']('broken')
Ah, I've not used Python 3.5, and I can't find any documentation on
this cr_frame business, but I've added cr_frame and f_back to the
disallowed attributes list.
> Also, if the point of giving me a subclass of datetime is to prevent access
> to the actual class, that can be circumvented:
>
> >>> real_datetime = datetime.datetime.mro()[1]
> >>> real_datetime
>
>
> But I'm not sure what good that is.
It means you can alter the datetime class that is used by the
containing application, which is bad - you could lie to it about
what day it is for example ;-)
I've made it so instead of a direct subclass it now makes an
intermediate subclass which makes mro() return an empty list.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, 8 Apr 2016 at 09:05 Chris Barker wrote: > On Fri, Apr 8, 2016 at 2:50 AM, Nick Coghlan wrote: > >> On 7 April 2016 at 03:26, Brett Cannon wrote: >> > > >> > Method or attribute? (changes what kind of one-liner you might use in >> > libraries, but I think historically all protocols have been methods and >> the >> > serialized string representation might be costly to build) >> > > couldn't it be a property? > A property is a method pretending to be an attribute, so yes. :) > > >> Method, as long as there's a helper function somewhere > > > what has the helper function got to do with whether it's a method or > attribute (would we call a property an attribute here?) > Yes, a property is an attribute in this instance. And it somewhat tweaks how simple of a one-liner is needed which in turn makes the function either nearly redundant or helpful. With an attribute: getattr(path, '__ospath__', path) With a method: path.__ospath__() if hasattr(path, '__ospath__') else path > > > Built-in? (name is dependent on #1 if we add one) >> >> os.fspath (alongside os.fsencode and os.fsdecode) >> >> (Putting this in a module low in the dependency stack makes it easy >> for other modules to access without pulling in all of pathlib's >> dependencies) > > > Iike that -- though still =0.5 on having one at all -- this is only going > to be used by the stdlib and other path-using libraries, not user code -- > is that that hard to call obj.__fspath__() ? > With a function we can add some type checking so that you know you are getting back a string and not something else like an file descriptor int or something. > > > Add the method/attribute to str? (I assume so, much like __index__() is >> on >> > int, but I have not seen it explicitly stated so I would rather clarify >> it) >> > > I thought the whole point off all this is that not any old string can be a > path! (whereas any int can be an index). Unless we go with Chris A's > suggestion that this be a more generic lossless string protocol, rather > than just for paths. > The whole point is to not treat a path object like any old string. We still have to support a string someone created that is a valid path. Remember, what we're trying to avoid is people simply doing `str(path)` everywhere since that works with e.g. None. > > >> It's worth summarising in a PEP at least for communications purposes - >> very easy for folks that don't follow python-dev to miss otherwise. >> > > I'd say add it to the existing pathlib PEP -- along with the extra > discussion of why Path does not inherit from str. > That's the plan. -Brett > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > [email protected] > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, 8 Apr 2016 at 09:13 Koos Zevenhoven wrote: > Nick Coghlan wrote: > > On 7 April 2016 at 03:26, Brett Cannon wrote: > >> > >> Name: __path__, __fspath__, or something else? > > > > __fspath__ > > > > I think I might like this dunder name because it does not clutter the > list of regular methods and attributes, and is perhaps more pythonic. > > >> Method or attribute? (changes what kind of one-liner you might use in > >> libraries, but I think historically all protocols have been methods and > the > >> serialized string representation might be costly to build) > > > > Method, as long as there's a helper function somewhere > > As a further minor benefit of it being a method, it may be easier to > distinguish it from from `__path__`, which is an iterable attribute. > > >> Built-in? (name is dependent on #1 if we add one) > > > > os.fspath (alongside os.fsencode and os.fsdecode) > > > > (Putting this in a module low in the dependency stack makes it easy > > for other modules to access without pulling in all of pathlib's > > dependencies) > > Strong +1 on putting it in os. This should also be implemented in > DirEntry, instances of which are "yielded" by os.scandir. > > Also, you have a strong case regarding naming with the 'fs' prefix. It > is also easier to read fspath as f-s-path than it is to read ospath as > o-s-path, because ospath could also be pronounced as a single > (meaningless?) word. > > I'm still thinking a little bit about 'pathname', which to me sounds > more like a string than fspath does [1]. It would be nice to have the > string/path distinction especially when pathlib adoption grows larger. > But who knows, maybe somewhere in the far future, no-one will care > much about fspath, fsencode, fsdecode or os.path. > > >> Add the method/attribute to str? (I assume so, much like __index__() is > on > >> int, but I have not seen it explicitly stated so I would rather clarify > it) > > > > Makes sense > > If added to str, it should also be added to bytes. But will that then > return str or bytes? See also the next point. > > > Expand the C API to have something like PyObject_Path()? > > > > PyUnicode_FromFSPath, perhaps? The return type is well-defined here, > > so it can be done as an alternate constructor, and the C API > > counterparts of os.fsdecode and os.fsencode are PyUnicode functions > > (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) > > What about DirEntry, which may have a bytes representation? I would > expect the function return type of os.fspath to be Union[str, bytes], > unless bytes pathnames are decoded with surrogate escapes. > > [1] https://mail.python.org/pipermail/python-ideas/2016-April/039595.html > > > PS. I have been reading this list occasionally on the google groups > mirror, and I now subscribed to it just to send this. (BTW, I probably > broke the thread, as I did not have Nick's email in my inbox to reply > to. Sorry about that.) I'll have to mention that I was surprised, to > say the least, to find that the pathlib discussion had moved here from > python-ideas, where I had mentioned I was working on a proposal. Then, > I also found that the solution discussed here was seemingly an > improved version of what I had proposed on python-ideas somewhat > earlier [1], but did not get any reactions to. While I can only make > guesses about what happened, these kinds of things easily make you go > from "Hey, maybe I'll be able to do something to improve Python!" to > "These people don't seem to want me here or appreciate my efforts.". > Not to accuse anyone in particular; just to let people know. Anyway, I > somehow got sucked into thinking deeply about pathlib etc. (which I do > use). Not that I really have much at stake here, except spending > ridiculous amounts of time thinking about paths, mainly during my > Easter holidays and after that. I really had a hard time explaining to > friends and family what the heck I was doing ;). > Since I kicked up the discussion here on python-dev, I can explain what happened. After the python-ideas threads kicked up I realized I was not using pathlib in importlib and there were a handful of places it could be supported. But since pathlib is provisional I didn't want to have to start making the stdlib support it if we removed the whole module itself. So I simply asked over here on python-dev what it would take to remove the provisional label from pathlib. People then pulled over the python-ideas discussion of what people were upset about in regards to pathlib to help decide what it would require to remove the provisional label and the conversation forked (I also assumed Guido and others had muted the discussion over on python-ideas so it would have been a new thread somewhere regardless). And then when I realized what had happened I was going to reply to one of your emails on python-ideas to point out the bifurcation but someone beat me to it. So the whole thing just became a tangled mess of discussion. :
Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
On Sat, Apr 09, 2016 at 02:20:49AM +1000, Chris Angelico wrote: > On Sat, Apr 9, 2016 at 12:18 AM, Jon Ribbens > wrote: > > Anyway the code is at https://github.com/jribbens/unsafe > > It requires Python 3.4 or later (it could probably be made to work on > > Python 2.7 as well, but it would need some changes). > > Not being a security expert, I'm not the best one to try to break it > maliciously; but I can break things accidentally. Pull request sent > through. :) Thanks, I've merged that in. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Fri, 8 Apr 2016 at 09:39 Ethan Furman wrote: > On 04/08/2016 09:04 AM, Chris Barker wrote: > > On Fri, Apr 8, 2016 at 2:50 AM, Nick Coghlan wrote: > > >> Method, as long as there's a helper function somewhere > > > > what has the helper function got to do with whether it's a method or > > attribute (would we call a property an attribute here?) > > > >> Built-in? (name is dependent on #1 if we add one) > > > > os.fspath (alongside os.fsencode and os.fsdecode) > > > > [...] this is only going to be used by the stdlib and other > > path-using libraries, not user code -- is that that hard to > > call obj.__fspath__() ? > > 1) user code may call it > 2) folks who write libraries are still users ;) > 3) using __dunder__s directly is usually poor form. > > > I thought the whole point off all this is that not any old string can be > > a path! (whereas any int can be an index). Unless we go with Chris A's > > suggestion that this be a more generic lossless string protocol, rather > > than just for paths. > > That does seem to be a valid point against str.__fspath__. > Yep, and I'm expecting we won't want that at this point. The fact that paths need strings for low-level OS stuff is a historical and technical detail, so no need to drag the entire str type into it if we can provide a reasonable helper function (for either the ABC or magic method solution). ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On 04/08/2016 09:42 AM, Chris Barker wrote: On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: While I can only make guesses about what happened, these kinds of >> things easily make you go from "Hey, maybe I'll be able to do something >> to improve Python!" to "These people don't seem to want me here or >> appreciate my efforts.". Ouch, sorry about that. Glad to have you on -Dev, too. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: > On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >> >> I'm still thinking a little bit about 'pathname', which to me sounds >> more like a string than fspath does [1]. > > > I like that a lot - or even "__pathstr__" or "__pathstring__" > > after all, we're making a big deal out of the fact that a path is *not a > string*, but rather a string is a *representation* (or serialization) of a > path. For me, the point here is the reverse: that any str is not a path, and that it is misleading to call it *path* when whole point is to make it *not* a specialized path object but a plain string. I think it's ok to think of a path as special kind of string. For instance, an URI is explicitly defined as a *sequence of characters*, and URIs can be thought of as a more recent, improved and broadened concept than paths. This is the point of view I took in my recent proposal, but I don't think it's the only valid way to think about paths "in theory". I like the "serialization" interpretation as well, but i tend to think that that string serialization is what is called a path. Anyway, I don't think these philosophical considerations should dictate how Python is implemented. But it is always good to also have a valid theoretical point of view to back up a design decision. > For the record, this is pretty rare -- and it was announced on -ideas that > the discussion had started up here -- maybe you missed that post? If you mean in Ethan's response to my proposal, I noticed that, but the discussions here had already gone quite far by that time. Even more so by the time I had time to see what was going on. I do have to say this is not the first time I felt there was some sort of hostility towards newcomers on python-ideas. Sure, it might be partly because those people don't know the culture on the list, but I'm not sure if that should be used as an excuse. -Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pathlib enhancments - method name only
On 04/08/2016 10:46 AM, Koos Zevenhoven wrote: > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >>> I'm still thinking a little bit about 'pathname', which to me sounds >>> more like a string than fspath does. >> >> >> I like that a lot - or even "__pathstr__" or "__pathstring__" >> after all, we're making a big deal out of the fact that a path is >> *not a string*, but rather a string is a *representation* (or >> serialization) of a path. That's a decent point. So the plausible choices are, I think: - __fspath__ # File System Path -- possible confusion with Path - __fsstr__ # File System String - __fspathstr__ # File System Path String -- zero ambiguity, but # what a mouthful -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 11:13 AM, Ethan Furman wrote: > So the plausible choices are, I think: > > - __fspath__ # File System Path -- possible confusion with Path > > - __fsstr__ # File System String I think we really need "path" in there somewhere > > - __fspathstr__ # File System Path String -- zero ambiguity, but > # what a mouthful > we rejected plain old __path__ because this is already ued in another context, but if we add "str" on the end, that's not longer an issue, so do we need the "fs"? __pathstr__ # pathstring -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, 8 Apr 2016 at 11:13 Ethan Furman wrote: > On 04/08/2016 10:46 AM, Koos Zevenhoven wrote: > > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: > >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: > > >>> I'm still thinking a little bit about 'pathname', which to me sounds > >>> more like a string than fspath does. > >> > >> > >> I like that a lot - or even "__pathstr__" or "__pathstring__" > >> after all, we're making a big deal out of the fact that a path is > >> *not a string*, but rather a string is a *representation* (or > >> serialization) of a path. > > That's a decent point. > > So the plausible choices are, I think: > > - __fspath__ # File System Path -- possible confusion with Path > +1 > > - __fsstr__ # File System String > -1 Looks like a cat walked across my keyboard or someone trying to come up with a trendy startup name. > > - __fspathstr__ # File System Path String -- zero ambiguity, but > # what a mouthful > -1 See above. I personally still like __ospath__ as well. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 9:20 PM, Chris Barker wrote: > > we rejected plain old __path__ because this is already ued in another > context, but if we add "str" on the end, that's not longer an issue, so do > we need the "fs"? > > __pathstr__ # pathstring > Or perhaps __pathstring__ in case it may be or return byte strings. -Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven wrote: > > > > __pathstr__ # pathstring > > > > Or perhaps __pathstring__ in case it may be or return byte strings. > I'm fine with __pathstring__ , but I thought it was already decided that it would NOT return a bytestring! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker wrote: > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven wrote: >> >> > >> > __pathstr__ # pathstring >> > >> >> Or perhaps __pathstring__ in case it may be or return byte strings. > > > I'm fine with __pathstring__ , but I thought it was already decided that it > would NOT return a bytestring! I sincerely hope that's been settled on. There's no reason to have this ever return anything other than a str. (Famous last words, I know.) ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, 8 Apr 2016 at 12:10 Chris Angelico wrote: > On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker > wrote: > > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven > wrote: > >> > >> > > >> > __pathstr__ # pathstring > >> > > >> > >> Or perhaps __pathstring__ in case it may be or return byte strings. > > > > > > I'm fine with __pathstring__ , but I thought it was already decided that > it > > would NOT return a bytestring! > > I sincerely hope that's been settled on. There's no reason to have > this ever return anything other than a str. (Famous last words, I > know.) > It has been settled: pathlib.Path itself won't accept bytes anyway so there's no reason to expect this to ever return anything but str. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, 08 Apr 2016 19:24:44 -, Brett Cannon wrote: > On Fri, 8 Apr 2016 at 12:10 Chris Angelico wrote: > > > On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker > > wrote: > > > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven > > wrote: > > >> > > >> > > > >> > __pathstr__ # pathstring > > >> > > > >> > > >> Or perhaps __pathstring__ in case it may be or return byte strings. But there are other paths than OS file system paths. I prefer __fspath__ or __os_path__ myself. I think the fact that it is a string is implied by the fact that it is getting us the thing we can pass to the os (since Python3 deals with os paths as strings unless you specify otherwise, only converting them back to bytes, on unix, at the last moment). Heh, although I suppose one could make the argument that it should return whatever the native OS wants, and save the low level code from having to do that? Pass the path object all the way down to that "final step" in the C layer? (Just ignore me, I'm sure I'm only making trouble :) --David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 11:39 PM, R. David Murray wrote: > On Fri, 08 Apr 2016 19:24:44 -, Brett Cannon wrote: >> On Fri, 8 Apr 2016 at 12:10 Chris Angelico wrote: >> >> > On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker >> > wrote: >> > > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven >> > wrote: >> > >> >> > >> > >> > >> > __pathstr__ # pathstring >> > >> > >> > >> >> > >> Or perhaps __pathstring__ in case it may be or return byte strings. > > But there are other paths than OS file system paths. I prefer > __fspath__ or __os_path__ myself. I think the fact that it is a string > is implied by the fact that it is getting us the thing we can pass > to the os (since Python3 deals with os paths as strings unless you > specify otherwise, only converting them back to bytes, on unix, at the last > moment). > > Heh, although I suppose one could make the argument that it should > return whatever the native OS wants, and save the low level code > from having to do that? Pass the path object all the way down > to that "final step" in the C layer? (Just ignore me, I'm sure > I'm only making trouble :) My favorites are fspath and pathname, and since this is a dunder methdod, it is not as crucial what it is called. I have the feeling the consensus is converging towards fspath? I'll comment on the bytes issue in the other thread. Boy these threads are all over the place! -Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Incomplete Internationalization in Argparse Module
Hello, all. I was wondering if the following string was left untouched by gettext for a purpose (from line 720 of argparse.py, in class ArgumentError): 'argument %(argument_name)s: %(message)s' There may be other untranslatable strings in the argparse module, but I have yet to encounter them in the wild. Thank you. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Incomplete Internationalization in Argparse Module
On Fri, 8 Apr 2016 at 14:05 Grady Martin wrote: > Hello, all. I was wondering if the following string was left untouched by > gettext for a purpose (from line 720 of argparse.py, in class > ArgumentError): > > 'argument %(argument_name)s: %(message)s' > > There may be other untranslatable strings in the argparse module, but I > have yet to encounter them in the wild. > Probably so that anyone introspecting on the error message can count on somewhat of a consistent format (comes into play with doctest typically). ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Fri, Apr 8, 2016 at 8:34 PM, Brett Cannon wrote: > On Fri, 8 Apr 2016 at 09:39 Ethan Furman wrote: >> > I thought the whole point off all this is that not any old string can be >> > a path! (whereas any int can be an index). Unless we go with Chris A's >> > suggestion that this be a more generic lossless string protocol, rather >> > than just for paths. >> >> That does seem to be a valid point against str.__fspath__. > > Yep, and I'm expecting we won't want that at this point. The fact that paths > need strings for low-level OS stuff is a historical and technical detail, so > no need to drag the entire str type into it if we can provide a reasonable > helper function (for either the ABC or magic method solution). I'm not sure I understand what these points are about. Anyway, disallowing str or bytes as pathnames will break backwards compatibility if done at some point in the future. There's no way around that. But regarding all this talk of mine about bytes is because it has not been completely clear to me if something can break when converting a bytes path to str. I did originally propose guaranteeing a str, but I am so far only 85% convinced that that does not cause any problems. I understand that fsencode(fsdecode(bytes_path)) should always be equal to bytes_path. But can some other path operations fail when there are surrogates in the strings? And again, not to forget DirEntry, which may have a byte string path. Either way, I suppose os.fspath should accept anything that has __fspath__ or is a str or bytes (whether these have the dunder method or not). Then the options are either to return Union[str, bytes] or to always return str. And if the latter does not cause any problems, I like it way better, and it seems others would do too. And in that case it would probably be time to deprecate bytes paths on posix too (on Windows, this is already the case). But do we know that converting all paths to str does not cause any problems? -Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Fri, 8 Apr 2016 at 14:23 Koos Zevenhoven wrote: > On Fri, Apr 8, 2016 at 8:34 PM, Brett Cannon wrote: > > On Fri, 8 Apr 2016 at 09:39 Ethan Furman wrote: > >> > I thought the whole point off all this is that not any old string can > be > >> > a path! (whereas any int can be an index). Unless we go with Chris A's > >> > suggestion that this be a more generic lossless string protocol, > rather > >> > than just for paths. > >> > >> That does seem to be a valid point against str.__fspath__. > > > > Yep, and I'm expecting we won't want that at this point. The fact that > paths > > need strings for low-level OS stuff is a historical and technical > detail, so > > no need to drag the entire str type into it if we can provide a > reasonable > > helper function (for either the ABC or magic method solution). > > I'm not sure I understand what these points are about. It means we most likely won't add a new method to str in regards to this proposal. > Anyway, > disallowing str or bytes as pathnames will break backwards > compatibility if done at some point in the future. There's no way > around that. > No one is proposing disallowing str or bytes for a pre-existing API that supports either. The whole point of this is to make APIs work with strings and pathlib. > > But regarding all this talk of mine about bytes is because it has not > been completely clear to me if something can break when converting a > bytes path to str. I did originally propose guaranteeing a str, but I > am so far only 85% convinced that that does not cause any problems. Depends on your definition of "problem". If you somehow blindly converted a bytes object representing a path to a str without knowing its encoding you will definitely break someone silently (and even os.fsdecode() isn't fool-proof thanks to multiple encodings on a single file system). > I > understand that fsencode(fsdecode(bytes_path)) should always be equal > to bytes_path. But can some other path operations fail when there are > surrogates in the strings? And again, not to forget DirEntry, which > may have a byte string path. > At this point no one wants to touch bytes paths. If you need that level of control because of multiple encodings within a single file system then you will probably have to stick with managing bytes paths on your own to get the encoding right. And just because DirEntry supports bytes doesn't mean that any magic method it gains has to carry that forward (it can always raise a TypeError if necessary). > > Either way, I suppose os.fspath should accept anything that has > __fspath__ or is a str or bytes (whether these have the dunder method > or not). Maybe. I'm not sure if we will want to down that route of both bytes and str being supported out of the same function as that gets messy quickly. The main reason os.scandir() supports it is so it can be a drop-in replacement for os.listdir(). It really depends on how we choose to structure the function in terms of just doing the right thing for objects that follow the protocol or if we want to introduce some required structure for the resulting path and implement some type guarantees so you have a better idea of what you will be working with after calling the function. > Then the options are either to return Union[str, bytes] or to > always return str. And if the latter does not cause any problems, I > like it way better, and it seems others would do too. You don't have to convert byte paths to str, you can simply raise an exception in the face of them. > And in that case > it would probably be time to deprecate bytes paths on posix too (on > Windows, this is already the case). > Can't do that as Stephen Turnbull will tell you. :) At best we can marginalize the support of bytes-based paths to only low-level APIs exposed through the os package. -Brett ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: > I personally still like __ospath__ as well. Same here. The strings are essentially an OS-dependent serialization, rather than related to a particular file system. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
> On Apr 8, 2016, at 3:00 PM, Eric Snow wrote: > >> On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: >> I personally still like __ospath__ as well. > > Same here. The strings are essentially an OS-dependent serialization, > rather than related to a particular file system. Huh? I though the strings were a OS-independent, human readable serialization and interchange format. Bytes would be the OS-dependent serialization. But yes, I suppose the file-system-level version would be inodes or something. But this is a string that represents a path, thus __pathstr__. And the term "path" is used all over the place (including os.path and pathlib) for this particular type of path, so I don't see why we need the "fs" or "os", other than the fact that __path__ is already taken. But I'm looking forward to using this bike shed regardless of its color, so that's the last I'll comment on that. -CHB ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, 8 Apr 2016 at 15:21 Chris Barker - NOAA Federal < [email protected]> wrote: > > On Apr 8, 2016, at 3:00 PM, Eric Snow > wrote: > > > >> On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: > >> I personally still like __ospath__ as well. > > > > Same here. The strings are essentially an OS-dependent serialization, > > rather than related to a particular file system. > > Huh? I though the strings were a OS-independent, human readable > serialization and interchange format. > Depends if you use `/` or `\` as your path separator if they are truly OS-independent. :) -Brett > > Bytes would be the OS-dependent serialization. > > But yes, I suppose the file-system-level version would be inodes or > something. > > But this is a string that represents a path, thus __pathstr__. And the > term "path" is used all over the place (including os.path and pathlib) > for this particular type of path, so I don't see why we need the "fs" > or "os", other than the fact that __path__ is already taken. > > But I'm looking forward to using this bike shed regardless of its > color, so that's the last I'll comment on that. > > -CHB > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 3:57 PM, Eric Snow wrote: > On Fri, Apr 8, 2016 at 12:25 PM, Brett Cannon wrote: >> I personally still like __ospath__ as well. > > Same here. The strings are essentially an OS-dependent serialization, > rather than related to a particular file system. Hmm. It's important to note the distinction between a standardized representation of a path and the OS-dependent representation. That is essentially the same distinction as provided by Go's "path" vs. "path/fliepath" packages. pathlib provides an abstraction of FS paths, but does it provide a standardized representation? From what I can tell you only ever get some OS-dependent representation. All this matters because it impacts the value returned from __ospath__(). Should it return the string representation of the path for the current OS or some standardized representation? I'd expect the former. However, if that is the expectation then something like pathlib.PureWindowsPath will give you the wrong thing if your current OS is linux. pathlib.PureWindowsPath.__ospath__() would have to fail or first internally convert to pathlib.PurePosixPath? On the other hand, it seems like the caller should be in charge of deciding the required meaning. That implies that returning a standardized representation or even something like a pathlib.PureGenericPath would be more appropriate. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Sat, Apr 9, 2016 at 12:53 AM, Brett Cannon wrote: > On Fri, 8 Apr 2016 at 14:23 Koos Zevenhoven wrote: > > At this point no one wants to touch bytes paths. If you need that level of > control because of multiple encodings within a single file system then you > will probably have to stick with managing bytes paths on your own to get the > encoding right. What does this mean? I assume you don't mean os.path.* would stop dealing with bytes? And if not, then you seem to mean that os.fspath would do nothing except call .__fspath__(). In that case, I think we should go back to it being an attribute (or property) and a variation of the now very famous idiom getattr(path, '__fspath__', path) and perhaps have os.fspath do exactly that. > And just because DirEntry supports bytes doesn't mean that any magic method > it gains has to carry that forward (it can always raise a TypeError if > necessary). No, but what if some code gets pathnames from whatever other places and passes them on to os.scandir. Whenever it happens to get a bytes path, a TypeError gets raised, but only when it picks one of the DirEntry objects and for instance tries to open(...) it. Of course, I'm not sure how common this is. > It really depends on how we choose to structure the > function in terms of just doing the right thing for objects that follow the > protocol or if we want to introduce some required structure for the > resulting path and implement some type guarantees so you have a better idea > of what you will be working with after calling the function. Do you have an example of potential 'required structure'? >> Then the options are either to return Union[str, bytes] or to >> always return str. And if the latter does not cause any problems, I >> like it way better, and it seems others would do too. > > You don't have to convert byte paths to str, you can simply raise an > exception in the face of them. > I thought the point was for existing APIs to start supporting path objects, wouldn't raising an exception break the API? -Koos ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On 4/8/2016 11:25 AM, Brett Cannon wrote: I personally still like __ospath__ as well. +1. Because they aren't always files... but what else they might be is os dependent. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On 4/8/2016 3:28 PM, Eric Snow wrote: All this matters because it impacts the value returned from __ospath__(). Should it return the string representation of the path for the current OS or some standardized representation? I'd expect the former. However, if that is the expectation then something like pathlib.PureWindowsPath will give you the wrong thing if your current OS is linux. pathlib.PureWindowsPath.__ospath__() would have to fail or first internally convert to pathlib.PurePosixPath? Now that Windows 10++ will run Ubuntu apps, will Python be able to tell the difference for when it should return Windows-format paths and Posix-format paths? (I'm sure the answer is yes, the Python-for-Ubuntu running on Windows would do the latter, and the Python-for-Windows would do the former. Although, it is not clear what sys.platform will return, yet...) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On 04/08/2016 04:05 PM, Koos Zevenhoven wrote:
On Sat, Apr 9, 2016 at 12:53 AM, Brett Cannon wrote:
On Fri, 8 Apr 2016 at 14:23 Koos Zevenhoven wrote:
At this point no one wants to touch bytes paths. If you need that level of
control because of multiple encodings within a single file system then you
will probably have to stick with managing bytes paths on your own to get the
encoding right.
What does this mean? I assume you don't mean os.path.* would stop
dealing with bytes?
No, it does not mean that. It means the stuff in place won't change,
but the stuff we're adding now to integrate with Path will only support
str (which is one reason why os.path isn't going to die).
And if not, then you seem to mean that os.fspath
would do nothing except call .__fspath__().
Fair point. So it should be something like this:
def fspath(thing):
# look for path attribute
string = getattr(thing, '__fspath__', None)
if string is not None:
return string
# not found, do we have a str or bytes object?
if isinstance(thing, (str, bytes)):
return thing
raise TypeError('`thing` must implement the __fspath__ protocol or
be an instance of str or bytes')
And just because DirEntry supports bytes doesn't mean that any magic method
it gains has to carry that forward (it can always raise a TypeError if
necessary).
No, but what if some code gets pathnames from whatever other places
and passes them on to os.scandir. Whenever it happens to get a bytes
path, a TypeError gets raised, but only when it picks one of the
DirEntry objects and for instance tries to open(...) it. Of course,
I'm not sure how common this is.
Yeah, I don't think this is a good idea. Given that fspath() should be
able to return bytes if bytes are passed in, DirEntry's __fspath__
could return bytes to no ill effect.
I realize this may not be ideal, but throwing bytes to the wind is going
to bite us in the end.
After all, the idea is to make these things work with the stdlib, and
the stdlib accepts bytes for path strings.
--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Question about the current implementation of str
I have a straightforward question about the str object, specifically the PyUnicodeObject. I've tried reading the source to answer the question myself but it's nearly impenetrable. So I was hoping someone here who understands the current implementation could answer it for me. Although the str object is immutable from Python's perspective, the C object itself is mutable. For example, for dynamically-created strings the hash field may be lazy-computed and cached inside the object. I was wondering if there were other fields like this. For example, are there similar lazy-computed cached objects for the different encoded versions (utf8 utf16) of the str? What would really help an exhaustive list of the fields of a str object that may ever change after the object's initial creation. Thanks! We now return you to the debate about the pathlib module, //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Incomplete Internationalization in Argparse Module
That string looks like it is aimed at the developer, not the user of the program, so it makes sense not to translate it. On Fri, Apr 8, 2016 at 2:07 PM, Brett Cannon wrote: > > > On Fri, 8 Apr 2016 at 14:05 Grady Martin wrote: >> >> Hello, all. I was wondering if the following string was left untouched by >> gettext for a purpose (from line 720 of argparse.py, in class >> ArgumentError): >> >> 'argument %(argument_name)s: %(message)s' >> >> There may be other untranslatable strings in the argparse module, but I >> have yet to encounter them in the wild. > > > Probably so that anyone introspecting on the error message can count on > somewhat of a consistent format (comes into play with doctest typically). > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016, 16:13 Glenn Linderman wrote: > On 4/8/2016 3:28 PM, Eric Snow wrote: > > All this matters because it impacts the value returned from > __ospath__(). Should it return the string representation of the path > for the current OS or some standardized representation? I'd expect > the former. However, if that is the expectation then something like > pathlib.PureWindowsPath will give you the wrong thing if your current > OS is linux. pathlib.PureWindowsPath.__ospath__() would have to fail > or first internally convert to pathlib.PurePosixPath? > > Now that Windows 10++ will run Ubuntu apps, will Python be able to tell > the difference for when it should return Windows-format paths and > Posix-format paths? > All the bits of code in Python accept / as a separator on Windows so it doesn't matter (but Ubuntu on Windows is Linux, so it will be / just like any other Linux install). > (I'm sure the answer is yes, the Python-for-Ubuntu running on Windows > would do the latter, and the Python-for-Windows would do the former. > Although, it is not clear what sys.platform will return, yet...) > It should return Linux. -Brett ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 7:41 PM, Brett Cannon wrote: > > > On Fri, Apr 8, 2016, 16:13 Glenn Linderman wrote: >> >> On 4/8/2016 3:28 PM, Eric Snow wrote: >> >> All this matters because it impacts the value returned from >> __ospath__(). Should it return the string representation of the path >> for the current OS or some standardized representation? I'd expect >> the former. However, if that is the expectation then something like >> pathlib.PureWindowsPath will give you the wrong thing if your current >> OS is linux. pathlib.PureWindowsPath.__ospath__() would have to fail >> or first internally convert to pathlib.PurePosixPath? >> >> Now that Windows 10++ will run Ubuntu apps, will Python be able to tell >> the difference for when it should return Windows-format paths and >> Posix-format paths? > > > All the bits of code in Python accept / as a separator on Windows so it > doesn't matter (but Ubuntu on Windows is Linux, so it will be / just like > any other Linux install). Technically it isn't linux. :) It's the Ubuntu user-space using the linux syscalls (like normal), and those syscalls are implemented as light wrappers around the Windows kernel. They even implemented fork. On Windows. There's no linux kernel involved. > >> >> (I'm sure the answer is yes, the Python-for-Ubuntu running on Windows >> would do the latter, and the Python-for-Windows would do the former. >> Although, it is not clear what sys.platform will return, yet...) > > > It should return Linux. >From screenshots it looks like lsb_release -a returns the normal Ubuntu info [1] and uname -a says linux (don't know if that will change). [2] So yeah, sys.platform should return Linux. -eric [1] https://blogs.windows.com/buildingapps/2016/03/30/run-bash-on-ubuntu-on-windows/ [2] https://insights.ubuntu.com/2016/03/30/ubuntu-on-windows-the-ubuntu-userspace-for-windows-developers/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib (was: Defining a path protocol)
On 8 April 2016 at 00:25, Jim J. Jewett wrote: > (1) I think the "built-in" should instead be a module-level function > in the pathlib. If you aren't already expecting pathlib paths, then > you're just expecting strings to work anyhow, and a builtin isn't > likely to be helpful. Concrete data in relation to "Why not put the helper function in pathlib?": >>> import sys >>> orig_modules = set(sys.modules) >>> "os" in orig_modules True >>> import pathlib >>> extra_dependencies = set(sys.modules) - orig_modules >>> print(sorted(extra_dependencies)) ['_collections', '_functools', '_heapq', '_operator', '_sre', 'collections', 'contextlib', 'copyreg', 'fnmatch', 'functools', 'heapq', 'itertools', 'keyword', 'ntpath', 'operator', 'pathlib', 're', 'reprlib', 'sre_compile', 'sre_constants', 'sre_parse', 'urllib', 'urllib.parse', 'weakref'] We want to be able to readily use the protocol helper in builtin modules like os and low level Python modules like os.path, which means we want it to be much lower down in the import hierarchy than pathlib. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
