Re: [Python-Dev] Why does IOBase.__del__ call .close?
On 11 Jun 2014 12:31, "Antoine Pitrou" wrote: > > Le 10/06/2014 21:30, Nikolaus Rath a écrit : > >> >> For me, having __del__ call close() does not qualify as a reasonable >> default implementation unless close() is required to be idempotent >> (which one could deduce from the documentation if one tries to, but it's >> far from clear). > > > close() should indeed be idempotent on all bundled IO class implementations (otherwise it's a bug), and so should it preferably on third-party IO class implementations. > > If you want to improve the documentation on this, you're welcome to provide a patch! We certainly assume idempotent close() behaviour in various places, so if that expectation isn't currently clear in the docs, suggestions for improved wording would definitely be appreciated! Cheers, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Returning Windows file attribute information via os.stat()
>> What would be the next steps to get this to happen? Open an issue on >> bugs.python.org and submit a patch with tests? > > Yep! Okay, I've done step one (opened an issue on bugs.python.org), and hope to provide a patch in the next few weeks if no-one else does (I've never compiled CPython on Windows before): http://bugs.python.org/issue21719 -Ben ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue #21205: add __qualname__ to generators
Hi, I'm working on asyncio and it's difficult to debug code because @asyncio.coroutine decorator removes the name of the function if the function is not a generator (if it doesn't use yield from). I propose to add new gi_name and gi_qualname fields to the C structure PyGenObject, add a new __qualname__ (= gi_qualname) attribute to the Python API of generator, and change how the default value of __name__ (= gi_name) of generators. Instead of getting the name from the code object, I propose to get the name from the function (if the generator was created from a function). So if the function name was modified, you get the new name instead of getting the name from the code object (as done in Python 3.4). I also propose to display the qualified name in repr(generator) instead of the name. All these changes should make my life easier to debug asyncio, but it should help any project using generators. Issues describing the problem, I attached a patch implementing my ideas: http://bugs.python.org/issue21205 Would you be ok with these (minor) incompatible changes? By the way, it looks like generator attributes were never documented :-( My patch also adds a basic documentation (at least, it lists all attributes in the documentation of the inspect module). Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #21205: add __qualname__ to generators
Le 11/06/2014 10:28, Victor Stinner a écrit : Hi, I'm working on asyncio and it's difficult to debug code because @asyncio.coroutine decorator removes the name of the function if the function is not a generator (if it doesn't use yield from). I propose to add new gi_name and gi_qualname fields to the C structure PyGenObject, add a new __qualname__ (= gi_qualname) attribute to the Python API of generator, and change how the default value of __name__ (= gi_name) of generators. Instead of getting the name from the code object, I propose to get the name from the function (if the generator was created from a function). So if the function name was modified, you get the new name instead of getting the name from the code object (as done in Python 3.4). I also propose to display the qualified name in repr(generator) instead of the name. All these changes should make my life easier to debug asyncio, but it should help any project using generators. Issues describing the problem, I attached a patch implementing my ideas: http://bugs.python.org/issue21205 Would you be ok with these (minor) incompatible changes? +1 from me. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Returning Windows file attribute information via os.stat()
On 6/11/2014 9:27 AM, Ben Hoyt wrote: What would be the next steps to get this to happen? Open an issue on bugs.python.org and submit a patch with tests? Yep! Okay, I've done step one (opened an issue on bugs.python.org), and hope to provide a patch in the next few weeks if no-one else does (I've never compiled CPython on Windows before): http://bugs.python.org/issue21719 If you have problems compiling, the core-mentorship list is one place to ask. For 3.4+, I believe the devguide instructions are correct. If not, say something. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
I am banned from tracker, so I post the bug here: Normal Windows behavior: >hg status --rev ".^1" M mercurial\commands.py ? pysptest.py >hg status --rev .^1 abort: unknown revision '.1'! So, ^ is an escape character. See http://www.tomshardware.co.uk/forum/35565-45-when-special-command-line But subprocess doesn't escape it, making cross-platform command fail on Windows. ---[cut pysptest.py]-- import subprocess as sp # this fails with # abort: unknown revision '.1'! cmd = ['hg', 'status', '--rev', '.^1'] # this works #cmd = 'hg status --rev ".^1"' # this works too #cmd = ['hg', 'status', '--rev', '.^^1'] try: print sp.check_output(cmd, stderr=sp.STDOUT, shell=True) except Exception as e: print e.output -- -- anatoly t. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
Of course! And, why not escape everything else, too? abc -> ^a^b^c echo %PATH% -> ^e^c^h^o^ ^%^P^A^T^H^% In all seriousness, to me this is obvious. When you pass a command to the shell, naturally, certain details are shell-specific. -1. Bad idea. Very bad idea. If you want the ^ to be escaped, do it yourself. Or better yet, don't pass shell=True. anatoly techtonik wrote: >I am banned from tracker, so I post the bug here: > >Normal Windows behavior: > > >hg status --rev ".^1" > M mercurial\commands.py > ? pysptest.py > > >hg status --rev .^1 > abort: unknown revision '.1'! > >So, ^ is an escape character. See >http://www.tomshardware.co.uk/forum/35565-45-when-special-command-line > > >But subprocess doesn't escape it, making cross-platform command fail on >Windows. > >---[cut pysptest.py]-- >import subprocess as sp > ># this fails with ># abort: unknown revision '.1'! >cmd = ['hg', 'status', '--rev', '.^1'] ># this works >#cmd = 'hg status --rev ".^1"' ># this works too >#cmd = ['hg', 'status', '--rev', '.^^1'] > >try: > print sp.check_output(cmd, stderr=sp.STDOUT, shell=True) >except Exception as e: > print e.output >-- > >-- >anatoly t. > > > > >___ >Python-Dev mailing list >[email protected] >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 7:58 AM, Ryan wrote: > In all seriousness, to me this is obvious. When you pass a command to the > shell, naturally, certain details are shell-specific. > > -1. Bad idea. Very bad idea. If you want the ^ to be escaped, do it > yourself. Or better yet, don't pass shell=True. Definitely the latter. Why pass shell=True when executing a single command? I don't get it. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Wed, Jun 11, 2014, at 13:26, anatoly techtonik wrote: > I am banned from tracker, so I post the bug here: Being banned from the tracker is not an invitation to use python-dev@ as one. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
Also notice that using a list with shell=True is using the API incorrectly. It wouldn't even work on Linux, so that torpedoes the cross-platform concern already :) This kind of confusion is why I opened http://bugs.python.org/issue7839. On Wed, 11 Jun 2014 16:58:30 -0500, Ryan wrote: > Of course! And, why not escape everything else, too? > > abc -> ^a^b^c > > echo %PATH% -> ^e^c^h^o^ ^%^P^A^T^H^% > > In all seriousness, to me this is obvious. When you pass a command to the > shell, naturally, certain details are shell-specific. > > -1. Bad idea. Very bad idea. If you want the ^ to be escaped, do it > yourself. Or better yet, don't pass shell=True. > > anatoly techtonik wrote: > >I am banned from tracker, so I post the bug here: > > > >Normal Windows behavior: > > > > >hg status --rev ".^1" > > M mercurial\commands.py > > ? pysptest.py > > > > >hg status --rev .^1 > > abort: unknown revision '.1'! > > > >So, ^ is an escape character. See > >http://www.tomshardware.co.uk/forum/35565-45-when-special-command-line > > > > > >But subprocess doesn't escape it, making cross-platform command fail on > >Windows. > > > >---[cut pysptest.py]-- > >import subprocess as sp > > > ># this fails with > ># abort: unknown revision '.1'! > >cmd = ['hg', 'status', '--rev', '.^1'] > ># this works > >#cmd = 'hg status --rev ".^1"' > ># this works too > >#cmd = ['hg', 'status', '--rev', '.^^1'] > > > >try: > > print sp.check_output(cmd, stderr=sp.STDOUT, shell=True) > >except Exception as e: > > print e.output > >-- > > > >-- > >anatoly t. > > > > > > > > > >___ > >Python-Dev mailing list > >[email protected] > >https://mail.python.org/mailman/listinfo/python-dev > >Unsubscribe: > >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 1:30 AM, Chris Angelico wrote: > Why pass shell=True when executing a single > command? I don't get it. > I don't know about Linux, but on Windows programs are not directly available as /usr/bin/python, so you need to find command in PATH directories. Passing shell=True makes this lookup done by shell and not manually. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 1:30 AM, Chris Angelico wrote: > On Thu, Jun 12, 2014 at 7:58 AM, Ryan wrote: > > In all seriousness, to me this is obvious. When you pass a command to the > > shell, naturally, certain details are shell-specific. > On Windows cmd.exe is used by default: http://hg.python.org/cpython/file/38a325c84564/Lib/subprocess.py#l1108 so it makes sense to make default behavior cross-platform. > > -1. Bad idea. Very bad idea. If you want the ^ to be escaped, do it > > yourself. Or better yet, don't pass shell=True. > > Definitely the latter. Why pass shell=True when executing a single > command? I don't get it. > This is a complete use case using Rietveld upload script: http://techtonik.rainforce.org/2013/07/code-review-with-rietveld-and-mercurial.html I am interested to know how to modify upload script without kludges: https://code.google.com/p/rietveld/source/browse/upload.py#1056 I expect many people are facing with the same problem trying to wrap Git and HG with Python scripts. -- anatoly t. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does IOBase.__del__ call .close?
MRAB writes:
> On 2014-06-11 02:30, Nikolaus Rath wrote:
>> Hello,
>>
>> I recently noticed (after some rather protacted debugging) that the
>> io.IOBase class comes with a destructor that calls self.close():
>>
>> [0] nikratio@vostro:~/tmp$ cat test.py
>> import io
>> class Foo(io.IOBase):
>> def close(self):
>> print('close called')
>> r = Foo()
>> del r
>> [0] nikratio@vostro:~/tmp$ python3 test.py
>> close called
>>
>> To me, this came as quite a surprise, and the best "documentation" of
>> this feature seems to be the following note (from the io library
>> reference):
>>
>> "The abstract base classes also provide default implementations of some
>> methods in order to help implementation of concrete stream classes. For
>> example, BufferedIOBase provides unoptimized implementations of
>> readinto() and readline()."
>>
>> For me, having __del__ call close() does not qualify as a reasonable
>> default implementation unless close() is required to be idempotent
>> (which one could deduce from the documentation if one tries to, but it's
>> far from clear).
>>
>> Is this behavior an accident, or was that a deliberate decision?
>>
> To me, it makes sense. You want to make sure that it's closed, releasing
> any resources it might be holding, even if you haven't done so
> explicitly.
I agree with your intentions, but I come to the opposite conclusion:
automatically calling close() in the destructor will hide that there's a
problem in the code. Without that automatic cleanup, there's at least a
good chance that a ResourceWarning will be emitted so the problem gets
noticed. "Silently work around bugs in caller's code" doesn't seem like
a very useful default to me...
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 2:00 AM, R. David Murray wrote: > Also notice that using a list with shell=True is using the API > incorrectly. It wouldn't even work on Linux, so that torpedoes > the cross-platform concern already :) > > This kind of confusion is why I opened http://bugs.python.org/issue7839. I thought exactly about that. Usually separate arguments are used to avoid problems with escaping of quotes and other stuff. I'd deprecate subprocess and split it into separate modules. One is about shell execution and another one is for secure process control. shell execution module then could build on top of process control and be insecure by design. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does IOBase.__del__ call .close?
On Wed, Jun 11, 2014, at 17:11, Nikolaus Rath wrote:
> MRAB writes:
> > On 2014-06-11 02:30, Nikolaus Rath wrote:
> >> Hello,
> >>
> >> I recently noticed (after some rather protacted debugging) that the
> >> io.IOBase class comes with a destructor that calls self.close():
> >>
> >> [0] nikratio@vostro:~/tmp$ cat test.py
> >> import io
> >> class Foo(io.IOBase):
> >> def close(self):
> >> print('close called')
> >> r = Foo()
> >> del r
> >> [0] nikratio@vostro:~/tmp$ python3 test.py
> >> close called
> >>
> >> To me, this came as quite a surprise, and the best "documentation" of
> >> this feature seems to be the following note (from the io library
> >> reference):
> >>
> >> "The abstract base classes also provide default implementations of some
> >> methods in order to help implementation of concrete stream classes. For
> >> example, BufferedIOBase provides unoptimized implementations of
> >> readinto() and readline()."
> >>
> >> For me, having __del__ call close() does not qualify as a reasonable
> >> default implementation unless close() is required to be idempotent
> >> (which one could deduce from the documentation if one tries to, but it's
> >> far from clear).
> >>
> >> Is this behavior an accident, or was that a deliberate decision?
> >>
> > To me, it makes sense. You want to make sure that it's closed, releasing
> > any resources it might be holding, even if you haven't done so
> > explicitly.
>
> I agree with your intentions, but I come to the opposite conclusion:
> automatically calling close() in the destructor will hide that there's a
> problem in the code. Without that automatic cleanup, there's at least a
> good chance that a ResourceWarning will be emitted so the problem gets
> noticed. "Silently work around bugs in caller's code" doesn't seem like
> a very useful default to me...
Things which actually hold system resources (like FileIO) give
ResourceWarning if they close in __del__, so I don't understand your
point.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 10:00 AM, anatoly techtonik wrote: > I thought exactly about that. Usually separate arguments are used to avoid > problems with escaping of quotes and other stuff. > > I'd deprecate subprocess and split it into separate modules. One is about > shell execution and another one is for secure process control. ISTM what you want is not shell=True, but a separate function that follows the system policy for translating a command name into a path-to-binary. That's something that, AFAIK, doesn't currently exist in the Python 2 stdlib, but Python 3 has shutil.which(). If there's a PyPI backport of that for Py2, you should be able to use that to figure out the command name, and then avoid shell=False. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Thu, Jun 12, 2014 at 12:07 PM, Chris Angelico wrote:
> ISTM what you want is not shell=True, but a separate function that
> follows the system policy for translating a command name into a
> path-to-binary. That's something that, AFAIK, doesn't currently exist
> in the Python 2 stdlib, but Python 3 has shutil.which(). If there's a
> PyPI backport of that for Py2, you should be able to use that to
> figure out the command name, and then avoid shell=False.
Huh. Next time, Chris, search the web before you post. Via a
StackOverflow post, learned about distutils.spawn.find_executable().
Python 2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.spawn
>>> distutils.spawn.find_executable("python")
'C:\\Program Files\\LilyPond\\usr\\bin\\python.exe'
So that would be the way to go. Render the short-form into an
executable name, then skip the shell.
ChrisA
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On 06/11/2014 07:12 PM, Chris Angelico wrote: On Thu, Jun 12, 2014 at 12:07 PM, Chris Angelico wrote: ISTM what you want is not shell=True, but a separate function that follows the system policy for translating a command name into a path-to-binary. That's something that, AFAIK, doesn't currently exist in the Python 2 stdlib, but Python 3 has shutil.which(). If there's a PyPI backport of that for Py2, you should be able to use that to figure out the command name, and then avoid shell=False. Huh. Next time, Chris, search the web before you post. Via a StackOverflow post, learned about distutils.spawn.find_executable(). --> import sys --> sys.executable '/usr/bin/python' ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On Wed, Jun 11, 2014 at 9:43 PM, Ethan Furman wrote: > On 06/11/2014 07:12 PM, Chris Angelico wrote: >> >> On Thu, Jun 12, 2014 at 12:07 PM, Chris Angelico wrote: >>> >>> ISTM what you want is not shell=True, but a separate function that >>> follows the system policy for translating a command name into a >>> path-to-binary. That's something that, AFAIK, doesn't currently exist >>> in the Python 2 stdlib, but Python 3 has shutil.which(). If there's a >>> PyPI backport of that for Py2, you should be able to use that to >>> figure out the command name, and then avoid shell=False. >> >> >> Huh. Next time, Chris, search the web before you post. Via a >> StackOverflow post, learned about distutils.spawn.find_executable(). > > > --> import sys > --> sys.executable > '/usr/bin/python' For finding the Python executable, yes, but the discussion and example are about a 2.x version of shutil.which ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
* anatoly techtonik [2014-06-12 02:00:55 +0300]: > On Thu, Jun 12, 2014 at 1:30 AM, Chris Angelico wrote: > > > Why pass shell=True when executing a single > > command? I don't get it. > > > > I don't know about Linux, but on Windows programs are not directly > available as /usr/bin/python, so you need to find command in PATH > directories. Passing shell=True makes this lookup done by shell and not > manually. As it's been said, the whole *point* of shell=True is to be able to use shell features, so ^ being escaped automatically just would be... broken. How would I escape > then, for example ;) You basically have two options: - Do the lookup in PATH yourself, it's not like that's rocket science. I haven't checked if there's a ready function for it in the stdlib, but even when not: Get os.environ['PATH'], split it by os.pathsep, then for every directory check if your binary is in there. There's also some environment variable on Windows which contains the possible extensions for a binary in PATH, add that, and that's all. - Use shell=True and a cross-platform shell escape function. I've wrote one for a project of mine: [1] I've written some tests[2] but I haven't checked all corner-cases, so I can't guarantee it'll always work, as the interpretation of special chars by cmd.exe *is* black magic, at least to me. Needless to say this is probably the worse choice of the two. [1] http://git.the-compiler.org/qutebrowser/tree/qutebrowser/utils/misc.py?id=dffec73db76c867d261ec3416de011becb209f13#n154 [2] http://git.the-compiler.org/qutebrowser/tree/qutebrowser/test/utils/test_misc.py?id=dffec73db76c867d261ec3416de011becb209f13#n195 Florian -- http://www.the-compiler.org | [email protected] (Mail/XMPP) GPG 0xFD55A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ pgpy7W5wzDW9z.pgp Description: PGP signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character
On 12 June 2014 05:34, Florian Bruhin wrote: > Do the lookup in PATH yourself, it's not like that's rocket science. Am I missing something here? I routinely do subprocess.check_call(['hg', 'update']) or whatever, and it finds the hg executable fine. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
