[Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
see the following:
lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py
lac@smartwheels:~/junk$ idle-python3.5
hello there
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/idlelib/run.py", line 10, in
from idlelib import CallTips
File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in
from idlelib.HyperParser import HyperParser
File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in
_ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_")
AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess
por personal firewall software is blocking the connection.
I think that life would be a whole lot easier for people if instead we got
a message:
Warning: local file /u/lac/junk/string.py shadows module named string in the
Standard Library
I think that it is python exec that would have to do this -- though of
course the popup could also warn about shadowing in general, instead of
sending people on wild goose chases over their firewalls.
Would this be hard to do?
Laura
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On Thu, 29 Oct 2015 at 09:00 Laura Creighton wrote:
>
> see the following:
> lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py
> lac@smartwheels:~/junk$ idle-python3.5
> hello there
> Traceback (most recent call last):
> File "", line 1, in
> File "/usr/lib/python3.5/idlelib/run.py", line 10, in
> from idlelib import CallTips
> File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in
> from idlelib.HyperParser import HyperParser
> File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in
> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_")
> AttributeError: module 'string' has no attribute 'ascii_letters'
>
> IDLE then produces a popup that says:
>
> IDLE's subprocess didn't make connection. Either IDLE can't stat a
> subprocess por personal firewall software is blocking the connection.
>
>
>
> I think that life would be a whole lot easier for people if instead we got
> a message:
>
> Warning: local file /u/lac/junk/string.py shadows module named string in
> the
> Standard Library
>
> I think that it is python exec that would have to do this -- though of
> course the popup could also warn about shadowing in general, instead of
> sending people on wild goose chases over their firewalls.
>
> Would this be hard to do?
>
It would require a custom importer or overriding __import__ but it's
doable.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On Oct 29, 2015 10:22 AM, "Brett Cannon" wrote:
>
>
>
> On Thu, 29 Oct 2015 at 09:00 Laura Creighton wrote:
>>
>>
>> see the following:
>> lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py
>> lac@smartwheels:~/junk$ idle-python3.5
>> hello there
>> Traceback (most recent call last):
>> File "", line 1, in
>> File "/usr/lib/python3.5/idlelib/run.py", line 10, in
>> from idlelib import CallTips
>> File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in
>> from idlelib.HyperParser import HyperParser
>> File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in
>> _ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits +
"_")
>> AttributeError: module 'string' has no attribute 'ascii_letters'
>>
>> IDLE then produces a popup that says:
>>
>> IDLE's subprocess didn't make connection. Either IDLE can't stat a
subprocess por personal firewall software is blocking the connection.
>>
>>
>>
>> I think that life would be a whole lot easier for people if instead we
got
>> a message:
>>
>> Warning: local file /u/lac/junk/string.py shadows module named string in
the
>> Standard Library
>>
>> I think that it is python exec that would have to do this -- though of
>> course the popup could also warn about shadowing in general, instead of
>> sending people on wild goose chases over their firewalls.
>>
>> Would this be hard to do?
>
>
> It would require a custom importer or overriding __import__ but it's
doable.
Is there any reason not to issue such warnings by default in the standard
importer? The issue of accidentally shadowing stdlib modules isn't
restricted to idle, it's difficult for idle to handle correctly (how do you
define a custom importer if you don't yet have access to the stdlib?), and
it's not like there's any legitimate reason to want string.py in the
working directory to auto-monkeypatch stdlib string.
(I know saying that last part out loud will probably just cause someone to
pop out of the woodwork and explain how shadowing the sys module is a great
idea and they do it all the time or whatever, but I guess I'll take that
risk :-).)
-n
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On October 29, 2015 at 1:32:31 PM, Nathaniel Smith ([email protected]) wrote: > > (I know saying that last part out loud will probably just cause > someone to pop out of the woodwork and explain how shadowing the > sys module is a great idea and they do it all the time or whatever, > but I guess I'll take that risk :-).) How about someone saying that I wish the standard library was more easily shadow-able? ;) In 2.x it’s easy to do it by accident because of the implicit relative imports which is of course, crummy. I think it’d be nice if a package could override the standard library in a sane way though. Like how pdb++ does. This is already possible if you install stuff as .eggs and the world hasn’t burned down, it’s just not easily possible if you don’t install as eggs. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 29 October 2015 at 17:36, Donald Stufft wrote: > On October 29, 2015 at 1:32:31 PM, Nathaniel Smith ([email protected]) wrote: >> > (I know saying that last part out loud will probably just cause >> someone to pop out of the woodwork and explain how shadowing the >> sys module is a great idea and they do it all the time or whatever, >> but I guess I'll take that risk :-).) > > How about someone saying that I wish the standard library was more easily > shadow-able? ;) In 2.x it’s easy to do it by accident because of the implicit > relative imports which is of course, crummy. I think it’d be nice if a > package could override the standard library in a sane way though. Like how > pdb++ does. > > This is already possible if you install stuff as .eggs and the world hasn’t > burned down, it’s just not easily possible if you don’t install as eggs. The idle issues seem to me to demonstrate that shadowing the stdlib is a bad idea. Of course, consenting adults, and if you override you're responsible for correctly replacing the functionality, and all that, but honestly, I don't think it needs to be *easy* to shadow the stdlib - there's nothing wrong with it being an "advanced" technique that people have to understand in order to use. And I don't see pdb++ as a good motivating use case. When are you going to find code with pdb calls in, that you can't modify to add "import pdbpp as pdb" at the top, if you want the advanced functionality? (And if you can't modify the code, what's the point of debugging it anyway?) I think the reason the world hasn't burned down because you can shadow the stdlib with eggs, is because people *don't*. Or to rephrase, because people don't actually need to shadow the stdlib. But I have no hard data to support that contention, so if you do, I'll concede the point (but I don't think there are many packages that *have* to be shipped as eggs these days, if they did they wouldn't work with pip, so doesn't that mean that nobody is using the ability of eggs to shadow the stdlib?) Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On October 29, 2015 at 2:28:02 PM, Paul Moore (p.f.moore=40gmail.com) wro= te: > On 29 October 2015 at 17:36, Donald Stufft wrote: > > On October 29, 2015 at 1:32:31 PM, Nathaniel Smith (njs=40pobox.com) = wrote: > >> > (I know saying that last part out loud will probably just cause > >> someone to pop out of the woodwork and explain how shadowing the > >> sys module is a great idea and they do it all the time or whatever, > >> but I guess I'll take that risk :-).) > > > > How about someone saying that I wish the standard library was more ea= sily shadow-able=3F =20 > ;) In 2.x it=E2=80=99s easy to do it by accident because of the implici= t relative imports which =20 > is of course, crummy. I think it=E2=80=99d be nice if a package could o= verride the standard library =20 > in a sane way though. Like how pdb++ does. > > > > This is already possible if you install stuff as .eggs and the world = hasn=E2=80=99t burned down, =20 > it=E2=80=99s just not easily possible if you don=E2=80=99t install as e= ggs. > =20 > The idle issues seem to me to demonstrate that shadowing the stdlib is > a bad idea. Of course, consenting adults, and if you override you're > responsible for correctly replacing the functionality, and all that, > but honestly, I don't think it needs to be *easy* to shadow the stdlib > - there's nothing wrong with it being an =22advanced=22 technique that > people have to understand in order to use. I think the idle problem demonstrates that shadowing the stdlib accidenta= lly is a bad idea, and the old implicit relative imports made it extremel= y trivial to do so. Having =60.=60 first on the path still makes it trivi= al to do so, but less so than the old implicit relative imports did. > =20 > And I don't see pdb++ as a good motivating use case. When are you > going to find code with pdb calls in, that you can't modify to add > =22import pdbpp as pdb=22 at the top, if you want the advanced > functionality=3F (And if you can't modify the code, what's the point of= > debugging it anyway=3F) Every test runner that includes a =E2=80=94pdb flag that will automatical= ly invoke pdb at the point of failure. If pdb++ didn=E2=80=99t force the = shadowing of stdlib, then every single test runner would need an option l= ike =E2=80=94pdb++ instead of =E2=80=94pdb (or it would need to unconditi= onally prefer pdb++ over pdb if you had =E2=80=94pdb). But what if I don=E2= =80=99t like pdb++ and I want to use some other pdb replacement=3F Possib= ly one I=E2=80=99m writing myself=3F Either I can convince every test run= ner to support every pdb replacement or I need some sort of central regis= try where I can insert my pdb as a replacement to the =E2=80=9Creal=E2=80= =9D pdb. The sys.path and import system is one such registry that already= exists and is (my opinion) the logical choice. An example close to home for me where it would have been immensely useful= is for adding pip to the standard library. I was opposed to doing that b= ecause I wanted people to be able to upgrade their pip without having to = upgrade their Python (because we=E2=80=99ve seen how well it worked for d= istutils). Because there=E2=80=99s not a great way to do that, we ended u= p having to go with ensurepip instead which is more complicated and cause= d policy problems for a lot of downstream in a way that just adding pip t= o the standard library wouldn=E2=80=99t have. Instead we could have simpl= y added pip to the standard library, and then upgrading pip would have dr= opped it into site-packages and that could have taken precedence, allowin= g people to upgrade easily. > =20 > I think the reason the world hasn't burned down because you can shadow > the stdlib with eggs, is because people *don't*. Or to rephrase, > because people don't actually need to shadow the stdlib. But I have no > hard data to support that contention, so if you do, I'll concede the > point (but I don't think there are many packages that *have* to be > shipped as eggs these days, if they did they wouldn't work with pip, > so doesn't that mean that nobody is using the ability of eggs to > shadow the stdlib=3F) > =20 People route around technical limitations. In pdb++ case they used to bre= ak the ability of pip to prevent installation of egg by munging sys.argv = to remove the =E2=80=94single-version-externally-managed. Now they instal= l a .pth hack to force it to the front of sys.path.=C2=A0Other examples o= f ways people route around this is by using monkey patching (either expli= citly invoked or implicitly via shenagins), an example of this is setupto= ols monkeypatching distutils on import (which pip uses to override what d= istutils is). Another example is a more explicit shadowing which is done = by having some slightly differently named thing available and expecting a= ll users to do something like try: import unittest2 as unittest except Im= portError: import unittest. So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t sha
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
In a message of Thu, 29 Oct 2015 18:27:59 +, Paul Moore writes: >The idle issues seem to me to demonstrate that shadowing the stdlib is >a bad idea. Of course, consenting adults, and if you override you're >responsible for correctly replacing the functionality, and all that, >but honestly, I don't think it needs to be *easy* to shadow the stdlib >- there's nothing wrong with it being an "advanced" technique that >people have to understand in order to use. I am actually sick of the 'consenting adults' argument. I am dealing with '11 year old children trying to write their first, third and tenth python programs'. For the life of me I cannot see how convenience for the sort of person who has a legitimate reason to shadow the syslib should get a higher priority over these mites who are doing their damndest to write python despite natural language barriers and the fact that their peers and parents think they are nuts to want to do so. (a grumpy comment from a teacher at a Swedish 'coding for kids' club. Disregard if too grumpy.) Laura ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 29 October 2015 at 18:46, Laura Creighton wrote: > In a message of Thu, 29 Oct 2015 18:27:59 +, Paul Moore writes: >>The idle issues seem to me to demonstrate that shadowing the stdlib is >>a bad idea. Of course, consenting adults, and if you override you're >>responsible for correctly replacing the functionality, and all that, >>but honestly, I don't think it needs to be *easy* to shadow the stdlib >>- there's nothing wrong with it being an "advanced" technique that >>people have to understand in order to use. > > I am actually sick of the 'consenting adults' argument. > I am dealing with '11 year old children trying to write their > first, third and tenth python programs'. For the life of me > I cannot see how convenience for the sort of person who has a > legitimate reason to shadow the syslib should get a higher priority > over these mites who are doing their damndest to write python > despite natural language barriers and the fact that their peers > and parents think they are nuts to want to do so. That's actually a very good point, and I agree totally. To my mind, the point about "consenting adults" (and when I referred to that I was anticipating others using that argument, not proposing it myself) is that we don't *prevent* people from doing weird and wonderful things. But conversely, it's not a reason for making it *easy* to do such things. Quite the opposite - a "consenting adult" should be assumed to be capable of writing an import hook, or manipulating sys.path, or whatever. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On Oct 29, 2015 11:45 AM, "Donald Stufft" wrote: > > Every test runner that includes a =E2=80=94pdb flag that will automatical= > ly invoke pdb at the point of failure. If pdb++ didn=E2=80=99t force the = > shadowing of stdlib, then every single test runner would need an option l= > ike =E2=80=94pdb++ instead of =E2=80=94pdb (or it would need to unconditi= > onally prefer pdb++ over pdb if you had =E2=80=94pdb). But what if I don=E2= > =80=99t like pdb++ and I want to use some other pdb replacement=3F Possib= > ly one I=E2=80=99m writing myself=3F Either I can convince every test run= > ner to support every pdb replacement or I need some sort of central regis= > try where I can insert my pdb as a replacement to the =E2=80=9Creal=E2=80= > =9D pdb. The sys.path and import system is one such registry that already= > exists and is (my opinion) the logical choice. This strikes me as an argument for adding some sort of explicit plugin support to pdb, or defining a standard interface that test tools could use to query for installed debuggers. (Presumably via some entry point that alternative debugger packages would export?) In any case it's orthogonal to the issue of accidental shadowing. It seems unlikely that at this late date we'll be swapping the order of sys.path so site-packages comes before the stdlib, and I doubt there are a lot of people who are using pdb++ via the mechanism of making sure that it's always present in their working directory... -n ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
In a message of Thu, 29 Oct 2015 19:13:08 +, Paul Moore writes: >> I am actually sick of the 'consenting adults' argument. >> I am dealing with '11 year old children trying to write their >> first, third and tenth python programs'. For the life of me >> I cannot see how convenience for the sort of person who has a >> legitimate reason to shadow the syslib should get a higher priority >> over these mites who are doing their damndest to write python >> despite natural language barriers and the fact that their peers >> and parents think they are nuts to want to do so. > >That's actually a very good point, and I agree totally. To my mind, >the point about "consenting adults" (and when I referred to that I was >anticipating others using that argument, not proposing it myself) is >that we don't *prevent* people from doing weird and wonderful things. >But conversely, it's not a reason for making it *easy* to do such >things. Quite the opposite - a "consenting adult" should be assumed to >be capable of writing an import hook, or manipulating sys.path, or >whatever. > >Paul Hmmm, I think the set of 'consenting adults who cannot write an import hook' is rather large. But all I am asking for is a warning -- and it would be good if Idle noticed the warning and before it fell over dead with its message of firewalls it would mention the warning again, as a problem to look for. It will bugger up doctests for people who legitimately shadow the stdlib and now get a new warning. Anybody else be seriously inconvenienced if we do this? I cannot think of any, but then legitimately shadowing the stdlib in not on the list of things I have done. Perhaps Dstufft has ideas on this line. Laura ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 29 October 2015 at 18:45, Donald Stufft wrote: > So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= > ow the standard library, they just have various ways to do it that have s= > everal gotchas and require people to generally hack around the limitation= > .=C2=A0 (Your mailer or mine seems to have gone weird with encoding...) There's a difference between the opting into using (say) pdb++ at runtime, and at install time. The problem with the sort of shadowing eggs did (and people try to emulate) is that simply *installing* a package changes the behaviour of programs that don't use that package, by virtue of the fact that it's there at all. I'd much rather pdb++ or anything similar allowed the user to opt in on demand. For example, it could put a file pdb.py containing "from pdbpp import *" into a subdirectory, and direct users who want to replace pdb with pdbpp to add that directory to PYTHONPATH. Just don't do that at install time, leaving the user needing to uninstall pdbpp to opt out of the behaviour. As I said, I'm not against people being *able* to shadow the stdlib, I just don't want to see it being *easy*, nor do I want it to be the norm. Paul PS My experience with a similar case is pyreadline on Windows, which hooks into the stdlib readline functionality when installed. That's a PITA, because mostly I don't want pyreadline (I prefer the default Windows command line editing, despite it having less functionality) but I want pyreadline when using ipython. There's no way for me to get that behaviour because (py)readline is enabled by installing it, not by user choice at runtime. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
In a message of Thu, 29 Oct 2015 19:30:09 +, Paul Moore writes: >On 29 October 2015 at 18:45, Donald Stufft wrote: >> So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= >> ow the standard library, they just have various ways to do it that have s= >> everal gotchas and require people to generally hack around the limitation= >> .=C2=A0 > >(Your mailer or mine seems to have gone weird with encoding...) Dstuffts. I see this problem too Laura ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
> On Oct 29, 2015, at 11:46, Laura Creighton wrote: > > In a message of Thu, 29 Oct 2015 18:27:59 +, Paul Moore writes: >> The idle issues seem to me to demonstrate that shadowing the stdlib is >> a bad idea. Of course, consenting adults, and if you override you're >> responsible for correctly replacing the functionality, and all that, >> but honestly, I don't think it needs to be *easy* to shadow the stdlib >> - there's nothing wrong with it being an "advanced" technique that >> people have to understand in order to use. > > I am actually sick of the 'consenting adults' argument. > I am dealing with '11 year old children trying to write their > first, third and tenth python programs'. For the life of me > I cannot see how convenience for the sort of person who has a > legitimate reason to shadow the syslib should get a higher priority > over these mites who are doing their damndest to write python > despite natural language barriers and the fact that their peers > and parents think they are nuts to want to do so. > > (a grumpy comment from a teacher at a Swedish 'coding for > kids' club. Disregard if too grumpy.) +1 on **warning**, warning still allow people to shadow stdlib, and for people who have **legitimate** reasons to shadow, we can always find a solution to to tag a module as “yes I know I am shadowing, I am doing that on purpose”. StdlibShadowWarning and warning filter ? Also a warning would be useful for people to discover that some Stdlib modules exist, and maybe explore them. -- M ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
Laura, I think what you want should actually be more-or-less doable in IDLE. The main routine that starts IDLE should be able to detect if it starts correctly (something unlikely to happen if a significant stdlib module is shadowed), watch for an attribute error of that form and try to determine if shadowing is the cause, and if so, reissue a saner error message. The subprocess/firewall error is occurring because the ‘string’ problem in your example isn’t being hit right away so a few startup things already are happening. The point where we’re showing that error (as a result of a timeout) should be able to check as per the above that IDLE was able to start alright, and if not, change or ignore the timeout error. There’ll probably be some cases (depending on exactly what gets shadowed) that may be difficult to get to work, but it should be able to handle most things. Mark ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
In a message of Thu, 29 Oct 2015 13:26:08 -0700, Mark Roseman writes: >Laura, I think what you want should actually be more-or-less doable in IDLE. > >The main routine that starts IDLE should be able to detect if it starts >correctly (something unlikely to happen if a significant stdlib module is >shadowed), watch for an attribute error of that form and try to determine if >shadowing is the cause, and if so, reissue a saner error message. > >The subprocess/firewall error is occurring because the ‘string’ problem in >your example isn’t being hit right away so a few startup things already are >happening. The point where we’re showing that error (as a result of a timeout) >should be able to check as per the above that IDLE was able to start alright, >and if not, change or ignore the timeout error. > >There’ll probably be some cases (depending on exactly what gets shadowed) that >may be difficult to get to work, but it should be able to handle most things. > >Mark Mark, how splendid. Need I submit a bug report/feature request to get this happening? Very, very pleased to have mentioned it ... Laura ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 10/29/2015 12:30 PM, Paul Moore wrote: On 29 October 2015 at 18:45, Donald Stufft wrote: So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad= ow the standard library, they just have various ways to do it that have s= everal gotchas and require people to generally hack around the limitation= .=C2=A0 (Your mailer or mine seems to have gone weird with encoding...) There's a difference between the opting into using (say) pdb++ at runtime, and at install time. The problem with the sort of shadowing eggs did (and people try to emulate) is that simply *installing* a package changes the behaviour of programs that don't use that package, by virtue of the fact that it's there at all. I'd much rather pdb++ or anything similar allowed the user to opt in on demand. For example, it could put a file pdb.py containing "from pdbpp import *" into a subdirectory, and direct users who want to replace pdb with pdbpp to add that directory to PYTHONPATH. Just don't do that at install time, leaving the user needing to uninstall pdbpp to opt out of the behaviour. As I said, I'm not against people being *able* to shadow the stdlib, I just don't want to see it being *easy*, nor do I want it to be the norm. Paul PS My experience with a similar case is pyreadline on Windows, which hooks into the stdlib readline functionality when installed. That's a PITA, because mostly I don't want pyreadline (I prefer the default Windows command line editing, despite it having less functionality) but I want pyreadline when using ipython. There's no way for me to get that behaviour because (py)readline is enabled by installing it, not by user choice at runtime. +1 Although, I wouldn't mind if it were easy & documented, like say having both shadow-packages before stdlib and site-packages after it, so that multiple, super-tricky and annoying ways of doing the shadowing were not needed. But, say, packages shadow-packages would only be used if there were an extra directive somewhere to enable it, or a subset of the packages that reside there. Glenn ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
Why not just check the path of the imported modules and compare it with the Python library directory? On October 29, 2015 3:26:08 PM CDT, Mark Roseman wrote: >Laura, I think what you want should actually be more-or-less doable in >IDLE. > >The main routine that starts IDLE should be able to detect if it starts >correctly (something unlikely to happen if a significant stdlib module >is shadowed), watch for an attribute error of that form and try to >determine if shadowing is the cause, and if so, reissue a saner error >message. > >The subprocess/firewall error is occurring because the ‘string’ problem >in your example isn’t being hit right away so a few startup things >already are happening. The point where we’re showing that error (as a >result of a timeout) should be able to check as per the above that IDLE >was able to start alright, and if not, change or ignore the timeout >error. > >There’ll probably be some cases (depending on exactly what gets >shadowed) that may be difficult to get to work, but it should be able >to handle most things. > >Mark > >___ >Python-Dev mailing list >[email protected] >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
> Need I submit a bug report/feature request to get this happening? > Very, very pleased to have mentioned it … I took care of the bug report. Glad you mentioned it. My personal opinion is that many of the error messages in IDLE (to say nothing of large parts of the overall user experience) are a bit of a horror show, but it’s slowly fixable! :-) Don’t hesitate to share other snags, as the situation you’re in is what I’d like to see much better supported. Mark ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
In a message of Thu, 29 Oct 2015 15:50:30 -0500, Ryan Gonzalez writes: >Why not just check the path of the imported modules and compare it with the >Python library directory? My friend Åsa who is 12 years old suggested exactly this at the club. If this works then I will be certain to mention this to her. I said that I would ask 'how hard is this?' Laura ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
Well, tell your friend that that means middle and high schoolers must think alike! :D On Thu, Oct 29, 2015 at 4:18 PM, Laura Creighton wrote: > In a message of Thu, 29 Oct 2015 15:50:30 -0500, Ryan Gonzalez writes: > >Why not just check the path of the imported modules and compare it with > the Python library directory? > > My friend Åsa who is 12 years old suggested exactly this at the club. If > this > works then I will be certain to mention this to her. I said that I would > ask 'how hard is this?' > > Laura > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On Thu, Oct 29, 2015 at 1:50 PM, Ryan Gonzalez wrote: > Why not just check the path of the imported modules and compare it with the > Python library directory? It works, but it requires that everyone who could run into this problem carefully add some extra guard code to every stdlib import statement, and in practice nobody will (or at least, not until after they've already gotten bitten by this at least once... at which point they no longer need it). Given that AFAICT there's no reason this couldn't be part of the default import system's functionality and "just work" for everyone, if I were going to spend time on trying to fix this I'd probably target that :-). (I guess the trickiest bit would be to find an efficient and maintainable way to check whether a given package name is present in the stdlib.) -n -- Nathaniel J. Smith -- http://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On Thu, 29 Oct 2015 16:56:38 -0700, Nathaniel Smith wrote: > On Thu, Oct 29, 2015 at 1:50 PM, Ryan Gonzalez wrote: > > Why not just check the path of the imported modules and compare it with the > > Python library directory? > > It works, but it requires that everyone who could run into this > problem carefully add some extra guard code to every stdlib import > statement, and in practice nobody will (or at least, not until after > they've already gotten bitten by this at least once... at which point > they no longer need it). > > Given that AFAICT there's no reason this couldn't be part of the > default import system's functionality and "just work" for everyone, if > I were going to spend time on trying to fix this I'd probably target > that :-). > > (I guess the trickiest bit would be to find an efficient and > maintainable way to check whether a given package name is present in > the stdlib.) For Idle, though, it sounds like a very viable strategy, and that's what Laura is concerned about. --David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
Well, this works on Python 2 only (I'm on a phone with only access to 2 right
now), but this is a start (apologies for the screwy syntax highlighting):
import sys, imp, logging, os
modules = '''
imp
string
...
'''.split()
class StdlibTester(object):
base = os.path.dirname(os.__file__) # hack; is there a better way to do
this?
def find_module(self, fullname, path=None):
if fullname in modules:
self.path = path
return self
return None
def load_module(self, name):
if name in sys.modules:
return sys.modules[name]
module_info = imp.find_module(name, self.path)
module = imp.load_module(name, *module_info)
sys.modules[name] = module
if hasattr(module, '__file__') and not
os.path.dirname(module.__file__).startswith(self.base):
logging.warning('stdlib module %s was overriden', name)
return module
sys.meta_path.append(StdlibTester())
import string
On October 29, 2015 7:06:51 PM CDT, "R. David Murray"
wrote:
>On Thu, 29 Oct 2015 16:56:38 -0700, Nathaniel Smith
>wrote:
>> On Thu, Oct 29, 2015 at 1:50 PM, Ryan Gonzalez
>wrote:
>> > Why not just check the path of the imported modules and compare it
>with the
>> > Python library directory?
>>
>> It works, but it requires that everyone who could run into this
>> problem carefully add some extra guard code to every stdlib import
>> statement, and in practice nobody will (or at least, not until after
>> they've already gotten bitten by this at least once... at which point
>> they no longer need it).
>>
>> Given that AFAICT there's no reason this couldn't be part of the
>> default import system's functionality and "just work" for everyone,
>if
>> I were going to spend time on trying to fix this I'd probably target
>> that :-).
>>
>> (I guess the trickiest bit would be to find an efficient and
>> maintainable way to check whether a given package name is present in
>> the stdlib.)
>
>For Idle, though, it sounds like a very viable strategy, and that's
>what Laura is concerned about.
>
>--David
>___
>Python-Dev mailing list
>[email protected]
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
--
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 10/29/2015 4:53 PM, Mark Roseman wrote: Need I submit a bug report/feature request to get this happening? Very, very pleased to have mentioned it … I took care of the bug report. The idle issue is https://bugs.python.org/issue25514 As I said there, I think that removing '' from sys.path, so that IDLE can run, is better than a nicer warning that it cannot run. This, of course, requires that sys not be shadowed, so that sys.path can be accessed. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 10/29/2015 11:59 AM, Laura Creighton wrote:
see the following:
lac@smartwheels:~/junk$ echo "print ('hello there')" >string.py
lac@smartwheels:~/junk$ idle-python3.5
hello there
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/idlelib/run.py", line 10, in
from idlelib import CallTips
File "/usr/lib/python3.5/idlelib/CallTips.py", line 16, in
from idlelib.HyperParser import HyperParser
File "/usr/lib/python3.5/idlelib/HyperParser.py", line 14, in
_ASCII_ID_CHARS = frozenset(string.ascii_letters + string.digits + "_")
AttributeError: module 'string' has no attribute 'ascii_letters'
IDLE then produces a popup that says:
IDLE's subprocess didn't make connection. Either IDLE can't stat a subprocess por
personal firewall software is blocking the connection.
I think that life would be a whole lot easier for people if instead we got
a message:
Warning: local file /u/lac/junk/string.py shadows module named string in the
Standard Library
I think that it is python exec that would have to do this -- though of
course the popup could also warn about shadowing in general, instead of
sending people on wild goose chases over their firewalls.
Would this be hard to do?
Leaving IDLE aside, the reason '' is added to sys.path is so that people
can import their own modules. This is very useful. Shadowing is the
result of putting it at the front. I have long thought this a dubious
choice. If '' were instead appended, people could still import modules
that did not duplicate stdlib names. Anyone who wanted shadowing could
move '' to the front. But then shadowing would be intentional, not an
accident.
--
Terry Jan Reedy
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen
On 10/29/2015 5:18 PM, Laura Creighton wrote: In a message of Thu, 29 Oct 2015 15:50:30 -0500, Ryan Gonzalez writes: Why not just check the path of the imported modules and compare it with the Python library directory? My friend Åsa who is 12 years old suggested exactly this at the club. This was my first idea, until I realized that it would be even better to avoid shadowing in the first place. If this works then I will be certain to mention this to her. As far as I can tell, comparison in not foolproof, even if done carefully. This is a proper stdlib import. >>> import string >>> string.__file__ 'C:\\Programs\\Python35\\lib\\string.py' If we look at suffixes, the only part guaranteed, after changing Windows' '\\' to '/', is '/lib/string.py'. Now suppose someone runs python in another 'lib' directory containing string.py. >>> import string >>> string.__file__ 'C:\\Users\\Terry\\lib\\string.py' Same suffix. Let's try prefixes. >>> import os.path >>> import sys >>> os.path.dirname(string.__file__) in sys.path False This is True for the stdlib import. Hooray. But this requires more imports, which also might be shadowed. Having '' at the front of sys.path is a real nuisance when one wants to guaranteed authentic stdlib imports. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
