Re: [Python-Dev] Cleaning-up the new unittest API
C. Titus Brown writes: > p.s. Seriously? I can accept that there's a rational minimalist argument > for this "feature", It is a feature, even if you aren't gonna need it. I want it. Many programmers do know that sets are partially ordered by inclusion, preordered by size, and (in Python) totally ordered by memory address. There's nothing wrong with not knowing that -- these are rather abstract mathematical concepts. But it's very useful that sorted() or .sort() use <=, and it's very useful that Python so often obeys simple consistent rules, and it would be quite confusing to those who do understand that "in Python the set type is partially ordered by inclusion" if sorted() used some other relation to order collections of sets. It's not so hard to change this: class SizedSet (set): def __lt__(a, b): return length(a) < length(b) def __le__(a, b): return length(a) <= length(b) def __gt__(a, b): return length(a) > length(b) def __ge__(a, b): return length(a) >= length(b) # These two are arguable, which makes size comparison not so # great as a candidate for the OOWTDI of set.__lt__(). def __eq__(a, b): return length(a) == length(b) def __ne__(a, b): return length(a) != length(b) If there were an obvious way to compare sets for use in sorting, that way would very likely be the most useful definition for <=, too. But there isn't, really (it's pretty obvious that comparing memory addresses is implausible, but otherwise, there are lots of candidates that are at least sometimes useful). Do you think otherwise? If so, what do you propose for the OOWTDI of sorting a collection of sets? > but arguing that it's somehow the responsibility of a programmer to > *expect* this seems kind of whack. I don't quite agree that everyone should "expect exactly the implemented behavior", but I do think it's a Python *programmer's* responsibility to refrain from expecting something else in this case. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85902 - in python/branches/py3k/Lib: os.py test/test_os.py
I don't know how to ignore the BytesWarning without importing warning. How can I do that? Victor Le vendredi 29 octobre 2010 04:31:42, Benjamin Peterson a écrit : > 2010/10/28 victor.stinner : > > Author: victor.stinner > > Date: Fri Oct 29 02:38:58 2010 > > New Revision: 85902 > > > > Log: > > Issue #10210: os.get_exec_path() ignores BytesWarning warnings > > > > > > Modified: > > python/branches/py3k/Lib/os.py > > python/branches/py3k/Lib/test/test_os.py > > > > Modified: python/branches/py3k/Lib/os.py > > = > > = --- python/branches/py3k/Lib/os.py (original) > > +++ python/branches/py3k/Lib/os.py Fri Oct 29 02:38:58 2010 > > @@ -382,18 +382,32 @@ > > *env* must be an environment variable dict or None. If *env* is > > None, os.environ will be used. > > """ > > +# Use a local import instead of a global import to avoid bootstrap > > issue: +# the os module is used to build Python extensions. > > +import warnings > > This sort of function import should be avoided. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On Tue, Nov 02, 2010 at 05:28:43PM +0900, Stephen J. Turnbull wrote: > C. Titus Brown writes: > > > p.s. Seriously? I can accept that there's a rational minimalist argument > > for this "feature", > > It is a feature, even if you aren't gonna need it. I want it. > > Many programmers do know that sets are partially ordered by inclusion, > preordered by size, and (in Python) totally ordered by memory address. > There's nothing wrong with not knowing that -- these are rather > abstract mathematical concepts. But it's very useful that sorted() or > .sort() use <=, and it's very useful that Python so often obeys simple > consistent rules, and it would be quite confusing to those who do > understand that "in Python the set type is partially ordered by > inclusion" if sorted() used some other relation to order collections > of sets. > > It's not so hard to change this: [ ... ] > If there were an obvious way to compare sets for use in sorting, that > way would very likely be the most useful definition for <=, too. But > there isn't, really (it's pretty obvious that comparing memory > addresses is implausible, but otherwise, there are lots of candidates > that are at least sometimes useful). Do you think otherwise? If so, > what do you propose for the OOWTDI of sorting a collection of sets? I don't have one... > > but arguing that it's somehow the responsibility of a programmer to > > *expect* this seems kind of whack. > > I don't quite agree that everyone should "expect exactly the > implemented behavior", but I do think it's a Python *programmer's* > responsibility to refrain from expecting something else in this case. ...but, as someone who has to figure out how to teach stuff to CSE undergrads (and biology grads) I hate the statement "...any programmer should expect this..." because (unless you're going to disqualify a huge swathe of people from being programmers) it's *just not true*. I don't expect Python to cater to the lowest common denominator but we should be mindful of our audience, too. I think Python has a great advantage in not being too surprising much of the time, which helps quite a bit with learning. I hope people keep that in mind for future features. cheers, --t -- C. Titus Brown, [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Resource leaks warnings
Sorry for late post. On 2010/09/29 20:01, Antoine Pitrou wrote: Furthermore, it can produce real bugs, especially under Windows when coupled with refererence cycles created by traceback objects I think this can be relaxed with the patch in #9815. ;-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On 11/2/2010 10:05 AM, C. Titus Brown wrote: ...but, as someone who has to figure out how to teach stuff to CSE undergrads (and biology grads) I hate the statement "...any programmer should expect this..." And indeed I (intentionally) did not say that. People who are ignorant and inexperienced about something should avoid making expectations in any direction until they have read the doc and experimented a bit. What I did say in the post you responded to is "Any programmer who sorts (or uses functions that depend on proper sorting) should know and respect the difference between partial orders, such as set inclusion, and total orders, such as lex order of sequences." I should hope that you teach the difference, or rather, help students to notice what they already know. Tell them to consider that difference between sorting people by a totally ordered characteristic like height or weight and a characteristic that is at best partially ordered, like hair color or ethical character. Or have them consider the partial order dependencies between morning get-ready-for-class activities (socks before shoes versus pants and shirt in either order). They already do topological sorting every day, even if the name seems fancy. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On 02/11/2010 16:23, Terry Reedy wrote: On 11/2/2010 10:05 AM, C. Titus Brown wrote: ...but, as someone who has to figure out how to teach stuff to CSE undergrads (and biology grads) I hate the statement "...any programmer should expect this..." And indeed I (intentionally) did not say that. People who are ignorant and inexperienced about something should avoid making expectations in any direction until they have read the doc and experimented a bit. Expectations come from consistent behaviour. sorted behaves consistently for *most* of the built-in types and will also work for custom types that provide a 'standard' (total ordering) implementation of __lt__. It is very easy to *not realise* that a consequence of sets (and frozensets) providing partial ordering through operator overloading is that sorting is undefined for them. Particularly as it still works for other mutable collections. Worth being aware that custom implementations of standard operators will break expectations of users who aren't intimately aware of the problem domains that the specific type may be created for. All the best, Michael Foord What I did say in the post you responded to is "Any programmer who sorts (or uses functions that depend on proper sorting) should know and respect the difference between partial orders, such as set inclusion, and total orders, such as lex order of sequences." I should hope that you teach the difference, or rather, help students to notice what they already know. Tell them to consider that difference between sorting people by a totally ordered characteristic like height or weight and a characteristic that is at best partially ordered, like hair color or ethical character. Or have them consider the partial order dependencies between morning get-ready-for-class activities (socks before shoes versus pants and shirt in either order). They already do topological sorting every day, even if the name seems fancy. -- http://www.voidspace.org.uk/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On Tue, Nov 2, 2010 at 11:23 AM, Terry Reedy wrote: > What I did say in the post you responded to is "Any programmer who sorts (or > uses functions that depend on proper sorting) should know and respect the > difference between partial orders, such as set inclusion, and total orders, > such as lex order of sequences." FWIW (i.e. not much): before this thread if you'd asked me about partial and total orders I'd have had to run to Wikipedia real quick to figure it out. Hopefully I'm still allowed to use Python. Jacob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On Tue, Nov 2, 2010 at 12:37 PM, Jacob Kaplan-Moss wrote: > Hopefully I'm still allowed to use Python. Definitely! Python's a great place to learn about all these things. :-) -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
Terry Reedy writes: > ethical character. Or have them consider the partial order dependencies > between morning get-ready-for-class activities (socks before shoes > versus pants and shirt in either order). They already do topological > sorting every day, even if the name seems fancy. Augment the example a bit, perhaps: socks and pants before shoes, socks and pants in either order. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On 04:29 pm, [email protected] wrote: On 02/11/2010 16:23, Terry Reedy wrote: On 11/2/2010 10:05 AM, C. Titus Brown wrote: ...but, as someone who has to figure out how to teach stuff to CSE undergrads (and biology grads) I hate the statement "...any programmer should expect this..." And indeed I (intentionally) did not say that. People who are ignorant and inexperienced about something should avoid making expectations in any direction until they have read the doc and experimented a bit. Expectations come from consistent behaviour. sorted behaves consistently for *most* of the built-in types and will also work for custom types that provide a 'standard' (total ordering) implementation of __lt__. It is very easy to *not realise* that a consequence of sets (and frozensets) providing partial ordering through operator overloading is that sorting is undefined for them. Perhaps. The documentation for sets says this, though: Since sets only define partial ordering (subset relationships), the output of the list.sort() method is undefined for lists of sets. Particularly as it still works for other mutable collections. Worth being aware that custom implementations of standard operators will break expectations of users who aren't intimately aware of the problem domains that the specific type may be created for. I can't help thinking that most of this confusion is caused by using < for determining subsets. If < were not defined for sets and people had to use "set.issubset" (which exists already), then sorting a list with sets would raise an exception, a much more understandable failure mode than getting back a list in arbitrary order. Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On 02/11/2010 17:17, [email protected] wrote: On 04:29 pm, [email protected] wrote: On 02/11/2010 16:23, Terry Reedy wrote: On 11/2/2010 10:05 AM, C. Titus Brown wrote: ...but, as someone who has to figure out how to teach stuff to CSE undergrads (and biology grads) I hate the statement "...any programmer should expect this..." And indeed I (intentionally) did not say that. People who are ignorant and inexperienced about something should avoid making expectations in any direction until they have read the doc and experimented a bit. Expectations come from consistent behaviour. sorted behaves consistently for *most* of the built-in types and will also work for custom types that provide a 'standard' (total ordering) implementation of __lt__. It is very easy to *not realise* that a consequence of sets (and frozensets) providing partial ordering through operator overloading is that sorting is undefined for them. Perhaps. The documentation for sets says this, though: Since sets only define partial ordering (subset relationships), the output of the list.sort() method is undefined for lists of sets. Right, I did quote that exact text earlier in the thread. False expectations come when there are exceptions to otherwise-consistent behaviour. Particularly as it still works for other mutable collections. Worth being aware that custom implementations of standard operators will break expectations of users who aren't intimately aware of the problem domains that the specific type may be created for. I can't help thinking that most of this confusion is caused by using < for determining subsets. If < were not defined for sets and people had to use "set.issubset" (which exists already), then sorting a list with sets would raise an exception, a much more understandable failure mode than getting back a list in arbitrary order. I agree. This is a cost of overloading operators with domain specific meanings. All the best, Michael Foord Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
On 11/2/2010 1:23 PM, Michael Foord wrote: Right, I did quote that exact text earlier in the thread. False expectations come when there are exceptions to otherwise-consistent behaviour. Particularly as it still works for other mutable collections. Worth being aware that custom implementations of standard operators will break expectations of users who aren't intimately aware of the problem domains that the specific type may be created for. I can't help thinking that most of this confusion is caused by using < for determining subsets. If < were not defined for sets and people had to use "set.issubset" (which exists already), then sorting a list with sets would raise an exception, a much more understandable failure mode than getting back a list in arbitrary order. I agree. This is a cost of overloading operators with domain specific meanings. I disagree. In mathematics, total ordering is a special case of partial ordering, not the other way around. Set inclusion is a standard example of non-total ordering. In everyday life, another example (other than action dependencies) are ancestry relationships. In general, acyclic directed graphs model sets with partial orders. Totally ordered linear chains, as with integers, are a special case. A Python program, for instance, is usually a non-unique topological sort of a set a statements with a non-total dependency order. This is related to a topological sort of a set of actions with a non-total dependency order. A NameError, if not due to a misspelling, is typically a result of violating one of the space or time order constraints. So I stick with my statement that a programmer should have some understanding (at least at a gut level) of non-total orders and non-unique sorts. They are a major part of what programming is. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon wrote: > So basically it seems like we have learned a lesson: we prefer to have > our code structured in files that match the public API. I think that > is a legitimate design rule for the stdlib to follow from now on, but > in the case of unittest it's too late to change it back (and it's a > minor price to pay to learn this lesson and to have Michael > maintaining unittest like he has been, plus we could consider using > the new structure so that the public API matches the file structure > when the need arises). Something to note in PEP 8, perhaps? Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cleaning-up the new unittest API
[email protected] wrote: I can't help thinking that most of this confusion is caused by using < for determining subsets. If < were not defined for sets and people had to use "set.issubset" (which exists already), then sorting a list with sets would raise an exception, a much more understandable failure mode than getting back a list in arbitrary order. Personally I think it was premature to throw out __cmp__. What should have happened instead is for __cmp__ to be augmented with a fourth outcome, "not equal but unordered". Then operations such as sorting that require a total ordering could use __cmp__ and complain if they get an unordered result. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85902 - in python/branches/py3k/Lib: os.py test/test_os.py
On Tue, Nov 2, 2010 at 10:55 PM, Victor Stinner wrote: > I don't know how to ignore the BytesWarning without importing warning. How can > I do that? I was suggesting trying to fix the bootstrap issue so you could use a top-level import, instead of working around it with a function level import (which we've learned from experience is a recipe for later reports from users of programs deadlocking on the import lock - we've made lots of improvement to avoid such deadlocks, but still prefer to avoid function level imports anyway). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 15:33, Nick Coghlan wrote: > On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon wrote: >> So basically it seems like we have learned a lesson: we prefer to have >> our code structured in files that match the public API. I think that >> is a legitimate design rule for the stdlib to follow from now on, but >> in the case of unittest it's too late to change it back (and it's a >> minor price to pay to learn this lesson and to have Michael >> maintaining unittest like he has been, plus we could consider using >> the new structure so that the public API matches the file structure >> when the need arises). > > Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes. -Brett > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: > > I think the issue here is that the file structure of the code no > longer matches the public API documented by unittest. Personally I, > like most people it seems, prefer source files to be structured in a > way to match the public API. In the case of unittest Michael didn't. > He did ask python-dev if it was okay to do what he did, we all kept > quiet, and now we have realized that most of us prefer to have files > that mirror the API; lesson learned. But Python 2.7 shipped with this > file layout so we have to stick with it lest we break any imports out > there that use the package-like file structure Michael went with > (which we could actually document and use if we wanted now that > Michael has already broken things up). Reversing the trend by sticking > all the code into unittest/__init__.py and then sticking import shims > into the existing modules would be a stupid waste of time, especially > considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Raymond___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 3:33 PM, Nick Coghlan wrote: > On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon wrote: >> So basically it seems like we have learned a lesson: we prefer to have >> our code structured in files that match the public API. I think that >> is a legitimate design rule for the stdlib to follow from now on, but >> in the case of unittest it's too late to change it back (and it's a >> minor price to pay to learn this lesson and to have Michael >> maintaining unittest like he has been, plus we could consider using >> the new structure so that the public API matches the file structure >> when the need arises). > > Something to note in PEP 8, perhaps? I'll propose some PEP 8 wording in the bug tracker (essentially advice on when and how to use packaging), and everyone can offer their assent, dissent, and word-smithing. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 02, 2010, at 03:43 PM, Brett Cannon wrote: >On Tue, Nov 2, 2010 at 15:33, Nick Coghlan wrote: >> On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon wrote: >>> So basically it seems like we have learned a lesson: we prefer to have >>> our code structured in files that match the public API. I think that >>> is a legitimate design rule for the stdlib to follow from now on, but >>> in the case of unittest it's too late to change it back (and it's a >>> minor price to pay to learn this lesson and to have Michael >>> maintaining unittest like he has been, plus we could consider using >>> the new structure so that the public API matches the file structure >>> when the need arises). >> >> Something to note in PEP 8, perhaps? > >If everyone agrees with making this policy, then yes. If SHOULD not MUST, then +0 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger wrote: > I'm not sure I follow where we're stuck with the current package. > AFAICT, the module is still used with "import unittest". > The file splitting was done badly, so I don't think there any of the > components are usable directly, i.e. "from unitest.case import SkipTest". > Also, I don't think the package structure was documented or announced. > > This is in contrast to the logging module which does have a > clean separation of components and where it isn't unusual > to import just part of the package. > > What is it you're seeing as a risk that I'm not seeing? > Are we permanently locked into the exact ten filenames > that are currently used: utils, suite, loader, case, result, main, signals, > etc? > Is the file structure now frozen? To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 15:47 -0700, Raymond Hettinger a écrit : > > What is it you're seeing as a risk that I'm not seeing? > Are we permanently locked into the exact ten filenames > that are currently used: utils, suite, loader, case, result, main, > signals, etc? > Is the file structure now frozen? I don't think it's frozen. It's just that Michael and Benjamin (perhaps others too) prefer it like that, and given who does most of the maintenance and improvement work it is reasonable to respect that decision. If one day someone else becomes maintainer of unittest, then, sure, they can undo the splitting or do it differently. But right now there's no reason to change. Oh, and I much prefer a splitting without any impact on the public API. I *hate* writing "urllib.request.urlopen" and I really wish we hadn't done that; "urllib.urlopen" was so much easier to remember it isn't funny :/ Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger wrote: > On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: > > I think the issue here is that the file structure of the code no > longer matches the public API documented by unittest. Personally I, > like most people it seems, prefer source files to be structured in a > way to match the public API. In the case of unittest Michael didn't. > He did ask python-dev if it was okay to do what he did, we all kept > quiet, and now we have realized that most of us prefer to have files > that mirror the API; lesson learned. But Python 2.7 shipped with this > file layout so we have to stick with it lest we break any imports out > there that use the package-like file structure Michael went with > (which we could actually document and use if we wanted now that > Michael has already broken things up). Reversing the trend by sticking > all the code into unittest/__init__.py and then sticking import shims > into the existing modules would be a stupid waste of time, especially > considering the head maintainer of the package likes it the way it is. > > I'm not sure I follow where we're stuck with the current package. > AFAICT, the module is still used with "import unittest". Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. > The file splitting was done badly, so I don't think there any of the > components are usable directly, i.e. "from unitest.case import SkipTest". I wouldn't say it was done badly, just non-standard. I was able to figure out where all the key classes were based on the file names. I personally would have no trouble doing ``from unittest.case import TestCase`` if more test case classes came along for various needs. > Also, I don't think the package structure was documented or announced. Announced publicly? No, not that I know of. > This is in contrast to the logging module which does have a > clean separation of components and where it isn't unusual > to import just part of the package. > What is it you're seeing as a risk that I'm not seeing? Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module. > Are we permanently locked into the exact ten filenames > that are currently used: utils, suite, loader, case, result, main, signals, > etc? > Is the file structure now frozen? Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
2010/11/2 Raymond Hettinger : > On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: > > I think the issue here is that the file structure of the code no > longer matches the public API documented by unittest. Personally I, > like most people it seems, prefer source files to be structured in a > way to match the public API. In the case of unittest Michael didn't. > He did ask python-dev if it was okay to do what he did, we all kept > quiet, and now we have realized that most of us prefer to have files > that mirror the API; lesson learned. But Python 2.7 shipped with this > file layout so we have to stick with it lest we break any imports out > there that use the package-like file structure Michael went with > (which we could actually document and use if we wanted now that > Michael has already broken things up). Reversing the trend by sticking > all the code into unittest/__init__.py and then sticking import shims > into the existing modules would be a stupid waste of time, especially > considering the head maintainer of the package likes it the way it is. > > I'm not sure I follow where we're stuck with the current package. > AFAICT, the module is still used with "import unittest". > The file splitting was done badly, so I don't think there any of the > components are usable directly, i.e. "from unitest.case import SkipTest". > Also, I don't think the package structure was documented or announced. > This is in contrast to the logging module which does have a > clean separation of components and where it isn't unusual > to import just part of the package. See http://docs.python.org/whatsnew/2.7.html#updated-module-unittest -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 22:43, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:33, Nick Coghlan wrote: On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes. I'd like to reply a bit further, I'll do it as a reply to your earlier email though. Michael -Brett Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 3:58 PM, Guido van Rossum wrote: > To spout a somewhat contrarian opinion, I just browsed the new > unittest package, and the structure seems reasonable to me, even if > its submodules are not particularly reusable. I've used this kind of > style for development myself. What is so offensive about it? I don't find anything offensive about it. The issues have to do with being able to find and analyze code. For example, to find-out what assert.ItemsEqual does, I have to figure-out that it was put in the case.py file. In Py2.6, you code use IDLE's Open Module tool to immediately bring up all the source for unittest. Then you could fire-up the class browser to quickly see and navigate the structure, but that also no longer works in Py2.7. Also, it used to be the just knowing the module name was sufficient to find the code with http://svn.python.org/view/python/branches/release26-maint/Lib/unittest.py?view=markup All you needed to study the code was a web browser and its find function. Now you need to open ten tabs to be able to browse this code. IOW, the packaging broke a read-the-source-luke style of research that I've been teaching people to use for years. I probably didn't articulate the above very well, but I think Martin said it more succinctly in this same thread. The other issue that Brett pointed out is that the file names now become part of the API, "from unittest.utils import safe_repr". In the logging module, packaging was done well. The files fell along natural lines in the API, some of the components we usable separately and testable separately. Likewise with the xml packages. In contrast, the unittest module is full of cross-imports and tightly coupled pieces (like suite and case) have been separated. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 16:20 -0700, Raymond Hettinger a écrit : > > For example, to find-out what assert.ItemsEqual does, I have > to figure-out that it was put in the case.py file. Well, it's a TestCase method, so it seems rather intuitive to look for it in case.py. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote: >> Are we permanently locked into the exact ten filenames >> that are currently used: utils, suite, loader, case, result, main, signals, >> etc? >> Is the file structure now frozen? > > Somewhat, yes. That's a bummer. Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 16:32 -0700, Raymond Hettinger a écrit : > On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote: > >> Are we permanently locked into the exact ten filenames > >> that are currently used: utils, suite, loader, case, result, main, > >> signals, > >> etc? > >> Is the file structure now frozen? > > > > Somewhat, yes. > > That's a bummer. > > Sounds like a decision to split a module into a package is a big > commitment. Each of the individual file names becomes a permanent > part of the API. Even future additional splits are precluded because > it might break someones dotted import (i.e. not a single function can > be moved between those files -- once in unittest.utils, alway in > unittest.utils). I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 23:00, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger wrote: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.) Much better times to discuss this would be when it was proposed or when it was done, not months after it has been shipped in a production release. [snip...] This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module. I *know* that some people are using the new package structure directly, because the topic has come up on the Testing in Python mailing list. Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back. Absolutely, that would be the worst of all possible worlds - a monolithic huge module but still embedded in a package. People *are* using the existing package structure to import directly from (against my advice!) as this particular topic has been discussed on the Testing In Python mailing list. Although Raymond has been vociferous in his detestation of this particular split he does not represent a "clear consensus" in favour of merging back. Both Fred Drake and Barrry Warsaw voiced their approval and on the "Clean up unittest API" issue both yourself (Brett) and Antoine have supported keeping the existing structure. Alexander Belopolsky and Martin Loewis expressed difficulties with the new structure, but that was in response to the original email from Raymond that didn't seem (on my reading) to expressly suggest re-merging unittest back into a module but was instead seemed to be using it as an example. I am aware of the costs of having a package rather than single (however large it may be) module, but to my mind the benefits to maintenance still outweigh these cost. I'm happy to again discuss these benefits at great length, but having had the same conversation in person with Raymond twice and at repeated most of the points (but by no means all) in this thread on the mailing list it really feels like going round in circles. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwr
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 22:58, Guido van Rossum wrote: On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger wrote: I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. "from unitest.case import SkipTest". Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? Amen. Although not that contrarian, others have spoken up in favour. The split is pretty obvious (in general - I'm sure its not perfect) and divided along major functionality. TestCase - case.py TestResult - result.py TestSuite - suite.py TextTestRunner - runner.py TestLoader - loader.py main function - main.py signal handling - signals.py The utils module is somewhat an odd one out as it is only used by case.py, but this is hardly the most egregious error in the world. If you can't guess where a class lives, __init__.py shows you explicitly (a clear advantage of exporting the public API at the top level ;-) Due to the original design of unittest (and I have many thoughts on that) the modules aren't strictly "reusable" (i.e. isolated from each other) - but many test frameworks (for example) just use the TestCase without using other components. I find it difficult to believe that this package structure is only acceptable if we make people import the TestCase from unittest.case and not expose it at the top level. As mentioned in another email, but this thread has many long and tedious emails, there is no clear consensus that there should be a remerger and I am aware that there are already some consumers of the new package structure. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord wrote: > As the maintainer of unittest I'd like to say that in the absence of clear > consensus that the merger should happen, or a fiat from the BDFL, the merger > won't happen. I believe that this is standard Python development process. I don't think that anybody seriously expected the unittest package would be restructured again. The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. Personally I don't think it should -- it should by now be clear that this is not an area where one style will fit all. I also think that the convenience of one style over another might have something to do with the tools being used. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Question on imports and packages
Brett, Does the import mechanism for importing packages preserve enough information to be able to figure-out where all the components are defined? I'm wondering if it is possible for the class browser to be built-out to scan/navigate class structure across a module that has been split into a package. Raymod ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 4:43 PM, Guido van Rossum wrote: > The remaining thrust of the thread seems > to be whether PEP 8 should advise against breaking code up into many > little modules. I was thinking of PEP 8 wording that listed the forces for and against. For example, ply.yacc and ply.lex was a very useful split (separately testable, natural division of concerns, no nested of cross-imports). The xml.sax, xml.dom, and xml.minidom was a nice split because it separated distinct tools. The xml packaging also worked well because it is easy to substitute in alternate parsers implementing the same API. I think we also want to recommend against putting much if any code in __init__.py. Some forces against packaging are that it breaks the class browser. As you say, different users of different toolsets are affected differently. For me, the unittest split broke my usual ways of finding out how the new methods were implemented. Another force against is what Brett pointed-out, that the package file structure becomes a permanent and unchangeable part of the API. It's a one-way street. In general, I think the advice should be that packaging should be done when there is some clear benefit beyond "turning one big file into lots of smaller files". Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 23:34, Michael Foord wrote: On 02/11/2010 23:00, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger wrote: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with "import unittest". Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.) unittest2 dammit. Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages
Raymond Hettinger writes: > >> Are we permanently locked into the exact ten filenames that are > >> currently used: utils, suite, loader, case, result, main, signals, > >> etc? […] > Sounds like a decision to split a module into a package is a big > commitment. Each of the individual file names becomes a permanent part > of the API. Even future additional splits are precluded because it > might break someones dotted import (i.e. not a single function can be > moved between those files -- once in unittest.utils, alway in > unittest.utils). Is this a case where it would be better if the package names had the leading underscore: ‘_utils’, ‘_suite’, etc.? Does the convention on single-leading-underscore identifiers as “don't rely on this name staying the same in future versions” hold for package names? -- \ “Alternative explanations are always welcome in science, if | `\ they are better and explain more. Alternative explanations that | _o__) explain nothing are not welcome.” —Victor J. Stenger, 2001-11-05 | Ben Finney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 5:03 PM, Raymond Hettinger wrote: > Some forces against packaging are that it breaks the class browser. As you > say, different users of different toolsets are affected differently. For me, > the unittest split broke my usual ways of finding out how the new methods > were implemented. Maybe the IDLE class browser can be fixed; there is plenty of code with this structure that can't or won't be restructured, no matter how strongly PEP 8 is worded. FWIW, personally I don't use the IDLE class browser -- I use Emacs, grep, and find. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Question on imports and packages
If you are importing the code, the __module__ attribute on each class should tell you where it is actually defined (as opposed to where you imported it from). Then sys.modules gives you the module object which has a __file__ attribute, etc. On Tue, Nov 2, 2010 at 4:44 PM, Raymond Hettinger wrote: > Brett, Does the import mechanism for importing packages preserve enough > information to be able to figure-out where all the components are defined? > I'm wondering if it is possible for the class browser to be built-out to > scan/navigate class structure across a module that has been split into a > package. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Antoine Pitrou writes: > I don't agree with this. Until it's documented, it's an implementation > detail and should be able to change without notice. If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)? > If someone wants to depend on some undocumented detail of the > directory layout it's their problem (like people depending on bytecode > and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. -- \“Your [government] representative owes you, not his industry | `\ only, but his judgment; and he betrays, instead of serving you, | _o__)if he sacrifices it to your opinion.” —Edmund Burke, 1774 | Ben Finney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Question on imports and packages
On 02/11/2010 23:44, Raymond Hettinger wrote: Brett, Does the import mechanism for importing packages preserve enough information to be able to figure-out where all the components are defined? I'm wondering if it is possible for the class browser to be built-out to scan/navigate class structure across a module that has been split into a package. Can it not do that through static analysis - just look at the classes / functions defined in the sub-modules. I mean, you could do it from the ast, right. Relying on importing code to analyse it is unpleasant if the code has top level side-effects (which no good code does of course). There may be *some* cases where magic makes things weird (__package__), but how common are those in practise? If you build up a data-structure representing definitions in a package, working out where any individual class / function used in a module is defined is a matter of looking at where it is imported (assuming it hasn't been aliased or fetched dynamically) and matching the import to a package you have analysed (or analyse on the fly). A project that attempts to do something like this is pysmell: http://github.com/orestis/pysmell/ All the best, Michael Raymod ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 12:47 am, [email protected] wrote: Antoine Pitrou writes: I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)? If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. And if that isn't the rule, then what the heck is? Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 8:47 PM, Ben Finney wrote: > I would say that names without a single leading underscore are part of > the public API, whether documented or not. I don't recall this ever being the standard library's policy. There are many modules using leading underscores as hints, and many others which don't. Other people consider the presence of a docstring on a non-underscored name significant, while still others refer to the out-of-line documentation as definitive. For modules, an __all__ attribute is generally agreed on as definitive, if present, but that's a fairly limited case. At this point, there isn't a single clear way to determine if something is public API. I doubt it will be likely to agree on a single definition now without engendering a lengthy discussion on whether names can be changed to reflect such a policy (where backward compatibility is sure to be lost). I'm sticking to the out-of-line documentation to determine what's public. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 16:43, Guido van Rossum wrote: > On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord > wrote: >> As the maintainer of unittest I'd like to say that in the absence of clear >> consensus that the merger should happen, or a fiat from the BDFL, the merger >> won't happen. I believe that this is standard Python development process. > > I don't think that anybody seriously expected the unittest package > would be restructured again. The remaining thrust of the thread seems > to be whether PEP 8 should advise against breaking code up into many > little modules. Personally I don't think it should -- it should by now > be clear that this is not an area where one style will fit all. I also > think that the convenience of one style over another might have > something to do with the tools being used. This is not what I am suggesting for PEP 8. I want to say that a package's file structure should reflect the public API. I personally have no trouble with modules in packages that do not have a ton of objects in them. I just think if you have pkg/mod.py, pkg.mod should be exposed in the API, else name the file _mod.py. In the case of unittest that would just mean documenting that it's unittest.case.TestCase and that unittest.TestCase is for legacy reasons, much like os.path is just blindly added on to os even though it is a separate module(s). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 02:35, Brett Cannon wrote: On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou wrote: On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward. [snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly. It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score. As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post. All the best, Michael Foord I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Question on imports and packages
On Tue, Nov 2, 2010 at 17:35, Guido van Rossum wrote: > If you are importing the code, the __module__ attribute on each class > should tell you where it is actually defined (as opposed to where you > imported it from). Then sys.modules gives you the module object which > has a __file__ attribute, etc. What Guido said. It's the equivalent of browsing an object that a function returned to you. Working backwards to where something is defined has nothing to do with imports and more to do with __module__, __class__, etc. Import has nothing to do with introspection for things that you access off of a module that happened to have imported the object. > > On Tue, Nov 2, 2010 at 4:44 PM, Raymond Hettinger > wrote: >> Brett, Does the import mechanism for importing packages preserve enough >> information to be able to figure-out where all the components are defined? >> I'm wondering if it is possible for the class browser to be built-out to >> scan/navigate class structure across a module that has been split into a >> package. > > -- > --Guido van Rossum (python.org/~guido) > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 19:50, Michael Foord wrote: > On 02/11/2010 02:35, Brett Cannon wrote: >> >> On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou wrote: >>> >>> On Tue, 26 Oct 2010 22:06:37 -0400 >>> Alexander Belopolsky wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single "big-ball-of-mud" namespace, only rather than implemented in a single file, it is now swept in from eight different files. >>> >>> Are you saying that it has become a pile of medium-sized balls of mud? >>> I would like to say thanks for the mud, Michael! It's high quality mud >>> for sure. >> >> I realize I am a little late in this reply but issue 10273 linked to >> this and so now I am actually bothering to read this thread since it >> felt like bikeshedding when the thread began. >> >> I think the issue here is that the file structure of the code no >> longer matches the public API documented by unittest. Personally I, >> like most people it seems, prefer source files to be structured in a >> way to match the public API. In the case of unittest Michael didn't. > > Well the structure *does* match the API (which is primarily why I disagree > with Raymond that this is a 'bad split'). But not the public API as documented, e.g., it's documented as unittest.TestCase, not unittest.case.TestCase as the file structure suggests. > > How could we have split the module into a package in a way that matched the > API, whilst still retaining backwards compatibility with the old API? We had > no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -Brett > >> He did ask python-dev if it was okay to do what he did, we all kept >> quiet, and now we have realized that most of us prefer to have files > > Most of us? Raymond, Alexander and Martin are the only ones I *recall* > complaining about the split specifically in this thread and not all of those > were on the grounds you mention. Several people supported the split. Guido > didn't object to it on these grounds and Antoine noted that finding core > classes was generally straightforward. > >> [snip...] >> So basically it seems like we have learned a lesson: we prefer to have >> our code structured in files that match the public API. > > When designing packages from the ground up that is a sensible rule of thumb > to follow, but usually follows naturally from good design. This doesn't > necessarily mean that all the sub-modules will export public APIs for > consumers, so this is where I disagree with this. Python's package system is > a very useful way of providing internal structure for projects. That doesn't > mean that this structure must always be exposed publicly. > > It should be as easy to navigate as possible (and there is plenty about the > old unittest.py module that wasn't easy to navigate I can assure you), but I > *don't* think that the new package fails in a substantially greater way on > that score. > > As Guido points out, this may depend a lot on which tools you use. I could > write more about the role and value of packages, I guess I'll save it for a > blog post. > > All the best, > > Michael Foord > >> I think that >> is a legitimate design rule for the stdlib to follow from now on, but >> in the case of unittest it's too late to change it back (and it's a >> minor price to pay to learn this lesson and to have Michael >> maintaining unittest like he has been, plus we could consider using >> the new structure so that the public API matches the file structure >> when the need arises). >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > -- > > http://www.voidspace.org.uk/ > > READ CAREFULLY. By accepting and reading this email you agree, > on behalf of your employer, to release me from all obligations > and waivers arising from any and all NON-NEGOTIATED agreements, > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > confidentiality, non-disclosure, non-compete and acceptable use > policies (”BOGUS AGREEMENTS”) that I have entered into with your > employer, its partners, licensors, agents and assigns, in > perpetuity, without prejudice to my ongoing rights and privileges. > You further represent that you have the authority to release me > from any BOGUS
Re: [Python-Dev] Question on imports and packages
FWIW, I also agree with Michael that static analysis would be much preferred. You never know what side effects importing a module has. (This could even be construed as an attack vector.) --Guido On Tue, Nov 2, 2010 at 7:54 PM, Brett Cannon wrote: > On Tue, Nov 2, 2010 at 17:35, Guido van Rossum wrote: >> If you are importing the code, the __module__ attribute on each class >> should tell you where it is actually defined (as opposed to where you >> imported it from). Then sys.modules gives you the module object which >> has a __file__ attribute, etc. > > What Guido said. It's the equivalent of browsing an object that a > function returned to you. Working backwards to where something is > defined has nothing to do with imports and more to do with __module__, > __class__, etc. Import has nothing to do with introspection for things > that you access off of a module that happened to have imported the > object. > >> >> On Tue, Nov 2, 2010 at 4:44 PM, Raymond Hettinger >> wrote: >>> Brett, Does the import mechanism for importing packages preserve enough >>> information to be able to figure-out where all the components are defined? >>> I'm wondering if it is possible for the class browser to be built-out to >>> scan/navigate class structure across a module that has been split into a >>> package. >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, 03 Nov 2010 11:47:55 +1100 Ben Finney wrote: > > > If someone wants to depend on some undocumented detail of the > > directory layout it's their problem (like people depending on bytecode > > and other stuff). > > I would say that names without a single leading underscore are part of > the public API, whether documented or not. That's not what we are talking about; we are talking about their locations. If the official location is the unittest package, then I don't see why we should also support undocumented locations just because they happen to work. Otherwise we should also support e.g. "unittest.unlink" if the unittest package happens to have "from os import unlink" at its top. I don't think it's reasonable. Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 2 Nov 2010 19:57:48 -0700 Brett Cannon wrote: > > > > How could we have split the module into a package in a way that matched the > > API, whilst still retaining backwards compatibility with the old API? We had > > no choice but to export the public names at the top level. > > I'm not disagreeing with that. What I am saying is can now document > that it's unittest.case.TestCase instead of having that just be an > implementation detail. -1. unittest.TestCase is far simpler and more obvious that any javaesque qualified name. urllib.request and friends are already annoying enough. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 7:50 PM, Brett Cannon wrote: > This is not what I am suggesting for PEP 8. I want to say that a > package's file structure should reflect the public API. But what does that mean? I could argue that unittest's structure (TestCase in case.py, etc.) reflects its public API just fine. > I personally > have no trouble with modules in packages that do not have a ton of > objects in them. I just think if you have pkg/mod.py, pkg.mod should > be exposed in the API, else name the file _mod.py. In the case of > unittest that would just mean documenting that it's > unittest.case.TestCase and that unittest.TestCase is for legacy > reasons, much like os.path is just blindly added on to os even though > it is a separate module(s). I really don't think we should encourage the use as unittest.case.TestCase -- it's unnecessarily introducing structure. I think it's fine now that the cat is out of the bag to document unittest.case.TestCase as an alternative spelling, but I don't think it should be the preferred one. os.path is so old that should not be taken as an example for anything. (It predates packages!) But it should not be changed either, there'd be too much churn. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages
Antoine Pitrou writes: > On Wed, 03 Nov 2010 11:47:55 +1100 > Ben Finney wrote: > > > > > If someone wants to depend on some undocumented detail of the > > > directory layout it's their problem (like people depending on > > > bytecode and other stuff). > > > > I would say that names without a single leading underscore are part > > of the public API, whether documented or not. > > That's not what we are talking about; we are talking about their > locations. If the official location is the unittest package, then I > don't see why we should also support undocumented locations just > because they happen to work. So long as the names available for import are such that they indicate whether they're public or implementation-detail (i.e. without a leading single underscore or with one), I agree that this is distinct from the issue of locations on the filesystem. > Otherwise we should also support e.g. "unittest.unlink" if the > unittest package happens to have "from os import unlink" at its top. I > don't think it's reasonable. Hmm. That example does give me pause. I'm trying to think of a simple way that such imports are excluded from being “public interface”, but can't immediately think of one. The distinction is clear in my head, though, for what it's worth :-) -- \“I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.” —Douglas Adams | Ben Finney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages
Just a small input into this discussion: In EVE, for historical reasons, we implemented our own importing mechanism. I think it is because we started out with an ancient Python version that didn't support packages. Regardless, we still have a system where a hierarchy of files is scanned, and then code in each .py files determines where in the "namespace" it lands. This can be Declaratively (by using a __guid__ attribute on a class, for instance) or by defining a special __exports__ dict at the module level. The good thing about this system is that it allows us to separate code in a manner independent of the api. We can choose for example to group all network Code in a folder. Or have each class in the "game.entity" namespace be defined in its own file. It unhooks file structure from name structure. Now, this has its own problems of course, the biggest of it being that it is non-standard. Off the shelf IDEs have problems with it. And we have to implement dynamic reloading on our own. The list goes on, and for that reason, we are moving away from it in favor of standard python import. However, I am personally not super happy about how this will force one to think in "api" terms when creating source files. As has been mentioned, files cannot be moved and restructured once in general use, and when writing new code, one has to think long and hard about "where" to put the source, not "what" to put in it. What is more, a hierarchy, while a convenient system for storing files, does not, IMHO, always map to problem domain. But we're having a go at it. Time will tell if "forcing us to think inside the hierarchy" will be beneficial in the long run. Cheers, K ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
