Re: [Python-Dev] Best practices for Enum
On May 12, 2013, at 8:26 PM, Eli Bendersky wrote: > Thanks for the insights, Raymond. I don't think anyone is planning on rushing > anything. We still have to get the enum module itself committed and a serious > review process has just started for that, so it will take time. > > There's no general "let's replace all constants with enums" TODO item that I > know of. It's my hope that such changes will happen very gradually and only > when deemed important and useful by core developers. Ethan's email suggests that against my advice he is in-fact going to go through the standard library, applying enums quite broadly. That is somewhat at odds with the notions of holistic refactoring and gradual change. Nor does it reflect sufficient appreciation for concerns about maintenance issues, code stability, the effect on 2-to-3 migration, doctests, performance, the wishes of the module authors, or whether users will see any actual benefits (particularly for internal constants). I fully understand the enthusiasm to take the car out for a spin, but the standard library isn't really a great place for experimentation. And "trying to learn the standard library" isn't a good rationale for making extensive changes to it. So, please do help make sure there is some restraint and careful consideration. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Best practices for Enum
Raymond Hettinger writes: > whether users will see any actual benefits (particularly for > internal constants). I don't understand the parenthetical remark. It seems to me that changing internal constants should have the benefits that Ethan points to for understanding, debugging, and interactive exploration, with the least risk of destabilizing external APIs. I agree with you that the *net* benefit is still likely to be negative due to effects of code churn and the potential for new bugs, but (considering benefit separately from cost) the advertised benefits would be achieved. The point is that I think it's clear that using enums to provide name spaces for the large number of constants whose names are well-known, but not so much the values, in the os module and other OS library wrappers is highest priority, and IMO a net plus. But I would say the next place to look would be exactly these internal constants, where they have similar characteristics (many of them, well-known names, hard-to-remember values: the fds for stdin, stdout, and stderr are non-candidates!) As I said, *I* don't think it's worth doing internal constants, but I couldn't defend that opinion well to somebody who thinks it is worth doing. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Best practices for Enum
Le Mon, 13 May 2013 00:06:52 -0700, Raymond Hettinger a écrit : > > On May 12, 2013, at 8:26 PM, Eli Bendersky wrote: > > > Thanks for the insights, Raymond. I don't think anyone is planning > > on rushing anything. We still have to get the enum module itself > > committed and a serious review process has just started for that, > > so it will take time. > > > > There's no general "let's replace all constants with enums" TODO > > item that I know of. It's my hope that such changes will happen > > very gradually and only when deemed important and useful by core > > developers. > > Ethan's email suggests that against my advice he is in-fact going to > go through the standard library, applying enums quite broadly. It probably won't go in without reviews, so there's no need to be too concerned IMHO. The fact that one of the enum classes is an int subclass should make the behaviour changes minimal, if any. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Best practices for Enum
Raymond Hettinger wrote: > I would like to ask for a little restraint and for there to > be individual cost/benefit evaluations for each case. +1 > For external constants, some thought needs to be given to: > * is the current API working just fine (i.e. decimal's ROUND_DOWN) For compatibility with the Python version, I recently changed the rounding constants of the C version to strings. This was at the request of a user who wanted to exchange (Decimal, ROUNDING) pickles between the versions. I think the strings are working fine and personally I have no plans to change the type again. The episode shows that pickling backwards compatibility is one thing to consider, but I'm probably stating the obvious here. :) Stefan Krah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tightening up the specification for locals()
On Sun, May 12, 2013 at 2:01 PM, Antoine Pitrou wrote: > On Fri, 03 May 2013 12:43:41 +1000 > Steven D'Aprano wrote: >> On 03/05/13 11:29, Nick Coghlan wrote: >> > An exchange in one of the enum threads prompted me to write down >> > something I've occasionally thought about regarding locals(): it is >> > currently severely underspecified, and I'd like to make the current >> > CPython behaviour part of the language/library specification. (We >> > recently found a bug in the interaction between the __prepare__ method >> > and lexical closures that was indirectly related to this >> > underspecification) >> >> Fixing the underspecification is good. Enshrining a limitation as the >> one correct way, not so good. > > I have to say, I agree with Steven here. Mutating locals() is currently > an implementation detail, and it should IMHO stay that way. Only > reading a non-mutated locals() should be well-defined. > > Regards > > Antoine. Like it or not, people rely on this behavior. I don't think CPython (or PyPy) can actually afford to change it. If so, documenting it sounds like a better idea than leaving it undocumented only known to the "inner shrine" Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] weak refs in descriptors (http://bugs.python.org/issue17950)
Hello python-dev. I'm working on a patch to remove reference cycles from heap-allocated classes: http://bugs.python.org/issue17950 Part of the patch involves making sure that descriptors in the class dictionary don't contain strong references to the class itself. This is item 2) in the defect description. I have implemented this via weak references and hit no issues at all when running the test suite. But I'd like to ask the oracle if there is anything I may be overlooking with this approach? Any hidden problems we might encounter? K ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
Okay, I've renamed my "BetterWalk" module to "scandir" and updated it as per our discussion: https://github.com/benhoyt/scandir/#readme It's not yet production-ready, and is basically still in API and performance testing stage. For instance, the underlying scandir_helper functions don't even return iterators yet -- they're just glorified versions of os.listdir() that return an additional d_ino/d_type (Linux) or stat_result (Windows). In any case, I really like the API (thanks mostly to Nick Coghlan), and performance is great, even with DirEntry being written in Python. PERFORMANCE: On Windows I'm seeing that scandir.walk() on a large test tree (see benchmark.py) is 8-9 times faster than os.walk(), and on Linux it's 3-4 times faster. Yes, it is that much faster, and yes, those numbers are real. :-) Please critique away. At this stage it'd be most helpful to critique any API or performance-related issues rather than coding style or minor bugs, as I'm expecting the code itself will change quite a bit still. Todos: * Make _scandir.scandir_helper functions return real iterators instead of lists * Move building of DirEntry objects into C module, so basically the entire scandir() is in C * Add tests -Ben ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
Hi Ben, Am 13.05.13 14:25, schrieb Ben Hoyt: ...It's not yet production-ready, and is basically still in API and performance testing stage. ... In any case, I really like the API (thanks mostly to Nick Coghlan), and performance is great, even with DirEntry being written in Python. PERFORMANCE: On Windows I'm seeing that scandir.walk() on a large test tree (see benchmark.py) is 8-9 times faster than os.walk(), and on Linux it's 3-4 times faster. Yes, it is that much faster, and yes, those numbers are real. :-) Please critique away. At this stage it'd be most helpful to critique any API or performance-related issues ... you asked for critique, but the performance seems to be also 2-3 times speedup (as stated by benchmark.py) on mac osx 10.8.3 (on MacBook Pro 13 inch, start of 2011, solid state disk) with python 2.7.4 (the homebrew one): $> git clone git://github.com/benhoyt/scandir.git $> cd scandir && python setup.py install $> python benchmark.py USING FAST C version Creating tree at benchtree: depth=4, num_dirs=5, num_files=50 Priming the system's cache... Benchmarking walks on benchtree, repeat 1/3... Benchmarking walks on benchtree, repeat 2/3... Benchmarking walks on benchtree, repeat 3/3... os.walk took 0.104s, scandir.walk took 0.031s -- 3.3x as fast $> python benchmark.py -s USING FAST C version Priming the system's cache... Benchmarking walks on benchtree, repeat 1/3... Benchmarking walks on benchtree, repeat 2/3... Benchmarking walks on benchtree, repeat 3/3... os.walk size 226395000, scandir.walk size 226395000 -- equal os.walk took 0.246s, scandir.walk took 0.125s -- 2.0x as fast So for now, all well and thank you. All the best, Stefan. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
On Mon, May 13, 2013 at 10:25 PM, Ben Hoyt wrote: > Okay, I've renamed my "BetterWalk" module to "scandir" and updated it > as per our discussion: > > https://github.com/benhoyt/scandir/#readme Nice! > PERFORMANCE: On Windows I'm seeing that scandir.walk() on a large test > tree (see benchmark.py) is 8-9 times faster than os.walk(), and on > Linux it's 3-4 times faster. Yes, it is that much faster, and yes, > those numbers are real. :-) I'd to see the numbers for NFS or CIFS - stat() can be brutally slow over a network connection (that's why we added a caching mechanism to importlib). > Please critique away. At this stage it'd be most helpful to critique > any API or performance-related issues rather than coding style or > minor bugs, as I'm expecting the code itself will change quite a bit > still. I initially quite liked the idea of not offering any methods on DirEntry, only properties, to make it obvious that they don't touch the file system, but just report info from the scandir call. However, I think that it ends up reading strangely, and would be confusing relative to the os.path() APIs. What you have now seems like a good, simple alternative. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weak refs in descriptors (http://bugs.python.org/issue17950)
On Mon, May 13, 2013 at 1:49 PM, Kristján Valur Jónsson wrote: > Hello python-dev. > > I‘m working on a patch to remove reference cycles from heap-allocated > classes: http://bugs.python.org/issue17950 > > Part of the patch involves making sure that descriptors in the class > dictionary don’t contain strong references to the class itself. > > This is item 2) in the defect description. > > I have implemented this via weak references and hit no issues at all when > running the test suite. > > But I’d like to ask the oracle if there is anything I may be overlooking > with this approach? Any hidden problems we might encounter? > > > > K Hi Kristjan The strong reference there is a feature. Descriptors keep the class alive if somehow the class disappears and the descriptor itself does not. Please don't change language semantics (yes, this is a change in semantics), just because the test suite passes - I can assure you there are people doing convoluted stuff that expect this to work. Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weak refs in descriptors (http://bugs.python.org/issue17950)
On 13 May 2013 10:20, Maciej Fijalkowski wrote: > On Mon, May 13, 2013 at 1:49 PM, Kristján Valur Jónsson > wrote: >> Hello python-dev. >> >> I‘m working on a patch to remove reference cycles from heap-allocated >> classes: http://bugs.python.org/issue17950 >> >> Part of the patch involves making sure that descriptors in the class >> dictionary don’t contain strong references to the class itself. >> >> This is item 2) in the defect description. >> >> I have implemented this via weak references and hit no issues at all when >> running the test suite. >> >> But I’d like to ask the oracle if there is anything I may be overlooking >> with this approach? Any hidden problems we might encounter? >> >> >> >> K > > Hi Kristjan > > The strong reference there is a feature. Descriptors keep the class > alive if somehow the class disappears and the descriptor itself does > not. Please don't change language semantics (yes, this is a change in > semantics), just because the test suite passes - I can assure you > there are people doing convoluted stuff that expect this to work. > +1 for it being an expected behavior. So I think it would be a nice thing to write a test that breaks under this condition js -><- > Cheers, > fijal > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tightening up the specification for locals()
> Like it or not, people rely on this behavior. I don't think CPython > (or PyPy) can actually afford to change it. If so, documenting it > sounds like a better idea than leaving it undocumented only known to > the "inner shrine" > +1. I am relying on this. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Best practices for Enum
On May 12, 2013, at 04:49 PM, Raymond Hettinger wrote: >After the long design effort for the enum module, I'm sure there will be a >forthcoming effort to apply them pervasively throughout the standard library. We usually, explicitly, try not to do such wholesale adoptions in the stdlib when new features land. This is almost always a good idea in order to gain more experience with the new feature, reduce code churn (and thus the introduction of bugs), and aid in back/forward porting. It seems entirely reasonable to me to be just as conservative about adoption of enums in the stdlib. As Eli mentions, making the socket constants enums seems like a good test case. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
Am 13.05.2013 02:21, schrieb Ben Hoyt: > Are you suggesting just accessing .cached_lstat could call os.lstat()? > That seems very bad to me. It's a property access -- it looks cheap, > therefore people will expect it to be. From PEP 8 "Avoid using > properties for computationally expensive operations; the attribute > notation makes the caller believe that access is (relatively) cheap." > > Even worse is error handling -- I'd expect the expression > "entry.cached_lstat" to only ever raise AttributeError, not OSError in > the case it calls stat under the covers. Calling code would have to > have a try/except around what looked like a simple attribute access. > > For these two reasons I think lstat() should definitely be a function. OK, you got me! I'm now convinced that a property is a bad idea. I still like to annotate that the function may return a cached value. Perhaps lstat() could require an argument? def lstat(self, cached): if not cached or self._lstat is None: self._lstat = os.lstat(...) return self._lstat > True. My isdir/isfile/islink implementations should catch any OSError > from the lstat() and return False (like os.path.isdir etc do). But > then calling code still doesn't need try/excepts around the isdir() > calls. This is how os.walk() is implemented -- there's no extra error > handling around the isdir() call. You could take the opportunity and take the 'file was deleted' case into account. I admit it has a very low priority. Please regard the case for bonus points only. ;) > Sure. I'm primarily a Windows dev, so not too familiar with all the > fancy stat* functions. But what you're saying makes sense. I'm glad to be of assistance! The feature is new (added in 3.3) and is available on most POSIX platforms. http://docs.python.org/3/library/os.html#dir-fd If you need any help or testing please feel free to ask me. I really like to get this feature into 3.4. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Best practices for Enum
On 05/13/2013 12:06 AM, Raymond Hettinger wrote: Ethan's email suggests that against my advice he is in-fact going to go through the standard library, applying enums quite broadly. I think you are falling victim to Wizard's First Rule: people will believe what they want to be true, or are afraid is true. What I said was I'd like to apply them where it makes sense. Which does not mean quite broadly, unless you think it would make sense to have them everywhere? Because that's not the impression I have from your posts. Furthermore, what's wrong with going through the stdlib, examining the various modules, and then asking questions to see if, indeed, it does "make sense" to use enums in that module? How else will I know? Are you going to give me list of which ones are acceptable? I don't know you well enough to guess at your motivations for misrepresenting me, but please stop. If you have a question, ask me. -- ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weak refs in descriptors (http://bugs.python.org/issue17950)
On 5/13/2013 9:20 AM, Maciej Fijalkowski wrote: The strong reference there is a feature. Descriptors keep the class alive if somehow the class disappears and the descriptor itself does Is this feature stated or implied in the reference manual? 3.3.2.1. Implementing Descriptors 3.3.2.2. Invoking Descriptors ??? or is it an implementation detail that people have come to rely on? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weak refs in descriptors (http://bugs.python.org/issue17950)
On Mon, 13 May 2013 13:21:17 -0400 Terry Jan Reedy wrote: > On 5/13/2013 9:20 AM, Maciej Fijalkowski wrote: > > > The strong reference there is a feature. Descriptors keep the class > > alive if somehow the class disappears and the descriptor itself does > > Is this feature stated or implied in the reference manual? > 3.3.2.1. Implementing Descriptors > 3.3.2.2. Invoking Descriptors > ??? > or is it an implementation detail that people have come to rely on? Any reference that is not documentedly weak is strong by definition; this is Python's basic semantics, there's no need to ask about documentation pointers. The only question is whether some people rely on this particular one. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
> I'd to see the numbers for NFS or CIFS - stat() can be brutally slow > over a network connection (that's why we added a caching mechanism to > importlib). How do I know what file system Windows networking is using? In any case, here's some numbers on Windows -- it's looking pretty good! This is with default DEPTH/NUM_DIRS/NUM_FILES on a LAN: Benchmarking walks on \\anothermachine\docs\Ben\bigtree, repeat 3/3... os.walk took 11.345s, scandir.walk took 0.340s -- 33.3x as fast And this is on a VPN on a remote network with the benchmark.py values cranked down to DEPTH = 3, NUM_DIRS = 3, NUM_FILES = 20 (because otherwise it was taking far too long): Benchmarking walks on \\ben1.titanmt.local\c$\dev\scandir\benchtree, repeat 3/3... os.walk took 122.310s, scandir.walk took 5.452s -- 22.4x as fast If anyone can run benchmark.py on Linux / NFS or similar, that'd be great. You'll probably have to lower DEPTH/NUM_DIRS/NUM_FILES first and then move the "benchtree" to the network file system to run it against that. > I initially quite liked the idea of not offering any methods on > DirEntry, only properties, to make it obvious that they don't touch > the file system, but just report info from the scandir call. However, > I think that it ends up reading strangely, and would be confusing > relative to the os.path() APIs. > > What you have now seems like a good, simple alternative. Thanks. Yeah, I kinda liked the "DirEntry doesn't make any OS calls" at first too, but then as I got into it I realized it make for a really nasty API for most use cases. I like how it's ended up. -Ben ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
> OK, you got me! I'm now convinced that a property is a bad idea. Thanks. :-) > I still like to annotate that the function may return a cached value. > Perhaps lstat() could require an argument? > > def lstat(self, cached): > if not cached or self._lstat is None: > self._lstat = os.lstat(...) > return self._lstat Hmm, I'm just not sure I like the API. Setting cached to True to me would imply it's only ever going to come from the cache (i.e., just return self._lstat). Also, isdir() etc have the same issue, so if you're going this route, their signatures would need this too. The DirEntry instance is really a cached value in itself. ".name" is cached, ".dirent" is cached, and the methods return cached if they can. That's more or less the point of the object. But you have a fair point, and this would need to be explicit in the documentation. -Ben > > >> True. My isdir/isfile/islink implementations should catch any OSError >> from the lstat() and return False (like os.path.isdir etc do). But >> then calling code still doesn't need try/excepts around the isdir() >> calls. This is how os.walk() is implemented -- there's no extra error >> handling around the isdir() call. > > You could take the opportunity and take the 'file was deleted' case into > account. I admit it has a very low priority. Please regard the case for > bonus points only. ;) > >> Sure. I'm primarily a Windows dev, so not too familiar with all the >> fancy stat* functions. But what you're saying makes sense. > > I'm glad to be of assistance! The feature is new (added in 3.3) and is > available on most POSIX platforms. > http://docs.python.org/3/library/os.html#dir-fd > > If you need any help or testing please feel free to ask me. I really > like to get this feature into 3.4. > > Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 - ref impl disc 2
On 05/10/2013 10:15 PM, Glenn Linderman wrote:
So it is quite possible to marry the two, as Ethan helped me figure out using
an earlier NamedInt class:
class NIE( IntET, Enum ):
x = ('NIE.x', 1)
y = ('NIE.y', 2)
z = ('NIE.z', 4)
and then expressions involving members of NIE (and even associated integers)
will be tracked... see demo1.py.
But the last few lines of demo1 demonstrate that NIE doesn't like, somehow,
remember that its values, deep down under
the covers, are really int. And doesn't even like them when they are wrapped
into IntET objects. This may or may not
be a bug in the current Enum implementation.
[demo1.py excerpt]
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))
So the questions are:
1) Is there a bug in ref435 Enum that makes demo1 report errors instead of
those lines working?
Nope.
2) Is something like demo2 interesting to anyone but me? Of course, I think it
would be great for reporting flag values
using names rather than a number representing combined bit fields.
No idea. ;)
3) I don't see a way to subclass the ref435 EnumMeta except by replacing the
whole __new__ method... does this mechanism
warrant a slight refactoring of EnumMeta to make this mechanism easier to
subclass with less code redundancy?
I've broken it down to make subclassing easier.
4) Or is it simple enough and useful enough to somehow make it a feature of
EnumMeta, enabled by a keyword parameter?
Probably not.
5) All this is based on "IntET"... which likely suffices for API flags
parameters... but when I got to __truediv__ and
__rtruediv__, which don't return int, then I started wondering how to write a
vanilla ET class that inherits from
"number" instead of "int" or "float"? One could, of course, make cooperating
classes FloatET and DecimalET is this
a language limitation, or is there more documentation I haven't read? :) (I
did read footnote [1] of
,
and trembled.)
Sounds like a fun project (for some value of fun ;)
Okay, sorry for the long delay.
What it comes down to is if you want to marry two complex types together, you
may have to be the counselor as well. ;)
Here's your code, revamped. I did make a slight change in the meta -- I moved the name assignment above the __init__
call so it's available in __init__.
--8<
from ref435 import Enum
from flags import IntET
class NIE1( IntET, Enum ):
x = 1
y = 2
z = 4
def __new__(cls, value):
member = IntET.__new__(cls, 'temp', value)
member._value = value
return member
def __init__(self, value):
self._etname = self._name
print( repr( NIE1.x.value ))
print( repr( NIE1.x + NIE1.y ))
print( repr( NIE1.x + ~ NIE1.y))
print( repr( NIE1.x + ~ 2 ))
print( repr( NIE1.z * 3 ))
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))
--8<
and my results:
1
IntET('(x + y)', 3)
IntET('(x + ~y)', -2)
IntET('(x + -3)', -2)
IntET('(z * 3)', 12)
IntET('(x + y)', 3)
IntET('(x + y)', 3)
Oh, and if you really wanted the 'NEI' in the _etname, change the name
assignment:
self._etname = 'NIE.' + self._name
--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 - ref impl disc 2
On 05/13/2013 07:36 PM, Ethan Furman wrote:
Here's your code, revamped. I did make a slight change in the meta -- I moved
the name assignment above the __init__
call so it's available in __init__.
--8<
from ref435 import Enum
from flags import IntET
class NIE1( IntET, Enum ):
x = 1
y = 2
z = 4
def __new__(cls, value):
member = IntET.__new__(cls, 'temp', value)
member._value = value
return member
def __init__(self, value):
self._etname = self._name
print( repr( NIE1.x.value ))
print( repr( NIE1.x + NIE1.y ))
print( repr( NIE1.x + ~ NIE1.y))
print( repr( NIE1.x + ~ 2 ))
print( repr( NIE1.z * 3 ))
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))
--8<
and my results:
1
IntET('(x + y)', 3)
IntET('(x + ~y)', -2)
IntET('(x + -3)', -2)
IntET('(z * 3)', 12)
IntET('(x + y)', 3)
IntET('(x + y)', 3)
Forget to mention the good part -- in the custom __new__ you are able to set the value to whatever you want (not a big
deal in this case, but if you had several parameters going in you could still make _value be a single, simple int).
--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 - ref impl disc 2
On 5/13/2013 7:36 PM, Ethan Furman wrote:
On 05/10/2013 10:15 PM, Glenn Linderman wrote:
So it is quite possible to marry the two, as Ethan helped me figure
out using an earlier NamedInt class:
class NIE( IntET, Enum ):
x = ('NIE.x', 1)
y = ('NIE.y', 2)
z = ('NIE.z', 4)
and then expressions involving members of NIE (and even associated
integers) will be tracked... see demo1.py.
But the last few lines of demo1 demonstrate that NIE doesn't like,
somehow, remember that its values, deep down under
the covers, are really int. And doesn't even like them when they are
wrapped into IntET objects. This may or may not
be a bug in the current Enum implementation.
[demo1.py excerpt]
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))
So the questions are:
1) Is there a bug in ref435 Enum that makes demo1 report errors
instead of those lines working?
Nope.
Well, if it isn't a bug, it will be interesting to read the
documentation that explains the behavior, when the documentation is written:
The "obvious" documentation would be that Enum names values of any type,
particularly the first type in the multiple-inheritance list. The values
assigned to the enumeration members are used as parameters to the
constructor of that first type, but the value of the enumeration member
itself is an item of the type, created by the constructor.
The __call__ syntax [ EnumDerivation( value ) ] looks up enumeration
members by value.
The obvious documentation would stop there. But if demo1 doesn't
demonstrate a bug, it would have to continue, saying something like:
However, if you have a complex type, you can't look up by value, but
rather have to resupply the constructor parameters used to create the
item. This means that for simple types
EnumDerivation( EnumerationMember.value ) is EnumerationMember
but that doesn't hold for complex types. I think it should.
2) Is something like demo2 interesting to anyone but me? Of course, I
think it would be great for reporting flag values
using names rather than a number representing combined bit fields.
No idea. ;)
3) I don't see a way to subclass the ref435 EnumMeta except by
replacing the whole __new__ method... does this mechanism
warrant a slight refactoring of EnumMeta to make this mechanism
easier to subclass with less code redundancy?
I've broken it down to make subclassing easier.
Thanks... I'll take a look, eventually, but I'll be offline until next week.
4) Or is it simple enough and useful enough to somehow make it a
feature of EnumMeta, enabled by a keyword parameter?
Probably not.
5) All this is based on "IntET"... which likely suffices for API
flags parameters... but when I got to __truediv__ and
__rtruediv__, which don't return int, then I started wondering how to
write a vanilla ET class that inherits from
"number" instead of "int" or "float"? One could, of course, make
cooperating classes FloatET and DecimalET is this
a language limitation, or is there more documentation I haven't read?
:) (I did read footnote [1] of
,
and trembled.)
Sounds like a fun project (for some value of fun ;)
Not sure I'll get there, for a few years... such might be useful in
certain debugging scenarios, but not sure it is useful enough to
implement, given the footnote, except, perhaps, to truly become an
expert in the Python object model.
Okay, sorry for the long delay.
What it comes down to is if you want to marry two complex types
together, you may have to be the counselor as well. ;)
:) I assume by "counselor" you mean the code for __new__ and __init__
below, which, when I get a chance to understand them, will probably
explain some of your earlier remarks about it maybe being easier to
implement in such a manner. Of course, I don't particularly want to
marry the types, just have XxxEnum work for IntET as well as it does for
int... I was bumping into name conflicts between Nick's implementation
and yours, that weren't immediately obvious to me, because I haven't
done multiple inheritance much — Enum is dragging me into that and
metaclasses, though, which is a good thing for me, likely.
The one piece of "marriage" that is interesting is to avoid specifying
the name twice, and it seems your code
Here's your code, revamped. I did make a slight change in the meta
-- I moved the name assignment above the __init__ call so it's
available in __init__.
That's handy, thanks.
--8<
from ref435 import Enum
from flags import IntET
class NIE1( IntET, Enum ):
x = 1
y = 2
z = 4
def __new__(cls, value):
member = IntET.__new__(cls, 'temp', value)
member._value = value
return member
def __init__(self, value):
self._etname = self._name
print( repr( NIE1.x.value ))
prin
[Python-Dev] PEP 435 doesn't help with bitfields [Was: Re: PEP 435 - ref impl disc 2]
On 5/13/2013 7:36 PM, Ethan Furman wrote: 2) Is something like demo2 interesting to anyone but me? Of course, I think it would be great for reporting flag values using names rather than a number representing combined bit fields. No idea. ;) There's been some talk of Enum-ing constants in the Socket library... I'm no socket programmer, so I'd have to go read the APIs to know if any of them are for bitfields which are typically combined together with | or + being the typical operators... and which would convert them to plain integers, and lose the reporting by name. That's the problem I see with IntEnum used for bitfields. For simple selection of choices, one choice per parameter, Enum will be great. But for bitfields, it is lacking. Sorry if this sounds repetitious, but all the other times I've mentioned it, it has been in a big discussion of other stuff too. Glenn ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 doesn't help with bitfields [Was: Re: PEP 435 - ref impl disc 2]
On 05/13/2013 10:01 PM, Glenn Linderman wrote: Sorry if this sounds repetitious, but all the other times I've mentioned it, it has been in a big discussion of other stuff too. It's a while 'til 3.4. A bitfield-type enum may show up in the docs, if no where else. ;) -- ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 - ref impl disc 2
On 05/13/2013 10:01 PM, Glenn Linderman wrote:
On 5/13/2013 7:36 PM, Ethan Furman wrote:
On 05/10/2013 10:15 PM, Glenn Linderman wrote:
So it is quite possible to marry the two, as Ethan helped me figure out using
an earlier NamedInt class:
class NIE( IntET, Enum ):
x = ('NIE.x', 1)
y = ('NIE.y', 2)
z = ('NIE.z', 4)
and then expressions involving members of NIE (and even associated integers)
will be tracked... see demo1.py.
But the last few lines of demo1 demonstrate that NIE doesn't like, somehow,
remember that its values, deep down under
the covers, are really int. And doesn't even like them when they are wrapped
into IntET objects. This may or may not
be a bug in the current Enum implementation.
[demo1.py excerpt]
print( repr( NIE1( 1 ) + NIE1(2)))
print( repr( NIE1( IntET('NIE1.x', 1 )) + NIE1(2)))
So the questions are:
1) Is there a bug in ref435 Enum that makes demo1 report errors instead of
those lines working?
Nope.
Well, if it isn't a bug, it will be interesting to read the documentation that
explains the behavior, when the
documentation is written:
The "obvious" documentation would be that Enum names values of any type,
particularly the first type in the
multiple-inheritance list. The values assigned to the enumeration members are
used as parameters to the constructor of
that first type, but the value of the enumeration member itself is an item of
the type, created by the constructor.
The __call__ syntax [ EnumDerivation( value ) ] looks up enumeration members
by value.
The obvious documentation would stop there.
The one piece of "marriage" that is interesting is to avoid specifying the name
twice, and it seems your code
I'd expect NIE1.x.value to be IntET('x', 1) but I'll have to look more
carefully at what you've done, when I have
some time next week. You may have made some "simplifying assumptions", and
things _should_ be as simple as possible, but
no simpler... especially not if it leads to unexpected results.
This will take more thought than I have time for tonight, also. Right now, I
think I want the value for NIE.x to be
IntET('NIE.x', 1 ). And your code isn't achieving that at present, but maybe I
just need to tweak __new__ and then
can... and maybe it cures the discrepancy in expectations mentioned earlier
too...
Thank you for being persistent. You are correct, the value should be an IntET
(at least, with a custom __new__ ;).
I'll look into it.
--
~Ethan~
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
On Sun, May 12, 2013 at 3:04 PM, Ben Hoyt wrote: > > And if we're creating a custom object instead, why return a 2-tuple > > rather than making the entry's name an attribute of the custom object? > > > > To me, that suggests a more reasonable API for os.scandir() might be > > for it to be an iterator over "dir_entry" objects: > > > > name (as a string) > > is_file() > > is_dir() > > is_link() > > stat() > > cached_stat (None or a stat object) > > Nice! I really like your basic idea of returning a custom object > instead of a 2-tuple. And I agree with Christian that .stat() would be > clearer called .lstat(). I also like your later idea of simply > exposing .dirent (would be None on Windows). > > One tweak I'd suggest is that is_file() etc be called isfile() etc > without the underscore, to match the naming of the os.path.is* > functions. > > > That would actually make sense at an implementation > > level anyway - is_file() etc would check self.cached_lstat first, and > > if that was None they would check self.dirent, and if that was also > > None they would raise an error. > > Hmm, I'm not sure about this at all. Are you suggesting that the > DirEntry object's is* functions would raise an error if both > cached_lstat and dirent were None? Wouldn't it make for a much simpler > API to just call os.lstat() and populate cached_lstat instead? As far > as I'm concerned, that'd be the point of making DirEntry.lstat() a > function. > > In fact, I don't think .cached_lstat should be exposed to the user. > They just call entry.lstat(), and it returns a cached stat or calls > os.lstat() to get the real stat if required (and populates the > internal cached stat value). And the entry.is* functions would call > entry.lstat() if dirent was or d_type was DT_UNKNOWN. This would > change relatively nasty code like this: > > files = [] > dirs = [] > for entry in os.scandir(path): > try: > isdir = entry.isdir() > except NotPresentError: > st = os.lstat(os.path.join(path, entry.name)) > isdir = stat.S_ISDIR(st) > if isdir: > dirs.append(entry.name) > else: > files.append(entry.name) > > Into nice clean code like this: > > files = [] > dirs = [] > for entry in os.scandir(path): > if entry.isfile(): > dirs.append(entry.name) > else: > files.append(entry.name) > > This change would make scandir() usable by ordinary mortals, rather > than just hardcore library implementors. > > In other words, I'm proposing that the DirEntry objects yielded by > scandir() would have .name and .dirent attributes, and .isdir(), > .isfile(), .islink(), .lstat() methods, and look basically like this > (though presumably implemented in C): > > class DirEntry: > def __init__(self, name, dirent, lstat, path='.'): > # User shouldn't need to call this, but called internally by > scandir() > self.name = name > self.dirent = dirent > self._lstat = lstat # non-public attributes > self._path = path > > def lstat(self): > if self._lstat is None: > self._lstat = os.lstat(os.path.join(self._path, self.name)) > return self._lstat > > def isdir(self): > if self.dirent is not None and self.dirent.d_type != DT_UNKNOWN: > return self.dirent.d_type == DT_DIR > else: > return stat.S_ISDIR(self.lstat().st_mode) > > def isfile(self): > if self.dirent is not None and self.dirent.d_type != DT_UNKNOWN: > return self.dirent.d_type == DT_REG > else: > return stat.S_ISREG(self.lstat().st_mode) > > def islink(self): > if self.dirent is not None and self.dirent.d_type != DT_UNKNOWN: > return self.dirent.d_type == DT_LNK > else: > return stat.S_ISLNK(self.lstat().st_mode) > > Oh, and the .dirent would either be None (Windows) or would have > .d_type and .d_ino attributes (Linux, OS X). > > This would make the scandir() API nice and simple to use for callers, > but still expose all the information the OS provides (both the > meaningful fields in dirent, and a full stat on Windows, nicely cached > in the DirEntry object). > > Thoughts? > I like the sound of this (which sounds like what you've implemented now though I haven't looked at your code). -gps > > -Ben > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Best practices for Enum
On Sun, May 12, 2013 at 4:49 PM, Raymond Hettinger < [email protected]> wrote: > > * will enums break doctests or any existing user code > Those are already broken by design. We shouldn't be limited just because someone wrote a bad test that assumed a particular repr of a value. We've already broken that assumption several times in the past from the recent hash randomization change to removing the evil trailing Ls on the old long type, changing the float str vs repr, particular information and wording of exception error messages, etc. This sounds like a feature request for doctest. doctest could be educated about enums and automatically compare to the integer value for such cases. Regardless, it sounds like the consensus agrees with your overall sentiment: refrain from mass converting things "just because" and start with an obvious improvement like the socket module. -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
