Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Sun, Mar 21, 2010 at 10:50 PM, Greg Ewing
wrote:
> Raymond Hettinger wrote:
>
>> Since decimal also allows arbitrary sizes, all long ints can be
>> exactly represented (this was even one of the design goals
>> for the decimal module).
>
> There may be something we need to clarify here. I've been
> imagining that the implicit conversions to Decimal that
> we're talking about would be done to whatever precision
> is set in the context. Am I wrong about that? Is the intention
> to always use enough digits to get an exact representation?
I've been thinking about this, too.
Currently, Decimal + integer -> Decimal converts the integer
losslessly, with no reference to the Decimal context.
But the Fraction type is going to mess this up: for Decimal +
Fraction -> Decimal, I don't see any other sensible option than to
convert the Fraction using the current context, since lossless
conversion isn't generally possible.
So with the above two conventions, we'd potentially end up with
Decimal('1.23') + 314159 giving a different result from
Decimal('1.23') + Fraction(314159, 1) (depending on the context).
It may be that we should reconsider Decimal + int interactions, making
the implicit int->Decimal conversion lossy. Decimal would then match
the way that float behaves: float + int and float + Fraction both do
a lossy conversion of the non-Fraction argument to Fraction. But then
we're changing established behaviour of the Decimal module, which
could cause problems for existing users.
Note that comparisons are a separate issue: those always need to be
done exactly (at least for equality, and once you're doing it for
equality it makes sense to make the other comaprisons exact as well),
else the rule that x == y implies hash(x) == hash(y) would become
untenable. Again, this is the pattern that already exists for
int<->float and Fraction<->float interactions: comparisons are exact,
but arithmetic operations involve a lossy conversion.
Mark
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Mark Dickinson wrote: > But the Fraction type is going to mess this up: for Decimal + > Fraction -> Decimal, I don't see any other sensible option than to > convert the Fraction using the current context, since lossless > conversion isn't generally possible. Be able to duck this question was precisely why I put Decimal to the left of Fraction in my pragmatic tower, btw. There are all sorts of good reasons why that order was mathematically wrong (Real vs Rational, representability of NaN/Inf, etc), but from the point of view of providing clearly defined behaviour for an atypical operation, it should be a lot easier. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Greg Ewing wrote: > Raymond Hettinger wrote: >> >> Remember, the notion of inexactness is a taint, >> not an intrinsic property of a type. Even the Scheme numeric >> tower recognizes this. LIkewise, the decimal specification also >> spells-out this notion as basic to its design. > > I'm not sure it really does, otherwise every decimal > value would have a flag indicating whether it was > tainted with inexactness, and this flag would propagate > through calculations. http://docs.python.org/library/decimal.html#decimal.Inexact (Part of the thread context rather than the individual decimal values, but if you use it properly it tells you whenever an inexact operation has occurred in the current thread) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Raymond Hettinger wrote: > On Mar 21, 2010, at 3:35 PM, Guido van Rossum wrote: >>> It seems to me that Decimals and floats should be considered at >>> the same level (i.e. both implement Real). >> Agreed, but doesn't help. (Except against the idea that Decimal goes >> on the "integer" side of Fraction, which is just wrong.) > > Woohoo! Glad you agree. > I was concerned that idea > was gathering a following. Heck no, it was just a random late night thought from me, and even I thought it was a somewhat dubious idea. I don't mind at all that it since has been knocked soundly (and deservedly) on the head :) > Reasoning for emitting a warning by default: > > 1) Real actual use cases for mixed decimal / float operations are rare. > 2) But accidental mixed decimal / float is an easy mistake to make. > 3) Implicit coercion hides the error. > 4) A warning flag gives you a chance to catch your error. > 5) A warning is educational (it makes sure that you understand > what your program is doing) > 6). A warning is easily silenced either through a) the warnings module, > b) setting a context flag in decimal, or c) by making the coercion > explicit > using Decimal.from_float(). I'll add another one to that list: (7) For backwards compatible changes, it is easy to go from exception -> warning -> no warning (if we later decide to take that second step). Going from exception -> no warning -> warning (if we were to change our minds the other way) is a lot less user-friendly. (I was going to try to play devil's advocate and argue in favour of float results and/or no warning, but I got nuthin' - Raymond's points made too much sense to me). A warning is nice in that you can mix decimals and floats at the interpreter prompt with a single warning per session, but the warning can still act as a pointer into the weird and wonderful world of binary vs decimal floating point. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 3147, __pycache__ directories and umask
I have a pretty good start on PEP 3147 implementation [1], but I've
encountered a situation that I'd like to get some feedback on. Here's the
test case illustrating the problem. From test_import.py:
def test_writable_directory(self):
# The umask is not conducive to creating a writable __pycache__
# directory.
with umask(0o222):
__import__(TESTFN)
self.assertTrue(os.path.exists('__pycache__'))
self.assertTrue(os.path.exists(os.path.join(
'__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag
The __pycache__ directory does not exist before the import, and the import
machinery creates the directory, but the umask leaves the directory unwritable
by anybody. So of course when the import machinery goes to write the .pyc
file inside __pycache__, it fails. This does not cause an ImportError though,
just like if today the package directory were unwritable.
This might be different than today's situation though because once the
unwritable __pycache__ directory is created, nothing is going to change that
without explicit user interaction, and that might be difficult after the
fact.
I'm not sure what the right answer is. Some possible choices:
* Tough luck
* Force the umask so that the directory is writable, but then the question is,
by whom? ugo+w or something less?
* Copy the permissions from the parent directory and ignore umask
* Raise an exception or refuse to create __pycache__ if it's not writable
(again, by whom?)
Or maybe you have a better idea? What's the equivalent situation on Windows
and how would things work there?
-Barry
[1] https://edge.launchpad.net/~barry/python/pep3147
P.S. I'm down to only 8 unit test failures.
signature.asc
Description: PGP signature
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask
Barry Warsaw python.org> writes: > > I'm not sure what the right answer is. Some possible choices: > > * Tough luck > * Force the umask so that the directory is writable, but then the question is, > by whom? ugo+w or something less? > * Copy the permissions from the parent directory and ignore umask > * Raise an exception or refuse to create __pycache__ if it's not writable > (again, by whom?) Welcome to a problem PHP has known for years. The problem with solution 3 is that a non-root user can't change the owner of a directory, therefore copying the permissions doesn't ensure the directory will be writable by the right users. Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode files in a root-owned application. Less laxist chmods have the same problem as solution 3, and can still be insecure. I think solution 4 would be the best compromise. In the face of ambiguity, refuse to guess. But, rather than : refuse to create __pycache__ if it's not writable, the test should be : refuse to create __pycache__ if I can't create it with the same ownership and permissions as the parent directory (and raise an ImportWarning). In light of this issue, I'm -0.5 on __pycache__ becoming the default caching mechanism. The directory ownership/permissions issue is too much of a mess, especially for Web applications (think __pycache__ files created by the Apache user). __pycache__ should only be created if explicitly activated (for example by distutils when installing stuff). Otherwise, if not present, the "legacy" mechanism (writing an untagged pyc file along the py file) should be used. Actually, __pycache__ creation doesn't have to be part of the import mechanism. It can be part of distutils instead (or whatever third-party tool such as distribute, or distro-specific packaging scripts). This would relax complexity of core Python a bit. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 16, 2010, at 07:44 PM, Nick Coghlan wrote: >In Python 3.1, *invoking* py_compile.compile() will create 2.x style >bytecode. Similarly, when force==False, compileall.compile_dir() and >compileall.compile_path() will check for 2.x style bytecode in order to >decide whether or not to compile the module. > >The question for 3.2 is what bytecode layout py_compile.compile() should >generate. For the precompile-a-system-library use case it should clearly >generate a PEP 3147 layout and this probably makes sense as the default >behaviour in 3.2. > >However, for production of bytecode-only packages, it would be >convenient to be able to explicitly invoke the 2.x style behaviour >without having to specify the target filename explicitly using the >'cfile' parameter (which isn't exposed at the compileall layer anyway). My working branch modifies py_compile to produce PEP 3147 layout by default. I agree that it should support traditional pyc output as an option, and compileall should support this as well. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
> Or maybe you have a better idea? What's the equivalent situation on Windows > and how would things work there? On Windows, this problem is easy: create the directory with no specification of an ACL, and it will (usually) inherit the ACL of the parent directory. IOW, the same users will have write access to __pycache__ as to the parent directory. It is possible to defeat this machinery, by setting ACL entries on the parent that explicitly don't get inherited, or that only apply to subdirectories. In this case, I would say "tough luck": this is what the users requested, so it may have a reason. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
> refuse to create __pycache__ if I can't create it with the same ownership > and permissions as the parent directory (and raise an ImportWarning). I don't think an ImportWarning should be raised: AFAICT, we are not raising one, either, when the pyc file cannot be created (and it may well be the case that you don't have write permission at all, which would trigger lots of ImportWarnings). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mar 22, 2010, at 2:23 AM, Mark Dickinson wrote: > Note that comparisons are a separate issue: those always need to be > done exactly (at least for equality, and once you're doing it for > equality it makes sense to make the other comaprisons exact as well), > else the rule that x == y implies hash(x) == hash(y) would become > untenable. Again, this is the pattern that already exists for > int<->float and Fraction<->float interactions: comparisons are exact, > but arithmetic operations involve a lossy conversion. My instinct says that we're asking for trouble if comparisons have different coercion rules than arithmetic operations. That would suggest that we follow the chain of lossless conversions: Fraction + float --> Fraction Fraction + decimal --> Fraction Decimal + float --> Decimal That way, arithmetic coercions match comparison coercions. We preserve get Mark's fast, clean new universal hash function. The equality/hash homomorphism is preserved. And a decimal context is not needed for any of the coercions (it is needed for the addition between two decimals once coercion has happened, but that is completely normal for Decimal so there are no surprises). Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 8:39 AM, Raymond Hettinger wrote: > My instinct says that we're asking for trouble if comparisons have > different coercion rules than arithmetic operations. Sorry, no, this is a station we passed long ago when float-long comparison was fixed to do the comparison exactly (but other mixed ops continue to convert the long to a float). Otherwise you'll get a violation of "if x < y and y < z then it follows that x < z". (FWIW, my instinct used to be the same as yours, but I got better. :-) > That would suggest that we follow the chain of lossless conversions: > > Fraction + float --> Fraction Besides the obvious backwards compatibility problem, I also just really think that this would be a mistake. Fraction corresponds to Rational in PEP 3141's numeric tower and float corresponds to Real (as does Decimal). The convention is to only move to the more general type in the numeric tower. Now we all know that floats (being a fixed number of bits) are actually representable as rationals as well, but *that's not how we think about them!* > Fraction + decimal --> Fraction This also violates the natural direction in the numeric tower. > Decimal + float --> Decimal If everybody associated with the Decimal implementation wants this I won't stop you; as I repeatedly said my intuition about this one (as opposed to the other two above) is very weak. > That way, arithmetic coercions match comparison coercions. Which is not a requirement. > We preserve get Mark's fast, clean new universal hash function. That hash function (or another that matches the requirements) is a given. > The equality/hash homomorphism is preserved. That is also a given (except I don't know if homomorphism is the right word to use). > And a decimal context is not needed for any of the coercions > (it is needed for the addition between two decimals once coercion > has happened, but that is completely normal for Decimal so there > are no surprises). Doesn't strike me as a deciding argument. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __pycache__ creation
Oh, and by the way, there can be a race condition between __pycache__ creation and deletion (if it fails the test), where an attacker can stuff a hostile pyc file in the directory in the meantime (and the deletion then fails because the directory isn't empty). IMO, all these issues militate for putting __pycache__ creation out of the interpreter core, and in the hands of third-party package-time/ install-time tools (or distutils). Le Mon, 22 Mar 2010 14:30:12 +, Antoine Pitrou a écrit : > > __pycache__ should only be created if explicitly activated (for example > by distutils when installing stuff). Otherwise, if not present, the > "legacy" mechanism (writing an untagged pyc file along the py file) > should be used. > > Actually, __pycache__ creation doesn't have to be part of the import > mechanism. It can be part of distutils instead (or whatever third-party > tool such as distribute, or distro-specific packaging scripts). This > would relax complexity of core Python a bit. > > Regards > > Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > >> Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. That's my vote. I believe Nick chimed-in in agreement. Mark, do you concur? Raymond___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On Mon, 22 Mar 2010, Antoine Pitrou wrote:
Oh, and by the way, there can be a race condition between __pycache__
creation and deletion (if it fails the test), where an attacker can stuff
a hostile pyc file in the directory in the meantime (and the deletion
then fails because the directory isn't empty).
Would creating it under a different name and then renaming help with this?
IMO, all these issues militate for putting __pycache__ creation out of
the interpreter core, and in the hands of third-party package-time/
install-time tools (or distutils).
Speaking only for myself, but really for anybody who likes tidy source
directories, I hope some version of the __pycache__ proposal becomes part
of standard Python, by which I ideally mean it's enabled by default but if
that is just not a good idea then at most it should be required to set a
command-line option to get this feature.
If I just want to write some .py code and run it, I don't see why my
directories need to clutter up with .pyc files. I've previously suggested
a Python version of javac's -d ("destination directory") option, but
putting all the .pyc's in a __pycache__ directory per source directory is
good enough to make me happy (and is Pythonically simple, in my opinion).
Isaac Morland CSCF Web Guru
DC 2554C, x36650WWW Software Specialist
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Attribute lookup ambiguity
Michael Foord a écrit : On 20/03/2010 12:00, Pascal Chambon wrote: But the point which for me is still unclear, is : does the default implementation of __getattribute__ (the one of "object") call __getattr__ by himself, or does it rely on its caller for that, by raising an AttributeError ? For Python2, it's blatantly the latter case which is favoured, but since it looks like an implementation detail at the moment, I propose we settle it (and document it) once for all. Ah right, my apologies. So it is still documented behaviour - __getattr__ is obviously called by the Python runtime and not by __getattribute__. (It isn't just by getattr as the same behaviour is shown when doing a normal attribute lookup and not via the getattr function.) I really don't see the docs you're referring to ; until I tested myself, I think I had no obvious reasons to guess that __getattribute__ relied on the upper level caller instead of finishing the hard job himself. Nick Coghlan a écrit : Michael Foord wrote: Well, the documentation you pointed to specifies that __getattr__ will be called if __getattribute__ raises an AttributeError, it just doesn't specify that it is done by object.__getattribute__ (which it isn't). And as for why not: because __getattribute__ implementations need to be able to call object.__getattribute__ without triggering the fallback behaviour. Cheers, Nick. I guess there are cases in which it is beneficial indeed. Michael Foord wrote: Well, the documentation you pointed to specifies that __getattr__ will be called if __getattribute__ raises an AttributeError, it just doesn't specify that it is done by object.__getattribute__ (which it isn't). If __getattribute__ raises an exception, it won't get a chance to do anything else, so something outside of __getattribute__ must catch the AttributeError and calling __getattr__. So I think the docs *are* specifying the behaviour here, if only by implication. I don't follow you there - in my mind, the default __getattribute__ could simply have wrapped all its operations inside soem kind of "try..catch AttributeError:" mechanism, and thus been able to fallback to __getattr__ in any way. If I sum it up properly the semantic is : -A.obj and getattr(A, "obj") are exactly the same -They trigger the calling of __getattribute__ on the object (or it's python core equivalent) -By default, this __getattribute__ browse the whole object hierarchy according to well known rules (__dict__, type, type's ancestors..), handling descriptor protocols and the like. But it doesn't fallback to __getattr__ - it raises an AttributeError instead. -getattr() falls back to __getattr__ if __getattribute__ fails -customized __getattribute__ methods have the choice between calling __getattr__ by themselves, or delegating it to getattr() by raising an exception. Wouldn't it be worth completing the doc with these point ? They really didn't seem obvious to me basically (even though, after analysis, some behaviours make more sense than others). I might submit a patch. regards, Pascal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Isaac Morland uwaterloo.ca> writes: > > > IMO, all these issues militate for putting __pycache__ creation out of > > the interpreter core, and in the hands of third-party package-time/ > > install-time tools (or distutils). > > Speaking only for myself, but really for anybody who likes tidy source > directories, I hope some version of the __pycache__ proposal becomes part > of standard Python, by which I ideally mean it's enabled by default but if > that is just not a good idea then at most it should be required to set a > command-line option to get this feature. This doesn't contradict by my proposal. What I am proposing is that the creation of __pycache__ /directories/ be put outside of the core. It can be part of distutils, or of a separate module, or delegated to third-party tools. It could even be as simple as "python -m compileall --pycache", if someone implements it. Creation of the __pycache__ /contents/ (files inside the directory) would still be part of core Python, but only if the directory exists and is writable by the current process. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > > Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. > > That's my vote. I've been lurking on this thread so far, but let me add my +1 to this option. My reasoning is that Decimal is a "better" model of Real than float and mixed operations should not degrade the result. "Better" can mean different things to different people, but to me the tie breaker is the support for contexts. I would not want precision to suddenly change in the middle of calculation I add 1.0 instead of 1. This behavior will also be familiar to users of other "enhanced" numeric types such as NumPy scalars. Note that in the older Numeric, it was the other way around, but after considerable discussion, the behavior was changed. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 10:22 AM, Alexander Belopolsky wrote: > On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger > wrote: >> >> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: >> >> Decimal + float --> Decimal >> >> If everybody associated with the Decimal implementation wants this I >> won't stop you; as I repeatedly said my intuition about this one (as >> opposed to the other two above) is very weak. >> >> That's my vote. > > I've been lurking on this thread so far, but let me add my +1 to this > option. My reasoning is that Decimal is a "better" model of Real than > float and mixed operations should not degrade the result. "Better" > can mean different things to different people, but to me the tie > breaker is the support for contexts. I would not want precision to > suddenly change in the middle of calculation I add 1.0 instead of 1. > > This behavior will also be familiar to users of other "enhanced" > numeric types such as NumPy scalars. Note that in the older Numeric, > it was the other way around, but after considerable discussion, the > behavior was changed. Thanks, "better" is a great way to express this. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: > > Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. > > That's my vote. > I believe Nick chimed-in in agreement. > Mark, do you concur? Yes; Decimal + float -> Decimal works for me; the greater flexibility of the Decimal type is the main plus for me. I don't think my own intuition is particularly strong here, either. It would be interesting to hear from major users of the decimal module (I have to confess to not actually using it myself very much). I argued earlier for Decimal + float -> Decimal on the basis that the float->Decimal conversion can be done exactly; but having thought about Decimal + Fraction it's no longer clear to me that this makes sense. Having Decimal + float -> Decimal round the float using the current Decimal context still seems like a reasonable option. Just for the record, I'd also prefer Decimal + Fraction -> Decimal. I don't want to let the abstractions of the numeric tower get in the way of the practicalities: we should modify the abstractions if necessary! In particular, it's not clear to me that all numeric types have to be comparable with each other. It might make sense for Decimal + complex mixed-type operations to be disallowed, for example. Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger > wrote: >> >> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote: >> >> Decimal + float --> Decimal >> >> If everybody associated with the Decimal implementation wants this I >> won't stop you; as I repeatedly said my intuition about this one (as >> opposed to the other two above) is very weak. >> >> That's my vote. >> I believe Nick chimed-in in agreement. >> Mark, do you concur? > > Yes; Decimal + float -> Decimal works for me; the greater flexibility > of the Decimal type is the main plus for me. > On Mar 22, 2010, at 11:22 AM, Alexander Belopolsky wrote: > >> I've been lurking on this thread so far, but let me add my +1 to this >> option. On Mar 22, 2010, at 11:24 AM, Guido van Rossum wrote: >> > > Thanks, "better" is a great way to express this. OMG ;-) It looks like there is a consensus on: Decimal + float --> Decimal We have progress. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 12:26, Mark Dickinson wrote: > I don't want to let the abstractions of the numeric tower get in the > way of the practicalities: we should modify the abstractions if > necessary! In particular, it's not clear to me that all numeric types > have to be comparable with each other. It might make sense for > Decimal + complex mixed-type operations to be disallowed, for example. Only until a needy user breaks out the duck tape and builds a ComplexDecimal type. ;) The nature of the beast is more will be added on later. So long as Decimal == complex works right I don't see a problem with Decimal + complex raising an exception. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 10:26 AM, Mark Dickinson wrote:
> On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger
> wrote:
>>
>> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>>
>> Decimal + float --> Decimal
>>
>> If everybody associated with the Decimal implementation wants this I
>> won't stop you; as I repeatedly said my intuition about this one (as
>> opposed to the other two above) is very weak.
>>
>> That's my vote.
>> I believe Nick chimed-in in agreement.
>> Mark, do you concur?
>
> Yes; Decimal + float -> Decimal works for me; the greater flexibility
> of the Decimal type is the main plus for me. I don't think my own
> intuition is particularly strong here, either. It would be
> interesting to hear from major users of the decimal module (I have to
> confess to not actually using it myself very much).
You're unlikely to hear from them here... :-( Though maybe Raymond qualifies.
> I argued earlier for Decimal + float -> Decimal on the basis that the
> float->Decimal conversion can be done exactly; but having thought
> about Decimal + Fraction it's no longer clear to me that this makes
> sense. Having Decimal + float -> Decimal round the float using the
> current Decimal context still seems like a reasonable option.
So now we have a second-order decision to make -- whether
Decimal+float should convert the float to Decimal using the current
context's precision, or do it exactly. I think we should just follow
Decimal.from_float() here, which AFAIK does the conversion exactly --
the operation will already round properly if needed, and I don't think
it is a good idea to round twice in the same operation. (Even though
this is what long+float does -- but the situation is different there
since float has no variable precision.)
> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.
I think that is pretty much uncontested. Though there are second-order
details to decide here too: Fraction + float converts the Fraction to
a float first and then just adds the floats; but but Fraction /
Decimal cannot always be computed exactly as a Decimal so here I'd
suggest rounding first because we have to in some cases (it's better
to be consistent, except for comparisons, which should always be done
exactly if at all possible).
> I don't want to let the abstractions of the numeric tower get in the
> way of the practicalities: we should modify the abstractions if
> necessary! In particular, it's not clear to me that all numeric types
> have to be comparable with each other. It might make sense for
> Decimal + complex mixed-type operations to be disallowed, for example.
One of the reasons for having a numeric tower is actually that unless
you do at least comparisons right, you get into inconsistencies like a
== b, b ==c, a != c (and similar for orderings). Of course complex
doesn't play along with the orderings, but it does play along with ==.
Real and Complex are distinct stations in the numeric tower, and the
tower defines mixed Real-Complex operations as returning Complex
results, but it allows raising exceptions as well. Since Decimal(-1)
** Decimal('0.5') already raises an exception instead of returning a
complex number, I'm fine with following that example -- complex
numbers are definitely a specialty area.
PS. PEP 3141 defines some miscellaneous operations that any number in
the numeric tower ought to implement, e.g. x.imag, x.real and
x.conjugate().
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > > Just for the record, I'd also prefer Decimal + Fraction -> Decimal. Guido was persuasive on why float + Fraction --> float, so this makes sense for the same reasons. For the implementation, is there a way to avoid the double rounding in myfloat + myfrac.numerator / myfrac.denominator? Perhaps translate it to: f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact addition return f.numerator / f.denominator # Only one decimal context rounding applied. Elsewhere in the decimal module, there is a fundamental notion that numbers are exact and only the results of operations are rounded. For example, it is possible in decimal to add together two high precision numbers but do so in a low precision context. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Antoine Pitrou wrote: Isaac Morland uwaterloo.ca> writes: IMO, all these issues militate for putting __pycache__ creation out of the interpreter core, and in the hands of third-party package-time/ install-time tools (or distutils). Speaking only for myself, but really for anybody who likes tidy source directories, I hope some version of the __pycache__ proposal becomes part of standard Python, by which I ideally mean it's enabled by default but if that is just not a good idea then at most it should be required to set a command-line option to get this feature. This doesn't contradict by my proposal. What I am proposing is that the creation of __pycache__ /directories/ be put outside of the core. It can be part of distutils, or of a separate module, or delegated to third-party tools. It could even be as simple as "python -m compileall --pycache", if someone implements it. Creation of the __pycache__ /contents/ (files inside the directory) would still be part of core Python, but only if the directory exists and is writable by the current process. +1 If I understand correctly, we would have the current mode as the default, and can trigger __pycache__ behavior simply by manually creating a __pycache__ directory and deleting any byte-code files in the module/program directory. I like this, it is easy to understand and can be used without messing with flags or environment variables. Ron ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 11:00 AM, Raymond Hettinger wrote: > > On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: >> >> Just for the record, I'd also prefer Decimal + Fraction -> Decimal. > > > Guido was persuasive on why float + Fraction --> float, > so this makes sense for the same reasons. > > For the implementation, is there a way to avoid the double rounding > in myfloat + myfrac.numerator / myfrac.denominator? > > Perhaps translate it to: > > f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact > addition > return f.numerator / f.denominator # Only one decimal context > rounding applied. > > Elsewhere in the decimal module, there is a fundamental notion > that numbers are exact and only the results of operations are > rounded. For example, it is possible in decimal to add together > two high precision numbers but do so in a low precision context. Works for me. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mar 22, 2010, at 11:54 AM, Guido van Rossum wrote: > > So now we have a second-order decision to make -- whether > Decimal+float should convert the float to Decimal using the current > context's precision, or do it exactly. I think we should just follow > Decimal.from_float() here, which AFAIK does the conversion exactly -- > the operation will already round properly if needed, and I don't think > it is a good idea to round twice in the same operation. (Even though > this is what long+float does -- but the situation is different there > since float has no variable precision.) I concur. That is consistent with the basic design of the decimal module which treats inputs as exact and only applies rounding to the results of operations. FWIW, Mark and I have both been bitten severely by doubling rounding in the world of binary floats. Avoiding double rounding is a darned good idea. Raymond___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Guido van Rossum wrote: > > Decimal + float --> Decimal > > If everybody associated with the Decimal implementation wants this I > won't stop you; as I repeatedly said my intuition about this one (as > opposed to the other two above) is very weak. I've been following the discussion only passively so far, but I also think this is the most logical solution. Stefan Krah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 7:00 PM, Raymond Hettinger
wrote:
>
> On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:
>>
>> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.
>
>
> Guido was persuasive on why float + Fraction --> float,
> so this makes sense for the same reasons.
>
> For the implementation, is there a way to avoid the double rounding
> in myfloat + myfrac.numerator / myfrac.denominator?
>
> Perhaps translate it to:
>
> f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact
> addition
> return f.numerator / f.denominator # Only one decimal context
> rounding applied.
I'm not sure; I see a couple of problems with this. (1) It's fine
for the basic arithmetic operations that Fraction already supports,
but what about all the other Decimal methods that don't have Fraction
counterparts. (2) It bothers me that the Decimal -> Fraction
conversion can be inefficient in cases like Decimal('1e');
currently, all Decimal operations are relatively efficient (no
exponential-time behaviour) provided only that the coefficients don't
get too large; large exponents aren't a problem.
I think getting this to work would involve a lot of extra code and
significant 'cleverness'. I'd prefer the simple-to-implement and
simple-to-explain option of rounding the Fraction before performing
the operation, even if this means that the whole operation involves
two rounding operations. It's not so different from what currently
happens for Fraction+float, or even int+float.
Mark
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
One other thought. The Decimal constructor should now accept floats as a possible input type. Formerly, we separated that out to Decimal.from_float() because decimals weren't interoperable with floats. This will put decimal and float back on equal footing so that we have both: float(some_decimal)# coerce to binary float and Decimal(some_float) # coerce to decimal float That will also save us from odd idioms like: d = some_float + Decimal(0) # coerce to decimal then apply context rounding This also matches the behavior of other constructors: int(some_float) or int(some_decimal) or str(some_float) or str(some_decimal) Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Raymond Hettinger wrote: > > On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote: > > > > Just for the record, I'd also prefer Decimal + Fraction -> Decimal. > > > Guido was persuasive on why float + Fraction --> float, > so this makes sense for the same reasons. > > For the implementation, is there a way to avoid the double rounding > in myfloat + myfrac.numerator / myfrac.denominator? > > Perhaps translate it to: > > f = Fractions.from_decimal(myfloat) + myfract # Lossless, exact > addition > return f.numerator / f.denominator # Only one decimal context > rounding applied. > > Elsewhere in the decimal module, there is a fundamental notion > that numbers are exact and only the results of operations are > rounded. For example, it is possible in decimal to add together > two high precision numbers but do so in a low precision context. I don't think this will be practical for huge decimal exponents. Also, at first glance I wonder how to integrate this cleanly into the control flow. convert_other() will not work, so there would need to be a special case for fractions in each function. Or do you mean to outsource the whole computation to the fractions module, which calls decimal only for the final division? Stefan Krah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 11:36 AM, Raymond Hettinger wrote: > One other thought. > > The Decimal constructor should now accept floats as a possible input type. > Formerly, we separated that out to Decimal.from_float() because > decimals weren't interoperable with floats. Not sure this follows; Fraction(1.1) raises an exception, you have to use Fraction.from_float(). > This will put decimal and float back on equal footing so that we have both: > float(some_decimal) # coerce to binary float > and > Decimal(some_float) # coerce to decimal float If you really want this I won't stop you. > That will also save us from odd idioms like: > > d = some_float + Decimal(0) # coerce to decimal then apply context > rounding Although this isn't too bad: d = Decimal.from_float(some_float) > This also matches the behavior of other constructors: > int(some_float) > or > int(some_decimal) > or > str(some_float) > or > str(some_decimal) True, I don't care strongly. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Sorry to intervene out of the blue, but I find the suggested rule for fractional to decimal conversion not as clean as I'd expect. If fractions are converted to decimals when doing arithmetics, would it be worthwhile to at least provide a minimum of fractional conversion integrity? What I have in mind is the following rule: When doing conversion from fraction to decimal, always generate a whole number of repeating digits, always at least twice. Examples, with a precision of 5 in Decimal: 1/2 -> 0.5 1/3 -> 0.3 1/11 -> 0.090909 # Note that we produced 6 digits, because # the repeating pattern contains 2 digits. 1/7 -> 0.142857142857 # Always at least two full patterns. The benefits I see are that: 1. If a number can be represented exactly it will be converted exactly. 2. The minimum precision requested is respected. 3. The conversion yields something that will convert back more precisely. Not perfectly, but see the next point. 4. Since the repeating pattern is present at least twice at the end, one can augment the precision of the conversion by detecting the repetition and adding more. This detection is trivial. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 8:02 PM, Pierre B. wrote: > Sorry to intervene out of the blue, but I find the suggested rule for > fractional to decimal conversion not as clean as I'd expect. > > If fractions are converted to decimals when doing arithmetics, would it be > worthwhile to at least provide a minimum of fractional conversion integrity? > What I have in mind is the following rule: > > When doing conversion from fraction to decimal, always generate a whole number > of repeating digits, always at least twice. > > Examples, with a precision of 5 in Decimal: > > 1/2 -> 0.5 > > 1/3 -> 0.3 > > 1/11 -> 0.090909 > # Note that we produced 6 digits, because > # the repeating pattern contains 2 digits. > > 1/7 -> 0.142857142857 > # Always at least two full patterns. And for 1/123123127? The decimal expansion of this fraction has a period of over 15 million! Sorry, but this doesn't seem like a feasible or desirable strategy. Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Pierre B. hotmail.com> writes: > > 4. Since the repeating pattern is present at least twice at the end, > one can augment the precision of the conversion by detecting the > repetition and adding more. This detection is trivial. Wrong of me. This point is acutally invalid, since it is impossible to generally differentiate where the pattern ends. The counter example is: 0.177177177177... vs 0.177.1777... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
On Mar 22, 2010, at 02:30 PM, Antoine Pitrou wrote: >Barry Warsaw python.org> writes: >> >> * Tough luck >> * Force the umask so that the directory is writable, but then the question >> is, >> by whom? ugo+w or something less? >> * Copy the permissions from the parent directory and ignore umask >> * Raise an exception or refuse to create __pycache__ if it's not writable >> (again, by whom?) > >Welcome to a problem PHP has known for years. Yay. >The problem with solution 3 is that a non-root user can't change the owner of >a directory, therefore copying the permissions doesn't ensure the directory >will be writable by the right users. > >Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode >files in a root-owned application. Less laxist chmods have the same problem as >solution 3, and can still be insecure. > >I think solution 4 would be the best compromise. In the face of ambiguity, >refuse to guess. But, rather than : >refuse to create __pycache__ if it's not writable, >the test should be : >refuse to create __pycache__ if I can't create it with the same ownership >and permissions as the parent directory (and raise an ImportWarning). I actually think "tough luck" might be the right answer and that it's not really all that serious. Certainly not serious enough not to enable this by default. When Python is being installed, either by a from-source 'make install' or by the distro packager, then you'd expect the umask not to be insane. In the latter case, it's a bug and in the former case you screwed up so you should be able to delete and reinstall, or at the very least execute the same `find` command that Python's own Makefile calls (in my branch) for 'make clean'. When you're installing packages, again, I would expect that the system installer, or you via `easy_install` or whatever, would not have an insane umask. As with above, if you *did* have a bad umask, you could certainly also have a bad umask whenever you ran the third party tool as when you ran Python. You could also have a bad umask when you're doing local development, but that should be easy enough to fix, and besides, I think you'd have other problems if that were the case. One possibility would be to instrument compileall to remove __pycache__ directories with a flag. >In light of this issue, I'm -0.5 on __pycache__ becoming the default caching >mechanism. The directory ownership/permissions issue is too much of a mess, >especially for Web applications (think __pycache__ files created by the Apache >user). > >__pycache__ should only be created if explicitly activated (for example by >distutils when installing stuff). Otherwise, if not present, the "legacy" >mechanism (writing an untagged pyc file along the py file) should be used. I don't particularly like this much, because most people won't get the benefit of it in a local dev environment. While not the primary use case, it's a useful enough side benefit, and unless Guido changes his mind, we want to enable this by default. >Actually, __pycache__ creation doesn't have to be part of the import >mechanism. It can be part of distutils instead (or whatever third-party tool >such as distribute, or distro-specific packaging scripts). This would relax >complexity of core Python a bit. It's not really that complicated. :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote: >What I am proposing is that the creation of __pycache__ /directories/ be put >outside of the core. It can be part of distutils, or of a separate module, or >delegated to third-party tools. It could even be as simple as >"python -m compileall --pycache", if someone implements it. Except that most people won't do this, they'll just run Python. I wouldn't count this as "enabled by default". -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: >If I understand correctly, we would have the current mode as the default, and >can trigger __pycache__ behavior simply by manually creating a __pycache__ >directory and deleting any byte-code files in the module/program directory. > >I like this, it is easy to understand and can be used without messing with >flags or environment variables. Well, for a package with subpackages, it gets more complicated. Definitely not something you're likely to do manually. Antoine's suggestion of 'python -m compileall --pycache' would work, but I think it's also obscure enough that most Python users won't get the benefit. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
While we're on the topic, I think you should consider allowing the Fraction()
constructor to accept a decimal input.
This corresponds to common schoolbook problems and simple client requests:
"Express 3.5 as a fraction".
>>> Fraction(Decimal('3.5'))
Fraction(7, 2)
Unlike typical binary floats which use full precision, it is not uncommon
to have decimal floats with only a few digits of precision where the
expression as a fraction is both useful and unsurprising.
Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask
Barry Warsaw python.org> writes: > > When Python is being installed, either by a from-source 'make install' or by > the distro packager, then you'd expect the umask not to be insane. In the > latter case, it's a bug and in the former case you screwed up so you should be > able to delete and reinstall, or at the very least execute the same `find` > command that Python's own Makefile calls (in my branch) for 'make clean'. > > When you're installing packages, again, I would expect that the system > installer, or you via `easy_install` or whatever, would not have an insane > umask. Well, precisely. That's why I suggest that creating the __pycache__ directories be done *at install time* (or packaging time), and not via the core import machinery (that is, not at import time). That is, when you *know* you are the right user, with the right umask. Which also means that it can and probably should be pulled out of the core (and written in pure Python, which is also much more practical to maintain, test and debug). > I don't particularly like this much, because most people won't get the benefit > of it in a local dev environment. AFAICT it has no benefit in a local dev environment (apart from supposedly "cleaner" directories, which is not something anyone seems to consider a problem with the current CPythons). And again, this could be part of the distutils standard "setup.py install" (or "setup.py develop" with distutils2/distribute) if people care :) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw wrote: > On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: > >>If I understand correctly, we would have the current mode as the default, and >>can trigger __pycache__ behavior simply by manually creating a __pycache__ >>directory and deleting any byte-code files in the module/program directory. Huh? Last time I looked weren't we going to make __pycache__ the default (and eventually only) behavior? >>I like this, it is easy to understand and can be used without messing with >>flags or environment variables. > > Well, for a package with subpackages, it gets more complicated. Definitely > not something you're likely to do manually. Antoine's suggestion of 'python > -m compileall --pycache' would work, but I think it's also obscure enough that > most Python users won't get the benefit. I see only two reasonable solutions for __pycache__ creation -- either we change all setup/install scripts (both for core Python and for 3rd party packages) to always create a __pycache__ subdirectory for every directory (including package directories) installed; or we somehow create it the first time it's needed. But creating it as needed runs into at least similar problems with ownership as creating .pyc files when first needed (if the parent directory is root-owned a mere mortal can't create it at all). So even apart from the security issue (which I haven't thought about deeply) I think precreation should at least be an easily accessible option both for the core (where it can be done by compileall) and for 3rd party packages (where I guess it's up to distutils or whatever install mechanism is used). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On 22 March 2010 19:32, Mark Dickinson wrote: > I think getting this to work would involve a lot of extra code and > significant 'cleverness'. I'd prefer the simple-to-implement and > simple-to-explain option of rounding the Fraction before performing > the operation, even if this means that the whole operation involves > two rounding operations. It's not so different from what currently > happens for Fraction+float, or even int+float. My instinct throughout this discussion has been that a_decimal + a_fraction is in effect a_decimal + a_fraction.numerator / a_fraction.denominator and as the numerator and denomination can be converted into exact decimals, my instincts "expect" a double rounding. Anything else effectively means implementing partial cases of (a+b/c) treated as a ternary operation. So I guess I'm +1 on Mark's suggestion. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger
wrote:
> While we're on the topic, I think you should consider allowing the Fraction()
> constructor to accept a decimal input.
>
> This corresponds to common schoolbook problems and simple client requests:
> "Express 3.5 as a fraction".
>
> >>> Fraction(Decimal('3.5'))
> Fraction(7, 2)
>
> Unlike typical binary floats which use full precision, it is not uncommon
> to have decimal floats with only a few digits of precision where the
> expression as a fraction is both useful and unsurprising.
There is already a Fraction.from_decimal() class method. I understand
the reasons for not allowing floats in the unadorned Fraction
constructor don't quite hold for Decimal, but at the same time I'm not
sure how compelling the use case is for such downcasts given that it
can be done easily with an explicit call. (Same actually for requiring
Decimal.from_float().)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger
wrote:
> While we're on the topic, I think you should consider allowing the Fraction()
> constructor to accept a decimal input.
>
> This corresponds to common schoolbook problems and simple client requests:
> "Express 3.5 as a fraction".
>
> >>> Fraction(Decimal('3.5'))
> Fraction(7, 2)
>
> Unlike typical binary floats which use full precision, it is not uncommon
> to have decimal floats with only a few digits of precision where the
> expression as a fraction is both useful and unsurprising.
Sounds fine to me. Fraction already accepts decimal floating-point
strings, so the implementation of this would be trivial (convert to
string, then call the Fraction constructor).
Mark
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Barry Warsaw wrote: > On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote: > >> What I am proposing is that the creation of __pycache__ /directories/ be put >> outside of the core. It can be part of distutils, or of a separate module, or >> delegated to third-party tools. It could even be as simple as >> "python -m compileall --pycache", if someone implements it. > > Except that most people won't do this, they'll just run Python. I wouldn't > count this as "enabled by default". Therefore, I'm in favor of having it on by default. If certain use cases make it problematic (e.g. Apache creating directories which you then cannot delete), there should be a way to turn it *off*. Perhaps the existing machinery to turn of byte code generation at all might be sufficient. As for the original question (funny umasks), I think my proposal is "tough luck". Don't try to be super-smart; as Antoine explains, it gets worse, not better. If the user has arranged that Python will create unusable directories, the user better changes his setup. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 12:45 PM, Mark Dickinson wrote:
> On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger
> wrote:
>> While we're on the topic, I think you should consider allowing the Fraction()
>> constructor to accept a decimal input.
>>
>> This corresponds to common schoolbook problems and simple client requests:
>> "Express 3.5 as a fraction".
>>
>> >>> Fraction(Decimal('3.5'))
>> Fraction(7, 2)
>>
>> Unlike typical binary floats which use full precision, it is not uncommon
>> to have decimal floats with only a few digits of precision where the
>> expression as a fraction is both useful and unsurprising.
>
> Sounds fine to me. Fraction already accepts decimal floating-point
> strings, so the implementation of this would be trivial (convert to
> string, then call the Fraction constructor).
Better, Fraction.from_decimal() already exists. ;-)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
On Mon, Mar 22, 2010 at 8:44 PM, Guido van Rossum wrote:
> On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger
> wrote:
>> While we're on the topic, I think you should consider allowing the Fraction()
>> constructor to accept a decimal input.
>>
>> This corresponds to common schoolbook problems and simple client requests:
>> "Express 3.5 as a fraction".
>>
>> >>> Fraction(Decimal('3.5'))
>> Fraction(7, 2)
>
> There is already a Fraction.from_decimal() class method.
So there is; I'd failed to notice that! So decimal -> fraction
conversion is implemented twice in the Fractions module---once in
__new__ and once in from_decimal. Hmm.
Mark
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
For the record, I thought I would take a stab at making a single post that recaps the trade-offs and reasoning behind the decision to have Fraction + decimal/float --> decimal/float. Pros: * While we know that both decimal and binary floats have a fixed internal precision and can be converted losslessly to a rational, that doesn't correspond to the way we think about them. We tend to think of floating point values as real numbers, not as rationals. * There is a notion of fractions being used for unrounded arithmetic and floats operations being rounded arithmetic. So, it doesn't make sense to create the illusion of an unrounded result from inputs that we already subject to rounding. * Backward compatibility. That is what the fractions module already does and we haven't have any problems with it. Cons: * The coercion logic for comparisons won't match the coercion logic for arithmetic operations. The former strives to be exact and to be consistent with hashing while the latter goes in the opposite direction. * Operations such as some_float + some_fraction are subject to double rounding. The potentially produces a different result than the single rounding in: float(Fraction.from_float(some_float) + some_fraction) Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Martin v. Löwis v.loewis.de> writes: > > If certain use cases > make it problematic (e.g. Apache creating directories which you then > cannot delete), there should be a way to turn it *off*. Perhaps the > existing machinery to turn of byte code generation at all might be > sufficient. Except that you notice the problem when it happens, so turning it off is always too late. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Antoine Pitrou wrote: > Martin v. Löwis v.loewis.de> writes: >> If certain use cases >> make it problematic (e.g. Apache creating directories which you then >> cannot delete), there should be a way to turn it *off*. Perhaps the >> existing machinery to turn of byte code generation at all might be >> sufficient. > > Except that you notice the problem when it happens, so turning it off is > always > too late. Why is it too late? Fix it, and get on. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
On Mon, Mar 22, 2010 at 06:56, Barry Warsaw wrote:
> I have a pretty good start on PEP 3147 implementation [1], but I've
> encountered a situation that I'd like to get some feedback on. Here's the
> test case illustrating the problem. From test_import.py:
>
> def test_writable_directory(self):
> # The umask is not conducive to creating a writable __pycache__
> # directory.
> with umask(0o222):
> __import__(TESTFN)
> self.assertTrue(os.path.exists('__pycache__'))
> self.assertTrue(os.path.exists(os.path.join(
> '__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag
>
> The __pycache__ directory does not exist before the import, and the import
> machinery creates the directory, but the umask leaves the directory unwritable
> by anybody. So of course when the import machinery goes to write the .pyc
> file inside __pycache__, it fails. This does not cause an ImportError though,
> just like if today the package directory were unwritable.
>
> This might be different than today's situation though because once the
> unwritable __pycache__ directory is created, nothing is going to change that
> without explicit user interaction, and that might be difficult after the
> fact.
>
> I'm not sure what the right answer is. Some possible choices:
>
> * Tough luck
> * Force the umask so that the directory is writable, but then the question is,
> by whom? ugo+w or something less?
> * Copy the permissions from the parent directory and ignore umask
> * Raise an exception or refuse to create __pycache__ if it's not writable
> (again, by whom?)
I say tough luck. At the moment if you can't write a .pyc it is a
silent failure; don't see the difference as significant enough to
warrant changing the practice.
-Brett
>
> Or maybe you have a better idea? What's the equivalent situation on Windows
> and how would things work there?
>
> -Barry
>
> [1] https://edge.launchpad.net/~barry/python/pep3147
>
> P.S. I'm down to only 8 unit test failures.
>
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On Mon, 22 Mar 2010 18:15:01 -, Antoine Pitrou wrote: > Isaac Morland uwaterloo.ca> writes: > > > > > IMO, all these issues militate for putting __pycache__ creation out of > > > the interpreter core, and in the hands of third-party package-time/ > > > install-time tools (or distutils). > > > > Speaking only for myself, but really for anybody who likes tidy source > > directories, I hope some version of the __pycache__ proposal becomes part > > of standard Python, by which I ideally mean it's enabled by default but if > > that is just not a good idea then at most it should be required to set a > > command-line option to get this feature. > > This doesn't contradict by my proposal. > > What I am proposing is that the creation of __pycache__ /directories/ be put > outside of the core. It can be part of distutils, or of a separate module, or > delegated to third-party tools. It could even be as simple as > "python -m compileall --pycache", if someone implements it. Or even as simple as 'mkdir __pycache__', if you are working in your own library and don't want .pyc clutter. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Martin v. Löwis v.loewis.de> writes: > > Antoine Pitrou wrote: > > Martin v. Löwis v.loewis.de> writes: > >> If certain use cases > >> make it problematic (e.g. Apache creating directories which you then > >> cannot delete), there should be a way to turn it *off*. Perhaps the > >> existing machinery to turn of byte code generation at all might be > >> sufficient. > > > > Except that you notice the problem when it happens, so turning it off is always > > too late. > > Why is it too late? Fix it, and get on. Sure, but it is annoying, and since it's the kind of things that noone (including sysadmins) ever thinks about in advance, it's bound to repeat itself quite often. It's especially annoying, of course, if you have to ask someone else to remove the directories for you (or if you have to write custom code and get it executed by the Apache or WSGI handler...). Really, it's unfriendly to users and it's certainly not outweighed by the "benefit" of having "cleaner" source directories. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
>> Why is it too late? Fix it, and get on.
>
> Sure, but it is annoying, and since it's the kind of things that noone
> (including sysadmins) ever thinks about in advance, it's bound to repeat
> itself
> quite often.
>
> It's especially annoying, of course, if you have to ask someone else to remove
> the directories for you (or if you have to write custom code and get it
> executed
> by the Apache or WSGI handler...).
>
> Really, it's unfriendly to users and it's certainly not outweighed by the
> "benefit" of having "cleaner" source directories.
Whether it is outweighed would also depend on how likely and frequent
the presumed problem is, no?
If Apache creates a folder for me that I cannot remove, most likely,
there was a configuration error in the first place: common practice
tells that you should execute user code under user permissions, not as
www-data. If your code does get run as Apache, this also opens a way of
not asking for help: just put "os.system('chmod +w /.../__pycache__')"
into your code, and have Apache run it again.
So I don't think this is any more unfriendly than creating .pyc files in
the first place, and the advantages of uniformity of this new approach
certainly outweigh the disadvantages.
Regards,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Raymond Hettinger wrote: > * The coercion logic for comparisons won't match the > coercion logic for arithmetic operations. The former > strives to be exact and to be consistent with hashing > while the latter goes in the opposite direction. Although Guido pointed out that float/long comparisons and coercions blazed the trail on this particular discrepancy years ago. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Le lundi 22 mars 2010 à 23:18 +0100, "Martin v. Löwis" a écrit :
> If Apache creates a folder for me that I cannot remove, most likely,
> there was a configuration error in the first place: common practice
> tells that you should execute user code under user permissions, not as
> www-data.
I'm sure there can be reasons not to do so.
One of them is mass hosting of Python apps: you don't want to create a
separate process per Python user, and therefore you can't run the code
under the "right" user.
> If your code does get run as Apache, this also opens a way of
> not asking for help: just put "os.system('chmod +w /.../__pycache__')"
> into your code, and have Apache run it again.
This is what I meant by "write custom code and get it executed by the
Apache or WSGI handler". Having the Web server execute ad hoc system
administration code is far from elegant and user-friendly.
> So I don't think this is any more unfriendly than creating .pyc files in
> the first place, and the advantages of uniformity of this new approach
> certainly outweigh the disadvantages.
I don't see the "advantages of uniformity".
What we had initially was a way to solve problems which are specific to
Ubuntu and Debian packagers, and which was only to be activated by those
people on those systems.
We now end with an alleged complete solution to a problem which doesn't
seem to exist, or is at least vastly overblown (the idea that having pyc
files along their source counterparts is a nuisance doesn't seem to be a
common grief against Python).
I would really recommend reexamining it, rather than falling for the
shiny new thing.
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Mark Dickinson wrote: But the Fraction type is going to mess this up: for Decimal + Fraction -> Decimal, I don't see any other sensible option than to convert the Fraction using the current context, since lossless conversion isn't generally possible. You could convert the Decimal to a Fraction, do the arithmetic as Fractions, and convert the result back to Decimal using the current precision. I think this would satisfy the principle of performing the calculation as though infinite precision were available, and the final rounding is justified, since seen from the outside we are performing a single operation on two numbers. Another approach would be to convert the numerators and denominators separately to Decimal, and then do a/b + c/d = (a*d + b*c) / (b*d) using the usual rules of Decimal arithmetic. This would incur more than one rounding, but the rules for Decimals concern operations between two Decimals, and we have a Decimal and a Fraction here, so all bets could be considered off. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
> We now end with an alleged complete solution to a problem which doesn't > seem to exist, or is at least vastly overblown (the idea that having pyc > files along their source counterparts is a nuisance doesn't seem to be a > common grief against Python). > > I would really recommend reexamining it, rather than falling for the > shiny new thing. I think the appropriate action at this point is to record this specific objection in the PEP. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Nick Coghlan wrote: http://docs.python.org/library/decimal.html#decimal.Inexact (Part of the thread context rather than the individual decimal values, but if you use it properly it tells you whenever an inexact operation has occurred in the current thread) My problem was that the statement "All numbers are exact; inexactness is a taint" appears to be self-contradictory. Numbers *can* become tainted with inexactness whether you explicitly keep track of that or not. However, the excerpt by Cowlishaw posted earlier reveals where the confusion is coming from, I think. What Cowlishaw appears to mean by "numbers are exact" is that Decimals represent particular values, not *intervals*. This is not really the same thing as the notion of inexact numbers in the numeric tower. There, it means more like "this number may not quite represent the value the programmer had in mind". -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
Antoine Pitrou wrote: In light of this issue, I'm -0.5 on __pycache__ becoming the default caching mechanism. The directory ownership/permissions issue is too much of a mess, especially for Web applications (think __pycache__ files created by the Apache user). Doesn't the existing .pyc mechanism have the same problem? Seems to me it's just as insecure to allow the Apache user to create .pyc files, since an attacker could overwrite them with arbitrary bytecode. The only safe way is to pre-compile under a different user and make everything read-only to Apache. The same thing would apply under the __pycache__ regime. Actually, __pycache__ creation doesn't have to be part of the import mechanism. It can be part of distutils instead (or whatever third-party tool What about development, or if a user installs by dragging into site-packages instead of using an installer? I don't like the idea of being required to use an installation tool in order to get .pyc files. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask
Greg Ewing canterbury.ac.nz> writes: > > Doesn't the existing .pyc mechanism have the same problem? Seems > to me it's just as insecure to allow the Apache user to create > .pyc files, since an attacker could overwrite them with arbitrary > bytecode. The problem is that you can't delete the __pycache__ directory if it doesn't have the right ownership and if it's non-empty. This problem doesn't exist with a pyc file situated in a directory you own. > > Actually, __pycache__ creation doesn't have to be part of the import mechanism. > > It can be part of distutils instead (or whatever third-party tool > > What about development, The main point of the __pycache__ proposal is to solve the needs of Ubuntu/Debian packagers. If you are developing (rather than deploying or building packages), you shouldn't have these needs AFAICT. > or if a user installs by dragging into > site-packages instead of using an installer? Well... do people actually do this? "python setup.py install" is simpler than finding the right place to drag your package to, and doing the dragging. It also gives you metadata for free. And there's less risk of screwing up. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Antoine Pitrou wrote: Oh, and by the way, there can be a race condition between __pycache__ creation and deletion (if it fails the test) You can check whether the directory would be created with the right user beforehand, and if not, don't create one at all. To exploit a race condition there, the attacker would have to be capable of either changing the owner of the parent directory or removing it and replacing it with a different one, and if he can do that, he can do whatever he wants anyway. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Attribute lookup ambiguity
Pascal Chambon wrote: I don't follow you there - in my mind, the default __getattribute__ could simply have wrapped all its operations inside soem kind of "try..catch AttributeError:" mechanism, and thus been able to fallback to __getattr__ in any way. But then it would be incorrect to say that "__getattribute__ raises an exception". When we say that a function raises an exception, we normally mean that the exception propagates out of the function and can be seen by the caller, not that it was raised and caught somewhere inside the function. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mixing float and Decimal -- thread reboot
Mark Dickinson wrote: It might make sense for Decimal + complex mixed-type operations to be disallowed, for example. As long as you're allowing Decimal-float comparisons, Decimal-complex comparison for equality has an obvious interpretation. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
On 22Mar2010 09:56, Barry Warsaw wrote: | I have a pretty good start on PEP 3147 implementation [1], but I've | encountered a situation that I'd like to get some feedback on. Here's the | test case illustrating the problem. From test_import.py: | | def test_writable_directory(self): | # The umask is not conducive to creating a writable __pycache__ | # directory. | with umask(0o222): [...] | The __pycache__ directory does not exist before the import, and the import | machinery creates the directory, but the umask leaves the directory unwritable | by anybody. So of course when the import machinery goes to write the .pyc | file inside __pycache__, it fails. This does not cause an ImportError though, | just like if today the package directory were unwritable. | | This might be different than today's situation though because once the | unwritable __pycache__ directory is created, nothing is going to change that | without explicit user interaction, and that might be difficult after the | fact. Like any bad/suboptimal permission. | I'm not sure what the right answer is. Some possible choices: | | * Tough luck +1 I'd go with this one myself. | * Force the umask so that the directory is writable, but then the question is, | by whom? ugo+w or something less? -2 Racy and dangerous. The umask is a UNIX process global, and other threads may get bad results if they're active during this window. | * Copy the permissions from the parent directory and ignore umask -1 Maybe. But consider that you may not be the owner of the parent: then the new child will have different ownership than the parent but the same permission mask. Potentially a bad mix. This approach is very hard to get right. | * Raise an exception or refuse to create __pycache__ if it's not writable | (again, by whom?) -3 Bleah. My python program won't run because an obscure (to the user) directory had unusual permissions? Tough ove, it's the only way:-) -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ When asked what would I most want to try before doing it, I said Death. - Michael Burton, [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
On 23Mar2010 11:40, I wrote: | | * Raise an exception or refuse to create __pycache__ if it's not writable | | (again, by whom?) | | -3 | Bleah. My python program won't run because an obscure (to the user) | directory had unusual permissions? Clarification: I'm -3 on the exception. Silent failure to make the __pycache__ would do, and may be better than silently making a useless (unwritable) one. How about: made_it = False ok = False try: if not isdir(__pycache__dir): mkdir(__pycache__dir) made_it = True write pyc content ... ok = True except OSError, IOerror etc: if not ok: os.remove pyc content file if made_it: rmdir(__pycache__dir) but be quiet if this fails, eg if it is not empty because another process or thread added stuff So silent tidyup attempt on failure, but no escaping exception. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ The govt MUST regulate the Net NOW! We can't have average people saying what's on their minds! - [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Antoine Pitrou wrote: Having the Web server execute ad hoc system administration code is far from elegant and user-friendly. With the right piece of code, you could create yourself a setuid-apache shell and solve this problem once and for all. :-) -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Barry Warsaw wrote: On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: If I understand correctly, we would have the current mode as the default, and can trigger __pycache__ behavior simply by manually creating a __pycache__ directory and deleting any byte-code files in the module/program directory. I like this, it is easy to understand and can be used without messing with flags or environment variables. Well, for a package with subpackages, it gets more complicated. Definitely not something you're likely to do manually. Antoine's suggestion of 'python -m compileall --pycache' would work, but I think it's also obscure enough that most Python users won't get the benefit. May be a bit more complicated, but it should be easy to write tools to handle the repetitive stuff. When I'm writing python projects I usually only work on one or two packages at a time at most. Creating a couple of directories when I first get started on a project is nothing compared with the other 10 or more files of > 1,000 lines of code each. This has a very low mental hurdle too. All the other packages I'm importing would probably already be preinstalled complete with __pycache__ directories and bytecode files. As Antoine was suggesting that it would be up to the installer's (scrips) to create the __pycache__ directories at install time, either directly or by possibly issuing a 'python -m compileall --pycache' command? Ron ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Greg Ewing canterbury.ac.nz> writes:
>
> Antoine Pitrou wrote:
> > Having the Web server execute ad hoc system
> > administration code is far from elegant and user-friendly.
>
> With the right piece of code, you could create yourself
> a setuid-apache shell and solve this problem once and for
> all.
Well, if I can create a setuid apache shell, I can probably su as root or apache
as well.
("su -c rm -r whatever")
Or are you talking about a Web-based shell?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
Guido van Rossum wrote: On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw wrote: On Mar 22, 2010, at 02:02 PM, Ron Adam wrote: If I understand correctly, we would have the current mode as the default, and can trigger __pycache__ behavior simply by manually creating a __pycache__ directory and deleting any byte-code files in the module/program directory. Huh? Last time I looked weren't we going to make __pycache__ the default (and eventually only) behavior? I expect that the __pycache__ directories would quickly become the recommended "defacto default" for writing and preinitializing modules and packages, with the current behavior of having bytecode in the same directory as the .py files only as the fall-back (what I meant by default) behavior when the __pycache__ directories do not exist. I see only two reasonable solutions for __pycache__ creation -- either we change all setup/install scripts (both for core Python and for 3rd party packages) to always create a __pycache__ subdirectory for every directory (including package directories) installed; or we somehow create it the first time it's needed. > But creating it as needed runs into at least similar problems with ownership as creating .pyc files when first needed (if the parent directory is root-owned a mere mortal can't create it at all). > So even apart from the security issue (which I haven't thought about deeply) I think precreation should at least be an easily accessible option both for the core (where it can be done by compileall) and for 3rd party packages (where I guess it's up to distutils or whatever install mechanism is used). Yes, I think that is what Antoine was also getting at. Is there a need for python to use __pycache__ directories 100% of the time? For 2.x it seems like being flexible would be best, and if 3.x is going to be strict about it, it should be strict sooner than later rather than have a lot of 3rd party packages break at some point down the road. Ron ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
Antoine Pitrou writes: > Barry Warsaw python.org> writes: > > > > When Python is being installed, either by a from-source 'make > > install' or by the distro packager, then you'd expect the umask not > > to be insane. In the latter case, it's a bug and in the former case > > you screwed up so you should be able to delete and reinstall, or at > > the very least execute the same `find` command that Python's own > > Makefile calls (in my branch) for 'make clean'. > > > > When you're installing packages, again, I would expect that the > > system installer, or you via `easy_install` or whatever, would not > > have an insane umask. > > Well, precisely. That's why I suggest that creating the __pycache__ > directories be done *at install time* (or packaging time), and not via > the core import machinery (that is, not at import time). That is, when > you *know* you are the right user, with the right umask. +1. Taking advantage of caching directories that have already been set up correctly in advance at install time is friendly. Littering the runtime directory with new subdirectories by default is not so friendly. Perhaps also of note is that the FHS recommends systems use ‘/var/cache/foo/’ for cached data from applications: /var/cache : Application cache data Purpose /var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system. http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA> This would suggest that Python could start using ‘/var/cache/python/’ for its cached bytecode tree on systems that implement the FHS. -- \ “Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__)appreciated.” —Edsger W. Dijkstra | Ben Finney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __pycache__ creation
On 3/22/2010 2:15 PM, Antoine Pitrou wrote: What I am proposing is that the creation of __pycache__ /directories/ be put outside of the core. It can be part of distutils, or of a separate module, or delegated to third-party tools. It could even be as simple as "python -m compileall --pycache", if someone implements it. Creation of the __pycache__ /contents/ (files inside the directory) would still be part of core Python, but only if the directory exists and is writable by the current process. -1 If, as I have done several times recently, I create a directory and insert an empty __init__.py and several real module.py files, I want the .pycs to go into __pycache__ *automatically, by default, without me also having to remember to create an empty __pycache__ *directory*, *each time*. Ugh. Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147, __pycache__ directories and umask
On 23.03.2010 02:28, Ben Finney wrote: Antoine Pitrou writes: Barry Warsaw python.org> writes: When Python is being installed, either by a from-source 'make install' or by the distro packager, then you'd expect the umask not to be insane. In the latter case, it's a bug and in the former case you screwed up so you should be able to delete and reinstall, or at the very least execute the same `find` command that Python's own Makefile calls (in my branch) for 'make clean'. When you're installing packages, again, I would expect that the system installer, or you via `easy_install` or whatever, would not have an insane umask. Well, precisely. That's why I suggest that creating the __pycache__ directories be done *at install time* (or packaging time), and not via the core import machinery (that is, not at import time). That is, when you *know* you are the right user, with the right umask. +1. Taking advantage of caching directories that have already been set up correctly in advance at install time is friendly. Littering the runtime directory with new subdirectories by default is not so friendly. Perhaps also of note is that the FHS recommends systems use ‘/var/cache/foo/’ for cached data from applications: /var/cache : Application cache data Purpose /var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system. http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA> This would suggest that Python could start using ‘/var/cache/python/’ for its cached bytecode tree on systems that implement the FHS. it reads *data*, not code. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
