[Python-Dev] exception too expensive?
i thought avoiding a second dict lookup should be faster, but it turned out to be completely wrong. it's only marginally faster, but if an exception occurs,it's x10 slower.## the code#>>> import time >>> b = dict((i, 7) for i in range(1000)) >>> def try_lookup(k):... try:... return b[k]... except KeyError:... return 17...>>> def if_lookup(k):... if k in b:... return b[k]... return 17 ...>>> def test(f, k, count=10):... t=time.time()... while count:... count -=1... f(k)... return time.time()-t...## try_lookup with valid key #>>> test(try_lookup, 56, 100)0.608950958252>>> test(try_lookup, 56, 100)0.6086757507324>>> test(try_lookup, 56, 100)0.608950958252~0.61 sec ## if_lookup with valid key#>>> test(if_lookup, 56, 100)0.6876376037598>>> test(if_lookup, 56, 100)0.6873623962402>>> test(if_lookup, 56, 100) 0.67200016975402832~0.68 sec## try_lookup with invalid key#>>> test(try_lookup, 9456, 100)7.06236239624>>> test(try_lookup, 9456, 100)6.81236239624 >>> test(try_lookup, 9456, 100)6.8440001010894775~6.90 sec## if_lookup with invalid key#>>> test(if_lookup, 9456, 100)0.625>>> test(if_lookup, 9456, 100) 0.6413242492676>>> test(if_lookup, 9456, 100)0.65599989891052246~0.64 sec==before someone says "why don't you use dict.get", it's not applicable inmy code. i have a cache -- either object already exists in the cache, or i create and store it in the cache -- so get() and setdefault() are not useful.so first of all, i'd recommend not using the try_lookup method; use the if_lookup instead... unless you can be sure the changes a KeyError will be raised are marginal.second, isn't there anything that can be done to improve the performanceof exceptions? imo, exceptions should be cheap. i understand it has to do with creating the exception instance everytime, but can't something be done to improve that? for example, hold a cache exception instances and just memcpy them when needed (just a wild idea)?-tomer ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] exception too expensive?
tomer filiba wrote: > i thought avoiding a second dict lookup should be faster, but it turned out > to be completely wrong. it's only marginally faster, but if an exception > occurs, it's x10 slower. maybe you should take this to comp.lang.python instead... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] exception too expensive?
> i thought avoiding a second dict lookup should be faster, but it turned out > to be completely wrong. Unless you have an expensive hash function, it is almost never worth it to try to avoid a second lookup. Because of memory cache effects, the second lookup is dirt cheap (a cache miss is about as expensive as a floating point division). See dictnotes.txt for a few thoughts on the subject. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] "Missing" 2.5 feature
Back in: http://mail.python.org/pipermail/python-dev/2005-March/051856.html I made a pitch for adding: sys._current_frames() to 2.5, which would return a dict mapping each thread's id to that thread's current (Python) frame. As noted there, an extension module exists along these lines that's used at least by the popular Zope DeadlockDebugger product, but it's not possible to do it correctly in an extension. The latter is why it needs to be in the core (so long as it's in an extension, it risks segfaulting, because the core's internal `head_mutex` lock isn't exposed). I forgot about this but was recently reminded. How much opposition would there be to sneaking this into 2.5b2? It would consist of adding a relatively simple new function, docs, and tests; since it wouldn't _change_ any existing code, it would have a hard time breaking anything that currently works. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
On 7/7/06, Talin <[EMAIL PROTECTED]> wrote: Brett Cannon wrote:> On 7/7/06, Guido van Rossum <[EMAIL PROTECTED]> wrote:> On 7/7/06, Brett Cannon <[EMAIL PROTECTED] > wrote:>> > I guess I am just not seeing where your approach is better than>> preventing>> > the constructor in 'file' and having open() return the 'file' object or>> > proxy object. With your approach 'file' would be flagged, but with the >> > other you just put the same check in 'file's constructor. With both>> you>> > would probably also want open() to be a factory function anyway. So I>> don't>> > see where you gain simplicity or more security. It seems like you are >> > pushing the check into the eval loop, but you still require the>> flagging>> of>> > objects as unsafe. Going with the other two proposals you don't burden>> the >> > eval loop with the check but the objects that you would have flagged in>> the>> > first place.>> >>> > It just seems like we are pushing around the flagging of unsafe stuff >> and>> > that doesn't feel like it buys us much. At the risk of repeating someone's point or making no sense (I'm only>> following this with half an ear) I would like to point out that as >> long as there's C code involved, we can have the equivalent of private>> constructors in C++/Java. This means classes that cannot be>> constructed by calling the class from Python. C code has to use some >> other API to create an instance, bypassing this check. It seems>> reasonable to me, even if most popular types *can* be constructed.>> There are other types that have this property, e.g. list iterators. >> Try type(iter(list()))(). Good point. C code could circumvent the bit check by doing all of the work> behind the scenes without pushing the object on the stack. But if the > check> is in the C code for the object itself it is much harder to get around.>> -BrettI may be confused (I usually am), but I think you are misinterpretingGuido's point. I think what he is saying is not "you should beware of C code getting around the checks" but rather he is pointing out that Ccode that gets around the checks can be a useful part of the system -thus the notion of "private constructors", in other words methods for creating an object that are normally inaccessible to Python code.As to your point: I think I am beginning to understand, so let mereiterate to see if I have it right.In the scenario you describe, the open() function is replaced by a function that returns a proxy that has limits on what it is allowed todo. (Ideally, it would return one of several different types of proxiesbased on the input file path - so a file handle to /tmp would have a different set of restrictions than a file handle to /home/me.)Yep. Somewhere inside this proxy is the 'real' file handle. We need to insurethat the proxy is air-tight, so that it can't 'leak' to the outsideworld. (The original motivation for my scheme was the belief thatair-tightness couldn't be achieved, however the point that Guido makes above is beginning to make me believe otherwise.)The proxy is air-tight by being written in C and holding on to the reference to the file object in the struct of the proxy. The proxy also has to be able to support all of the methods that theregular file handle can - because we're going to want to pass that proxyover to other subsystems that don't know that they are dealing with a proxy - such as XML parsers or config file readers. Because thepermission checks are implemented in the proxy, this means that librarycode using the proxy has exactly the same access restrictions as thesandboxed code. Yep. Or at least the methods that will be needed (e.g., a read-only proxy only needs the read()-like methods). If you want any library code to be able to have additional privilegesbeyond what the sandboxed code can do, you'll need to pass them the'real' file handle, something which can only be done by either theproxy, or by a C 'gateway' function. This is not a problem as long as the library code doesn't attempt to hold on to the file handle(otherwise then the sandboxed code could potentially grab it from thelibrary's objects.) So for any case where a library needs additionalprivileges, you'll need to add additional methods to the proxy or create the gateway functions to deal with that.Basically. A library shouldn't need additional abilities. If they do they are not following the security restrictions so it shouldn't work. It should be a rarity that a library really needs to sneak around the security of a proxy to the point of we don't even consider it. Duck typing should be enough for this not to be a problem. So if it truly is the case that the file handle cannot leak into theoutside world, then you are correct - there's no need to put a check into the interpreter. My motivation was based on the belief that therewould always be a way to get around that, so instead the notion was to'poison' these protected objects so that even if they did leak out, attempting to u
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
On 7/7/06, Guido van Rossum <[EMAIL PROTECTED]> wrote: On 7/8/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote:> The situation you're describing here is a classic case of one> component keeping a closely held authority while using it to > provide some limited capability to some other component. This> comes up quite often when you're trying to write secure code.>> If you want to be able to write that subsystem in Python, then > we will need a way to create airtight Python objects (i.e. objects> that only leak what they explicitly choose to leak).>> So this goes back to the big question of goals:>> Do we want to be able to protect one piece of Python code > from another piece of Python code?>> I'd like the answer to be yes. It sounded for a while like this> was not part of Brett's plan, though. Now i'm not so sure. It> sounds like you're also interested in having the answer be yes? >> Let's keep talking about and playing with more examples -- i think> they'll help us understand what goals we should aim for and what> pitfalls to anticipate before we nail down too many details. I'd like the answer to be no, because I don't believe that we cantrust the VM to provide sufficient barriers. The old pre-2.2restricted execution mode tried to do this but 2.2 punched a millionholes in it. Python isn't designed for this (it doesn't even enforce private attributes). I guess this is also the main reason I'mskeptical about capabilities for Python.My plan is no. As Guido said, getting this right is feasibly questionable. I do not plan on trying to have security proxies or such implemented in Python code; it will need to be in C. If someone comes along and manages to find a way to make Python work without significantly changing the languages, great, and we can toss out my security implementation for that. But as of right now, I am not planning on making Python code safe to run in Python code.-Brett Guido van Rossum (home page: http://www.python.org/~guido/)___Python-Dev mailing list [email protected]://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Missing" 2.5 feature
On 7/8/06, Tim Peters <[EMAIL PROTECTED]> wrote: Back in:http://mail.python.org/pipermail/python-dev/2005-March/051856.htmlI made a pitch for adding:sys._current_frames() to 2.5, which would return a dict mapping each thread's id to thatthread's current (Python) frame. As noted there, an extension moduleexists along these lines that's used at least by the popular ZopeDeadlockDebugger product, but it's not possible to do it correctly in an extension. The latter is why it needs to be in the core (so longas it's in an extension, it risks segfaulting, because the core'sinternal `head_mutex` lock isn't exposed).I forgot about this but was recently reminded. How much opposition would there be to sneaking this into 2.5b2? It would consist ofadding a relatively simple new function, docs, and tests; since itwouldn't _change_ any existing code, it would have a hard timebreaking anything that currently works. Well, my understanding is that beta is feature freeze, so I am going to say we shouldn't do this. Obviously this can go in day one when 2.6 is started.Anyway, this is a true test of how rigid the rules are! Can Neal and Anthony going to say no to Uncle Timmy? =) -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python]
Brett Cannon wrote:
> On 7/7/06, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>> On 7/8/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote:
>> > I'd like the answer to be yes. It sounded for a while like this
>> > was not part of Brett's plan, though. Now i'm not so sure. It
>> > sounds like you're also interested in having the answer be yes?
>> >
>> > Let's keep talking about and playing with more examples -- i think
>> > they'll help us understand what goals we should aim for and what
>> > pitfalls to anticipate before we nail down too many details.
>>
>> I'd like the answer to be no, because I don't believe that we can
>> trust the VM to provide sufficient barriers. The old pre-2.2
>> restricted execution mode tried to do this but 2.2 punched a million
>> holes in it. Python isn't designed for this (it doesn't even enforce
>> private attributes). I guess this is also the main reason I'm
>> skeptical about capabilities for Python.
>
> My plan is no. As Guido said, getting this right is feasibly
> questionable. I do not plan on trying to have security proxies or such
> implemented in Python code; it will need to be in C. If someone comes
> along
> and manages to find a way to make Python work without significantly
> changing
> the languages, great, and we can toss out my security implementation for
> that.
>
> But as of right now, I am not planning on making Python code safe to run in
> Python code.
It might be possible for the code *outside* the sandbox to create new
security policies written in Python.
Lets start with the concept of a generic "protection" wrapper - its a C
proxy object which can wrap around any Python object, and which can
restrict access to a specific set of methods. So for example:
protected_object = protect(myObject, methods=set('open','close'))
'protect' creates a C proxy which restricts access to the object,
allowing only those methods listed to be called.
Now, lets create a security policy, written in Python. The policy is
essentially a factory which creates wrapped objects:
class MyPolicy:
# Ask the policy to create a file object
def file( path, perms ):
if perms == 'r':
# Trivial example, a real proxy would be more
# sophisticated, and probably configurable.
return protect( file( path, perms ),
methods=set('open', 'read', 'close') )
raise SecurityException
Now, when we create our sandbox, we pass in the policy:
sb = Sandbox( MyPolicy() )
The sandbox calls 'protect' on the policy object, preventing it from
being inspected or called inappropriately.
-- Talin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Missing" 2.5 feature
> I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. +1 Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Missing" 2.5 feature
Tim Peters wrote: > Back in: > > http://mail.python.org/pipermail/python-dev/2005-March/051856.html > > I made a pitch for adding: > > sys._current_frames() > > to 2.5, which would return a dict mapping each thread's id to that > thread's current (Python) frame. As noted there, an extension module > exists along these lines that's used at least by the popular Zope > DeadlockDebugger product, but it's not possible to do it correctly in > an extension. The latter is why it needs to be in the core (so long > as it's in an extension, it risks segfaulting, because the core's > internal `head_mutex` lock isn't exposed). > > I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. If you just check it in, using an appropriate log message[1] I think nobody would object. Georg [1] Thinking of "Whitespace normalization"... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
