Re: [Python-Dev] Compact dict implementations (was: PEP 468
Now I fixed failing tests (some tests relying to underlying layout). Before posting it to bugs.python.org, I want to confirm I have chance to it merged. First big problem is language spec. If builtin dict in both of PyPy and CPython is ordered, many people will relying it. It will force other Python implementations to implement it for compatibility. In other words, it may be de-facto "Python Language", even if Python Language spec say it's an implementation detail. Is it OK? Second problem is performance. Quick benchmark on my laptop (Sorry, I don't have dedicated hardware for long running stable benchmarking), It reduces 3% memory usage and increase 3% cpu time. I'll run longer benchmark in next week. I think I can't avoid the penalty because index hashtable and (hash, key, value) is not in same cacheline. (I hope my thought is wrong and there is way to optimize more.) pybench: https://gist.github.com/methane/cfad1427d87ceff9310350e78a214880 benchmark: https://gist.github.com/methane/5eb11fdd93863813b222e795ca0bfc1f Is it acceptable? I have some other minor problems (e.g. How I can use 2byte integer? Using int16_t in stdint.h is OK?). I'll discuss them in core-mentor ML or bugs.python.org. Thanks -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] security SIG? (was: Discussion overload)
Brett Cannon writes: > Do we need a security SIG? E.g. would people like Christian and > Cory like to have a separate place to talk about the ssl stuff > brought up at the language summit? Besides what Barry brought up about the potential for attractive nuisance where people post security issues that should be confidential (I don't think it's that great, though), I don't see it solving the "clash of cultures" issue. The people who have invested in learning a lot of technical stuff related to security post as if they believe that "consenting adults" cannot be applied to security issues (more on that below), while RMs and working on distros tend to take the position that, of course, "consenting adults" covers security too. A SIG does help to address Christian's "ya gotta be this tall" to contribute to security discussions, at least in the early stages of discussion, but eventually it's going to arrive at python-dev.[1] ISTM that in this case sufficient behind the scenes discussion took place that the main contributors to the ultimate decision had a pretty good idea of where each other stood, and (I'm guessing here) Larry said "OK, we agree to disagree. I could say I'm RM, you lose, but to be fair I'll ask for a BDFL ruling." Even though there really wasn't anything for most of us to do but wait for that ruling (really -- Guido talks to Ted T'so and Theo de Raadt when he wants advice, there are very few among us who travel in those circles), it ended up that several of the security guys say they're not sure they can participate in Python development any more. I see the security issue as a backyard swimming pool. The law may say you must put a fence around it, but even 6 year olds can climb the fence, fall in the pool, and drown. The hard-line security advocate position then is "the risk is a *kid's life*, backyard pools must be banned". You have to sympathize with their honest and deep concern, but the community accepts that risk in the case of swimming pools. I suspect the Python community at large is going to be happy with Larry's decision and the strategy of emphasizing the secrets module starting with 3.6. If so, the hard-line security advocates are going to have to accept that, or stay painfully frustrated. That would be very unfortunate, because their knowledge is very much needed. Footnotes: [1] Keeping the BFDL ruling within the security group isn't going to work, either -- the news of a secret patch will become public quickly, and it will just seriously harm the trust the community has in its leaders. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
> On 18 Jun 2016, at 11:38, Stephen J. Turnbull wrote: > > I see the security issue as a backyard swimming pool. The law may say > you must put a fence around it, but even 6 year olds can climb the > fence, fall in the pool, and drown. The hard-line security advocate > position then is "the risk is a *kid's life*, backyard pools must be > banned". You have to sympathize with their honest and deep concern, > but the community accepts that risk in the case of swimming pools. I > suspect the Python community at large is going to be happy with > Larry's decision and the strategy of emphasizing the secrets module > starting with 3.6. I don’t think that’s really an accurate representation of any of the arguments put forward here. A better analogy is this: - Right now, we have a fence around the neighbourhood swimming pool. This fence has a gate with a guard, and the guard prevents 6 year olds from getting through the gate. - Kids can climb the fence (by using random.choice or something like it). The security community is mostly in agreement with the stdlib folks: we’re happy to say that this problem is best dealt with by educating children to not climb the fence (don’t use random.choice in a security context). - In Python 3.4 and earlier, the guard on this gate will, in some circumstances, turn up to work drunk. The guard is very good at pretending to be sober, so you cannot tell just by looking that he’s drunk, but in this state he will let anyone through the gate. He sobers up fast, so it only matters if a child tries to get in very shortly after you open the swimming pool. - In Python 3.5 we included a patch that, by accident, installed a breathalyser on the gate. Now the guard can only open the gate when he’s sober. - The problem is that he cannot open the gate *for anyone* while he’s drunk. All entry to the pool stops if he shows up drunk. - The security folks want to say “yes, this breathalyser is awesome, leave it in place, it should always have been there”. - The compat folks want to say “the gate hasn’t had a breathalyser for years, and it represents a genuine inconvenience to adults who want to swim, so we need to remove it”. We are not trying to take non-CSPRNGs away from you. We are not trying to remove the random module, we are not trying to say that everyone must use urandom for all cases. We totally agree with the consenting adults policy. We just believe that the number of people who have used os.urandom and actively wanted the Linux behaviour may well be zero, and is certainly a tiny fraction of the user base of os.urandom, whereas the number of people who have used os.urandom and expected it to produce safe random bytes is dramatically larger. We believe that invisible risks are bad. We believe that it is difficult to meaningfully consent to behaviour you do not know about. And we believe that when your expectations are violated, it is better for the system to fail hard than to subtly degrade to a behaviour that puts you at risk, because one of these provides a trigger for action and one does not. In the case of ‘consenting adults’: users cannot be said to have meaningfully consented to behaviours they do not understand. Consider urllib2 and PEP 476. Prior to Python 2.7.9, urllib2 did not validate TLS certificates. It *could*, if a user was willing to configure it to do so, but by default it did not. We could defend that behaviour under ‘consenting adults’: users *technically* consented to the behaviour by using the code. However, most of these users *passively* consented: their consent is inferred by their use of the API, but they have written no code that actively asserts their consent. In Requests, we allow consenting adults to turn off cert verification if they want to, but they have to *actively* consent: they *have* to say verify=False. And they have to say it every time: we deliberately provide no global switch to turn off cert validation in Requests, you have to set verify=False every single time. This is very deliberate. If a user wants to shoot themselves in the foot they are welcome to do so, but we don’t hand the gun to the user pointed at their foot. We’re arguing for the same here with os.urandom(). If you want the Linux default urandom behaviour, that’s fine, but we think that it’s surprising to people and that they should be forced to *actively ask* for that behaviour, rather than passively be given it. The TL;DR is: consent is not the absence of saying no, it’s the presence of saying yes. Cory signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
> On 18 Jun 2016, at 04:06, Brett Cannon wrote: > > Do we need a security SIG? E.g. would people like Christian and Cory like to > have a separate place to talk about the ssl stuff brought up at the language > summit? Honestly, I’m not sure what we would gain. Unless that SIG is empowered to take action, all it will be is a factory for generating arguments like this one. It will inevitably be either a toxic environment in itself, or a source of toxic threads on python-dev as the security SIG brings new threads like this one to the table. It should be noted that of the three developers that originally stepped forward on the security side of things here (myself, Donald, and Christian), only I am left subscribed to python-dev and nosy’d on the relevant issues. Put another way: each time we do this, several people on the security side burn themselves out in the thread and walk away (it’s possible that those on the other side of the threads do too, I just don’t know those people so well). It’s hard to get enthusiastic about signing people up for that. =) Cory signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact dict implementations (was: PEP 468
In the original discussion, I think they decided to reimplement set before dict. The original discussion is here, for anyone else: https://mail.python.org/pipermail/python-dev/2012-December/123028.html On Jun 18, 2016 3:15 AM, "INADA Naoki" wrote: > If builtin dict in both of PyPy and CPython is ordered, many people > will relying it. > It will force other Python implementations to implement it for compatibility. > In other words, it may be de-facto "Python Language", even if Python > Language spec > say it's an implementation detail. > > Is it OK? Ordered, or just initially ordered? I mean, "ordered if no deletion". They discussed scrambling the order. (Subdiscussion was here: https://mail.python.org/pipermail/python-dev/2012-December/123041.html) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact dict implementations (was: PEP 468
> > pybench: https://gist.github.com/methane/cfad1427d87ceff9310350e78a214880 > benchmark: https://gist.github.com/methane/5eb11fdd93863813b222e795ca0bfc1f > > Is it acceptable? latest result is here https://gist.github.com/methane/22cf5d1dadb62bc87a15e9244a9d0ab8 -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG?
On 06/18/2016 07:30 AM, Cory Benfield wrote: On 18 Jun 2016, at 04:06, Brett Cannon wrote: Do we need a security SIG? E.g. would people like Christian and Cory like >> to have a separate place to talk about the ssl stuff brought up at the >> language summit? Honestly, I’m not sure what we would gain. We would gain a place where security enhancements/fixes can be discussed by those interested, where the environment is "how do we fix/improve such-and-such while breaking as little as possible" (those that want backward-compatibility at all costs need not apply ;). Once a consensus has been reached (and possibly a PEP written, but hopefully that part will only rarely be necessary) then the proposal can be made to py-dev, complete with the "this portion is backwards incompatible, this is the expected impact, this is why it's important, here are the other far more painful alternatives". Unless that SIG is empowered to take action, all it will be is a factory for > generating arguments like this one. It will inevitably be either a toxic > environment in itself, or a source of toxic threads on python-dev as the > security SIG brings new threads like this one to the table. I suspect the resulting thread on py-dev will be far less painful when the initial discussions on ways to fix/improve this-or-that has already been done, the various options are being laid out, it's clear the new method will be in the next major release (unless incredibly serious, of course). It should be noted that of the three developers that originally stepped forward > on the security side of things here (myself, Donald, and Christian), only I am > left subscribed to python-dev and nosy’d on the relevant issues. Put another way: > each time we do this, several people on the security side burn themselves out in > the thread and walk away (it’s possible that those on the other side of the > threads do too, I just don’t know those people so well). It’s hard to get > enthusiastic about signing people up for that. =) One of the big advantages of a SIG is the much reduced pool of participants, and that those participants are usually interested in forward progress. It would also be helpful to have a single person both champion and act as buffer for the proposals (not necessarily the same person each time). I am reminded of the matrix-multiply PEP brought forward by Nathaniel a few months ago -- the proposal was researched outside of py-dev, presented to py-dev when ready, Nathaniel acted as the gateway between py-dev and those that wanted/needed the change, the discussion stayed (pretty much) on track, and it felt like the whole thing was very smooth. (If it was somebody else, my apologies for my terrible memory! ;) To sum up: I think it would be a good idea. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact dict implementations (was: PEP 468
> > Ordered, or just initially ordered? I mean, "ordered if no deletion". > I implemented "ordered". Because: * "orderd" is easier to explain than "ordered if no deletion". * I don't want to split sparse index hash and dense entry array. In case of very small dict, index hash (8byte) and first two entries (24*2=48byte) can be on one cache line. * Easy to implement "split dictionary" (aka. key sharing dictionary). You can see what I implemented in here. https://github.com/methane/cpython/pull/1/files -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
On Sat, 18 Jun 2016 at 07:30 Cory Benfield wrote: > > > On 18 Jun 2016, at 04:06, Brett Cannon wrote: > > > > Do we need a security SIG? E.g. would people like Christian and Cory > like to have a separate place to talk about the ssl stuff brought up at the > language summit? > > > Honestly, I’m not sure what we would gain. > > Unless that SIG is empowered to take action, all it will be is a factory > for generating arguments like this one. It will inevitably be either a > toxic environment in itself, or a source of toxic threads on python-dev as > the security SIG brings new threads like this one to the table. > > It should be noted that of the three developers that originally stepped > forward on the security side of things here (myself, Donald, and > Christian), only I am left subscribed to python-dev and nosy’d on the > relevant issues. Put another way: each time we do this, several people on > the security side burn themselves out in the thread and walk away (it’s > possible that those on the other side of the threads do too, I just don’t > know those people so well). It’s hard to get enthusiastic about signing > people up for that. =) > And this is the problem I'm trying to solve. As various people have pointed out, the conversation was pretty much cordial, but it did end up feeling like "you're not listening to me" on both sides on top of the volume, which is what I think burned people out on this thread. I think Nick brought up the point that we as a group need to come up with some guideline that we more-or-less stick with to help guide this kind of discussion or else we are going to burn out regularly any time security comes up; we can't keep holding security discussions like this or else we're going to end up in a bad place when everyone burns out and stops caring. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact dict implementations (was: PEP 468
> On Jun 18, 2016, at 9:57 AM, Franklin Lee wrote: > > In the original discussion, I think they decided to reimplement set before > dict. I ended-up going in a different direction with sets (using linear probes to reduce the cost of collisions). Also, after the original discussion, PyPy implemented the idea for dicts and achieved some nice improvements. So, I think Inada Naoki is going in the right direction by focusing on compact dicts. Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] JUMP_ABSOLUTE in nested if statements
Hi,
Could some one give a hand with explaining to me why we have a JUMP_ABSOLUTE
followed by a JUMP_FORWARD op code when this function is disassembled.
>>> def f1():
... a, b = 10, 11
... if a >= 10:
... if b >= 11:
... print("hello world")
…
The disassembled function is shown below.
>>> dis(f1)
2 0 LOAD_CONST 4 ((10, 11))
3 UNPACK_SEQUENCE 2
6 STORE_FAST 0 (a)
9 STORE_FAST 1 (b)
312 LOAD_FAST0 (a)
15 LOAD_CONST 1 (10)
18 COMPARE_OP 5 (>=)
21 POP_JUMP_IF_FALSE 47
424 LOAD_FAST1 (b)
27 LOAD_CONST 2 (11)
30 COMPARE_OP 5 (>=)
33 POP_JUMP_IF_FALSE 47
536 LOAD_CONST 3 ('hello world')
39 PRINT_ITEM
40 PRINT_NEWLINE
41 JUMP_ABSOLUTE 47
44 JUMP_FORWARD 0 (to 47)
>> 47 LOAD_CONST 0 (None)
50 RETURN_VALUE
From my understanding, once JUMP_ABSOLUTE is executed, then JUMP_FORWARD is
never gotten to so must be dead code so why is it being generated?
Furthermore why is JUMP_ABSOLUTE rather than JUMP_FORWARD used in this
particular case of nested if statements? I have tried other types of nested if
statements and it has always been JUMP_FORWARD that
is generated.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] JUMP_ABSOLUTE in nested if statements
Python has a peephole optimizer which does not remove dead code that it just created. Victor Le 18 juin 2016 23:14, "Obiesie ike-nwosu via Python-Dev" < [email protected]> a écrit : > Hi, > > Could some one give a hand with explaining to me why we have a > JUMP_ABSOLUTE followed by a JUMP_FORWARD op code when this function is > disassembled. > > >>> def f1(): > ... a, b = 10, 11 > ... if a >= 10: > ... if b >= 11: > ... print("hello world") > … > > The disassembled function is shown below. > >>> dis(f1) > 2 0 LOAD_CONST 4 ((10, 11)) > 3 UNPACK_SEQUENCE 2 > 6 STORE_FAST 0 (a) > 9 STORE_FAST 1 (b) > > 312 LOAD_FAST0 (a) > 15 LOAD_CONST 1 (10) > 18 COMPARE_OP 5 (>=) > 21 POP_JUMP_IF_FALSE 47 > > 424 LOAD_FAST1 (b) > 27 LOAD_CONST 2 (11) > 30 COMPARE_OP 5 (>=) > 33 POP_JUMP_IF_FALSE 47 > > 536 LOAD_CONST 3 ('hello world') > 39 PRINT_ITEM > 40 PRINT_NEWLINE > 41 JUMP_ABSOLUTE 47 > 44 JUMP_FORWARD 0 (to 47) > >> 47 LOAD_CONST 0 (None) > 50 RETURN_VALUE > > From my understanding, once JUMP_ABSOLUTE is executed, then JUMP_FORWARD > is never gotten to so must be dead code so why is it being generated? > Furthermore why is JUMP_ABSOLUTE rather than JUMP_FORWARD used in this > particular case of nested if statements? I have tried other types of nested > if statements and it has always been JUMP_FORWARD that > is generated. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] JUMP_ABSOLUTE in nested if statements
> On Jun 18, 2016, at 2:04 PM, Obiesie ike-nwosu via Python-Dev
> wrote:
>
> Hi,
>
> Could some one give a hand with explaining to me why we have a JUMP_ABSOLUTE
> followed by a JUMP_FORWARD op code when this function is disassembled.
> < snipped>
> From my understanding, once JUMP_ABSOLUTE is executed, then JUMP_FORWARD is
> never gotten to so must be dead code so why is it being generated?
> Furthermore why is JUMP_ABSOLUTE rather than JUMP_FORWARD used in this
> particular case of nested if statements? I have tried other types of nested
> if statements and it has always been JUMP_FORWARD that
> is generated.
The AST compilation step generates code with two JUMP_FORWARDs (see below).
Then, the peephole optimizer recognizes a jump-to-an-unconditional-jump and
replaces the first one with a JUMP_ABSOLUTE to save an unnecessary step.
The reason that it uses JUMP_ABSOLUTE instead of JUMP_FORWARD is that the
former is more general (it can jump backwards). Using the more general form
reduces the complexity of the optimizer.
The reason that the remaining jump-to-jump isn't optimized is that the
peepholer is intentionally kept simplistic, making only a single pass over the
opcodes. That misses some optimizations but gets the most common cases.
FWIW, the jump opcodes are very fast, so missing the final jump-to-jump isn't
much of a loss.
If you're curious, the relevant code is in Python/compile.c and
Python/peephole.c. The compile.c code generated opcodes in the most
straight-forward way possible and then the peephole optimizer gets some of the
low-hanging fruit by making a few simple transformations.
Raymond
AST generated code before peephole optimization -
5 0 LOAD_CONST 1 (10)
3 LOAD_CONST 2 (11)
6 BUILD_TUPLE 2
9 UNPACK_SEQUENCE 2
12 STORE_FAST 0 (a)
15 STORE_FAST 1 (b)
6 18 LOAD_FAST0 (a)
21 LOAD_CONST 1 (10)
24 COMPARE_OP 5 (>=)
27 POP_JUMP_IF_FALSE 53
7 30 LOAD_FAST1 (b)
33 LOAD_CONST 2 (11)
36 COMPARE_OP 5 (>=)
39 POP_JUMP_IF_FALSE 50
8 42 LOAD_CONST 3 ('hello world')
45 PRINT_ITEM
46 PRINT_NEWLINE
47 JUMP_FORWARD 0 (to 50)
>> 50 JUMP_FORWARD 0 (to 53)
>> 53 LOAD_CONST 0 (None)
56 RETURN_VALUE
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
On Jun 18, 2016, at 03:30 PM, Cory Benfield wrote: >Unless that SIG is empowered to take action It wouldn't be, but there *is* a private security mailing list that is. Christian was on it, and I'm sad that he got burned out. If you are willing and able to help out there, please contact security at python dot org. Cheers, -Barry pgp4uD9o3OigK.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
It's not just security discussions. The same thing happened with fspath, tzinfo, and many others that I have erased from my own memory. distutils-sig sees them often as well. The whole thing seems like a limitation of written communication. There's no way to indicate or define whether something should be nitpicked or not, and so everything gets line-by-line analysis whether it deserves it or not, which is what leads to such huge and fragmented threads, regardless of topic. At work, when we start seeing email or IM discussions going this way, we schedule a meeting. Perhaps we need a formal outlet for suspending discussion (and moderating incoming emails with a particular subject?) until an online call can be held and outcomes presented back to the list. Maybe we should schedule monthly online language summits and defer these discussions/decisions to that? I know that change won't be popular with some people. Honestly, if you haven't contributed more than the people who quit python-dev over these threads, you don't get to demand status quo. We need to change something, and I don't think more email or mute buttons (sorry Guido :) ) are the answer. Top-posted from my Windows Phone -Original Message- From: "Brett Cannon" Sent: 6/18/2016 11:13 To: "Cory Benfield" Cc: "Nick Coghlan" ; "Python Dev" Subject: Re: [Python-Dev] security SIG? (was: Discussion overload) On Sat, 18 Jun 2016 at 07:30 Cory Benfield wrote: > On 18 Jun 2016, at 04:06, Brett Cannon wrote: > > Do we need a security SIG? E.g. would people like Christian and Cory like to > have a separate place to talk about the ssl stuff brought up at the language > summit? Honestly, I’m not sure what we would gain. Unless that SIG is empowered to take action, all it will be is a factory for generating arguments like this one. It will inevitably be either a toxic environment in itself, or a source of toxic threads on python-dev as the security SIG brings new threads like this one to the table. It should be noted that of the three developers that originally stepped forward on the security side of things here (myself, Donald, and Christian), only I am left subscribed to python-dev and nosy’d on the relevant issues. Put another way: each time we do this, several people on the security side burn themselves out in the thread and walk away (it’s possible that those on the other side of the threads do too, I just don’t know those people so well). It’s hard to get enthusiastic about signing people up for that. =) And this is the problem I'm trying to solve. As various people have pointed out, the conversation was pretty much cordial, but it did end up feeling like "you're not listening to me" on both sides on top of the volume, which is what I think burned people out on this thread. I think Nick brought up the point that we as a group need to come up with some guideline that we more-or-less stick with to help guide this kind of discussion or else we are going to burn out regularly any time security comes up; we can't keep holding security discussions like this or else we're going to end up in a bad place when everyone burns out and stops caring. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] JUMP_ABSOLUTE in nested if statements
That is much clearer now.
Thanks a lot Raymond for taking the time out to explain this to me.
On a closing note, is this mailing list the right place to ask these kinds of
n00b questions?
Obi.
> On 18 Jun 2016, at 23:10, Raymond Hettinger
> wrote:
>
>
>> On Jun 18, 2016, at 2:04 PM, Obiesie ike-nwosu via Python-Dev
>> wrote:
>>
>> Hi,
>>
>> Could some one give a hand with explaining to me why we have a JUMP_ABSOLUTE
>> followed by a JUMP_FORWARD op code when this function is disassembled.
>> < snipped>
>> From my understanding, once JUMP_ABSOLUTE is executed, then JUMP_FORWARD is
>> never gotten to so must be dead code so why is it being generated?
>> Furthermore why is JUMP_ABSOLUTE rather than JUMP_FORWARD used in this
>> particular case of nested if statements? I have tried other types of nested
>> if statements and it has always been JUMP_FORWARD that
>> is generated.
>
> The AST compilation step generates code with two JUMP_FORWARDs (see below).
> Then, the peephole optimizer recognizes a jump-to-an-unconditional-jump and
> replaces the first one with a JUMP_ABSOLUTE to save an unnecessary step.
>
> The reason that it uses JUMP_ABSOLUTE instead of JUMP_FORWARD is that the
> former is more general (it can jump backwards). Using the more general form
> reduces the complexity of the optimizer.
>
> The reason that the remaining jump-to-jump isn't optimized is that the
> peepholer is intentionally kept simplistic, making only a single pass over
> the opcodes. That misses some optimizations but gets the most common cases.
>
> FWIW, the jump opcodes are very fast, so missing the final jump-to-jump isn't
> much of a loss.
>
> If you're curious, the relevant code is in Python/compile.c and
> Python/peephole.c. The compile.c code generated opcodes in the most
> straight-forward way possible and then the peephole optimizer gets some of
> the low-hanging fruit by making a few simple transformations.
>
>
> Raymond
>
>
> AST generated code before peephole optimization -
>
>
> 5 0 LOAD_CONST 1 (10)
> 3 LOAD_CONST 2 (11)
> 6 BUILD_TUPLE 2
> 9 UNPACK_SEQUENCE 2
> 12 STORE_FAST 0 (a)
> 15 STORE_FAST 1 (b)
>
> 6 18 LOAD_FAST0 (a)
> 21 LOAD_CONST 1 (10)
> 24 COMPARE_OP 5 (>=)
> 27 POP_JUMP_IF_FALSE 53
>
> 7 30 LOAD_FAST1 (b)
> 33 LOAD_CONST 2 (11)
> 36 COMPARE_OP 5 (>=)
> 39 POP_JUMP_IF_FALSE 50
>
> 8 42 LOAD_CONST 3 ('hello world')
> 45 PRINT_ITEM
> 46 PRINT_NEWLINE
> 47 JUMP_FORWARD 0 (to 50)
>>> 50 JUMP_FORWARD 0 (to 53)
>>> 53 LOAD_CONST 0 (None)
> 56 RETURN_VALUE
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] security SIG? (was: Discussion overload)
Like it or not, written communication is all we have. However, I do think we are running into some kind of limitation: the ancient concept of mailing lists (or newsgroups). I would like to continue the discussion of this limitation in the original thread. PS. I think it's somewhat ironic that Steve posted his idea to deal with discussions run amok in the forked thread that was meant specifically t discuss the proposal for a security-sig. Ditto that Cory used this same thread to bring up his philosophy about computer security -- that topic itself belongs clearly in the proposed SIG or on python-dev (if we don't create a SIG) but not (yet) in the discussion about whether we should create a SIG. On Sat, Jun 18, 2016 at 3:47 PM, Steve Dower wrote: > It's not just security discussions. The same thing happened with fspath, > tzinfo, and many others that I have erased from my own memory. > distutils-sig sees them often as well. > > The whole thing seems like a limitation of written communication. There's > no way to indicate or define whether something should be nitpicked or not, > and so everything gets line-by-line analysis whether it deserves it or not, > which is what leads to such huge and fragmented threads, regardless of > topic. > > At work, when we start seeing email or IM discussions going this way, we > schedule a meeting. Perhaps we need a formal outlet for suspending > discussion (and moderating incoming emails with a particular subject?) > until an online call can be held and outcomes presented back to the list. > Maybe we should schedule monthly online language summits and defer these > discussions/decisions to that? > > I know that change won't be popular with some people. Honestly, if you > haven't contributed more than the people who quit python-dev over these > threads, you don't get to demand status quo. We need to change something, > and I don't think more email or mute buttons (sorry Guido :) ) are the > answer. > > Top-posted from my Windows Phone > -- > From: Brett Cannon > Sent: 6/18/2016 11:13 > To: Cory Benfield > Cc: Nick Coghlan ; Python Dev > Subject: Re: [Python-Dev] security SIG? (was: Discussion overload) > > > > On Sat, 18 Jun 2016 at 07:30 Cory Benfield wrote: > >> >> > On 18 Jun 2016, at 04:06, Brett Cannon wrote: >> > >> > Do we need a security SIG? E.g. would people like Christian and Cory >> like to have a separate place to talk about the ssl stuff brought up at the >> language summit? >> >> >> Honestly, I’m not sure what we would gain. >> >> Unless that SIG is empowered to take action, all it will be is a factory >> for generating arguments like this one. It will inevitably be either a >> toxic environment in itself, or a source of toxic threads on python-dev as >> the security SIG brings new threads like this one to the table. >> >> It should be noted that of the three developers that originally stepped >> forward on the security side of things here (myself, Donald, and >> Christian), only I am left subscribed to python-dev and nosy’d on the >> relevant issues. Put another way: each time we do this, several people on >> the security side burn themselves out in the thread and walk away (it’s >> possible that those on the other side of the threads do too, I just don’t >> know those people so well). It’s hard to get enthusiastic about signing >> people up for that. =) >> > > And this is the problem I'm trying to solve. As various people have > pointed out, the conversation was pretty much cordial, but it did end up > feeling like "you're not listening to me" on both sides on top of the > volume, which is what I think burned people out on this thread. > > I think Nick brought up the point that we as a group need to come up with > some guideline that we more-or-less stick with to help guide this kind of > discussion or else we are going to burn out regularly any time security > comes up; we can't keep holding security discussions like this or else > we're going to end up in a bad place when everyone burns out and stops > caring. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion overload
Over on the "security SIG" thread, the point has been made that we seem to be hitting some limits in communication (Steve Dower said written communication, Guido said mailing lists/newsgroups). Based on the burnout we are seeing from these centi-threads we need to try and come up with some solution to this problem, else we are heading towards a bad place sue to communication burn-out. For me, I don't think we can give up written communication thanks to how worldwide we all are and thus make scheduling some monthly video chat very difficult. What I would like to consider, though, is something like Discourse where we at least have a chance to have tools available to us to manage discussions better than through federated email where everyone has different experiences in terms of delivery rate, ability to filter, splitting discussions, locking down out-of-control discussions, etc. I think harmonizing the experience along with better controls could help make all of this more manageable. On Fri, Jun 17, 2016, 18:13 Nick Coghlan wrote: > On 16 June 2016 at 19:00, Kevin Ollivier > wrote: > > Hi Guido, > > > > From: on behalf of Guido van Rossum > > > > Reply-To: > > Date: Thursday, June 16, 2016 at 5:27 PM > > To: Kevin Ollivier > > Cc: Python Dev > > Subject: Re: [Python-Dev] Discussion overload > > > > Hi Kevin, > > > > I often feel the same way. Are you using GMail? It combines related > messages > > in threads and lets you mute threads. I often use this feature so I can > > manage my inbox. (I presume other mailers have the same features, but I > > don't know if all of them do.) There are also many people who read the > list > > on a website, e.g. gmane. (Though I think that sometimes the delays > incurred > > there add to the noise -- e.g. when a decision is reached on the list > > sometimes people keep responding to earlier threads.) > > > > > > I fear I did quite a poor job of making my point. :( I've been on open > > source mailing lists since the late 90s, so I've learned strategies for > > dealing with mailing list overload. I've got my mail folders, my mail > rules, > > etc. Having been on many mailing lists over the years, I've seen many > > productive discussions and many unproductive ones, and over time you > start > > to see patterns. You also see what happens to those communities over > time. > > This is one of the major reasons we have the option of escalating > things to the PEP process (and that's currently in train for > os.urandom), as well as the SIGs for when folks really need to dig > into topics that risk incurring a relatively low signal-to-noise > ration on python-dev. It's also why python-ideas was turned into a > separate list, since folks without the time for more speculative > discussions and brainstorming can safely ignore it, while remaining > confident that any ideas considered interesting enough for further > review will be brought to python-dev's attention. > > But yes, one of the more significant design errors I've made with the > contextlib API was due to just such a draining pile-on by folks that > weren't happy the original name wasn't a 100% accurate description of > the underlying mechanics (even though it was an accurate description > of the intended use case), and "people yelling at you on project > communication channels without doing adequate research first" is the > number one reason we see otherwise happily engaged core developers > decide to find something else to do with their time. > > The challenge and art in community management in that context is > balancing telling both old and new list participants "It's OK to ask > 'Why is this so?', as sometimes the answer is that there isn't a good > reason and we may want to change it" and "Learn to be a good peer > manager, and avoid behaving like a micro-managing autocrat that chases > away experienced contributors". > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion overload
On Sat, Jun 18, 2016 at 6:17 PM, Brett Cannon wrote: > Over on the "security SIG" thread, the point has been made that we seem to > be hitting some limits in communication (Steve Dower said written > communication, Guido said mailing lists/newsgroups). Based on the burnout > we are seeing from these centi-threads we need to try and come up with some > solution to this problem, else we are heading towards a bad place [d]ue to > communication burn-out. > > For me, I don't think we can give up written communication thanks to how > worldwide we all are and thus make scheduling some monthly video chat very > difficult. What I would like to consider, though, is something like > Discourse where we at least have a chance to have tools available to us to > manage discussions better than through federated email where everyone has > different experiences in terms of delivery rate, ability to filter, > splitting discussions, locking down out-of-control discussions, etc. I > think harmonizing the experience along with better controls could help make > all of this more manageable. > Agreed that any form of real-time communication is out. First, I want to apologize to Kevin -- I only skimmed his message. I only saw that he had carefully qualified himself as a long-time open source contributor and list participant when I re-read his message. I also want to keep this short, so I'm proof-reading this before posting. Many projects on which I am currently working use one or more GitHub issue trackers as their main communication mechanism (mypy et al. don't even have a mailing list). I find that this works quite well to stay focused. We have quite a few issues that track important discussions over many days, weeks or months, and there is very little noise or cross-talk. It's easy to stay on topic, it's much easier to refer to other topics, it's easy to mute individual topics, and it's much less likely that a topic degenerates into a different discussion altogether (because it's easy to create a new issue for it). It's also easier to moderate, and you can even edit conversations (with restraint). I also like that it's possible to to do sentence-by-sentence quotation, but the extra effort required (copy/paste) encourages a linear thread of conversation within one issue. I did a quick check of my inbox and I think over the past week I had about as much mypy-related messages generated by GitHub as there were python-dev messages. And I felt much less bad for ignoring much of the mypy traffic while I was on vacation than I felt for ignoring python-dev, because it's easy to catch up using GitHub's web UI. (And no, I don't want to use gmane. I think it doesn't solve any of the other problems.) I don't know Discourse, but if it has a similar (or even better) feature set maybe we should give it a try. Or, now that we're going to migrate the CPython repo to GitHub, maybe we could just give GitHub's issue tracker a try? We could create a repo that has just a tracker (or a tracker plus a README.md explaining its purpose -- eventually we could add more resources and even a wiki). I'm sure that in the venerable python-dev tradition everyone is now jumping to give their opinion about Discourse, the GitHub tracker, their favorite alternative, the needs for free-form discussion, the need to have a GitHub account to participate, Slack, and the upcoming Mailman 3.0. But let's not do that, because it would be too self-referential (and defeat the purpose). I think we seriously need to rethink the way we have conversations here, and that includes the conversation about conversations. Here's my proposal: let's decide what to do about this roughly the same way we decided what to do with Mercurial. We don't have to take as long, but we'll use a similar process: a small committee run by a dedicated volunteer will compare alternatives and pick a strategy. If you're interested in serving on this committee, send me email off-list. If you want to head the committee, ditto. If you reply-all, you're automatically disqualified. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] JUMP_ABSOLUTE in nested if statements
On Sat, Jun 18, 2016 at 11:32:52PM +0100, Obiesie ike-nwosu via Python-Dev wrote: > That is much clearer now. > Thanks a lot Raymond for taking the time out to explain this to me. > On a closing note, is this mailing list the right place to ask these kinds > of n00b questions? That depends what sort of n00b question. If they are specifically related to the internals of the CPython interpreter, then this is certainly the right place. Code generation will count as an internal function of the interpreter. If they're general questions about Python the language, then the python-list mailing list is better. (Also available as comp.lang.python on Usenet.) Beware: it tends to be a high-volume, easily distracted forum where people often go off on long discussions which are only peripherally related to Python. -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact dict implementations (was: PEP 468
I've sent my patch to issue tracker, since I can't fix some remains TODOs by myself. http://bugs.python.org/issue27350 On Fri, Jun 17, 2016 at 6:15 PM, INADA Naoki wrote: > Hi, developers. > > I'm trying to implement compact dict. > https://github.com/methane/cpython/pull/1 > > Current status is passing most of tests. > Some tests are failing because of I haven't updated `sizeof` until layout fix. > And I haven't dropped OrderedDict has linked list. > > Before finishing implementation, I want to see comments and tests from core > developers. > Please come to core-mentorship ML or pull request and try it if you > interested in. > > Regards, > -- > INADA Naoki > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] frame evaluation API PEP
Hi Brett, I've got a few questions about the specific design. Probably you know the answers, it would be nice to have them in the PEP. First, why not have a global hook? What does a hook per interpreter give you? Would even finer granularity buy anything? Next, I'm a bit (but no more than a bit) concerned about the extra 8 bytes per code object, especially since for most people this is just waste (assuming most people won't be using Pyjion or Numba). Could it be a compile-time feature (requiring recompilation of CPython but not extensions)? Could you figure out some other way to store per-code-object data? It seems you considered this but decided that the co_extra field was simpler and faster; I'm basically pushing a little harder on this. Of course most of the PEP would disappear without this feature; the extra interpreter field is fine. Finally, there are some error messages from pep2html.py: https://www.python.org/dev/peps/pep-0523/#copyright --Guido On Fri, Jun 17, 2016 at 7:58 PM, Brett Cannon wrote: > I have taken PEP 523 for this: > https://github.com/python/peps/blob/master/pep-0523.txt . > > I'm waiting until Guido gets back from vacation, at which point I'll ask > for a pronouncement or assignment of a BDFL delegate. > > On Fri, 3 Jun 2016 at 14:37 Brett Cannon wrote: > >> For those of you who follow python-ideas or were at the PyCon US 2016 >> language summit, you have already seen/heard about this PEP. For those of >> you who don't fall into either of those categories, this PEP proposed a >> frame evaluation API for CPython. The motivating example of this work has >> been Pyjion, the experimental CPython JIT Dino Viehland and I have been >> working on in our spare time at Microsoft. The API also works for >> debugging, though, as already demonstrated by Google having added a very >> similar API internally for debugging purposes. >> >> The PEP is pasted in below and also available in rendered form at >> https://github.com/Microsoft/Pyjion/blob/master/pep.rst (I will assign >> myself a PEP # once discussion is finished as it's easier to work in git >> for this for the rich rendering of the in-progress PEP). >> >> I should mention that the difference from python-ideas and the language >> summit in the PEP are the listed support from Google's use of a very >> similar API as well as clarifying the co_extra field on code objects >> doesn't change their immutability (at least from the view of the PEP). >> >> -- >> PEP: NNN >> Title: Adding a frame evaluation API to CPython >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Brett Cannon , >> Dino Viehland >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 16-May-2016 >> Post-History: 16-May-2016 >> 03-Jun-2016 >> >> >> Abstract >> >> >> This PEP proposes to expand CPython's C API [#c-api]_ to allow for >> the specification of a per-interpreter function pointer to handle the >> evaluation of frames [#pyeval_evalframeex]_. This proposal also >> suggests adding a new field to code objects [#pycodeobject]_ to store >> arbitrary data for use by the frame evaluation function. >> >> >> Rationale >> = >> >> One place where flexibility has been lacking in Python is in the direct >> execution of Python code. While CPython's C API [#c-api]_ allows for >> constructing the data going into a frame object and then evaluating it >> via ``PyEval_EvalFrameEx()`` [#pyeval_evalframeex]_, control over the >> execution of Python code comes down to individual objects instead of a >> hollistic control of execution at the frame level. >> >> While wanting to have influence over frame evaluation may seem a bit >> too low-level, it does open the possibility for things such as a >> method-level JIT to be introduced into CPython without CPython itself >> having to provide one. By allowing external C code to control frame >> evaluation, a JIT can participate in the execution of Python code at >> the key point where evaluation occurs. This then allows for a JIT to >> conditionally recompile Python bytecode to machine code as desired >> while still allowing for executing regular CPython bytecode when >> running the JIT is not desired. This can be accomplished by allowing >> interpreters to specify what function to call to evaluate a frame. And >> by placing the API at the frame evaluation level it allows for a >> complete view of the execution environment of the code for the JIT. >> >> This ability to specify a frame evaluation function also allows for >> other use-cases beyond just opening CPython up to a JIT. For instance, >> it would not be difficult to implement a tracing or profiling function >> at the call level with this API. While CPython does provide the >> ability to set a tracing or profiling function at the Python level, >> this would be able to match the data collection of the profiler and >> quite possibly be faster for tracing by simply skipping per-line >> tracing support. >
