[Python-Dev] API design question: how to extend sys.settrace()?
Hi, In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead? Would it be possible to make the API "future-proof", so we can extend it again later? I almost never used sys.settrace(), so I prefer to ask on this python-dev list. Copy of the George King's message: https://bugs.python.org/issue29400#msg298441 """ After reviewing the thread, I'm reminded that the main design problem concerns preserving behavior of this idiom: "old=sys.gettrace(); ...; sys.settrace(old)" If we add more state, i.e. the `trace_instructions` bool, then the above idiom no longer correctly stores/reloads the full state. Here are the options that I see: 1. New APIs: * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable) -> None # no change.` * `gettraceinst() -> Callable # new.` * `settraceinst(tracefunc: Callable) -> None # new.` Behavior: * `settrace()` raises if `gettraceinst()` is not None. * `settraceinst()` raises if `gettrace()` is not None. 2. Add keyword arg to `settrace`. * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # added keyword.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `gettracestate()` returns the complete trace-related state: currently, `tracefunc` and `trace_instructions` fields. * `gettrace()` raises an exception if any of the trace state does not match the old behavior, i.e. if `trace_instructions` is set. 3. New API, but with extensible keyword args: * `settracestate(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # new.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `settrace()` raises if `gettracestate()` is not None. * `settracestate()` raises if `gettrace()` is not None. As I see it: * approach #1 is more conservative because it does not modify the old API. * approach #2 is more extensible because all future features can be represented by the state dictionary. * approach #3 has both of these strengths, but is also the most work. Examples of possible future features via keywords: * branch-level tracing, as briefly disscussed above. * instruction filtering set, which could be a more general version of the branch tracing. I intend to prototype one or both of these, but I'm also feeling a bit of time pressure to get the basic feature on track for 3.7. """ settraceinst() doesn't seem "future-proof" to me. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
On 09/27/2017 02:56 PM, Victor Stinner wrote: Hi, In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead? One possibility would be for settrace to query the capability on the part of the provided callable. For example: def settrace(tracefn): if getattr(tracefn, '__settrace_trace_instructions__', False): ... we need to trace instructions ... In general, the "trace function" can be upgraded to the concept of a "trace object" with (optional) methods under than __call__. This is extensible, easy to document, and fully supports the original "old=sys.gettrace(); ...; sys.settrace(old)" idiom. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
27.09.17 15:56, Victor Stinner пише: In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead? I afraid that this change breaks an assumption in frame_setlineno() about the state of the stack. This can corrupt the stack if you jump from the instruction which is a part of Python operation. For example FOR_ITER expects an iterator on the stack. If you jump to the end of the loop from the middle of an assignment operator and skip say STORE_FAST, you will left an arbitrary value on the stack. This can lead to unpredictable consequences. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
On Wed, Sep 27, 2017 at 8:10 AM, Serhiy Storchaka wrote: > I afraid that this change breaks an assumption in frame_setlineno() about > the state of the stack. This can corrupt the stack if you jump from the > instruction which is a part of Python operation. For example FOR_ITER > expects an iterator on the stack. If you jump to the end of the loop from > the middle of an assignment operator and skip say STORE_FAST, you will left > an arbitrary value on the stack. This can lead to unpredictable > consequences. > Well, probably OT but the solution for that would be to stop using a local stack and instead use explicit addressing. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New [email protected] mailing list
On 21/09/17 17:30, Barry Warsaw wrote: > https://mail.python.org/mailman/listinfo/security-announce "No such list security-announce". > https://mail.python.org/mailman/listinfo/security-sig "No such list security-sig". -- Jesús Cea Avión _/_/ _/_/_/_/_/_/ [email protected] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ Twitter: @jcea_/_/_/_/ _/_/_/_/_/ jabber / xmpp:[email protected] _/_/ _/_/_/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New [email protected] mailing list
On Sep 27, 2017, at 11:56, Jesus Cea wrote: > > On 21/09/17 17:30, Barry Warsaw wrote: >> https://mail.python.org/mailman/listinfo/security-announce > > "No such list security-announce". > >> https://mail.python.org/mailman/listinfo/security-sig > > "No such list security-sig". These lists were just migrated to Mailman 3. Please use: https://mail.python.org/mm3/mailman3/lists/security-announce.python.org/ https://mail.python.org/mm3/mailman3/lists/security-sig.python.org/ Cheers, -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New [email protected] mailing list
I can't see the lists either. Same message "No such list security-announce". Cheers. On Wed, Sep 27, 2017, 18:05 Jesus Cea wrote: > On 21/09/17 17:30, Barry Warsaw wrote: > > https://mail.python.org/mailman/listinfo/security-announce > > "No such list security-announce". > > > https://mail.python.org/mailman/listinfo/security-sig > > "No such list security-sig". > > -- > Jesús Cea Avión _/_/ _/_/_/_/_/_/ > [email protected] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ > Twitter: @jcea_/_/_/_/ _/_/_/_/_/ > jabber / xmpp:[email protected] _/_/ _/_/_/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertomartinezp%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New [email protected] mailing list
Can we add redirects to the old list locations? On Wed, Sep 27, 2017 at 9:17 AM, Roberto Martínez < [email protected]> wrote: > I can't see the lists either. Same message "No such list > security-announce". > > Cheers. > > On Wed, Sep 27, 2017, 18:05 Jesus Cea wrote: > >> On 21/09/17 17:30, Barry Warsaw wrote: >> > https://mail.python.org/mailman/listinfo/security-announce >> >> "No such list security-announce". >> >> > https://mail.python.org/mailman/listinfo/security-sig >> >> "No such list security-sig". >> >> -- >> Jesús Cea Avión _/_/ _/_/_/_/_/_/ >> [email protected] - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ >> Twitter: @jcea_/_/_/_/ _/_/_/_/_/ >> jabber / xmpp:[email protected] _/_/ _/_/_/_/ _/_/ _/_/ >> "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/ >> "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/ >> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> robertomartinezp%40gmail.com >> > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
On 27 September 2017 at 22:56, Victor Stinner wrote: > Hi, > > In bpo-29400, it was proposed to add the ability to trace not only > function calls but also instructions at the bytecode level. I like the > idea, but I don't see how to extend sys.settrace() to add a new > "trace_instructions: bool" optional (keyword-only?) parameter without > breaking the backward compatibility. Should we add a new function > instead? As part of investigating the current signal safety problems in with statements [1], I added the ability to trace lines, opcodes, both, or neither by setting a couple of flags on a per-frame basis: https://docs.python.org/dev/whatsnew/3.7.html#other-cpython-implementation-changes So the idea is that if you want per-opcode tracing, your trace function has to turn it on for each frame when handling the call event, and you accept the responsibility of not messing up the frame stack. Cheers, Nick. [1] See https://bugs.python.org/issue29988 if you're interested in the gory details -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New [email protected] mailing list
On Sep 27, 2017, at 12:24, Guido van Rossum wrote: > > Can we add redirects to the old list locations? I’m not sure we need it for security-announce given how new that list is and that we only have one message to it. I’ve forwarded this request to postmaster@ though. -Barry signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
Victor, thank you for starting this conversation. Nick, I just looked at your patch and I think it is a better solution than mine, because it does not involve adding to or changing the sys API. I will close my pull request. The reason for my interest in this area is that I’m experimenting with a code coverage tool that does per-opcode tracing. I just updated it to use the new f_trace_opcodes feature and it *almost* worked: Nick’s implementation calls the opcode trace before updating the line number, whereas my version updated line numbers first, so the event stream is slightly different. See: https://github.com/python/cpython/commit/5a8516701f5140c8c989c40e261a4f4e20e8af86#diff-7f17c8d8448b7b6f90549035d2147a9f From my perspective it makes more sense to do the update to f_lineno first, followed by the opcode trace, because then line events and opcode events corresponding to the same opcode will have the same line number; as it is they come out different. Is the current order intentional? Otherwise I’ll submit a patch. Either way, I’m really happy that this functionality has made it into master! Thanks, George ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] API design question: how to extend sys.settrace()?
On 28 September 2017 at 06:23, George King wrote: > Victor, thank you for starting this conversation. Nick, I just looked at your > patch and I think it is a better solution than mine, because it does not > involve adding to or changing the sys API. I will close my pull request. > > The reason for my interest in this area is that I’m experimenting with a code > coverage tool that does per-opcode tracing. > I just updated it to use the new f_trace_opcodes feature and it *almost* > worked: Nick’s implementation calls the opcode trace before updating the line > number, whereas my version updated line numbers first, so the event stream is > slightly different. > > See: > https://github.com/python/cpython/commit/5a8516701f5140c8c989c40e261a4f4e20e8af86#diff-7f17c8d8448b7b6f90549035d2147a9f > > From my perspective it makes more sense to do the update to f_lineno first, > followed by the opcode trace, because then line events and opcode events > corresponding to the same opcode will have the same line number; as it is > they come out different. > > Is the current order intentional? Otherwise I’ll submit a patch. Line numbers simply weren't relevant to my use case (injecting inconveniently timed exceptions), so I never checked whether or not f_lineno was accurate when the trace event was emitted. So yeah, a follow-up bug report and PR to fix that would be appreciated (no need for a NEWS entry, since it's a bug in a not-yet-released feature). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] bugs.python.org updated
Hi, yesterday I updated the version of Roundup used to run our bugs.python.org instance and 4 other instances (the meta, jython, setuptools, and roundup trackers). If everything went right you shouldn't have noticed anything different :) This update included ~300 changesets from upstream and required an additional ~30 to update our instances and our fork of Roundup. A number of features that we added to our fork over the years have been ported upstream and they have now been removed from our fork, which is now -- except for the github integration -- almost aligned with upstream. If you notice any issue with the bug tracker (/with/, not /in/ -- that's expected!), please report it to the meta tracker ( http://psf.upfronthosting.co.za/roundup/meta/) and/or to me. If there are no reports and everything works fine I still have a few more updates coming up ;) A big thanks to John Rouillard (from the Roundup team) and R. David Murray for the help! Best Regards, Ezio Melotti P.S. Roundup started moving towards Python 3. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bugs.python.org updated
On Wed, Sep 27, 2017 at 6:54 PM, Ezio Melotti wrote: > > This update included ~300 changesets from upstream and required an > additional ~30 to update our instances and our fork of Roundup. A number > of features that we added to our fork over the years have been ported > upstream and they have now been removed from our fork, which is now -- > except for the github integration -- almost aligned with upstream. > > > P.S. Roundup started moving towards Python 3. > Thanks a lot for your work on this, Ezio! ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
