[Python-Dev] Helpers for dynamic bytecode generation
Hi everyone, I've found myself recently writing Python code that dynamically generates bytecode.¹ I now have yet another case where I'm having to do this, in which my nice situation of being able to easily precompute all the jump addresses no longer holds. So I'm starting to write a helper to make it easy to write bytecode from Python, with its basic API calls being write(opcode, arg) and nextLine(optional label). The argument can be an int, name, local name, constant, label, etc., depending on the opcode, and it maintains all the appropriate tables and finally dumps a code object at the end. All of which is well and good and makes life much easier, but... I am *not* looking forward to writing the logic that basically duplicates that of assemble() in compile.c, of splitting all of this into basic blocks and computing the correct jump positions and so on before finally dumping out the bytecode. Has anyone already done this that people know of? (Searching the Internetz didn't turn anything up) Failing that, to what extent is it reasonable to either consider assemble() as some kind of sane API point into compile.c, and/or add some new API in compile.h which implements all of the stuff described above in C? (I'm fully expecting the answer to these latter questions to be "you have got to be kidding me," but figured it was wiser to check than to reinvent this particular wheel if it isn't necessary) Yonatan ¹ Not out of masochism, in case you're wondering; there was a real use case. A storage system would receive a read request that specified a bunch of (key, condition) pairs, where conditions where either return any value, return an exact value, or return values in a range. It would then receive between 1 and 1M (depending on the request parameters) candidate cells from the underlying storage layers, each of which had a tuple of bytes as its actual key values; it had to compare each of those tuples against the request parameters, and yield the values which matched. Because it's an inner loop and can easily be called 1M times, doing this in pure Python slows things down by a lot. Because it's also only called once, doing some really expensive overhead like synthesizing Python code and calling compile() on it would also slow things down a lot. But converting a bunch of (key, condition) pairs to a really efficient function from tuples of bytes to bools was pretty easy. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/E57DTM65LFEROFZLHKRV442JPPFAWNJU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
On Thu, Oct 24, 2019, 9:05 PM Yonatan Zunger wrote: > Hi everyone, > > Has anyone already done this that people know of? (Searching the > Internetz didn't turn anything up) Failing that, to what extent is it > reasonable to either consider assemble() as some kind of sane API point > into compile.c, and/or add some new API in compile.h which implements all > of the stuff described above in C? > > (I'm fully expecting the answer to these latter questions to be "you have > got to be kidding me," but figured it was wiser to check than to reinvent > this particular wheel if it isn't necessary) > There is byteplay (https://github.com/tallforasmurf/byteplay), but it is broken on Python 3.6+ due to the change to wordcode. There is also a newer one called simply "bytecode" (https://github.com/vstinner/bytecode) that appears to work through at least 3.7 (though I haven't personally tried it. > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/L7UWUOI22RNKJUUXDB6FV4XXBH3HP4QT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
Hi, On Fri, 25 Oct 2019 at 04:13, Jonathan Goble wrote: >> Has anyone already done this that people know of? (Searching the Internetz >> didn't turn anything up) Failing that, to what extent is it reasonable to >> either consider assemble() as some kind of sane API point into compile.c PyPy contains a complete rewrite of compile.c in Python, which should be relatively easy to extract. Here's the py3.6 version: https://bitbucket.org/pypy/pypy/src/py3.6/pypy/interpreter/astcompiler/ Independently, you may also want to benchmark your code on PyPy (*without* using any bytecode generation, just plain Python loops). A bientôt, Armin. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/NUPFLTIO7GB2DNBFHKVBMKGYYZE4PPO4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
Interesting that you bring this up. Just earlier this week I published the first version of a new package that lets you write compiled bytecode instructions inline with pure-Python syntax. The code's still a bit messy, being only a week old and all, but it works as advertised for CPython 3.6.2 through 3.9.0a0, and even includes neat features like labeled jumps, unused name/constant removal, stack size adjustments, etc... Perhaps it'll be useful to you (or at least you'll find it interesting): https://github.com/brandtbucher/hax Victor's Stinner's Bytecode package (already mentioned) is surely better for *dynamic* generation... I've never used it personally, but it looks great. Definitely not for the faint of heart, though! ;) Brandt ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/PAQUZZ4MHZ6TJFIBNCPZGZBW6DOZFUJG/ Code of Conduct: http://python.org/psf/codeofconduct/
