[Python-Dev] Expose stack effects to Python?

2013-07-03 Thread Larry Hastings 



I wrote my own assembler for Python bytecode called "Maynard".  I had to 
statically compute the stack effects for each bytecode instruction by 
hand; what I did was copied and pasted opcode_stack_effect() (which is 
static) out of Python/compile.c and into my own driver program, then I 
probed it with test values to produce a table.  I then coded up a 
function using that table, but hand-calculating the value sometimes as 
there are some opcodes whose stack effect varies based on the oparg.


It sure would be nice if this information was simply available to the 
Python interpreter; theoretically it can change between point releases.  
Would anybody mind if I added it somewhere?  I'd probably just expose 
opcode_stack_effect to Python, then add all this other junk to the dis 
module and make it available there.



//arry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Expose stack effects to Python?

2013-07-03 Thread Victor Stinner 
Hi,

For my registervm project (fork of CPython using register-based
bytecode, instead of stack-based bytecode), I implemented a
Instruction.use_stack() method which just checks if the stack is
"used": read the stack, exchange values in the stack (like "ROT"
instruction), push or pop a value.

Instruction.use_stack():
http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546

The method uses a dummy heuristic, just because it was quick to
implement it, and it's enough for my use case.

To answer your question: yes, I would like a opcode_stack_effect() function.

registervm has its own disassembler which creates objects, rather than
just generating a text representation of the bytecode. I'm using
objects because I rewrite most instructions and I need more
information and functions:

* disassembler (raw bytes => list of instructions)
* assembler (list of instructions => raw bytes)
* format an instruction (human readable assembler)
* is_terminal(): last instruction of a block
* is_cond_jump(): the instruction is a conditional jump? hesitate to
move this disassembler to CPython directly. I'm not sure that it would
be useful, its API is maybe too specific to my registervm project.
* is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on registers
* etc.

Would it be useful to have such high-level API in Python?

Victor

2013/7/3 Larry Hastings :
>
>
> I wrote my own assembler for Python bytecode called "Maynard".  I had to
> statically compute the stack effects for each bytecode instruction by hand;
> what I did was copied and pasted opcode_stack_effect() (which is static) out
> of Python/compile.c and into my own driver program, then I probed it with
> test values to produce a table.  I then coded up a function using that
> table, but hand-calculating the value sometimes as there are some opcodes
> whose stack effect varies based on the oparg.
>
> It sure would be nice if this information was simply available to the Python
> interpreter; theoretically it can change between point releases.  Would
> anybody mind if I added it somewhere?  I'd probably just expose
> opcode_stack_effect to Python, then add all this other junk to the dis
> module and make it available there.
>
>
> /arry
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Expose stack effects to Python?

2013-07-03 Thread Nick Coghlan 
On 3 July 2013 20:06, Victor Stinner  wrote:

> Hi,
>
> For my registervm project (fork of CPython using register-based
> bytecode, instead of stack-based bytecode), I implemented a
> Instruction.use_stack() method which just checks if the stack is
> "used": read the stack, exchange values in the stack (like "ROT"
> instruction), push or pop a value.
>
> Instruction.use_stack():
>
> http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546
>
> The method uses a dummy heuristic, just because it was quick to
> implement it, and it's enough for my use case.
>
> To answer your question: yes, I would like a opcode_stack_effect()
> function.
>
> registervm has its own disassembler which creates objects, rather than
> just generating a text representation of the bytecode. I'm using
> objects because I rewrite most instructions and I need more
> information and functions:
>
> * disassembler (raw bytes => list of instructions)
> * assembler (list of instructions => raw bytes)
> * format an instruction (human readable assembler)
>
* is_terminal(): last instruction of a block
> * is_cond_jump(): the instruction is a conditional jump? hesitate to
> move this disassembler to CPython directly. I'm not sure that it would
> be useful, its API is maybe too specific to my registervm project.
> * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on
> registers
> * etc.
>
> Would it be useful to have such high-level API in Python?
>

I finally committed a longstanding patch to add something like that a while
ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis

It's still fairly low level, but already far more programmatically useful
than the old disassembler text.

I'm still inclined to push higher level stuff out to external libraries -
this patch was mostly about making some of our compiler tests a bit more
maintainable, as well as giving third party libraries better building
blocks without changing the dis module too much.

To get back to Larry's question, though, I think exposing the stack effects
through dis.Instruction would be a good idea (since that will have access
to the oparg to calculate the variable effects).

As far as how to expose the data to Python goes, I suggest adding an
_opcode C module to back opcode.py and eliminate the manual duplication of
the opcode values while you're at it.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Expose stack effects to Python?

2013-07-03 Thread Maciej Fijalkowski 
I wrote a thing that adds more structure to dis (but is not finished)
https://bitbucket.org/pypy/pypy/src/15b0489c15d8150b22815312dd283aa5bafcdd67/lib_pypy/disassembler.py?at=default

On Wed, Jul 3, 2013 at 2:16 PM, Nick Coghlan  wrote:
> On 3 July 2013 20:06, Victor Stinner  wrote:
>>
>> Hi,
>>
>> For my registervm project (fork of CPython using register-based
>> bytecode, instead of stack-based bytecode), I implemented a
>> Instruction.use_stack() method which just checks if the stack is
>> "used": read the stack, exchange values in the stack (like "ROT"
>> instruction), push or pop a value.
>>
>> Instruction.use_stack():
>>
>> http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546
>>
>> The method uses a dummy heuristic, just because it was quick to
>> implement it, and it's enough for my use case.
>>
>> To answer your question: yes, I would like a opcode_stack_effect()
>> function.
>>
>> registervm has its own disassembler which creates objects, rather than
>> just generating a text representation of the bytecode. I'm using
>> objects because I rewrite most instructions and I need more
>> information and functions:
>>
>> * disassembler (raw bytes => list of instructions)
>> * assembler (list of instructions => raw bytes)
>> * format an instruction (human readable assembler)
>>
>> * is_terminal(): last instruction of a block
>> * is_cond_jump(): the instruction is a conditional jump? hesitate to
>> move this disassembler to CPython directly. I'm not sure that it would
>> be useful, its API is maybe too specific to my registervm project.
>> * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on
>> registers
>> * etc.
>>
>> Would it be useful to have such high-level API in Python?
>
>
> I finally committed a longstanding patch to add something like that a while
> ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis
>
> It's still fairly low level, but already far more programmatically useful
> than the old disassembler text.
>
> I'm still inclined to push higher level stuff out to external libraries -
> this patch was mostly about making some of our compiler tests a bit more
> maintainable, as well as giving third party libraries better building blocks
> without changing the dis module too much.
>
> To get back to Larry's question, though, I think exposing the stack effects
> through dis.Instruction would be a good idea (since that will have access to
> the oparg to calculate the variable effects).
>
> As far as how to expose the data to Python goes, I suggest adding an _opcode
> C module to back opcode.py and eliminate the manual duplication of the
> opcode values while you're at it.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Solving the import-deadlock case

2013-07-03 Thread Pascal Chambon 

Thanks for the comments,

in my particular case we're actually on a provisioning /framework/, so 
we chose the easy (lazy?) way, i.e initializing miscellaneous modules at 
loading times (like Django or others do, I think), rather than building 
an proper initialization dispatcher to be called from eg. a wsgi launcher.
It works pretty well actually, except that nasty (but fortunately very 
rare) import deadlock. ^^


Since module loading errors *might* occur for tons of reasons (i.e 
searching the disk for py files already IS a side effect...), and since 
the current behaviour (letting children module survive disconnected from 
their parent) is more harmful than useful, I guess that the cleanup that 
Nick evocated iwould be the path to follow, wouldn't it ?


thanks,
Regards,
Pascal

Le 02/07/2013 23:32, Nick Coghlan a écrit :



On 3 Jul 2013 04:34, "Pascal Chambon" > wrote:

>
> Hello everyone,
>
> I'd like to bring your attention to this issue, since it touches the 
fundamentals of python's import workflow:

> http://bugs.python.org/issue17716
>
> I've tried to post it on the python-import ML for weeks, but it must 
still be blocked somewhere in a moderation queue, so here I come ^^

>
> TLDR version: because of the way import current works, if importing 
a package "temporarily" fails whereas importing one of its children 
succeeded, we reach an unusable state, all subsequent attempts at 
importing that package will fail if a "from...import" is used 
somewhere. Typically, it makes a web worker broken, even though the 
typical behaviour of such process woudl be to retry loading, again and 
again, the failing view.

>
> I agree that a module loading should be, as much as possible, "side 
effects free", and thus shouldn't have temporary errors. But well, in 
practice, module loading is typically the time where process-wide 
initialization are done (modifying sys.path, os.environ, instantiating 
connection or thread pools, registering atexit handler, starting 
maintenance threads...), so that case has chances to happen at a 
moment or another, especially if accesses to filesystem or network 
(SQL...) are done at module loading, due to the lack of initialization 
system at upper levels.

>
> That's why I propose modifying the behaviour of module import, so 
that submodules are deleted as well when a parent module import fails. 
True, it means they will be reloaded as well when importing the parent 
will start again, but anyway we already have a "double execution" 
problem with the reloading of the parent module, so it shouldn't make 
a big difference.
> The only other solution I'd see would be to SYSTEMATICALLY perform 
name (re)binding when processing a from...import statement, to recover 
from the previously failed initialization. Dunno if it's a good idea.

>
> On a (separate but related) topic, to be safer on module reimports 
or reloadings, it could be interesting to add some "idempotency" to 
common initialization tasks ; for example the "atexit" registration 
system, wouldn't it be worth adding a boolean flag to explicitely skip 
registration if a callable with same fully qualified name is already 
registered.

>
> Do you have opinions on these subjects ?

Back on topic...

As I stated on the issue, I think purging the whole subtree when a 
package implicitly imports child modules is the least bad of the 
available options, and better than leaving the child modules in place 
in violation of the "all parent packages can be assumed to be in 
sys.modules" invariant (which is what we do now).


Cheers,
Nick.
>
> thanks,
> regards,
> Pascal
>
> ___
> Python-Dev mailing list
> [email protected] 
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

>



___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/chambon.pascal%40wanadoo.fr
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com