Re: [Python-Dev] Coroutines (PEP 342)
On Mon, 2005-11-14 at 15:46 -0700, Bruce Eckel wrote: [...] > What is not clear to me, and is not discussed in the PEP, is whether > coroutines can be distributed among multiple processors. If that is or > isn't possible I think it should be explained in the PEP, and I'd be > interested in know about it here (and ideally why it would or wouldn't > work). Even if different coroutines could be run on different processors, there would be nothing gained except extra overheads of interprocessor memory duplication and communication delays. The whole process communication via yield and send effectively means only one co-routine is running at a time, and all the others are blocked waiting for a yield or send. This was the whole point; it is a convenient abstraction that appears to do work in parallel, while actually doing it sequentially, avoiding the overheads and possible race conditions of threads. It has the problem that a single co-routine can monopolise execution, hence the other name "co-operative multi-tasking", where co-operation is the requirement for it to work. At least... that's the way I understood it... I could be totally mistaken... -- Donovan Baarda <[EMAIL PROTECTED]> http://minkirri.apana.org.au/~abo/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Memory management in the AST parser & compiler
Transferring part of the discussion of Thomas Lee's PEP 341 patch to
python-dev. . .
Neal Norwitz wrote in the SF patch tracker:
> Thomas, I hope you will write up this experience in coding
> this patch. IMO it clearly demonstrates a problem with the
> new AST code that needs to be addressed. ie, Memory
> management is not possible to get right. I've got a 700+
> line patch to ast.c to correct many more memory issues
> (hopefully that won't cause conflicts with this patch). I
> would like to hear ideas of how the AST code can be improved
> to make it much easier to not leak memory and be safe at the
> same time.
As Neal pointed out, it's tricky to write code for the AST parser and compiler
without accidentally letting memory leak when the parser or compiler runs into
a problem and has to bail out on whatever it was doing. Thomas's patch got to
v5 (based on Neal's review comments) with memory leaks still in it, my review
got rid of some of them, and we think Neal's last review of v6 of the patch
got rid of the last of them.
I am particularly concerned about the returns hidden inside macros in the AST
compiler's symbol table generation and bytecode generation steps. At the
moment, every function in compile.c which allocates code blocks (or anything
else for that matter) and then calls one of the VISIT_* macros is a memory
leak waiting to happen.
Something I've seen used successfully (and used myself) to deal with similar
resource-management problems in C code is to use a switch statement, rather
than getting goto-happy.
Specifically, the body of the entire function is written inside a switch
statement, with 'break' then used as the equivalent of "raise Exception". For
example:
PyObject* switchAsTry()
{
switch(0) {
default:
/* Real function body goes here */
return result;
}
/* Error cleanup code goes here */
return NULL;
}
It avoids the potential for labelling problems that arises when goto's are
used for resource cleanup. It's a far cry from real exception handling, but
it's the best solution I've seen within the limits of C.
A particular benefit comes when macros which may abort function execution are
used inside the function - if those macros are rewritten to use break instead
of return, then the function gets a chance to clean up after an error.
Cheers,
Nick.
P.S. Getting rid of the flow control macros entirely is another option, of
course, but it would make compile.c and symtable.c a LOT harder to follow.
Raymond Chen's articles notwithstanding, a preprocessor-based mini-language
does make sense in some situations, and I think this is one of them.
Particularly since the flow control macros are private to the relevant
implementation files.
--
Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia
---
http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str.dedent
Greg Ewing <[EMAIL PROTECTED]> writes: > James Y Knight wrote: > >> ITYM you mean "If only python were lisp". (macros, or even reader macros) > > No, I mean it would be more satisfying if there > were a syntax for expressing multiline string > literals that didn't force it to be at the left > margin. The lack of such in such an otherwise > indentation-savvy language seems a wart. Wasn't there a PEP about this? Yes, 295. But that was rejected, I presume[*] because it proposed changing all multi-string literals, a plainly doomed idea (well, it would make *me* squeal, anyway). Cheers, mwh (who finds the whole issue rather hard to care about) [*] The reason for rejection isn't in the PEP, grumble. -- I would hereby duly point you at the website for the current pedal powered submarine world underwater speed record, except I've lost the URL. -- Callas, cam.misc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
2005/11/15, Nick Coghlan <[EMAIL PROTECTED]>:
> Specifically, the body of the entire function is written inside a switch
> statement, with 'break' then used as the equivalent of "raise Exception". For
> example:
>
>PyObject* switchAsTry()
>{
> switch(0) {
>default:
> /* Real function body goes here */
> return result;
> }
> /* Error cleanup code goes here */
> return NULL;
>}
>
> It avoids the potential for labelling problems that arises when goto's are
> used for resource cleanup. It's a far cry from real exception handling, but
> it's the best solution I've seen within the limits of C.
do {
} while (0);
Same benefit and saves some typing :)
Now back to my usual hiding place.
--
{ Marek Baczyński :: UIN 57114871 :: GG 161671 :: JID [EMAIL PROTECTED] }
{ http://www.vlo.ids.gda.pl/ | imbaczek at poczta fm | http://www.promode.org }
.. .. .. .. ... ... .. evolve or face extinction .. ... ... .. .. .. ..
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
Marek Baczek Baczyński wrote:
> 2005/11/15, Nick Coghlan <[EMAIL PROTECTED]>:
>> It avoids the potential for labelling problems that arises when goto's are
>> used for resource cleanup. It's a far cry from real exception handling, but
>> it's the best solution I've seen within the limits of C.
>
>
> do {
>
>
> } while (0);
>
>
> Same benefit and saves some typing :)
Heh. Good point. I spend so much time working with a certain language I tend
to forget do/while loops exist ;)
Cheers,
Nick.
--
Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia
---
http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 341 patch & memory management (was: Memory management in the AST parser & compiler)
Interesting trick!
The PEP 341 patch is now using Marek's 'do ... while' resource cleanup
trick instead of the nasty goto voodoo.
I've also fixed the last remaining bug that Neal pointed out. I'm
running the unit tests right now, shall have the updated (and hopefully
final) PEP 341 patch up on sourceforge within the next 15 minutes.
If anybody has feedback/suggestions for the patch, please let me know.
I'm new to this stuff, so I'm still finding my way around :)
Cheers,
Tom
Nick Coghlan wrote:
>Marek Baczek Baczyński wrote:
>
>
>>2005/11/15, Nick Coghlan <[EMAIL PROTECTED]>:
>>
>>
>>>It avoids the potential for labelling problems that arises when goto's are
>>>used for resource cleanup. It's a far cry from real exception handling, but
>>>it's the best solution I've seen within the limits of C.
>>>
>>>
>>
>>do {
>>
>>
>>} while (0);
>>
>>
>>Same benefit and saves some typing :)
>>
>>
>
>Heh. Good point. I spend so much time working with a certain language I tend
>to forget do/while loops exist ;)
>
>Cheers,
>Nick.
>
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Gothenburg PyPy Sprint II: 7th - 11th December 2005
Gothenburg PyPy Sprint II: 7th - 11th December 2005 == (NOTE: internal EU-only sprint starts on the 5th!) The next PyPy sprint is scheduled to be in December 2005 in Gothenborg, Sweden. Its main focus is heading towards phase 2, which means JIT work, alternate threading models and logic programming (but there are also other possible topics). We'll give newcomer-friendly introductions. To learn more about the new PyPy Python-in-Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint -- We have released pypy-0.8.0_, which is officially a "research base" for future work. The goal of the Gothenburg sprint is to start exploring new directions and continue in the directions started at the Paris sprint. The currently scheduled main topics are: - The L3 interpreter, a small fast interpreter for "assembler-level" flow graphs. This is heading towards JIT work. - Stackless: write an app-level interface, which might be either Tasklets, as in "Stackless CPython", or the more limited Greenlets. - Porting C modules from CPython. (_socket is not finished) - Optimization/debugging work in general. In particular our thread support is far from stable at the moment and unaccountably slow. - Experimentation: logic programming in Python. A first step might be to try to add logic variables to PyPy. .. _`pypy-0.8.0`: http://codespeak.net/pypy/dist/pypy/doc/release-0.8.0.html Location & Accomodation The sprint will be held in the apartment of Laura Creighton and Jacob Halen which is in Gotabergsgatan 22. The location is central in Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, where many lines call. .. _tram: http://www.vasttrafik.se Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. (You can have a 10% discount there; ask in the pypy-sprint mailing list for details. We also have some possibilites of free accomodation.) .. _`SGS Veckobostader`: http://www.sgsveckobostader.com Exact times --- The public PyPy sprint is held Wednesday 7th - Sunday 11th December 2005. There is a sprint for people involved with the EU part of the project on the two days before the "official" sprint. Hours will be from 10:00 until people have had enough. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Network, Food, currency Sweden is not part of the Euro zone. One SEK (krona in singular, kronor in plural) is roughly 1/10th of a Euro (9.15 SEK to 1 Euro). The venue is central in Gothenburg. There is a large selection of places to get food around, from edible-and-cheap to outstanding. You normally need a wireless network card to access the network, but we can provide a wireless/ethernet bridge. Sweden uses the same kind of plugs as Germany. 230V AC. Registration etc.pp. Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Gothenburg people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/gothenburg-2005/people.txt .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`Gothenburg people`: http://codespeak.net/pypy/extradoc/sprintinfo/gothenburg-2005/people.html Cheers, mwh -- Speaking from personal experience, I can attest that the barrel of any firearm (or black powder weapon) pointed at one appears large enough to walk down, hands at full extension above the head, without touching the top. -- Mike Andrews, asr ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
> As Neal pointed out, it's tricky to write code for the AST parser > and compiler > without accidentally letting memory leak when the parser or > compiler runs into > a problem and has to bail out on whatever it was doing. Thomas's > patch got to > v5 (based on Neal's review comments) with memory leaks still in it, > my review > got rid of some of them, and we think Neal's last review of v6 of > the patch > got rid of the last of them. Another lurker's 2 cents: My experience with compilers in particular is that an arena is the way to go for memory management. I haven't looked at the AST code, but this can take a variety of forms: anything from linked lists of pointers to free from something which allocates memory in large blocks and parcels them out. The goal is just to be able to free the memory en-masse whatever happens and not have to track individual pointers. Generally, compilers have memory allocations which operate in phases and so are very amenable to arenas. You might have one memory pool for long lived representation, one that is freed and recreated between passes, etc. If you need to keep the AST around long term, then a mark-sweep garbage collector combined with a linked list might even be a good idea. Obviously, the whole thing is a tradeoff of peak memory size (which goes up) against correctness (which is basically ensured, and at least easily auditable). Niko ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On 11/15/05, Niko Matsakis <[EMAIL PROTECTED]> wrote: > > As Neal pointed out, it's tricky to write code for the AST parser > > and compiler > > without accidentally letting memory leak when the parser or > > compiler runs into > > a problem and has to bail out on whatever it was doing. Thomas's > > patch got to > > v5 (based on Neal's review comments) with memory leaks still in it, > > my review > > got rid of some of them, and we think Neal's last review of v6 of > > the patch > > got rid of the last of them. > > Another lurker's 2 cents: > > My experience with compilers in particular is that an arena is the > way to go for memory management. I haven't looked at the AST code, > but this can take a variety of forms: anything from linked lists of > pointers to free from something which allocates memory in large > blocks and parcels them out. The goal is just to be able to free the > memory en-masse whatever happens and not have to track individual > pointers. Thanks for the message. I was going to suggest the same thing. I think it's primarily a question of how to add an arena layer. The AST phase has a mixture of malloc/free and Python object allocation. It should be straightforward to change the malloc/free code to use an arena API. We'd probably need a separate mechanism to associate a set of PyObject* with the arena and have those DECREFed. Jeremy > > Generally, compilers have memory allocations which operate in phases > and so are very amenable to arenas. You might have one memory pool > for long lived representation, one that is freed and recreated > between passes, etc. > > If you need to keep the AST around long term, then a mark-sweep > garbage collector combined with a linked list might even be a good idea. > > Obviously, the whole thing is a tradeoff of peak memory size (which > goes up) against correctness (which is basically ensured, and at > least easily auditable). > > > Niko > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > On 11/15/05, Niko Matsakis <[EMAIL PROTECTED]> wrote: > > > As Neal pointed out, it's tricky to write code for the AST parser > > > and compiler > > > without accidentally letting memory leak when the parser or > > > compiler runs into > > > a problem and has to bail out on whatever it was doing. Thomas's > > > patch got to > > > v5 (based on Neal's review comments) with memory leaks still in it, > > > my review > > > got rid of some of them, and we think Neal's last review of v6 of > > > the patch > > > got rid of the last of them. > > > > Another lurker's 2 cents: > > > > My experience with compilers in particular is that an arena is the > > way to go for memory management. I haven't looked at the AST code, > > but this can take a variety of forms: anything from linked lists of > > pointers to free from something which allocates memory in large > > blocks and parcels them out. The goal is just to be able to free the > > memory en-masse whatever happens and not have to track individual > > pointers. > > Thanks for the message. I was going to suggest the same thing. I > think it's primarily a question of how to add an arena layer. The AST > phase has a mixture of malloc/free and Python object allocation. It > should be straightforward to change the malloc/free code to use an > arena API. We'd probably need a separate mechanism to associate a set > of PyObject* with the arena and have those DECREFed. > Might just need two lists; malloc'ed pointers and PyObject pointers. Could redefine Py_INCREF and Py_DECREF locally for ast.c and compile.c to use the arena API and thus hide the detail. Otherwise just a big, flashing "USE THIS API" sign will be needed. I have gone ahead and added this as a possible topic to sprint on at PyCon. -Brett > Jeremy > > > > > Generally, compilers have memory allocations which operate in phases > > and so are very amenable to arenas. You might have one memory pool > > for long lived representation, one that is freed and recreated > > between passes, etc. > > > > If you need to keep the AST around long term, then a mark-sweep > > garbage collector combined with a linked list might even be a good idea. > > > > Obviously, the whole thing is a tradeoff of peak memory size (which > > goes up) against correctness (which is basically ensured, and at > > least easily auditable). > > > > > > Niko > > ___ > > Python-Dev mailing list > > [email protected] > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > > Thanks for the message. I was going to suggest the same thing. I > think it's primarily a question of how to add an arena layer. The AST > phase has a mixture of malloc/free and Python object allocation. It > should be straightforward to change the malloc/free code to use an > arena API. We'd probably need a separate mechanism to associate a set > of PyObject* with the arena and have those DECREFed. Well good. It seems we all agree there is a problem and on the general solution. I haven't thought about Brett's idea to see if it could work or not. It would be great if we had someone start working to improve the situation. It could well be that we live with the current code for 2.5, but it would be great to use arenas for 2.6 at least. Niko, Marek, how would you like to lose your lurker status? ;-) n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str.dedent
Thanks for your examples. I understand tham sometimes it's a good idea
not to write the HTML inside the function (although it may be nice to
sometimes write it just before the function - and if it's a method,
then we get the same indentation problem.)
However, as you said, sometimes it is desired to write multilined
strings inside functions. You think it's ok to add white spaces to the
HTML code, I personally prefer not add varying indentation to my
output according to the level of indentation the code that generated
it.
I just wanted to add another use case: long messages. Consider those
lines from idlelib/run.py:133
msg = "IDLE's subprocess can't connect to %s:%d. This may be due "\
"to your personal firewall configuration. It is safe to "\
"allow this internal connection because no data is visible on "\
"external ports." % address
tkMessageBox.showerror("IDLE Subprocess Error", msg, parent=root)
and from idlelib/PyShell.py:734:
def display_port_binding_error(self):
tkMessageBox.showerror(
"Port Binding Error",
"IDLE can't bind TCP/IP port 8833, which is necessary to "
"communicate with its Python execution server. Either "
"no networking is installed on this computer or another "
"process (another IDLE?) is using the port. Run IDLE with the -n "
"command line switch to start without a subprocess and refer to "
"Help/IDLE Help 'Running without a subprocess' for further "
"details.",
master=self.tkconsole.text)
I know, of course, that it could be written using textwrap.dedent, but
I think that not having to load a module will encourage the use of
dedent; if I have to load a module, I might say, "oh, I can live with
all those marks around the text, there's no need for another module",
and then, any time I want to change that message, I have a lot of
editing work to do.
Noam
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Behavoir question.
Since I am fiddling with int/long conversions to/from string:
Is the current behavior intentional (or mandatory?):
v = int(' 5 ')
works, but:
v = int(' 5L ')
fails.
--Scott David Daniels
[EMAIL PROTECTED]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Conclusion: Event loops, PyOS_InputHook, and Tkinter
Thanks everybody for contributing to this discussion. I didn't expect it to become this extensive. I think that by now, everybody has had their chance to voice their opinion. It seems safe to conclude that there is no consensus on this topic. So what I'm planning to do is to write a small extension module that implements some of the ideas that came up in this discussion, and see how they perform in the wild. It will give us an idea of what works, what doesn't, and what the user demand is for such functionality, and will help us if this issue happens to turn up again at some point in the future. Thanks again, --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conclusion: Event loops, PyOS_InputHook, and Tkinter
At 06:48 PM 11/15/2005 -0500, Michiel Jan Laurens de Hoon wrote: >Thanks everybody for contributing to this discussion. I didn't expect it >to become this extensive. >I think that by now, everybody has had their chance to voice their opinion. >It seems safe to conclude that there is no consensus on this topic. Just a question: did you ever try using IPython, and confirm whether it does or does not address the issue you were having? As far as I could tell, you never confirmed or denied that point. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conclusion: Event loops, PyOS_InputHook, and Tkinter
Phillip J. Eby wrote: > At 06:48 PM 11/15/2005 -0500, Michiel Jan Laurens de Hoon wrote: > >> Thanks everybody for contributing to this discussion. I didn't expect it >> to become this extensive. >> I think that by now, everybody has had their chance to voice their >> opinion. >> It seems safe to conclude that there is no consensus on this topic. > > > Just a question: did you ever try using IPython, and confirm whether > it does or does not address the issue you were having? As far as I > could tell, you never confirmed or denied that point. > Yes I did try IPython. First of all, IPython, being pure Python code, does not affect the underlying Python's loop (at the C level). So just running Python through IPython does not fix our event loop problem. On Windows, for example, after importing IPython into IDLE (which most of our users will want to use), our graphics window still freezes. This leaves us with the possibility of using IPython's event loop, which it runs on top of regular Python. But if we use that, we'd either have to convince all our users to switch to IPython (which is impossible) or we have to maintain two mechanisms to hook our extension module into the event loop: one for Python and one for IPython. There are several other reasons why the alternative solutions that came up in this discussion are more attractive than IPython: 1) AFAICT, IPython is not intended to work with IDLE. 2) I didn't get the impression that the IPython developers understand why and how their event loop works very well (which made it hard to respond to their posts). I am primarily interested in understanding the problem first and then come up with a suitable mechanism for events. Without such understanding, IPython's event loop smells too much like a hack. 3) IPython adds another layer on top of Python. For IPython's purpose, that's fine. But if we're just interested in event loops, I think it is hard to argue that another layer is absolutely necessary. So rather than setting up an event loop in a layer on top of Python, I'd prefer to find a solution within the context of Python itself (be it threads, an event loop, or PyOS_InputHook). 4) Call me a sentimental fool, but I just happen to like regular Python. My apologies in advance to the IPython developers if I misunderstood how it works. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conclusion: Event loops, PyOS_InputHook, and Tkinter
Michiel Jan Laurens de Hoon wrote: > There are several other reasons why the alternative solutions that came > up in this discussion are more attractive than IPython: > 1) AFAICT, IPython is not intended to work with IDLE. Not so far, but mostly by accident. The necessary changes are fairly easy (mainly abstracting out assumptions about being in a tty). I plan on making ipython embeddable inside any GUI (including IDLE), as there is much demand for that. > 2) I didn't get the impression that the IPython developers understand > why and how their event loop works very well (which made it hard to > respond to their posts). I am primarily interested in understanding the > problem first and then come up with a suitable mechanism for events. > Without such understanding, IPython's event loop smells too much like a > hack. I said I did get that code off the ground by stumbling in the dark, but I tried to explain to you what it does, which is pretty simple: a. You find, for each toolkit, what its timer/idle mechanism is. This requires reading a little about each toolkit's API, as they all do it slightly differently. But the principle is always the same, only the implementation details change. b. You subclass threading.Thread, as you do for all threading code. The run method of this class manages a one-entry queue where code is put for execution from stdin. c. The timer you set up with the info from (a) calls the method which executes the code object from the queue in (b), with suitable locking. That's pretty much it. Following this same idea, just this week I implemented an ipython-for-OpenGL shell. All I had to do was look up what OpenGL uses for an idle callback. > 3) IPython adds another layer on top of Python. For IPython's purpose, > that's fine. But if we're just interested in event loops, I think it is > hard to argue that another layer is absolutely necessary. So rather than > setting up an event loop in a layer on top of Python, I'd prefer to find > a solution within the context of Python itself (be it threads, an event > loop, or PyOS_InputHook). I gave you a link to a 200 line script which implements the core idea for GTK without any ipython at all. I explained that in my message. I don't know how to be more specific, ipython-independent or clear with you. > 4) Call me a sentimental fool, but I just happen to like regular Python. That's fine. I'd argue that ipython is exceptionally useful in a scientific computing workflow, but I'm obviously biased. Many others in the scientific community seem to agree with me, though, given the frequency of ipython prompts in posts to the scientific computing lists. But this is free software in a free world: use whatever you like. All I'm interested in is in clarifying a technical issue, not in evangelizing ipython; that's why I gave you a link to a non-ipython example which implements the key idea using only the standard python library. > My apologies in advance to the IPython developers if I misunderstood how > it works. No problem. But your posts so far seem to indicate you hardly read what I said, as I've had to repeat several key points over and over (the non-ipython solutions, for example). Cheers, f ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Behavoir question.
On 11/15/05, Scott David Daniels <[EMAIL PROTECTED]> wrote:
> Since I am fiddling with int/long conversions to/from string:
>
> Is the current behavior intentional (or mandatory?):
>
> v = int(' 5 ')
> works, but:
> v = int(' 5L ')
> fails.
>
> --Scott David Daniels
> [EMAIL PROTECTED]
int(s) works where s is a string representing a number. 10L does not
represent a number directly, but is Python syntax for making an
integer constant a long, and not an int. (Consider that both are
representations of mathematical integers, tho in python we only call
one of them an integer by terminology).
So, what you're asking is like if list('[1,2]') returned [1,2]. If you
need this functionality, maybe you need a regex match and expr().
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Problems with the Python Memory Manager
I know (thanks to Google) that much has been said in the past about the Python Memory Manager. My purpose in posting is simply to given a use-case example of how the current memory manager (in Python 2.4.X) can be problematic in scientific/engineering code. Scipy core is a replacement for Numeric. One of the things scipy core does is define a new python scalar object for ever data type that an array can have (currently 21). This has many advantages and is made feasible by the ability of Python to subtype in C. These scalars all inherit from the standard Python types where there is a correspondence. More to the point, however, these scalar objects were allocated using the standard PyObject_New and PyObject_Del functions which of course use the Python memory manager.One user ported his (long-running) code to the new scipy core and found much to his dismay that what used to consume around 100MB now completely dominated his machine consuming up to 2GB of memory after only a few iterations. After searching many hours for memory leaks in scipy core (not a bad exercise anyway as some were found), the real problem was tracked to the fact that his code ended up creating and destroying many of these new array scalars. The Python memory manager was not reusing memory (even though PyObject_Del was being called). I don't know enough about the memory manager to understand why that was happening. However, changing the allocation from PyObject_New to malloc and from PyObject_Del to free, fixed the problems this user was seeing. Now the code runs for a long time consuming only around 100MB at-a-time. Thus, all of the objects in scipy core now use system malloc and system free for their memory needs. Perhaps this is unfortunate, but it was the only solution I could see in the short term. In the long term, what is the status of plans to re-work the Python Memory manager to free memory that it acquires (or improve the detection of already freed memory locations). I see from other postings that this has been a problem for other people as well. Also, is there a recommended way for dealing with this problem other than using system malloc and system free (or I suppose writing your own specialized memory manager). Thanks for any feedback, -Travis Oliphant ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
