Re: [Python-Dev] ast-objects branch created
On 12/5/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Jeremy Hylton wrote: > I would expect that you allocate in the process memory that needs to > outlive the arena, e.g. python strings. The fundamental problem is that > the arena deallocation cannot know whether such memory exists, and what > to do with it. I can see that problem occurring with an all-or-nothing solution, but not if you have the freedom to allocate from an arena or from some other mechanism. If there are multiple ways to allocate memory, there is some increased programming burden (you have to remember how each pointer was allocated) but you gain flexibility. The ast-arena branch allocates most memory from an arena, but allocates identifiers on the regular heap as PyObjects. It does keep a list of these PyObjects so that it can DECREF them later. > IOW, there would be no problem if you were *completely* done when > you throw away the arena. This is not the case, though: eventually > you end up with byte code that need to persist. Right. The bytecode can't be allocated from the arena, but the AST can. The AST is an internal abstraction. > > Sure does. It seems like the code generation from the AST description > > also makes this kind of experimentation easier. > > Indeed. I wish there was a way to generate ast.c from a transformation > description as well. I'm sure there's a way to generate a parser from the the description, but that seemed like a bigger project than I wanted to tackle. GIven how long it took to finish the AST without a new parser, I think it was a wise choice :-). Jeremy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] svn problem - can't get log info for a specific revision
I'm investigating some problems we're having at work with SIGUSR1 handling. Our apps tend to see KeyboardInterrupt when sent SIGUSR1 signals. We're still running 2.3.4 but are in the process of installing/testing 2.4.2. A couple "find ... | xargs egrep -i usr1" commands in the 2.3 and 2.4 trees showed me that between the two a test was added to test_threadsignals.py, apparently in svn revision 36760: % svn blame Lib/test/test_threadsignals.py ... 36760mwh 36760mwh def registerSignals((for_usr1, for_usr2, for_alrm)): 36760mwh usr1 = signal.signal(signal.SIGUSR1, for_usr1) 36760mwh usr2 = signal.signal(signal.SIGUSR2, for_usr2) 36760mwh alrm = signal.signal(signal.SIGALRM, for_alrm) 36760mwh return usr1, usr2, alrm 36760mwh 36760mwh 41566 fred.drake # The signal handler. Just note that the signal occurred and 36760mwh # from who. 36760mwh def handle_signals(sig,frame): 36792tim_one signal_blackboard[sig]['tripped'] += 1 36760mwh signal_blackboard[sig]['tripped_by'] = thread.get_ident() 36760mwh 36760mwh # a function that will be spawned as a separate thread. 36760mwh def send_signals(): 36760mwh os.kill(process_pid, signal.SIGUSR1) 36760mwh os.kill(process_pid, signal.SIGUSR2) 36760mwh signalled_all.release() 36760mwh ... Wanting to investigate that further, I checked the developer's FAQ and found this command to view the checkin history for that rev: svn log --verbose -r 36760 When I run it I get this error: svn: REPORT request failed on '/projects/!svn/bc/36760/python/branches/release24-maint' svn: '/projects/!svn/bc/36760/python/branches/release24-maint' path not found What is that error telling me? I ran the "svn log" command in the same directory as "svn blame" and "svn up", so it's not a basic problem connecting to the server. It appears to be something specific to this command or revision. Thx, Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast-objects branch created
On Dec 5, 2005, at 8:46 AM, Jeremy Hylton wrote: > On 12/5/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > >> Jeremy Hylton wrote: >> I would expect that you allocate in the process memory that needs to >> outlive the arena, e.g. python strings. The fundamental problem is >> that >> the arena deallocation cannot know whether such memory exists, and >> what >> to do with it. >> > > I can see that problem occurring with an all-or-nothing solution, but > not if you have the freedom to allocate from an arena or from some > other mechanism. If there are multiple ways to allocate memory, there > is some increased programming burden (you have to remember how each > pointer was allocated) but you gain flexibility. The ast-arena branch > allocates most memory from an arena, but allocates identifiers on the > regular heap as PyObjects. It does keep a list of these PyObjects so > that it can DECREF them later. ISTM that having to remember which pointers are arena-allocated and which are normally-refcounted-allocated removes the major gain that an arena method is supposed to bring: resistance to mistakes. I'd find having a single way to allocate and track memory easier than multiple. Then you just have to follow the single set of best practices for memory management, and you're all set. (and with PyObjects, the same practices the rest of python uses, another win.) I'd also like to parrot the concern others have had that if the AST nodes are not made of PyObjects, then a mirror hierarchy of PyObject- ified AST nodes will have to be created, which seems like quite a wasteful duplication. If it is required that there be a collection of AST python objects (which I think it is), is there really any good reason to make the "real" AST objects not be the _only_ AST objects? I've not heard one. James ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn problem - can't get log info for a specific revision
[EMAIL PROTECTED] ... > Wanting to investigate that further, I checked the developer's FAQ and found > this command to view the checkin history for that rev: > >svn log --verbose -r 36760 > > When I run it I get this error: > >svn: REPORT request failed on > '/projects/!svn/bc/36760/python/branches/release24-maint' >svn: '/projects/!svn/bc/36760/python/branches/release24-maint' path not > found > > What is that error telling me? I'm not sure, and I have no idea how you're getting strings like "/!svn/" and "/bc/" in your output, but it works fine here if I point it at the trunk instead: $ svn log -v -r36760 svn+ssh://svn.python.org/python/trunk r36760 | mwh | 2004-08-03 10:37:14 -0400 (Tue, 03 Aug 2004) | 2 lines Changed paths: A /python/trunk/Lib/test/test_threadsignals.py Argh! This was meant to be part of patch #960406. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast-objects branch created
On 12/5/05, James Y Knight <[EMAIL PROTECTED]> wrote: > > On Dec 5, 2005, at 8:46 AM, Jeremy Hylton wrote: > > I can see that problem occurring with an all-or-nothing solution, but > > not if you have the freedom to allocate from an arena or from some > > other mechanism. If there are multiple ways to allocate memory, there > > is some increased programming burden (you have to remember how each > > pointer was allocated) but you gain flexibility. The ast-arena branch > > allocates most memory from an arena, but allocates identifiers on the > > regular heap as PyObjects. It does keep a list of these PyObjects so > > that it can DECREF them later. > > ISTM that having to remember which pointers are arena-allocated and > which are normally-refcounted-allocated removes the major gain that > an arena method is supposed to bring: resistance to mistakes. I'd > find having a single way to allocate and track memory easier than > multiple. Then you just have to follow the single set of best > practices for memory management, and you're all set. (and with > PyObjects, the same practices the rest of python uses, another win.) It's a question of degree, right? If you can find a small number of rules that are easy to understand then you are still likely to avoid mistakes. For example, the current ast-arena branch uses two rules: All AST nodes are allocated from the arena. All PyObjects attached to an AST node (identifiers and constants) are associated with the arena, i.e. they are DECREFed when it is freed. > I'd also like to parrot the concern others have had that if the AST > nodes are not made of PyObjects, then a mirror hierarchy of PyObject- > ified AST nodes will have to be created, which seems like quite a > wasteful duplication. If it is required that there be a collection of > AST python objects (which I think it is), is there really any good > reason to make the "real" AST objects not be the _only_ AST objects? > I've not heard one. The PyObject-ified AST nodes are only needed if user code requests an AST from the compiler. That is, if we add a new feature that exposes AST, we would need AST objects represented in Python code. I think this feature would be great to add, but it doesn't seem like a primary concern for the internal compiler implementation. There is no need for PyObject-ified AST objects in the internal compiler. (I think this fact is obvious, since the compiler exists but PyObject-ified AST objects don't.) The question, then, is the simplest way to provide Python code with access to the AST objects. I still think that a set of pure Python classes to represent the AST nodes is a good approach. You define a simple serialization format for ASTs and the serialized AST can be passed from the interpreter to user code and back. The user code gets a mutable tree of AST nodes that it can reserialize for compilation to bytecode. This strategy is exactly like the existing parser module. One advantage of this approach is the AST objects in each language are simpler to use. The C AST nodes provide an easy API for C programmers and the Python AST nodes provide an easy API for Python programmers. Put another way, since the AST code is all generated from a high level description, the implementation doesn't matter at all. What matters is the API exposed in each programming language. If the best API happens to admit a shared implementation, that's great. If it doesn't, no loss. Jeremy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn problem - can't get log info for a specific revision
>> svn log --verbose -r 36760 >> >> When I run it I get this error: >> >> svn: REPORT request failed on '/projects/!svn/bc/36760/python/branches/release24-maint' >> svn: '/projects/!svn/bc/36760/python/branches/release24-maint' path not found Tim> I'm not sure, and I have no idea how you're getting strings like Tim> "/!svn/" and "/bc/" in your output, but it works fine here if I Tim> point it at the trunk instead: Tim> $ svn log -v -r36760 svn+ssh://svn.python.org/python/trunk ... Okay, I got it. I was asking for that revision on the release24-maint branch, but it was actually applied to the trunk. It shows up now in the maintenance branch, but still has to be queried for on the trunk. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast-objects branch created
On 12/5/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > On 12/5/05, James Y Knight <[EMAIL PROTECTED]> wrote: > > > > ISTM that having to remember which pointers are arena-allocated and > > which are normally-refcounted-allocated removes the major gain that > > an arena method is supposed to bring: resistance to mistakes. I'd > > find having a single way to allocate and track memory easier than > > multiple. Then you just have to follow the single set of best > > practices for memory management, and you're all set. (and with > > PyObjects, the same practices the rest of python uses, another win.) > > It's a question of degree, right? If you can find a small number of > rules that are easy to understand then you are still likely to avoid > mistakes. This is my understanding of the two approaches from what I've seen so far (Jeremy or Martin should correct me if I'm wrong). With current arena impl: * need to call PyArena_AddPyObject() for any allocated PyObject * need to call PyArena_AddMallocPointer() for any malloc()ed memory (there are current no manual calls like this, all the calls are in generated code?) With the PyObject imp: * need to init all PyObjects to NULL * need to Py_XDECREF() on exit * need to goto error if there is any failure Both impls have a bit more details, but those are the highlights. >From what I've seen of both, the arena is easier to deal with even though it is different from the rest of python. There is only one thing to remember. I didn't look at the changes much, but from what I saw I think it may be better to move the arenas off the branch and onto the head now. It appears to be much easier to get right since there is virtually no error handling code in line. It's all taken care of in a few central places. We can then decide between the arenas in the head vs PyObjects. > > I'd also like to parrot the concern others have had that if the AST > > nodes are not made of PyObjects, then a mirror hierarchy of PyObject- > > ified AST nodes will have to be created, which seems like quite a > > wasteful duplication. If it is required that there be a collection of > > AST python objects (which I think it is), is there really any good > > reason to make the "real" AST objects not be the _only_ AST objects? > > I've not heard one. > > The PyObject-ified AST nodes are only needed if user code requests an > AST from the compiler. That is, if we add a new feature that exposes > AST, we would need AST objects represented in Python code. I think > this feature would be great to add, but it doesn't seem like a primary > concern for the internal compiler implementation. FWIW, I agree with this approach. I don't care that much about the internal AST for its own sake. I want to consume the AST and I only care about the internals insofar as the result is correct and maintainable. So my view of the best approach is one that is easy to get right and maintain. That's why I think the arena should be moved to the head now. From what I saw it was much easier to get right, it removed a bunch of code and should be more maintainable. I will also probably work on the PyObject approach, since if that's more maintainable I'd prefer that in the end. I don't know which approach is best. I also really like Martin's idea about generating a lot more (all?) of the manually written Python/ast.c code. I'd prefer much less C code to maintain. n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ast-objects branch created
On 12/5/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > On 12/5/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: [SNIP] > I didn't look at the changes much, but from what I saw I think it may > be better to move the arenas off the branch and onto the head now. It > appears to be much easier to get right since there is virtually no > error handling code in line. It's all taken care of in a few central > places. > > We can then decide between the arenas in the head vs PyObjects. > I am also +1 with merging the arena into the trunk. The arena approach compared to the existing solution is a lot easier to use. With almost all calls to the arena in the auto-generated constructor code, one just has to make sure that key places have PyArena_Free() to free the arena and that errors propagate up to those points. But, as Neal is suggesting, this should not prevent the PyObject version from moving forward since it could still turn out to be the better solution. > > > I'd also like to parrot the concern others have had that if the AST > > > nodes are not made of PyObjects, then a mirror hierarchy of PyObject- > > > ified AST nodes will have to be created, which seems like quite a > > > wasteful duplication. If it is required that there be a collection of > > > AST python objects (which I think it is), is there really any good > > > reason to make the "real" AST objects not be the _only_ AST objects? > > > I've not heard one. > > > > The PyObject-ified AST nodes are only needed if user code requests an > > AST from the compiler. That is, if we add a new feature that exposes > > AST, we would need AST objects represented in Python code. I think > > this feature would be great to add, but it doesn't seem like a primary > > concern for the internal compiler implementation. > > FWIW, I agree with this approach. I don't care that much about the > internal AST for its own sake. I want to consume the AST and I only > care about the internals insofar as the result is correct and > maintainable. > It really comes down to how people expect to use the exposure of the AST. If we try to make sure there is no horrible overhead in getting the AST to Python code and then to the bytecode compiler then it can be used for optimizations (e.g., the existing peepholer could be rewritten in Python and just a default transformation that the AST is passed through). But if we don't want to make sure that AST access is used for optimization transformation but more for non-performance critical uses (e.g., error checking ala PyChecker or refactoring tools) then the simplest, easiest to maintain solution should win out. Personally I want the former abilities for academic experimentation reasons. I don't think that a bunch of optimizations are suddenly going to appear out of nowhere for Python code, but I still would like to be able to experiment with some without having to worry about a performance penalty for doing so. Granted, though, if we byte-compiled scripts passed in on the command-line we would definitely help minimize the performance impact. Interpreter input might be a little slower, but then again since it will be such bite-sized chunks of AST a couple more Python calls shouldn't be that significant. Plus I don't know if serialization will be that much slower than passing the AST itself out since doing a full transformation on an AST might be extremely more costly than just getting the AST to the Python code in the first place. > So my view of the best approach is one that is easy to get right and > maintain. That's why I think the arena should be moved to the head > now. From what I saw it was much easier to get right, it removed a > bunch of code and should be more maintainable. > > I will also probably work on the PyObject approach, since if that's > more maintainable I'd prefer that in the end. I don't know which > approach is best. > > I also really like Martin's idea about generating a lot more (all?) of > the manually written Python/ast.c code. I'd prefer much less C code > to maintain. > A new sprint topic for PyCon for Guido to give us a month deadline on after we have worked on it for three years! =) -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn problem - can't get log info for a specific revision
[EMAIL PROTECTED] wrote: > Okay, I got it. I was asking for that revision on the release24-maint > branch, but it was actually applied to the trunk. It shows up now in the > maintenance branch, but still has to be queried for on the trunk. My subversion (1.2.3) gives me a slightly more legible output: svn: File not found: revision 36760, path '/python/branches/release24-maint' It complains that, in revision 36760, there was no release24-main directory, so it can't know what the history is. I would consider it a bug: it could know that release24-maint was a copy of trunk which was made after 36760, so it could conclude I meant to perform the same operation on trunk, then. Anyway, to get this location-independent (and even without a working copy), do svn log http://svn.python.org/projects --verbose -r 36760 This works because revisions are repository revisions, so the version number uniquely identifies the change in the repository. Of course, this also builds on the knowledge that http://svn.python.org/projects is the same as svn+ssh://[EMAIL PROTECTED] (but more easy to type). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Patch reviews & request for patch review
Hi I attended the bug day on Sunday and reviewed six bugs/patches (1212287, 1215184, 1115886, 1372650, 1216942, 878275). So, I'm hoping one of those nice people who offered 'review 5 get 1 free' might look at a patch of mine. Test, documentation, and explanatory comments in the tracker are all there: http://python.org/sf/1157027 "cookielib mis-handles RFC 2109 cookies in Netscape mode" (It's an old SF patch tracker ID, but I have uploaded a new patch for Python 2.5 since the old patch was not applied in 2.4.1 / 2.4.2.) There's another patch I uploaded whose resolution was agreed upon back in March, but the simple patch (including added test) never got applied: http://python.org/sf/1117398 "cookielib LWPCookieJar and MozillaCookieJar exceptions" Thanks in advance to anybody who has time to look at these, John ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn problem - can't get log info for a specific revision
Martin v. Löwis wrote: > It complains that, in revision 36760, there was no release24-main > directory, so it can't know what the history is. I would consider it > a bug: it could know that release24-maint was a copy of trunk > which was made after 36760, so it could conclude I meant to > perform the same operation on trunk, then. I just asked on the #svn irc channel, and was told that it indeed should *not* follow copies when I do "svn log -r revno", instead, it should give me information on the path as it looked in revno (so in this case, it should tell the path didn't exist). Sometimes, you want to know "how did foo look in revision revno, even if it was called bar at that time". To do that, you refer to [EMAIL PROTECTED], and then can add a revision of that object. They call this syntax "peg revision". So to do what you want, you should be able to say svn log -r 36760 [EMAIL PROTECTED] (i.e. find out what . as it is in HEAD was in revision 36760). Unfortunately, svn log does not support peg revisions, so I filed a bug report at http://subversion.tigris.org/issues/show_bug.cgi?id=2460 To see an example where peg revisions do work, do [EMAIL PROTECTED]:~/work/py2.4$ LANG=C svn info -r 36760 [EMAIL PROTECTED] Path: trunk URL: svn+ssh://[EMAIL PROTECTED]/python/trunk Repository Root: svn+ssh://[EMAIL PROTECTED] Repository UUID: 6015fed2-1504-0410-9fe1-9d1591cc4771 Revision: 36760 Node Kind: directory Last Changed Author: mwh Last Changed Rev: 36760 Last Changed Date: 2004-08-03 16:37:14 +0200 (Tue, 03 Aug 2004) (But then, this is also what you get if you do "svn info -r 36760") Regards, Martin P.S. Instead of HEAD, you would normally use BASE: HEAD is the latest revision in the repository, whereas BASE is the one that your working copy is based on. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dynamic Link Library
Hi everyone, Apologies if my question is a bit novice-ish. I was wondering if there was a way of creating a Dynamic Link Library with Python. Please provide any answers you may have. Warmest Regards, musicdev ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dynamic Link Library
musicdev wrote: > Apologies if my question is a bit novice-ish. I was wondering if there > was a way of creating a Dynamic Link Library with Python. > > Please provide any answers you may have. Please don't use python-dev for such questions; this is a list for the development of Python, not the development with Python. Use the general [email protected] (news:comp.lang.python) for general question. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Broader iterable support for xmlrpclib
During the bug day Andrew Kuchling (I think) mentioned that he was working on some xmlrpclib issues. I then broached an idea I had a week or so ago to allow sets to be marshalled as XML-RPC arrays. This was met with some head scratching by those present. Forging ahead nonetheless, I then proposed the even wackier idea to simply allow all currently unsupported iterables (sets and arrays seem the most obvious candidates to me) to be marshalled as lists. After a couple messages back and forth and more head scratching by those in attendance I offered to whip up a patch and rationale. For that, see here: http://python.org/sf/1374063 Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
